Saturday, August 29, 2009

Does BI require business process re-engineering?

I met up with a senior executive of a large firm last week. This firm is in process of rolling out BI intiative enterprise wide. He asked us a question about changes required in their business processes to ensure sucessful roll out of BI initiative. This question took all of us by surprise. We all were ready with our answer of how BI can help them increase revenue, profitability and reduce cost. This type of question shows maturity of an organization to adopt technology like BI. After the meeting, I didn't have any doubt what so ever about readiness of this organization to roll out enterprise wide BI Initiative.

Today many BI initiatives fail as they do not systematically address the business process changes required to capture business value of BI. It is common for BI vendor value propositions to emphasize business benefits such as profitability, responsiveness, customer intimacy, information sharing, flexibility, and collaboration. But investing in BI to achieve such business benefits may actually destroy business value unless those attributes can be defined in operational terms and realized through business processes that affect revenues or costs.

Many companies use BI to improve customer segmentation, customer acquisition and customer retention. These improvements can be linked to reduced customer acquisitions costs, increased revenue, and increased customer lifetime value, which translate to increase in profitability. However, a BI investment that improves demand forecasting will not deliver business value unless the forecasts are actually incorporated into operational business processes that then deliver reduced inventory or some other tangible economic benefit. In other word, the business benefit "Improved forecasting" is useless unless it is somehow converted into incremental after-tax cash flow.

To capture the business value of BI requires organizations to go well beyond the technical implementation of a BI environment. Specifically, organizations must engage in effective process engineering and change management in order to capture business value from BI.

BI systems delivers lot of useful information such as most profitable customers of your organization, fast & slow moving inventories, Good & bad suppliers, product cost & profitability. If you do not integrate this information with your management processes and operational processes such as ERP & CRM then you will not derive any value out of your investment in BI systems.

Process engineering is very essential for successful BI roll out. Process engineering identifies how BI applications will be used within the context of key management and operational processes that drive increased revenue and/or reduce cost. It provides a map of which processes must change and how they must change in order to create business value with BI applications. Thus, it lays the foundation for change management because process changes drive changes in individual and organizational behavior.

Monday, August 24, 2009

BI as Business Enabler

I have started my sales/pre-sales journey with SAS about 3 1/2 year ago. My job is 100% customer facing. I have seen quite a few changes, in a way, people have started adopting business intelligence technology in past few years.

My first assignment at SAS was that of data quality. At that point in time there was not much awareness about data quality solutions in Indian market. It took almost a year to convince customer to invest in data quality solution. There were very few data quality solutions available in Indian market at that time. Today, when I look in market, there are plenty of data quality software vendors. Data quality has become integral part of data governance & compliance strategy for most of the organizations. We have worked on more than 5 enterprise data warehouse RFPs in last 3 months. All of them had included requirements for data quality. Data quality is not only used for basic data cleansing but it is also used for doing de-dup, house hold analysis and cross sell/up sell.

One of the major banks is using data quality to ensure that they give loan to right people and thereby ensuring low NPAs. One of the CPG company is using data quality to create single view of retail outlets across various product categories. This helps them in optimizing sales force & increasing cross sell/Up Sell opportunities within same outlet for different products. One of the manufacturing company is using data quality to create single view of customer across various business functions. This helps them to cross sell/up sell products across various business functions.

I have also seen enormous change in a way people are using business intelligence, now compared with what it was, four years ago. Earlier BI was used by very few people in organization. That's the reason some BI vendors are offering per-user licensing. Today BI reporting access is given to all users at all levels for decision making. User based licensing makes no sense in today's environment. Most of the organizations are now moving towards establishing enterprise-wide reporting framework. They are also standardizing on their reporting platform across various departments. Need for reporting user interface is also changing from one user group to another user group. Customers are expecting different user interfaces for users having different skill sets. There are some users , who are more comfortable with excel, requires excel interface. There are some users, who are more comfortable with web interface, requires web interface for reports. Earlier, reports used to get refreshed every month or week. Today customers are expecting to refresh reports multiple times in a day or atleast once in a day. In past few months I have seen several requirements for real time dashboarding & OLAP analysis.

One of the large private limited bank is using real time dashboards to monitor cash level & down time of each ATM. BI system sends out an alert to respective regional manager when cash drops below certain level or ATM is down for long period. This has reduced cash out situations drastically across all ATMs in country. One of the large manufacturing company is using BI reporting to measure supplier performance. They monitor quality and quantity of s upply of each supplier using this system. They also compare pricing from different supplier for thesame material and quantity. They use this information to better neogotiate material pricing with their suppliers. One of the retailer is using real time dashboard to monitor their fast moving items at store level and thereby reducing stock out situations.

Today customers are considering BI as a business enabler. Their expectations from BI systems have increased in last few years. They are looking for BI solutions which offer capabilities beyond Querying and reporting. They are looking for solutions which can help them forecast, predict and optimize. Lots of companies have started mentioning about data warehouse initiative as a strategic initiative in their annual reports. They believe that the implementation of the BI system will help them increase their sales and profitability. This represents a large opportunity for BI vendors. Hence we are seeing lots of consolidation in BI market.

Saturday, August 22, 2009

Customer Oriented Banking

I have had a bad experience with customer service of one of the large private bank last week. My netbanking id was locked as I didn't use it for past more than 3 months. When I approached their phone banking support to get it unlocked, they advised me to raise support ticket via web service request form. I did the same promptly as I wanted to transfer funds urgently to my some other accounts. They took 5 days to unlock my id. I missed my deadline to transfer the fund.

I am a loyal customer of this bank for more than 4 years now. I am also one of their premium customer and hold more than 4 products from the same bank. However when I submitted support request, they treated my request on the same priority as that of a non premium customer. I was expecting quick turnaround and differentiated service level, being a premium customer of bank. It didn't happen. Today I have decided to move to other bank. I am sure this bank must be losing lots of such customer in a year. This bank is known to be one of the most technology savvy bank in India, but unfortunately they do not use any system which helps them differentiate their premium customers from non premium customers. There are several advanced analytics solutions available in market which can help bank address such problems/challenges proactively. I have seen banks spending lots efforts and money on marketing campaigns to retain their premium customers when they are leaving. But if they invest in analytics systems then they can save lots of money on such marketing spend.

Similarly where a new loan to a non premium customer is given in say 3-4 days, can the same be given to a premium customer in just 1-2 days ? I had to wait for 4 working days to avail an Auto Loan last year through the same bank where I was a premium customer.

Some years ago banks were focussing on acquiring customers and quality of customer was not given lot of importance. In today's economic condition, you need to improve quality of new customers. You would like to increase wallet share per customer and at the same time you want to retain only those customers which are profitable to your organizations. Analytical solutions can help address all of such requirements. To survive in this competitive market conditions, we need to address the following questions pro-actively

1. Am I acquiring good customers?

2. Am I spending my marketing budget in right direction to increase wallet share per customer?

3. Do I know my most profitable customers? What am I doing to retain them?

4. Do I know my non profitable customers? What am I doing to terminate them?

5. Can I expediate the process of giving loans to the most profitable customers ?

Typically in banking when you submit any web request, it goes into queue. CRM system automatically routes your request basis information available in subject line of your request. Lots of time subject line is incorrect and that results in routing of a request to wrong customer support group. The request gets re-assigned to right group and then they start working on it. This is very tedious and time consuming process. Lot of precious time is lost in a assigning request to right support group. This can be avoided by implementing text analytics solutions. Text analytics solution can help you route the request based on text in the web form. If customer id is available in web form then it can also identify whether a particular customer is premium customer or non premium customer. Accordingly it can route it to queue dedicated to the premium customers.

Sunday, August 16, 2009

Change Data Capture - Real Time BI

I have heard need for change data capture from several customers in past few months.

Traditionally, ETL processes have been run periodically, on a monthly or weekly basis, and use a bulk approach that moves and integrates the entire data set from the operational source systems to the target data warehouse. Now, data integration requirements have changed. Customers would like to move the changes made to enterprise data while the operational systems are running, without the need for a downtime window. They do not want to degrade performance and service levels of their operation systems.

Business conditions have changed over a period of time. It requires a new way of integrating data in real time and efficient manner.

  1. Business globalization and 24x7 operations. In the past, enterprises could stop online systems during the night or weekend, to provide a window of time for running bulk ETL processes. Today, running a global business with 24x7 operations means smaller or no downtime windows.
  2. Need for up-to-date, current data. In today’s competitive environment & competitive pressure, organization cannot afford to have their managers’ work on last week or yesterday’s data. Today, decision-makers need data that is updated a few times a day or even in real time.
  3. Data volumes are increasing. Data is doubling every 9 months. The larger the data volumes become, the more resources and time are required by the ETL processes. This trend challenges the bulk extract windows that are getting smaller and smaller.
  4. Cost reduction. Bulk ETL operations are costly and inefficient, as they require more processing power, more memory and more network bandwidth. In addition, as bulk ETL processes run for long periods of time, they also require more administration and IT resources to manage.

The first step in change data capture is detecting the changes! There are four main ways to detect changes:

  • Audit columns. In most cases, the source system contains audit columns. Audit columns are appended to the end of each table to store the date and time a record was added or modified. Audit columns are usually populated via database triggers that are fired off automatically as records are inserted or updated.
  • Database log scraping. Log scraping effectively takes a snapshot of the database redo log at a scheduled point in time (usually midnight) and scours it for transactions that affect the tables you care about for your ETL load. Sniffing involves a “polling” of the redo log, capturing transactions on-the-fly. Scraping the log for transactions is probably the messiest of all techniques. It’s not rare for transaction logs to “blow-out,” meaning they get full and prevent new transactions from occurring. If you’ve exhausted all other techniques and find log scraping is your last resort for finding new or changed records, persuade the DBA to create a special log to meet your specific needs.
  • Timed extracts. With a timed extract you typically select all of the rows where the date in the Create or Modified date fields equal SYSDATE-1, meaning you’ve got all of yesterday’s records. Sounds perfect, right? Wrong. Loading records based purely on time is a common mistake made by most beginning ETL developers. This process is horribly unreliable. Time-based data selection loads duplicate rows when it is restarted from mid-process failures. This means that manual intervention and data cleanup is required if the process fails for any reason. Meanwhile, if the nightly load process fails to run and misses a day, a risk exists that the missed data will never make it into the data warehouse.
  • Full database “diff compare.” A full diff compare keeps a full snapshot of yesterday’s database, and compares it, record by record against today’s database to find what changed. The good news is that this technique is fully general: you are guaranteed to find every change. The obvious bad news is that in many cases this technique is very resource intensive. If you must do a full diff compare, then try to do the compare on the source machine so that you don’t have to transfer the whole database into the ETL environment. Also, investigate using CRC (cyclic redundancy checksum) algorithms to quickly tell if a complex record has changed.

CDC solutions are designed to maximize the efficiency of ETL processes, minimize resource usage by replicating/moving only changes to the data (i.e., the deltas) and minimize the latency in the delivery of timely business information to the potential consumers. Change data capture solutions comprises of the following key components

  • Change Capture Agents
  • Changed Data Services
  • Change Delivery

Change Capture Agents
Change capture agents are the software components that are responsible for the identification and capture of changes to the source operational data store. Change capture agent sits on source system and takes minimal power of source system. Typically it utilizes 1 to 2% of source system processing power. Change capture agents can be optimized and dedicated to the source system (i.e., typically using database journals, triggers or exit hooks) or by using generic methods such as data log comparison.

Change Data Services
Change data services provide a set of functions critical to achieving successful CDC, including but not limited to: filtering (e.g., receiving only committed changes), sequencing (e.g., receiving changes based on transaction/unit of work boundaries, by table or by timestamp), change data enrichment (e.g., add reference data to the delivered change for further processing purposes), life cycle management (i.e., how long will the changes be available for consuming applications) and auditing that enables monitoring of the system's end-to-end behavior, as well as the examination of trends over time.

Change Delivery
Change delivery mechanisms are responsible for the reliable delivery of changed data to change consumers -- typically an ETL program. Change delivery mechanisms can support one or more consumers and provide flexible ways by which the changes can be delivered including push and pull models. A pull model means that the change consumer asks for the changes on a periodic basis (as frequently as needed, typically every few minutes or hours), preferably using a standard interface such as ODBC or JDBC. A push model means that the change consumer listens and waits for changes, and those are delivered as soon as they are captured, typically using some messaging middleware. Another important function of change delivery is the ability to dynamically go back and ask for older changes for repeated, additional or recovery processing.

Following are two sample scenarios that highlight how organizations can leverage CDC.

Sample Scenario 1: Batch-Oriented CDC (pull CDC)
In this scenario, an ETL tool periodically requests the changes, each time receiving a batch of records that represent all the changes that were captured since the last request cycle. Change delivery requests can be done in low or high frequencies (e.g., twice a day or every 15 minutes). For many organizations, the preferred method of providing extracted changes is to expose them as records of a data source table. This approach enables the ETL tool to seamlessly access the changed records using standard interfaces such as ODBC. The CDC solution needs to take care of maintaining the position of the last change delivery and deliver new changes every time.
This scenario is very similar to traditional bulk ETL, except that it processes only the changes to the data instead of the entire source data store. This approach greatly reduces the required resources and eliminates the need for a downtime window for ETL operations.

When should organizations use this approach? This batch-oriented approach is very easy to implement, as it is similar to traditional ETL processes and capitalizes on existing skill sets. Organizations should use this method when their latency requirements are measured in hours or minutes.

Sample Scenario 2: Live/Real-Time CDC (push CDC)
In this scenario, which accommodates near real-time or real-time latency requirements, the change delivery mechanism pushes the changes to the ETL program as soon as changes are captured. This is typically done using a reliable transport such as an event-delivery mechanism or messaging middleware. Some CDC solutions use proprietary event delivery mechanisms, and some support standard messaging middleware (e.g., MQ Series).

Note that while message-oriented or event-driven integration is more common in EAI products (i.e., using tools such as Integration Brokers), many of the leading ETL tool vendors are offering such capabilities in their solutions to accommodate the demands of high-end, real-time BI applications. This real-time approach is required when the BI applications demand zero latency and the most up-to-date data.

Change Data Capture Technical Considerations
While CDC seems to offer significant advantages, there are several factors that need to be considered and evaluated, including:

Change Capture Technique. Change capture methods vary, and each has different implications on the overall solution latency, scalability and level of intrusion. Common techniques for capturing changes include reading database journals or log files, usage of database triggers or exit hooks, data comparison and programming custom event notifications within enterprise programs.

Level of Intrusion. All CDC solutions have a certain degree of system impact, making intrusion a critical evaluation factor. The highest degree of intrusion is "source code" intrusion that requires changes to be made to the enterprise applications that make the changes to the data stores. A lesser degree of intrusion is "in-process" or "address space" intrusion, which means that the change capture solution affects the operational system resources. This is the case when using database triggers and exit hooks because they run as part of the operational system and share its resources. Using database journals or archive logs is the least intrusive solution and it does not affect the operational data sources of applications.

Capture Latency. This factor is a key driver for choosing CDC in the first place. Latency is affected by the change capture method, the processing done to the changes and the choice of change delivery mechanism. As a result, changes can be streamed periodically, in high frequency or in real time. One should note that the more real-time the solution is, the more intrusive it typically is as well. Yet another point to consider is that different BI applications will have different latency requirements, and thus enterprises should look for CDC solutions that support a wide range of configurations.

Filtering and Sequencing Services. CDC solutions should provide various services to facilitate the filtering and sequencing of delivered changes. Filtering helps to guarantee that only the needed changes are indeed delivered, for example: an ETL process will typically need only the committed changes. Another example is the ability to discard redundant changes and deliver the last change to further reduce processing. Sequencing defines the order by which changes are delivered. For example, some ETL applications may need changes on a table by table basis, while others may want the changes based on units of work (i.e., across multiple tables).

Supporting Multiple Con-sumers. Captured changes may need to be delivered to more than one consumer, such as multiple ETL processes, data synchronization applications and business activity monitoring. CDC solutions need to support multiple consumers, each of which may have different latency requirements.

Failover and Recoverability. CDC solutions need to guarantee that changes will be delivered correctly, even when system, network or process failures occur. Recovery means that a change delivery stream can continue from its last position and that the solution keeps transactional integrity to the changes throughout the delivery cycle.

Mainframe and Legacy Data Sources. BI is only as good as the data it relies on. Analysts estimate that mainframe systems still store approximately 70 percent of corporate business information, and mainframes still process most of the business transactions in the world. Mainframe data sources also typically store higher volumes of data, further increasing the need for a more efficient approach to moving data such as change data capture. In addition, popular mainframe data sources such as VSAM, which are non-relational, present additional challenges when incorporating that data into BI solutions. As ETL and DW tools expect relational data, the non-relational data needs to somehow be mapped to a relational data model.

Seamless integration with ETL tools. When choosing a standalone CDC solution, enterprises should consider the ease of interoperability with its ETL program (off-the-shelf or homegrown). Standard interfaces and plug-ins can reduce risk and speed the data integration project.

Change data capture allows organization to deliver real-time business intelligence based on timely data while, at the same time, reducing the cost of data integration.

For organizations looking for ways to meet these demanding business needs, create an event-driven enterprise and provide real-time business intelligence, change data capture is a key component in the data integration architecture.

More and more organizations have started adopting change data capture solution. CDC has become integral part of data integration architecture.

Sunday, August 9, 2009

Applications of Text Analytics

In recent years, market research conducted by various software vendors and consulting firms has attempted to quantify the relative percentage split between structured and unstructured data in the average user organization. Most estimates name unstructured data the unqualified winner at 80–85%, leaving structured data in a distant second place at 15–20%.

The reality is that every hour of every day, directly and indirectly, customers place calls (that are transcribed), send direct emails, complete surveys and talk among themselves online in blogs, forums and social networks. They share their thoughts about products and services, their likes and dislikes, and their hopes for future features. Customers tell companies about product failures. They request help. They offer opinions about their experiences that contain insights for organizations that listen. This data is extremely valuable to customer-facing organizations as it’s in the form of first-person narrative – accounts from a single customer referring to himself or herself explicitly using words such as “I,” “me” or “we” that typically provide detailed opinions, issues, thoughts and sentiment about products and services, requirements and ideas.I have seen very few organizations who have started leveraging this information for decision making

Unstructured information can be used to answer the following questions effectively

  • Is our product launch going well?
  • Is there an emerging product issue?
  • Where should the product team focus its development dollars?
  • Is someone committing fraud?
  • Is my customer happy? Can I sell more to the same customer?
  • Is my customer unhappy? Will he stop using our services?
  • Is there a product defect in the market?

There are several applications of text analytics in various industries.

Sentiment Analysis:

One of the large multinational bank is using text analytics on a daily basis to review customer service requests, complaints and sentiment shifts. The bank's ability to retain and grow current customers is directly correlated to understanding and acting on sentiment shifts and their respective root causes. With text analytics, they monitor opinions and attitudes in order to determine where and how to spend on client initiatives. Text analytics gave them the ability to make decisions regarding expenses on marketing materials and viability of their online offerings. They were also able to produce accurate insights about customer preferences and indicators for what prompts customers to spend more.Each day, using text analytics to analyze customer emails, complaints and call agent notes, the organization looks for answers to questions such as:

  • Did customers like our new product?
  • What was their biggest issue?
  • Are my loyal customers angry about something?
  • Are new customers asking questions that might pose an opportunity for up-sell?
  • Is my most profitable customer unhappy about something?

Early Warning

One consumer electronics manufacturer uses text analytics to uncover product issues early, before they turn into expensive problems. When products are high-priced and marketed as the preeminent option, customers expect not only good quality, but also rapid and competent service when something goes wrong. To meet that expectation, customer loyalty managers at this company set up automatic alerts through their text analytics engine so they would know immediately when new product issues occurred. Once identified, proactive measures are taken to mitigate the issue and customer satisfaction is monitored and acted on. In one example, a product defect was found before the product came out of limited release, giving the company time to fix the issue and greatly reduce potential recall costs, not to mention customer satisfaction issues.

Call Center Optimization

A large cell phone carrier uses text analytics to stay on top of customer issues as they are being discussed online in web forums and blogs. In doing so, they’re able to leverage that knowledge to prepare their call center, proactively handle the customer issues, and possibly even deflect calls.In one instance, this company found a serious issue being discussed in web forums two weeks prior to it actually emerging in inbound calls and chats. Once the issue was identified (on a product that was released that same week), the call center took immediate action, posting remedies in an online FAQ, routing customers to agents who had been trained to handle the specific issue, and even proactively notifying customers about the problem. The company noticed a marked increase in customer satisfaction for the customers involved in this early action, which mitigated both a potential public relations problem and an influx of hard-to-manage inbound calls.

One of the large multinational bank is using text analytics to sort and route web customerservice request forms. This saves huge amount of manual efforts involved in manually routing the service requests. 30-40% of web customer service requests are routed to wrong customer service group by call center auto routing systems due to incorrect subject line. This leads to delay in response to customer. This bank has started using text analytics to auto route emails based in text in body of email & web forms.

Launch Monitoring

An industry-leading mobile phone manufacturer uses text analytics to keep a sharp eye on customer sentiment and any potential issues with products they release into the market. As new product introductions in the cell phone industry are frequent and expensive, and cell phones are some of the most discussed consumer electronic products on the Web, the company is committed to listening to, understanding and acting on feedback. Text analytics enables the manufacturer to identify issues early, improve quality, and increase customer satisfaction with each new product. In one example, this company identified a software flaw with a newly introduced phone within the first 24 hours of the product’s release. Discussion about the issue immediately hit online community forums and their text analytics engine discovered and summarized all of the data. The company was able to take immediate action: sending emails to customers with the solution, fixing new products in the queue for shipment, putting an FAQ on their site and notifying their partner carrier to fix new products sold. These steps turned a potential launch failure into a remarkable success.

Product Innovation and Quality

The largest appliance manufacturer in the world uses text analytics daily across its customer service, marketing, quality, engineering and development groups to identify product quality issues and to uncover new opportunities for innovation. This manufacturer uses the insights and ideas derived from customer feedback to drive product innovation.The company has also experienced “hundreds of millions of dollars” in cost savings resulting from early warning on issues. Had text analytics not identified some of these issues, immediate attention would not have been possible. The company has greatly benefited from the ability to understand the root cause behind product issues and respond quickly to manufacturing defects, as well as customer interactions and repair situations rather than having to react via expensive recalls.

Market Research Analysis

Do you regularly survey your customers? If the answer is yes, then you are among the best companies out there. But the real question is, do you take full advantage of that valuable research and augment it with what customers are telling you through “unaided” interactions? A large consumer high-tech company does this every day using text analytics. They go beyond scores and analyze the verbatims in their market research to get to the “why” behind their scores. Now they know what action to take. One of the things the company recently discovered using text analytics was a large disparity between scores and verbatims. Although customers reported that agents were courteous and provided good service, they explained in the verbatim that the issue wasn’t with the agent being nice, “I just couldn’t understand them.” In fact, the company found that for certain problem types their outsourced call center got good scores, but were actually generating call backs because language barriers prevented the agents from resolving the problem. For those call types, the company re-routed the calls to agents in a different locale and with different skills and were able to measure a material increase in their scores. Without the verbatim analysis the company wouldn’t have known what to do.

Competitive Analysis

One major airline uses text analytics not only to understand exactly why their customers are loyal and some are not, but to garner knowledge about their competitors as well. In an industry where fixed costs have risen dramatically and competitive data is transparent, staying on top of customers and their opinions is paramount. The airline analyzes survey responses and call center notes, but they also “harvest” the Internet for customer conversations about themselves and the competition – topics include everything from services, issues, products and prices to specific customer desires. In doing so, the airline is able to make better decisions such as where to invest to beat the competition, what marketing messages will resonate with customers, and what specific competitive differentiators should be promoted. Such insights enable the airline to truly understand how it compares to its fierce competitors, but more importantly, how it will win!

Fraud Detection

There are several goverment agencies who are using text analytics to detect fraud and monitor terrorist movements. There are several voice to text converters are available in market today. These tools help govt agencies convert voice into text and mine the text to detect terrorist movements.