Sentiment Analytics: Next Wave in BI

Business spends huge sums shaping brand image and promoting brand awareness. To gauge the effectiveness of particular campaigns, brand managers will study transactions, for instance sales made in response to direct mail or using coupons, web-page visits and ad click-through, etc. But study of past transactions is of limited use in understanding potential buyers who are not responding to market messaging, in understanding competitive positioning and in picking up on nascent trends. Surveys and social-media mining, especially for attitudinal indicators, can fill the gap.

Recently, I was talking to the marketing executive of a software firm. He said "We are spending huge amount of money and efforts to promote brand awareness. But we can't measure effiectiveness of the same. We have to heavily rely on surveys. By the time we receive survey results, it's too late to take any corrective action." I am sure there are lot of companies in the market who have similar pain.

There are several tools available in market today which can help you measure sentiment of people about your services, product & brand.What do customers, reviewers, the business community – thought leaders and the public – think about your company and your company's products and services – and about your competitors? What can you learn that will help you improve design and quality, positioning, and messaging and also respond quickly to complaints?

Recently, we did a PoV for a bank in India. We have found out that people are not very happy with their branch banking services. Things that they didn't like about bank were long queues & inadequet parking space around bank's branch. When we further drilled down to investigate, we found out that long queues were primarily due to very high number of customers in that area compared to number of branches. The bank took corrective action immediately. They have shifted branch to a location where in there is ample parking space, and opened one more branch.

One of the company is using sentiment analytics to determine which executives have the highest correlation to positively moving the stock price when they deliver positive news. They found that certain executives had a positive influence on the markets, while others actually had a negative influence because of the tone of their delivery. This is very interesting. Stock market is often driven by sentiments of people. I have noticed one thing since Obama took over as president of America. Whenever Obama delivers a public speech, there is a negative sentiment and decline in indian stock exchange index(SENSEX).

One of the bank is using sentiment analytics to determine whether their customers are happy with their services and products or not. If a customer is happy with bank's service then they are passing that lead to marketing dept so that they can cross sell/up sell more products.

One of the consumer electronics company is using sentiment manager to analyze feedback about their newly launched product on review sites such as Amazon & CNET. They use this feedback as input to their product development lifecycle.

There are several such examples of sentiment analytics. I think this is going to be next wave in business intelligence space.Sentiment analytics can help customers to make more sense out of their Business Intelligence reports and KPIs. If your sales is going down in one region then is it because
1. Your sales team is not efficient or
2. Your supply chain is weak which results into frequent stock outs at dealer place/store or
3. Your brand is percieved as expensive or low quality goods or bad customer service.

Successful BI Strategy - Part II

A BI initiative is of no use if it is not driven by the objectives of the enterprise. Implementing a BI solution should help an enterprise achieve the objective of advancing business by making the best use of information.

Dos and Don'ts for successful BI/DWH project implementation:

Do not start with a big bang implementation approach. Iterative implementation approach works well with BI project. Identify a business objective and deliver it via BI/DWH within first two-three months. Longer you take to deliver your first output from BI/DWH, higher is the possibility of failure. It is very important to deliver first output from BI/DWH on time with good quality. This will also help in selling BI/DWH vision to business teams. The shorter implementation cycles would be quite beneficial for the end users as well in terms of cost and time as they would have a much better feel of the end product, they would be able to modify the scope based on what is implemented after each cycle.

Do not try to roll out BI/DWH to many departments/groups at a time in first phase. If possible choose either Sales or Finance department for first phase as these areas are more closer to heart of CEO/CFO of the organization. It is easier to gain acceptability of an initiative if it has C-level executives acceptability & support.

Do not over burden end users with lot of trainings initially. If end users have to go through multiple days of trainings to use new BI/DWH system then there is a high probability that they will not use the system. Always look at maturity of a user group before delivering reports to them. If a user group has been using excel based static reports for past few years then give them reports which has drill down and parameter selection criteria. If someone has been using parameters based reports then give them OLAP based reports which will allow them to slice & dice the data on the fly. If someone has been using OLAP based reports then give them access to adhoc reporting tool.These will help in reducing training efforts that are required to use new BI/DWH system.Also, it makes transition to the new system easier and smooth. Lot of time static report users are given access to OLAP cubes which requires huge training efforts and time.Also, it requires steep learning curve, and it often demotivates them from using new BI/DWH system. Do not drastically change the way they are consuming information now. The change has to be gradual.

Successful BI Strategy - Part I

We often run into situations where major companies ask us to help develop a BI strategy. When we ask companies about the objective of implementing BI solution, we hear the following statements quite often

• “…produce enhanced organizational capabilities to manage data and information as organizational assets.”
• “…provide a single version of the truth.”
• “…enable consistent and reliable access to accurate corporate-wide data.”
• “…provide more sophisticated reporting and analysis, faster turnaround, improved accessibility and enhanced quality.”
• “…a single touch point where detailed financial transaction information can be filtered on user-entered selection criteria, viewed online, downloaded in standard file formats and used to generate real time reports.”

These objectives doesn't excite business executives and managers as it doesn't articulate how business intelligence will be used within specific business processes to improve business performance.As a result, they underfund business intelligence, which limits its business impact. Very recently, we have faced a situation at one of the large organizations where in Business teams refused fund BI project. They didn't see any compelling reason & business case to implement BI solution.

Lot of time, organizations also get into functional requirements such as the following during BI product evaluation cycle

• The system shall provide the ability to drill down, drill across, and slice-and-dice.
• The system shall provide the ability to specify organizational hierarchies and display performance scorecards for each organizational unit.
• The system shall enable role-based access to information.
• The system shall provide capabilities to route alerts to business users according to user-defined parameters.
• The system shall enable integration of data from multiple disparate sources.

BI functional requirements like those listed above are standard features of commercially available BI tools.While it is important to know what your company needs BI tools to do, BI functional requirements typically say little about the kinds of business information, analytical techniques and decision support that are required or the specific core business processes that the company seeks to improve via business intelligence.

As per Gatrner Report "Fatal Flaws in BI Implementation", it is very important to get buy in & active participation from business teams for a BI project to be successful. This requires a clear linkage between business strategies, the core business processes via which the strategies are executed, and BI-driven business improvement opportunities, which is the basis for a BI business case that is compelling to the business stakeholders.

Some examples of compelling BI system objectives can be as below,

• " BI system will help reduce transportation cost by 5%".
• "BI system will help reduce cash out situations at ATMs to less than 3".
• "BI System will help reduce idle cash in ATMs by 40%"
• "BI System will help increase private label sell by 5% in 80% of retail outlets".
• "BI System will help reduce stock out situations to less than 2 per outlet for premium or fast moving items".
• "BI System will help increase share of wallet by 10%".

Each of the above BI objectives are linked to a business process. It clearly tells business team how implementation of BI system can help them achieve their goals. It becomes easier to get buy in from business executives and managers when BI is directly linked to business process improvement.

When you develop a BI strategy, do not look at point solutions like reporting, data integration etc..It's always recommended to look at a business analytics framework which can help you improve your business processes and achieve your business goals. The framework will in turn comprises of set of solutions which can help you address your business problems.  Point solutions like reporting, data integration etc will help you gain short term benefits but it will not help you gain long term benefits & business support. 

Fatal Flaws in Business Intelligence Implementations

Lot of organizations, assumes that business intelligence(BI) projects are like any other project, are often surprised when their BI project spins out of control. The requirements appear to be a “moving target;” the schedule keeps slipping; the source data is much dirtier than expected and is impacting the ETL team; the staff does not have the necessary skills and is not properly trained; communication between staff members takes too long; traditional roles and responsibilities, and how they are assigned, seem to result in too much rework; the traditional methodology does not seem to work; and so on.

BI Projects are often political in nature as lot of people do not like when their performance is being tracked by their management. This requires culture change & creating awareness about benefits of BI within end user community. BI Project should be seen as business enabler rather than a performance tracking tool. They should use BI system to meet or exceed their KPIs.

I have been thinking about writing on this topic for a long time but then I came across a nice research paper on this topic from Gartner. I have shared below the details of Gartner report as is. I have personally experienced and seen some of the flaws mentioned below in lot of BI projects very recently.

Most failed business intelligence (BI) efforts suffer from one or more of nine fatal flaws, generally revolving around people and processes rather than technology, according to Gartner, Inc.

Gartner said the failure to achieve strategic results usually stems from one or more of nine common mistakes:

Flaw No. 1: Believing that “If you build it, they will come”

Often the IT organisation sponsors, funds and leads its BI initiatives from a technical, data-centric perspective. The danger with this approach is that its value is not obvious to the business, and so all the hard work does not result in massive adoption by business users — with the worst case being that more staff are involved in building a data warehouse than use it regularly.

Gartner recommends that the project team include significant representation from the business side. In addition, organisations should establish a BI competency centre (BICC) to drive adoption of BI in the business, as well as to gather the business, technology and communication skills required for successful BI initiatives.

Flaw No. 2: Managers “dancing with the numbers

Many companies are locked into an “Excel culture” in which users extract data from internal systems, load it to spreadsheets and perform their own calculations without sharing them companywide. The result of these multiple, competing frames of reference is confusion and even risk from unmanaged and unsecured data held locally by individuals on their PCs.

BI project instigators should seek business sponsors who believe in a transparent, fact-based approach to management and have the strength to cut through political barriers and change culture.

Flaw No. 3: “Data quality problem? What data quality problem”

Data quality issues are almost ubiquitous and the impact on BI is significant — people won’t use BI applications that are founded on irrelevant, incomplete or questionable data.

To avoid this, firms should establish a process or set of automated controls to identify data quality issues in incoming data and block low-quality data from entering the data warehouse or BI platform.

Flaw No. 4: “Evaluate other BI platforms? Why bother”

“One-stop shopping” or buying a BI platform from the standard corporate resource application vendor doesn’t necessarily lower the total cost of ownership or deliver the best fit for an organisation’s needs.

BI platforms are not commodities and all do not yet deliver all functions to the same level, so organisations should evaluate competitive offerings, rather than blindly taking the path of least resistance.

Integration between the application vendor’s ERP/data warehouse and BI offerings is not a compelling reason for ignoring alternatives, especially as many third-party BI platforms are as well integrated.

Flaw No. 5: “It’s perfect as it is. Don’t ever change “

Many organisations treat BI as a series of discrete (often departmental) projects, focused on delivering a fixed set of requirements. However, BI is a moving target — during the first year of any BI implementation, users typically request changes to suit their needs better or to improve underlying business processes. These changes can affect 35 per cent to 50 per cent of the application’s functions.

Organisations should therefore define a review process that manages obsolescence and replacement within the BI portfolio.

Flaw No. 6: “Let’s just outsource the whole darn BI thing”

Managers often try to fix struggling BI efforts by hiring an outsourcer that they expect will do a better job at a lower cost. Focusing too much on costs and development time often results in inflexible, poorly architected systems.

Organisations should outsource only what is not a core competency or business and rely on outsourcing only temporarily while they build skills within their own IT organisation.

Flaw No. 7: “Just give me a dashboard. Now”

Many companies press their IT organisations to buy or build dashboards quickly and with a small budget. Managers don’t want to fund expensive BI tools or information management initiatives that they perceive as lengthy and risky. Many of the dashboards delivered are of very little value because they are silo-specific and not founded on a connection to corporate objectives.

Gartner recommends that IT organisations make reports as pictorial as possible — for example, by including charting and visualisation — to forestall demands for dashboards, while including dashboarding and more-complex visualisation tools in the BI adoption strategy.

Flaw No. 8: “X + Y = Z, doesn’t it”

A BI initiative aims to create a “single version of the truth” but many organisations haven’t even agreed on the definition of fundamentals, such as “revenue” Achieving one version of the truth requires cross-departmental agreement on how business entities (customers, products, key performance indicators, metrics and so on) are defined.

Many organisations end up creating siloed BI implementations that perpetuate the disparate definitions of their current systems. IT organisations should start with their current master data definitions and performance metrics to ensure that BI initiatives have some consistency with existing vocabulary, and publicise these “standards”.

Flaw No. 9: “BI strategy? No thanks, we’ll just follow our noses”

The final and biggest flaw is the lack of a documented BI strategy, or the use of a poorly developed or implemented one. Gartner recommends creating a team tasked with writing or revising a BI strategy document, with members drawn from the IT organisation and the business, under the auspices of a BICC or similar entity.

“Simple departmental BI projects that pay an immediate return on investment can mean narrow projects that don’t adapt to changing requirements and that hinder the creation of companywide BI strategies,” said James Richardson, research director at Gartner.

Link to Gartner report:
http://www.gartner.com/it/page.jsp?id=774912

Intelligent Operational System

I just received an automated "telemarketing" SMS on my cell phone. Big deal and who cares, right? Well it isn't a big deal, and its doubtful anyone cares. But it did bring to mind an interesting reminder about the Intelligent Operational system when designing a customer experience.

The SMS was an automated SMS from Crossword. Those who do not know about Crossword, Crossword is a big book store chain in India. They didn't SMS me to sell anything. Instead, they were sending SMS to help me, which ultimately helps them.

The SMS was from the Crossword "Book Rewards" program. If you are not familiar with the Book Rewards Program, it is Crossword's customer loyalty program, where you scan a card at the point of sale and a percentage of your purchase counts towards an in-store credit or a gift voucher. Crossword mails you a gift voucher for the credit, and the credit can be used in the store.

The SMS was to alert me that my gift voucher, which I had forgotten about, would expire soon. It provided me with the details of my gift voucher, such as how much it was, and when it would expire. The information was provided a month ahead of the expiration date, which would allow me time to get a replacement voucher if I didn't receive the original, or provide me with ample time to schedule a trip or research a purchase. I wasn't thinking about going to Crossword in the near future, but I was considering buying a new book from Oxford Book Store which is located near our office. Now that I've been reminded that I have credit at Crossword, I'll buy it there.

It reminds us that we need to think about the "systems" in which our customers reside. These "systems" include technology, work, and social context components, and the interaction of these components provide opportunities, or limitations which we may not have considered. The automated SMS from Crossword provides a good example of the application of intelligent operational system, since it leveraged the capabilities of automated computer and telephony systems to reach into my "system" and gently provide a socially acceptable message indicating that "we miss your business, so please shop with us soon."

While designing applications, seriously consider the user's system, and the opportunities and limitations provided by the system. By applying these considerations to our designs, we can make applications which will ultimately be more helpful for our customers, as well as easier to use.

Information Overload & BI

In these difficult times we live in, when resources seem scarce, there is still one thing that is widely and abundantly available: information. According to the most recent statistics, the amount of information created annually by businesses and organizations, paper and digital combined, is growing at a rate of more than 65%. The amount of digital information being created in the world and distributed in emails, instant messages, blog posts, new Web pages, digital phone calls, podcasts and so on, will increase 10-fold over the next five years. The one fact that stands out is this: The growth of information is relentless.

There is too much of infornation available in various forms. Is it information Overload? or is it failure of Information filter? There is so much of information out there that one can't browse through every possible bit of information. Business Intelligence systems can play a very important role here. It can act as a information filter. It can provide information which is very critical and must need your attention. In today's world when someone have hundred's of KPIs to monitor, BI system can help to identify only those KPIs which needs immediate attention.One can start his day with BI portal. Typically, one follows the following routine

• Check Emails
• Check Calendar(Meeting Schedule)
• Check Important News/stocks
• Check Most critical KPIs
• Prepare To-Do List for a day
• Prepare/View status reports
• Collabrate with collegues using Enterprise messenger

BI Portal can integrate all of the above information and show them on single UI. One need not to log into 5 different systems to perform the above tasks. Business Users will start their day with Business Intelligence system. Business users will also be able to relate some of emails/news with status of most critical KPIs. BI system can be tightly integrated with content categorization tool which can ensure that only relavent information is delivered to users as per their role and choice. Content categorization tool can also categorized documents/news as per meaning of the document/news items. This can help users to weed out information which is not useful. It saves users a lot of time from browsing through every possible infomation.

Analytical MDM Vs Operational MDM

I was having interesting coversation about MDM with one of my customer last week. It's mid sized bank and they are in process of evaluating MDM. When i asked him  " How confident are you about quality of your data?" He said "Honestly, I do not know". Then i told him that for MDM one of prerequisite is to have good quality of data. If quality of your data is not good then MDM solution implementation is bound to fail.

I have seen quite a few organizations who would like to embark on MDM initiative without having good data quality system in place. Thanks to huge amount of marketing money spent by some of large IT product vendors. Quite often organizations fall in this trap and end up investing hugh amount of money and efforts.

There are two types of MDM solutions in market. Operational MDM and Analytical MDM.

Operational MDM is used to collect customer information at front desk. This solution is used to standardize the mechanism to capture customer information at various customer touch points in organization. Typically, organizations have 5-20 customer touch points. This solution provides customer information to various operational systems in organization.It ensures that any changes made in customer information at any of customer touch point are transferred to all operational systems. It will work in organizations which are still in process of implementing operational systems and have very few customer touch points. This approach requires discipline, and huge amount of training efforts. Currently most of banks in India have operational systems in place. These operational systems are built using old technology and captures customer information specific to their application.Enhancements to these oprational systems are very time consuming and lead to performance issues. Hence this solution will not be suitable for most of the large and mid sized organizations. It also requires huge amount of training efforts to train front desk staff on this solution.

Analytical MDM is used for historical and predictive analysis. This solution sources the data from transactional systems such as CRM, ERP, CBS, LOS etc...Analytical CRM can be updated once in a day or multiple times in a day. I have seen banks updating it once a day which is more than sufficient to cater to their current business requirements. This solution doesn't require retraining of front desk staff. However, it requires tight integration with transactional systems. Analytical MDM should be SOA enabled. This will enable source system to call web service and check whether new customer is already customer of bank or not. Analytical MDM will also provide information related to class of customer(Preffered, Gold etc) and behaviour based on past transactions. This will help to take decision about loan approval or issuing credit card or giving prefferential service to your customer.

The way Analytical MDM & operational MDM store the data is also different. Analytical MDM stores data in denormalized format so that it can be retrieved easily for analysis  whereas Operational MDM stores the data in normalized format so that it can be updated quickly.

Operational MDM stores demograhic details such as age, birthdate, name, address whereas Analytical MDM stores information related to profibility, behaviour score, credit score and propensity to buy product apart from demographic details of a customer.

Both types of MDM solutions require strong data quality engine in backend. This data quality engine should be capable enough to address peculiarities in Indian addresses and names.

Task Based Intelligence

I was working on "Task Based Intelligence" concept few years ago. The idea was to integrate BI with operational systems. E.g when someone is creating a purchase order in ERP system, he will be able to see scorecard of a supplier without going to a seperate interface or application. The supplier scorecard is embedded into PO application. This will not only prevent PO going to a black listed supplier, but also gives flexibility to users to select supplier based on priority at that point in time. The supplier can be selected based on score which is determined based on the various parameters such as lead time, On time delivery performance, price & quality of supply(Rejection Rate).

The same concept can be applicable to banking industry as well. While granting a loan or credit card to an individual, the bank officer will be able to see application and behaviour score on LOS system. This will help him take more informed decision.

Discount coupons can be printed @ATM machines based on the amount withdrawn from ATM at that point in time. Just imagine a scenario wherein a discount coupon for a digital camera is printed @ATM machine when 10000 Rs is withdrawn from ATM. Competition is increasing day by day in every industry vertical. Margin is going down day by day. I won't be surprised if the bank starts selling cricket match ticket or flight tickets in near future to share the cost of infrastructure and therefore increase profitability of each branch & ATM.

In retail, discount coupons can be printed based on items bought by customer at that point in time. This will not only increase customer satisfaction but also ensure that the customer returns to store for more purchases in near future. Customer loyalty program is in very nascent stage in India. Hence such intelligence embedded into operational system will definitely help retailer to increase revenue per customer.

Green Business Intelligence

Green Business Intelligence is a new buzz word in BI world nowadays. More than one third of Gartner Survey respondents plan on spending more than 15% of their IT dollars on Green IT projects. Most of these projects fall into the "improve energy efficiency" category for short-term, immediate cost savings.

So, with 15% of IT dollars going toward green projects, can BI initiatives be a part of that? Absolutely. Key to the success of green projects is measuring and monitoring. If a project claims to reduce energy usage, that usage must be measured before the project begins and monitored afterwards through the payback period (and potentially beyond). Web based BI systems can help move companies to a paperless environment.

Reducing power consumptions in servers is one way to contribute to green initiatives of a company.BI Systems can help companies to measure and monitor usage of hardware resources(CPU, RAM, Network, Storage Etc..). It can forecast hardware resource requirements based on historical data & events. This will help companies to consolidate hardware resources & there by reducing power consumption.One of pre-requisite for such BI system is to have centralized IT data mart which can collect & store data from various performance & monitoring tools like CA Unicenter or Tivoli or Open View.

Inventory management is perhaps the most important step where BI can improve efficiency. Not having the right product in store will lead to lost sales and unhappy shoppers or excess inventory. Either is bad for the environment. When consumers don’t find what they are looking for, they make additional trips, increasing the carbon footprint of their shopping. Excess inventory leads to waste (especially in the case of perishable products), affecting both the environment and margins. Most progressive retailers have implemented perpetual inventory systems to keep track of what is on the shelf, and advanced replenishment systems to forecast demand and generate orders based on past trends and current factors. While these systems have helped run a more efficient operation, they are not perfect. Business intelligence can help retailers get smart by analysing issues in forecasting, understanding their root causes and preventing future exceptions. Accurate inventory visibility in store is a key input for upstream operations such as manufacturing and distribution, to reduce waste, cost and carbon footprint.

In the near future green reporting is going to be as mandatory as any other financial reporting. When tight controls are in place, people discover and reinvent more creative and efficient ways to save money. There is no better time than now to take action and allocate a piece of the budget toward serious and productive green IT. Environmental issues will shape the information management landscape for decades to come, affecting areas like data management and data governance. It also will have significant impact on areas such as competitive strategy, business intelligence marketing and even a company’s ability to attract and retain people.

Application Data Warehouse

The definition of data warehouse is changing in Indian Market. Earlier people use to build data warehouse to cater to MIS reporting need of an organization nowadays data warehouse is built to support various business applications such as

• Cross Sell/Up Sell
• Retention
• Campaign Management
• Marketing Optimization
• Market Mix Modeling
• Basel II compliance
• Market Risk Analysis
• Op Risk Analysis
• Warranty Analysis
• Supply Chain Optimization etc...

I started my BI implementation career with a data warehouse implementation at one of the media company in India.  The objective of data warehouse project was to replace all excel based MIS reporting with automated reporting system. The sponsor of data warehouse project was IT director.  We designed our data warehouse schema based on reporting requirements of different business functions within the organization. It took 10 months to build a warehouse. We delivered some 25 odd reports using Business Objects. While the data warehouse project was appreciated well within IT department, end users didn't appreciated it. They felt  data warehouse is too rigid. They can't make changes in data warehouse easily. It used to take 4-6 weeks to include any new business requirement change in data warehouse. They also felt that data warehouse was not giving any value add or insight which can help them in their day to day activity. After a year, data warehouse project was scrapped by that organization because the ROI generated from data warehouse was not enough to justify it's investment.


Today scenario has changed. Very recently, we had worked on two enterprise data warehouse RFPs wherein the end goal of implementing data warehouse was to support various business applications. Prospect had clearly stated objective of data warehouse in RFP. They wanted to built a data warehouse to support the following business applications

• Basel II Compliance
• Market Risk
• Credit Scoring
• Cross Sell/Up Sell
• Retention
• Campaign Management 

They wanted to ensure that all variables that are required to do say e.g cross sell/Up sell analysis are included in data warehouse model. There are some 750+ variables to do only cross sell/up sell analysis. Similarly, there are thousands of variables available to support other applications. Lot of time the data warehouse is built keeping in mind only MIS reporting requirements. Hence whenever business users want to do business analysis, they end up creating a seperate mart for data specific to that analysis. This results in data duplication and system overhead. Last week I met up with a senior executive of one of the large bank. Currently they have three data marts. One data mart caters to MIS reporting requirement, second data mart caters to Risk compliance requirements and  third data mart caters to PM requirements. Soon, they are coming up with a RFP to consolidate all three data marts into a single data warehouse.


In today's economic conditions, it is very critical to build "Analytics" friendly data warehouse. Typically, you require historical data to do analytics. Hence you need to capture data related to analytical variables right from day one when data warehouse is implemented. I have seen lot of organization making a mistake of building MIS reporting data warehouse. There are several disadvantages of this approach.

1. It takes significant amount of time & efforts to build such data warehouse. Your reporting requirements change by the time data warehouse is implemented.
2. ROI generated from such data warehouse is not significant enough to justify it's investment.
3. If right variables are not captured in the data model then it takes significant amount of time and efforts to incorporate them in data model at later stage. It involves change in data model, ETL & BI Strategies. Lot of time, it is not possible to incorporate such changes due to complexity of data model and ETL routines, and you end up creating a data mart to cater to Analytical requirements. This results in data de-duplication.

In today's world, it is not just sufficient to know who is buying what & when. You will need to know what they will buy next, & whether they are profitable customer for you or not. This requires analytical capabilities built into your data warehouse. Hence "Application Data warehouse" is way to go.

New Products Forecasting

Yesterday Nokia officially launched X6 touch phone at Nokia world. This is a nice touch phone with features comparable to or better than iphone 3GS. It took them almost two years to release a phone which is comparable to or better than iphone. This phone is also very competitively priced (Rs 30,000). It's good to see that competition for Apple Iphone has finally arrived. Apple has been ignoring Indian market for long time now. Apple Iphone launch in India was a big failure due to very high pricing. India is suppose to be one of the largest mobile handset markets in the world. One can't afford to ignore them. I hope Apple learns from their past mistakes and launches Iphone 3GS at very competitive pricing in India.

Nokia comes out with a new product or model every month. Lot of time they launch a model which has overlapping features with existing model in market. I always wondered how they forecast inventory for their new phone.Typically, forecasting is done based on past data and events. There is no such data available for new products. Moreover, there are similar products available in market from the same manufacturer. Each of these products eat into each other's revenue. In high tech companies, typically there are 50-60% of mature or stable products, 35% of new products and 5% of "first-of-its-kind" products. How do you ensure that the new product doesn't impact sales of existing product in market?

Each product follows a particular life cycle. A new product launch should be decided in such a way that it doesn't cannablize revenue of other similar products in market. E,g Nokia X3 is a new music phone. Most probably, it will replace Nokia X5300 express music phone. Launch of Nokia X3 is planned when Nokia X5300 is reaching towards end of its life. If two models are going to co-exists than you need to ensure that messaging & target consumer audience is different for both the products. In case of Nokia, both Nokia N97 and N97 mini are going to co-exist. Both the products are meant for different consumer segment.

There are several new product forecasting techniques available. One of the most common one is a Bass diffusion technique. The bass diffusion technique requires 3 parameters

• Lifetime expected sales - the total amount of units sold in its product lifetime. Also known as market potential
• P(Mass media) - influenced by the technical aspects of a product that drives a consumer purchase. Also known as coefficient of innovation.
• Q(Word of Mouth) - Reflects the internal dynamics of the consumer. Also known as coefficient of imitation.

Typically, the following methodology is used for new product forecasting

• Find a cluster of like (similar) products. This is used to determine the historical data we can use from like products.
• Perform regression analysis on the cluster
• Use Bass diffusion model to determine the forecast of the new product
• Adjust or reforecast after we have some actual data

There are several high tech companies in world who are using sophisticated forecasting techniques to ensure that there is increase in profitability, market share & reduction in excess inventory.

I am ardent fan of Apple iphone. I personally believe that competition is always good for end consumer. With Nokia X6 launch, Apple will not be complacent. It will accelerate innovation at both the places. Finally end consumers like me will get benefited.

You can find more details about Nokia X6@
http://www.mobilenewshome.com/2009/09/nokia-launches-mobile-cum-music-device.html

Nokia Phone Comparision
http://europe.nokia.com/find-products/phone-comparison

Does BI require business process re-engineering?

I met up with a senior executive of a large firm last week. This firm is in process of rolling out BI intiative enterprise wide. He asked us a question about changes required in their business processes to ensure sucessful roll out of BI initiative. This question took all of us by surprise. We all were ready with our answer of how BI can help them increase revenue, profitability and reduce cost. This type of question shows maturity of an organization to adopt technology like BI. After the meeting, I didn't have any doubt what so ever about readiness of this organization to roll out enterprise wide BI Initiative.

Today many BI initiatives fail as they do not systematically address the business process changes required to capture business value of BI. It is common for BI vendor value propositions to emphasize business benefits such as profitability, responsiveness, customer intimacy, information sharing, flexibility, and collaboration. But investing in BI to achieve such business benefits may actually destroy business value unless those attributes can be defined in operational terms and realized through business processes that affect revenues or costs.

Many companies use BI to improve customer segmentation, customer acquisition and customer retention. These improvements can be linked to reduced customer acquisitions costs, increased revenue, and increased customer lifetime value, which translate to increase in profitability. However, a BI investment that improves demand forecasting will not deliver business value unless the forecasts are actually incorporated into operational business processes that then deliver reduced inventory or some other tangible economic benefit. In other word, the business benefit "Improved forecasting" is useless unless it is somehow converted into incremental after-tax cash flow.

To capture the business value of BI requires organizations to go well beyond the technical implementation of a BI environment. Specifically, organizations must engage in effective process engineering and change management in order to capture business value from BI.

BI systems delivers lot of useful information such as most profitable customers of your organization, fast & slow moving inventories, Good & bad suppliers, product cost & profitability. If you do not integrate this information with your management processes and operational processes such as ERP & CRM then you will not derive any value out of your investment in BI systems.

Process engineering is very essential for successful BI roll out. Process engineering identifies how BI applications will be used within the context of key management and operational processes that drive increased revenue and/or reduce cost. It provides a map of which processes must change and how they must change in order to create business value with BI applications. Thus, it lays the foundation for change management because process changes drive changes in individual and organizational behavior.

BI as Business Enabler

I have started my sales/pre-sales journey with SAS about 3 1/2 year ago. My job is 100% customer facing. I have seen quite a few changes, in a way, people have started adopting business intelligence technology in past few years.

My first assignment at SAS was that of data quality. At that point in time there was not much awareness about data quality solutions in Indian market. It took almost a year to convince customer to invest in data quality solution. There were very few data quality solutions available in Indian market at that time. Today, when I look in market, there are plenty of data quality software vendors. Data quality has become integral part of data governance & compliance strategy for most of the organizations. We have worked on more than 5 enterprise data warehouse RFPs in last 3 months. All of them had included requirements for data quality. Data quality is not only used for basic data cleansing but it is also used for doing de-dup, house hold analysis and cross sell/up sell.

One of the major banks is using data quality to ensure that they give loan to right people and thereby ensuring low NPAs. One of the CPG company is using data quality to create single view of retail outlets across various product categories. This helps them in optimizing sales force & increasing cross sell/Up Sell opportunities within same outlet for different products. One of the manufacturing company is using data quality to create single view of customer across various business functions. This helps them to cross sell/up sell products across various business functions.

I have also seen enormous change in a way people are using business intelligence, now compared with what it was, four years ago. Earlier BI was used by very few people in organization. That's the reason some BI vendors are offering per-user licensing. Today BI reporting access is given to all users at all levels for decision making. User based licensing makes no sense in today's environment. Most of the organizations are now moving towards establishing enterprise-wide reporting framework. They are also standardizing on their reporting platform across various departments. Need for reporting user interface is also changing from one user group to another user group. Customers are expecting different user interfaces for users having different skill sets. There are some users , who are more comfortable with excel, requires excel interface. There are some users, who are more comfortable with web interface, requires web interface for reports. Earlier, reports used to get refreshed every month or week. Today customers are expecting to refresh reports multiple times in a day or atleast once in a day. In past few months I have seen several requirements for real time dashboarding & OLAP analysis.

One of the large private limited bank is using real time dashboards to monitor cash level & down time of each ATM. BI system sends out an alert to respective regional manager when cash drops below certain level or ATM is down for long period. This has reduced cash out situations drastically across all ATMs in country. One of the large manufacturing company is using BI reporting to measure supplier performance. They monitor quality and quantity of s upply of each supplier using this system. They also compare pricing from different supplier for thesame material and quantity. They use this information to better neogotiate material pricing with their suppliers. One of the retailer is using real time dashboard to monitor their fast moving items at store level and thereby reducing stock out situations.

Today customers are considering BI as a business enabler. Their expectations from BI systems have increased in last few years. They are looking for BI solutions which offer capabilities beyond Querying and reporting. They are looking for solutions which can help them forecast, predict and optimize. Lots of companies have started mentioning about data warehouse initiative as a strategic initiative in their annual reports. They believe that the implementation of the BI system will help them increase their sales and profitability. This represents a large opportunity for BI vendors. Hence we are seeing lots of consolidation in BI market.

Customer Oriented Banking

I have had a bad experience with customer service of one of the large private bank last week. My netbanking id was locked as I didn't use it for past more than 3 months. When I approached their phone banking support to get it unlocked, they advised me to raise support ticket via web service request form. I did the same promptly as I wanted to transfer funds urgently to my some other accounts. They took 5 days to unlock my id. I missed my deadline to transfer the fund.

I am a loyal customer of this bank for more than 4 years now. I am also one of their premium customer and hold more than 4 products from the same bank. However when I submitted support request, they treated my request on the same priority as that of a non premium customer. I was expecting quick turnaround and differentiated service level, being a premium customer of bank. It didn't happen. Today I have decided to move to other bank. I am sure this bank must be losing lots of such customer in a year. This bank is known to be one of the most technology savvy bank in India, but unfortunately they do not use any system which helps them differentiate their premium customers from non premium customers. There are several advanced analytics solutions available in market which can help bank address such problems/challenges proactively. I have seen banks spending lots efforts and money on marketing campaigns to retain their premium customers when they are leaving. But if they invest in analytics systems then they can save lots of money on such marketing spend.

Similarly where a new loan to a non premium customer is given in say 3-4 days, can the same be given to a premium customer in just 1-2 days ? I had to wait for 4 working days to avail an Auto Loan last year through the same bank where I was a premium customer.

Some years ago banks were focussing on acquiring customers and quality of customer was not given lot of importance. In today's economic condition, you need to improve quality of new customers. You would like to increase wallet share per customer and at the same time you want to retain only those customers which are profitable to your organizations. Analytical solutions can help address all of such requirements. To survive in this competitive market conditions, we need to address the following questions pro-actively

1. Am I acquiring good customers?

2. Am I spending my marketing budget in right direction to increase wallet share per customer?

3. Do I know my most profitable customers? What am I doing to retain them?

4. Do I know my non profitable customers? What am I doing to terminate them?

5. Can I expediate the process of giving loans to the most profitable customers ?

Typically in banking when you submit any web request, it goes into queue. CRM system automatically routes your request basis information available in subject line of your request. Lots of time subject line is incorrect and that results in routing of a request to wrong customer support group. The request gets re-assigned to right group and then they start working on it. This is very tedious and time consuming process. Lot of precious time is lost in a assigning request to right support group. This can be avoided by implementing text analytics solutions. Text analytics solution can help you route the request based on text in the web form. If customer id is available in web form then it can also identify whether a particular customer is premium customer or non premium customer. Accordingly it can route it to queue dedicated to the premium customers.

Change Data Capture - Real Time BI

I have heard need for change data capture from several customers in past few months.

Traditionally, ETL processes have been run periodically, on a monthly or weekly basis, and use a bulk approach that moves and integrates the entire data set from the operational source systems to the target data warehouse. Now, data integration requirements have changed. Customers would like to move the changes made to enterprise data while the operational systems are running, without the need for a downtime window. They do not want to degrade performance and service levels of their operation systems.

Business conditions have changed over a period of time. It requires a new way of integrating data in real time and efficient manner.

1. Business globalization and 24x7 operations. In the past, enterprises could stop online systems during the night or weekend, to provide a window of time for running bulk ETL processes. Today, running a global business with 24x7 operations means smaller or no downtime windows.
2. Need for up-to-date, current data. In today’s competitive environment & competitive pressure, organization cannot afford to have their managers’ work on last week or yesterday’s data. Today, decision-makers need data that is updated a few times a day or even in real time.
3. Data volumes are increasing. Data is doubling every 9 months. The larger the data volumes become, the more resources and time are required by the ETL processes. This trend challenges the bulk extract windows that are getting smaller and smaller.
4. Cost reduction. Bulk ETL operations are costly and inefficient, as they require more processing power, more memory and more network bandwidth. In addition, as bulk ETL processes run for long periods of time, they also require more administration and IT resources to manage.

The first step in change data capture is detecting the changes! There are four main ways to detect changes:

• Audit columns. In most cases, the source system contains audit columns. Audit columns are appended to the end of each table to store the date and time a record was added or modified. Audit columns are usually populated via database triggers that are fired off automatically as records are inserted or updated.
• Database log scraping. Log scraping effectively takes a snapshot of the database redo log at a scheduled point in time (usually midnight) and scours it for transactions that affect the tables you care about for your ETL load. Sniffing involves a “polling” of the redo log, capturing transactions on-the-fly. Scraping the log for transactions is probably the messiest of all techniques. It’s not rare for transaction logs to “blow-out,” meaning they get full and prevent new transactions from occurring. If you’ve exhausted all other techniques and find log scraping is your last resort for finding new or changed records, persuade the DBA to create a special log to meet your specific needs.
• Timed extracts. With a timed extract you typically select all of the rows where the date in the Create or Modified date fields equal SYSDATE-1, meaning you’ve got all of yesterday’s records. Sounds perfect, right? Wrong. Loading records based purely on time is a common mistake made by most beginning ETL developers. This process is horribly unreliable. Time-based data selection loads duplicate rows when it is restarted from mid-process failures. This means that manual intervention and data cleanup is required if the process fails for any reason. Meanwhile, if the nightly load process fails to run and misses a day, a risk exists that the missed data will never make it into the data warehouse.
• Full database “diff compare.” A full diff compare keeps a full snapshot of yesterday’s database, and compares it, record by record against today’s database to find what changed. The good news is that this technique is fully general: you are guaranteed to find every change. The obvious bad news is that in many cases this technique is very resource intensive. If you must do a full diff compare, then try to do the compare on the source machine so that you don’t have to transfer the whole database into the ETL environment. Also, investigate using CRC (cyclic redundancy checksum) algorithms to quickly tell if a complex record has changed.

CDC solutions are designed to maximize the efficiency of ETL processes, minimize resource usage by replicating/moving only changes to the data (i.e., the deltas) and minimize the latency in the delivery of timely business information to the potential consumers. Change data capture solutions comprises of the following key components

• Change Capture Agents
• Changed Data Services
• Change Delivery

Change Capture Agents
Change capture agents are the software components that are responsible for the identification and capture of changes to the source operational data store. Change capture agent sits on source system and takes minimal power of source system. Typically it utilizes 1 to 2% of source system processing power. Change capture agents can be optimized and dedicated to the source system (i.e., typically using database journals, triggers or exit hooks) or by using generic methods such as data log comparison.

Change Data Services
Change data services provide a set of functions critical to achieving successful CDC, including but not limited to: filtering (e.g., receiving only committed changes), sequencing (e.g., receiving changes based on transaction/unit of work boundaries, by table or by timestamp), change data enrichment (e.g., add reference data to the delivered change for further processing purposes), life cycle management (i.e., how long will the changes be available for consuming applications) and auditing that enables monitoring of the system's end-to-end behavior, as well as the examination of trends over time.

Change Delivery
Change delivery mechanisms are responsible for the reliable delivery of changed data to change consumers -- typically an ETL program. Change delivery mechanisms can support one or more consumers and provide flexible ways by which the changes can be delivered including push and pull models. A pull model means that the change consumer asks for the changes on a periodic basis (as frequently as needed, typically every few minutes or hours), preferably using a standard interface such as ODBC or JDBC. A push model means that the change consumer listens and waits for changes, and those are delivered as soon as they are captured, typically using some messaging middleware. Another important function of change delivery is the ability to dynamically go back and ask for older changes for repeated, additional or recovery processing.

Following are two sample scenarios that highlight how organizations can leverage CDC.

Sample Scenario 1: Batch-Oriented CDC (pull CDC)
In this scenario, an ETL tool periodically requests the changes, each time receiving a batch of records that represent all the changes that were captured since the last request cycle. Change delivery requests can be done in low or high frequencies (e.g., twice a day or every 15 minutes). For many organizations, the preferred method of providing extracted changes is to expose them as records of a data source table. This approach enables the ETL tool to seamlessly access the changed records using standard interfaces such as ODBC. The CDC solution needs to take care of maintaining the position of the last change delivery and deliver new changes every time.
This scenario is very similar to traditional bulk ETL, except that it processes only the changes to the data instead of the entire source data store. This approach greatly reduces the required resources and eliminates the need for a downtime window for ETL operations.

When should organizations use this approach? This batch-oriented approach is very easy to implement, as it is similar to traditional ETL processes and capitalizes on existing skill sets. Organizations should use this method when their latency requirements are measured in hours or minutes.

Sample Scenario 2: Live/Real-Time CDC (push CDC)
In this scenario, which accommodates near real-time or real-time latency requirements, the change delivery mechanism pushes the changes to the ETL program as soon as changes are captured. This is typically done using a reliable transport such as an event-delivery mechanism or messaging middleware. Some CDC solutions use proprietary event delivery mechanisms, and some support standard messaging middleware (e.g., MQ Series).

Note that while message-oriented or event-driven integration is more common in EAI products (i.e., using tools such as Integration Brokers), many of the leading ETL tool vendors are offering such capabilities in their solutions to accommodate the demands of high-end, real-time BI applications. This real-time approach is required when the BI applications demand zero latency and the most up-to-date data.

Change Data Capture Technical Considerations
While CDC seems to offer significant advantages, there are several factors that need to be considered and evaluated, including:

Change Capture Technique. Change capture methods vary, and each has different implications on the overall solution latency, scalability and level of intrusion. Common techniques for capturing changes include reading database journals or log files, usage of database triggers or exit hooks, data comparison and programming custom event notifications within enterprise programs.

Level of Intrusion. All CDC solutions have a certain degree of system impact, making intrusion a critical evaluation factor. The highest degree of intrusion is "source code" intrusion that requires changes to be made to the enterprise applications that make the changes to the data stores. A lesser degree of intrusion is "in-process" or "address space" intrusion, which means that the change capture solution affects the operational system resources. This is the case when using database triggers and exit hooks because they run as part of the operational system and share its resources. Using database journals or archive logs is the least intrusive solution and it does not affect the operational data sources of applications.

Capture Latency. This factor is a key driver for choosing CDC in the first place. Latency is affected by the change capture method, the processing done to the changes and the choice of change delivery mechanism. As a result, changes can be streamed periodically, in high frequency or in real time. One should note that the more real-time the solution is, the more intrusive it typically is as well. Yet another point to consider is that different BI applications will have different latency requirements, and thus enterprises should look for CDC solutions that support a wide range of configurations.

Filtering and Sequencing Services. CDC solutions should provide various services to facilitate the filtering and sequencing of delivered changes. Filtering helps to guarantee that only the needed changes are indeed delivered, for example: an ETL process will typically need only the committed changes. Another example is the ability to discard redundant changes and deliver the last change to further reduce processing. Sequencing defines the order by which changes are delivered. For example, some ETL applications may need changes on a table by table basis, while others may want the changes based on units of work (i.e., across multiple tables).

Supporting Multiple Con-sumers. Captured changes may need to be delivered to more than one consumer, such as multiple ETL processes, data synchronization applications and business activity monitoring. CDC solutions need to support multiple consumers, each of which may have different latency requirements.

Failover and Recoverability. CDC solutions need to guarantee that changes will be delivered correctly, even when system, network or process failures occur. Recovery means that a change delivery stream can continue from its last position and that the solution keeps transactional integrity to the changes throughout the delivery cycle.

Mainframe and Legacy Data Sources. BI is only as good as the data it relies on. Analysts estimate that mainframe systems still store approximately 70 percent of corporate business information, and mainframes still process most of the business transactions in the world. Mainframe data sources also typically store higher volumes of data, further increasing the need for a more efficient approach to moving data such as change data capture. In addition, popular mainframe data sources such as VSAM, which are non-relational, present additional challenges when incorporating that data into BI solutions. As ETL and DW tools expect relational data, the non-relational data needs to somehow be mapped to a relational data model.

Seamless integration with ETL tools. When choosing a standalone CDC solution, enterprises should consider the ease of interoperability with its ETL program (off-the-shelf or homegrown). Standard interfaces and plug-ins can reduce risk and speed the data integration project.

Change data capture allows organization to deliver real-time business intelligence based on timely data while, at the same time, reducing the cost of data integration.

For organizations looking for ways to meet these demanding business needs, create an event-driven enterprise and provide real-time business intelligence, change data capture is a key component in the data integration architecture.

More and more organizations have started adopting change data capture solution. CDC has become integral part of data integration architecture.