Actionable BI - reducing time between event and action

Experts in various fields mention that the best time to correct a negative behaviour is as close to the behaviour being exhibited as possible.  This is true for children, pets and people.  Why isn't this so for organizations?  Why are they willing to wait days/weeks/months before knowing that a negative behaviour occurred?


What I'm referring to is latency and specifically latency between access to data and the ability to act on it.  I've found many organizations are okay with having data that is a day or more latent.  This latency has cost organizations significantly as they are effectively blind until the data is accessible.


An Example


To illustrate, I was working with an organization who gives commissions to its call center representatives based on their sales of various services/offerings.  On one occasion, the company offered a special incentive for a particular service it was promoting.  Each time a rep was able to provision said service, he/she would receive a larger commission than normally paid.  The promotion was a great success.  A few weeks later the campaign result report showed thousands of customers signed up for the service, higher than what was expected.  After two months an executive noted that expenses had increased significantly but revenue had not and he wondered why.


After some investigation, it turned out that the reps were adding the service to each account that they spoke to and would remove it shortly thereafter.  The commission report automatically increased a rep's count each time a service was added to an account but did not decrease a rep's count if they service was removed.  Since the service was removed prior to being provisioned, the customer did not see a difference on their monthly statement and therefore there were not any customer complaints about their bills.  Two months after the start of the incentive, once the abuse was discovered, the commission incentive was stopped and a few reps who were abusing the system quit.  Since commissions were being paid weekly based on the report and this abuse was not uncovered for two months and consequently the organization lost close to $500,000.  This may not have been completely prevented but could have been minimized if the company had shorter time between event and action (see diagram below).  

Framework for Right time Business Intelligence - Bolder Technology

The shorter the action time, the quicker an organization can act (or react) to changes in its environment.  In the previous example, while campaign results were published within weeks, the financial report is published monthly and it took two months for it to drive action.  With access to both views of the organization's data (campaign results vs. financial impact), the loss would have been significantly less if not mitigated.


This organization quickly saw the value in reducing latency.  I worked with a team to create an operational data store (ODS) and created alerts for behaviours outside of a "normal" range.  


Reducing latency


Since the data was directly loaded into the ODS from source systems, we significantly reduced data latency.  We reduced analysis latency by creating alerts around key metrics that would allow the organization to manage by exception.  Also since we were presenting data captured from all source systems, we could look at a variety of metrics and enable decisions to be made faster thus reducing decision latency.  


All three actions combined reduced action time to the point where we could get the right data to the right people at the right time.  The ODS could effectively provide a "real time" pulse of the organization's operations which allows managers to make better informed decisions.


"Real time" is a loaded term and means something different to all of us, especially in the context of BI. A topic that I will talk more about in my next post...

Waste - Why you shouldn't bring all your data into your Data Warehouse

In lean, there are 3 types of work: value creating work, incidental work and waste.  When applied to a data warehouse context, we have customer value add data, business value add data and waste.  


Customer Value Add Data
This is any data collected that is directly used in a business workflow to provide a product/service to a customer.  Without this data, you would not know who ordered what and how much, where to deliver, bill the correct person, manage inventory or know if you were ever paid.


Business Value Add Data
Any data collected that the business uses to better understand or know their customers.  This is data that helps the business create customer profiles, understand buying patterns, behaviors and preferences.  To understand your customers better, as a business, you need data to answer questions like: 

• How long has this person been a customer? 
• How valuable is this customer to me (is he/she in my best/next best/low value customer segment)?
• Are my products/services more attractive to teens/adults/singles/couples?
• Are customers buying products together?  Would I be better serving my customers by offering a bundle?
• What are my top sales regions?
• Are my customers only buying from me because I have the lowest prices, my loyalty program or because they like my customer service?
• How can I get better return on my marketing dollars?
• What can I offer this person so that they buy more of the same product or cross-sell him/her a complimentary product?

Without this information, you could still run your business as the data used to answer these questions aren't necessarily needed in my workflow to provision a product/service; you just wouldn't be able to better understand and meet the needs of your customers.  

Waste
Any data that's collected which isn't used in providing the customer with value or used by the business to understand or serve customers better.  Seems obvious yet a lot of this type of data ends up in data warehouses.  
On previous projects I have asked business stakeholders to define what data needs to be moved into the data warehouse.  All business stakeholders have said, "Move all data into the warehouse.  We don't need it now, but may need it in the future".  It is their belief that if we're moving data in from the various tables in the operational systems, it's easier and cheaper to do it all now.  In doing so, they don't have to wait for data to be brought into the warehouse before creating a new report if a field ever becomes important.  What's the impact of this statement?  From a Project Manager's perspective:

• Scope
• Create a plan that estimates what it would take to move all data, table by table, into the data warehouse
• Time
• Additional time required for the coding, processing and testing required of the waste data.
• Cost
• Development costs increase due to additional time and scope of waste data being migrated, integrated and validated after being inserted into the warehouse
• Need more storage to accommodate the waste data
• Processing costs increase - more data needs to be loaded in the overnight load window

Waste creates more waste 
Beyond the initial sticker price of getting your data warehouse to a point where you can finally begin reporting, there are also ongoing costs associated to adding and maintaining waste data.   Think of it as a waste tax (or several waste taxes).


Waste testing tax
Development of ETL code is a one time effort for the waste data, but every time your code is updated, you need to test the code to make sure you didn't break anything.  


Storage improvement tax
Waste data grows as fast as your value add data.  Since you're storing waste data on the same storage as your value add data, you will need to increase your storage more often.


Network traffic tax
Whether your organization performs nightly batch data loads or you load data in real time, waste data, along with value add data, is moved from your source system(s) over the network to the DW (or staging area if applicable).  The more your data grows, the more traffic you have on the network and the more waste you are moving.


Data load/refresh tax
If your organization performs nightly batch loads, the load window is a fixed length of time.  For most organizations, data loads, if all goes well, just fit within the window.  Now if you decide to add a new source of data, you will need to spend time optimizing current load times in order to fit the loading of the new source in the same window.  This takes time, people and ultimately money to optimize waste.




Nobody likes paying taxes or wants to wait 18+ months to see value.  So apply agile and lean principles to your BI and DW initiatives. Prioritize and focus on the high value data, build the reports you need and you will see a return on your investment sooner.

Aggravations with waterfall BI projects...need to do what works

A very good friend of mine once told me "Do what works for as long as it works.  If it stops working, do something else!".  Seems very simple doesn't it?  So if it's so simple, then why do we continue to run projects (especially BI projects) in a waterfall way when we have seen time and time again that it doesn't work?


Anyone in the BI world who has gone through implementing a new data mart/warehouse has experienced the pain...they have seen a project that had a ton of upfront design and requirements gathered and still fail to deliver once it's "ready".  


Why? 
Well there's many reasons...change in requirements, change in the competitive environment, change in technologies, change in people, change in <fill in the blank>...  


Some of the biggest issues:

1. Waterfall does not allow for change...well it does, but the request has to go through a long and difficult process before it is allowed.  Requirements and design are done upfront; everyone is now bound by the estimates and is expected to deliver on the agreed upon date.
2. Nothing is delivered until the end..."I don't know what's happening over the course of the project.  I sure hope my requirements are clear enough!"
3. Reams of documentation...sure it contains what the users want but because everyone was so busy creating the documentation, IT didn't have enough resources to complete the project in time.  The documentation also contains what the users wanted when it was written...what happens if things change? See 1.
4. Didn't meet what "the business" wanted because they weren't available...this one is from the IT side.  IT spends months with business users defining requirements and then those users are no where to be found during the project as they are too busy with their day-to-day work.  IT builds based on the "spec" and sometimes misses the mark.

Is there not a better way???

1. Why are our requirements written in stone?  We all agree, things change but for some reason changes to the project aren't allowed.  We don't have all the answers before we begin but we're asked to make decisions and stick with them.  Why not delay making decisions until we have to?  This will allow us to make decisions when we have enough/better information.  If we can write our requirements iteratively, we are more likely to be able to deliver what's needed and account for all the changes in <fill in the blank>...
2. Why can't we show progress throughout the project?  We need shorter feedback cycles than just at the end of the project.  Having regular showcases/demonstrations means our business will give valuable feedback to IT and IT knows it's delivering true value.
3. Why can't we document as we go?  We spend months before we start the project documenting.  Requirements, design and specification documents and the Project Manager won't start until it's all signed off.  If we don't know all the answers, then what good is all this documentation?  We would be far better off to document iteratively.
4. Why can't "the business" be available at all times throughout the project?  First off, I'd like to ask why do we in IT say "the business" instead of "our business"?  Secondly, our leaders need to make the project our business representatives' day job.  If we have our business users work with us throughout the project, IT is less likely to miss the mark as we will be given continuous feedback and our business stays engaged throughout.  

Enter Agile/Lean.  

These methodologies work.  They work in manufacturing, software development and yes you guessed it, Business Intelligence and data warehousing.

I can speak about the generic agile and lean methodologies but you can google/wikipedia that for yourselves.  What I'll be talking about is how these methodologies have helped me deliver complex BI/DW projects on time and on budget.  Feel free to share your experiences as well.

Welcome to my blog!!