Aggravations with waterfall BI projects...need to do what works

A very good friend of mine once told me "Do what works for as long as it works.  If it stops working, do something else!".  Seems very simple doesn't it?  So if it's so simple, then why do we continue to run projects (especially BI projects) in a waterfall way when we have seen time and time again that it doesn't work?


Anyone in the BI world who has gone through implementing a new data mart/warehouse has experienced the pain...they have seen a project that had a ton of upfront design and requirements gathered and still fail to deliver once it's "ready".  


Why? 
Well there's many reasons...change in requirements, change in the competitive environment, change in technologies, change in people, change in <fill in the blank>...  


Some of the biggest issues:

1. Waterfall does not allow for change...well it does, but the request has to go through a long and difficult process before it is allowed.  Requirements and design are done upfront; everyone is now bound by the estimates and is expected to deliver on the agreed upon date.
2. Nothing is delivered until the end..."I don't know what's happening over the course of the project.  I sure hope my requirements are clear enough!"
3. Reams of documentation...sure it contains what the users want but because everyone was so busy creating the documentation, IT didn't have enough resources to complete the project in time.  The documentation also contains what the users wanted when it was written...what happens if things change? See 1.
4. Didn't meet what "the business" wanted because they weren't available...this one is from the IT side.  IT spends months with business users defining requirements and then those users are no where to be found during the project as they are too busy with their day-to-day work.  IT builds based on the "spec" and sometimes misses the mark.

Is there not a better way???

1. Why are our requirements written in stone?  We all agree, things change but for some reason changes to the project aren't allowed.  We don't have all the answers before we begin but we're asked to make decisions and stick with them.  Why not delay making decisions until we have to?  This will allow us to make decisions when we have enough/better information.  If we can write our requirements iteratively, we are more likely to be able to deliver what's needed and account for all the changes in <fill in the blank>...
2. Why can't we show progress throughout the project?  We need shorter feedback cycles than just at the end of the project.  Having regular showcases/demonstrations means our business will give valuable feedback to IT and IT knows it's delivering true value.
3. Why can't we document as we go?  We spend months before we start the project documenting.  Requirements, design and specification documents and the Project Manager won't start until it's all signed off.  If we don't know all the answers, then what good is all this documentation?  We would be far better off to document iteratively.
4. Why can't "the business" be available at all times throughout the project?  First off, I'd like to ask why do we in IT say "the business" instead of "our business"?  Secondly, our leaders need to make the project our business representatives' day job.  If we have our business users work with us throughout the project, IT is less likely to miss the mark as we will be given continuous feedback and our business stays engaged throughout.  

Enter Agile/Lean.  

These methodologies work.  They work in manufacturing, software development and yes you guessed it, Business Intelligence and data warehousing.

I can speak about the generic agile and lean methodologies but you can google/wikipedia that for yourselves.  What I'll be talking about is how these methodologies have helped me deliver complex BI/DW projects on time and on budget.  Feel free to share your experiences as well.

Welcome to my blog!!