Agile – fragile?

19 10 2016

“Agility” is the mantra these days. We want all our processes to be agile to be able to deal with the fast changing environment. We want to see the outcome, even in a primitive shape, as soon as possible and then iteratively adjust/improve it taking the changing landscape into account. It all sounds very exciting. And the methodology may also be working very well for organizations. No more “being in the dark” until the end of the project to see and experience the end outcome. Users go along with the team (Scrum team)  in this journey. From organizations’ point of view, it all makes sense. The perpetual alignment among business user, product owner, developer and tester is the key.

But what does it mean it to people? Not the roles, but the actual people. Let me comment based on my observations and experiences. There is a frenzy of activity for every member of the team throughout the sprint. And there is the planning/grooming for the next sprint. On the surface (aka ideal world) it may seem that not all members are involved in each of the sprint activities and things can go on in parallel. Nevertheless, there is significant overlap in practice, especially if things are a little murky. As soon as you are done with one sprint, the next sprint starts and the cycle continues. It seems like you are on a continuous roll. It’s taxing and stressful. As far as I have observed, people in Bay Area generally seem over-worked. It is almost a norm. Speak to anyone, it is the same story. Too much work. Too much agility. :). Well, this maybe mainly because of unmanageable scope and too few resources. 😛

Ideally, “agile” doesn’t intend to be stressful. It all comes down to how we plan and orchestrate it. Understanding the methodology is one thing, and putting it in practice in the most efficient way is another thing. As with any change, shifting from Waterfall to Agile is a big deal. I didn’t realize how big of a deal it is until when my team has finally started embracing the model. To be frank, we are going through a lot of growing pains.

Traditionally, BI is notorious for its huge turnaround times. Given the underlying architecture, any BI deliverable – usually a standard or analytical report – takes a long time to deliver (2-3 months). The entire DW project is of course a monolith which may take 12-18 months. And hence we approach it incrementally, business process by business process as advocated by Kimball. But each business process may also take 3-4 months on average. And it is usually implemented as Waterfall.

Applying software agile development methodology to BI needs a little tweaking at best. There exist some sprint planning guidelines and best practices that can be leveraged and used as a starting point. And we made mistakes. And still are. First thing, in an attempt to provide business quick results (as is the tenet of agility), we took on too much and attempted to achieve too much in too little time. Consequences: burned out team and inefficient implementation resulting in a lot of after the fact corrections and technical debt. And so we changed tactics and decided that we take in only manageable scope, and approach things more methodically. Consequence: No quick results. Business may need to wait for a couple of release cycles in order to see the output. Sounds familiar? Yeah, sounds so much like Waterfall. So, we are indeed doing waterfall in the agile setup.

This brings up the question as to what “agility” means in the BI space and how Agile BI is different from the agile software development approach. The recent trends in BI world extol the need to transform the underlying architecture and integration patterns in order to achieve true agility., which is of course a much bigger discussion topic for another day. :).

But the point is, unless you change the way you think about BI and how it should be implemented, (which needs more agile architecture and framework), mere changing the project development methodology will not truly help and the whole attempt at being agile ends up making everything fragile. 🙂

Disclaimer: These are strictly my own personal views and do not reflect my team’s or company’s perspectives.


My first infographic

9 05 2016

Being a bit technical this time 😛 . My all-time favorite topic – Business Intelligence. 🙂




Created using