In our last posting we explained what agile data warehousing is and how in practice it differs from more traditional approaches to data warehouse design. In this post we look at how to get the most out of agile and where it works best. We ask the question “What do you need for agile data warehouse development to be effective?”
1) The agile data warehouse initiative needs to be strongly business driven
Business driven
Sometimes data warehouse initiatives are undertaken to harmonise data sitting in different reporting systems and make its management easier. That is often done silently by IT professionals and it’s unlikely to give you good results because business people rarely benefit directly from these tidy-up activities or from system upgrades. If a centralisation is being done to prevent disagreement between sales and finance so they have the same understanding of some figures, then that’s different. The initiative needs to have the resolution of specific business issues as its aim and activities that don’t contribute directly to those business aims generally will not add significant value.
Another way to obtain high value is to ensure a close relationship with the lead business representative or “product owner”. The strength of that relationship ensures you keep short accounts with one another because knowing that you are both striving for the same goal means you can have frank discussions. That two-way accountability will prevent the project from taking on any work that isn’t absolutely necessary.
One further element of ensuring the project is strongly business driven is having your data warehouse developers collect requirements face to face with business people, not having someone else write down a detailed specification that is then handed on to the development team. As anyone who has tried to work through a long supply chain of people with multiple hand-offs knows, so much of what is needed gets lost and also writing everything down to that degree takes a lot of time.
Overall this close relationship with your business will ensure that what you deliver is roughly right and not precisely wrong.
2) You need some experienced people who can both interact with your business people and create a good business data model
Collaborating on the business model
One of the mistakes we made on my first project was not to have enough really good experienced resource on the project, and this is where working with a good partner organisation like Red Olive can really help you. The people you need are people with good business-facing skills who are also equipped to do the development work itself. In the past, when people hand-coded data transformations to build the data warehouse, it was very difficult to find people with such a broad skill-set. Now that you can use data warehouse lifecycle tools such as RED from Wherescape to automate all the basic development it’s much easier to work in an agile way because an experienced dimensional modeller can not only work with your business people to understand what they really need, they can also build it using an intuitive front-end. We’ve worked with RED for about 5 years now and we find it speeds up implementations and makes handling change and documentation much easier.
3) You need real data to work with
Real messy data
As anyone who’s worked on a data warehouse project knows, you don’t really know whether your design is any good until real business people start trying it out on their own data. If you have a choice, and I realise often you don’t, then don’t choose to build a data warehouse at the same time as the main source of data for your target reports is undergoing a major overhaul. This is particularly an issue with agile data warehouseing because typically you’re delivering live functionality in short iterations and so your need for data comes that much earlier than in a waterfall project.
4) You need the right attitude to managing business risk
Managing risk
There are a couple of different ways that you need a supportive business culture when it comes to understanding and managing business risk.
The first criticism often levelled at agile development is that if you intend to deliver live functionality after months 2, 3, 4 etc, you can’t say at the start with any certainty what will be delivered to the business in 6 months time because the agile methodology doesn’t allow for it. The counter-argument is that the business risk is actually reduced because with agile, benefit is being delivered much quicker and some business benefit will have been delivered in months 2, 3, 4 and so on and what gets delivered in any iteration is whatever is viewed as the business prioritiy right then; in contrast, the success of a waterfall project with a longer plan leading to a bigger go-live all hinges on success in 6 months time. The cost of discovering a mis-understanding after 6 months of investment is much higher than discovering it following user review after a month.
5) It helps if you have backing of the IT department – but if you don’t, all is not lost
A second issue arises in an organisation’s approach to management of its core systems. To ensure the stability of systems used for sales order processing, invoice payment and so on, many organisations only promote changes to core systems between 2 and 4 times a year; so how do you benefit from agile in this scenario?
One option is to educate your organisation that your analytical data warehouse has a completely different risk profile from these core systems and so should not be governed under the same regime. After all, making an incorrect change to a sales report can be very visible to management, but it won’t lead to your bank account becoming empty in the way that an error on your banking systems can. In an era when innovation and smarter analytics than your competitors are becoming one of the only remaining ways of competing, your analytical systems need to be able to respond at the pace of your business people’s latest thinking. This can be easier said than done!
If your enterprise data warehouse is classed as a core system then another way around the problem is to develop individual data marts in a parallel prototyping track and to migrate these live developments into the main data warehouse as and when the core system policy allows. Many large companies do this, for example Vodafone UK described carrying out exactly this process at a recent seminar and Red Olive has also implemented RED in this capacity at a large pharmaceuticals company where the “main” data warehouse was built using IBM Data Stage.
I hope that’s been a useful view from the trenches, if you have any questions please feel free to comment on the blog or call us on +44 1256 831100. In the next few postings we’re going to be looking at some specific applications of data models. We’ll be starting with Single View of Customer.
