Red Olive Blog

Agile BI, the death-knell of the traditional Data Warehouse…?

Published: Jun 10th, 2016 | Author: Mark Fulgoni

So, if you have been reading recently, you’ll know that I’ve been looking at Agile BI. (see A journey from Football to Agile BI – part 1 and part 2). For those coming straight to this article, by this I mean the new breed of end-user tools such as Microsoft’s Power BI, MicroStrategy 10, Tableau, QlikView etc. which allow business users to quickly pull together and cleanse unprepared data, known in the parlance of these tools as data wrangling, without the need for complex Extract, Transform and Load (ETL) code. All of this in addition to offering powerful visualisation, mapping and reporting capabilities.

Now that these tools have reached a reasonable level of maturity it’s worth giving some though to the impact that these tools will have on the existing, traditional Business and Management Information reporting.

Using this new breed of BI tool, Business Users can acquire, wrangle (conform, cleanse and transform), join and aggregate a very wide variety of data sources and formats, be they more traditional, well-structured sources such as from an Enterprise Data Warehouse or web pages, RSS feeds, social media or unstructured big data.

Based on this you might reasonably draw the conclusion that business people no longer needs a traditional Data Warehouse. They no longer need to wait for their IT, MI or BI department to go through lengthy analysis, design and development cycles in order to report on new areas of interest; they can “do it themselves” with these new tools that require less technical or programming knowledge.

The counter argument is that business people should not be using such tools as they lead to a wild west culture where everyone with a tool develops their own reports and tweaks them according to their own agendas, the complaint that many IT functions have had in the past about reporting via Microsoft Excel. There would no longer be any agreement between departments about what figures mean, no “one version of the truth” and lots of arguing about what this week’s sales figures really are as every department is calculating “sales” in the way that puts the best spin on it from their perspective.

But is there a better way…?

Well, the first point that I would make is that we should think twice before throwing out all the wisdom that has been accumulated regarding the reasons for building a Data Warehouse. Many of these reasons still hold true for our frequently used and/or key data.

Cleansing and conforming this data; its ineffective and inefficient for everyone wanting to use a piece of data to have to apply cleansing and conforming rules to every time they want to use it.

Consistency, “one version of the truth”; one of the driving reasons for building a centralised Data Warehouse is to ensure that key attributes and measures are agreed across an organisation ensuring a consistent reporting language and avoiding disputes regarding the lineage of key business data.

Can, and should, the traditional Data Warehouse coexist with Agile BI?

The New World…

I think that it not only can, but that it very definitely should.

There is still a need to clearly define and strongly govern the key metrics which drive any business, these belong in a structured data repository where they are clearly defined, well understood and served up in a consistent manner.

However, if we look at Agile BI alongside the traditional Data Warehouse we can start to realise its true value. If governed well it can become a significant boon to both business users and the IT functions which support them.

Agile BI can be used as a reporting platform, sourcing data from a traditional Data Warehouse but also capable of supplementing this (with data gleaned from other sources, internal data which hasn’t yet been prioritised for inclusion in the warehouse, external market data etc.). In this case, business users are able to deliver reporting to meet their own needs without needing to wait for long IT development cycles. IT should then monitor these additional data sources, some will prove to be experimental or short term, but many will need to be used regularly in order to fill an ongoing reporting need. In these instances, the data sources need to be assessed and prioritised for inclusion in the centralised data warehouse.

Mark Fulgoni is a Principal Consultant in Red Olive’s Data Practice.

Want to learn more?

Do you want to learn more about Agile BI and Data Warehousing? Why not contact the Red Olive team, we’d be happy to chat with you about how we can help you with your aims. You can reach us here.