There has never been a time that data hasn’t been recognised to be as important as it is today. It seems that the world has woken up to the potential value that an organisation’s data can offer – not only for external monetisation, but also for understanding the internal organisations’s workings. The discussions around machine learning and artificial intelligence using data to automatically respond to situations and events offers even greater potential for the future.

However, the challenges to deliver that value are substantial, and today I wanted to focus on one issue that I first encountered when I worked as a Chief Architect at BT some 15 years ago, but I still see organisations struggling with today? Where should Business Intelligence (or reporting, or dashboarding, or DSS, or MIS, or whatever you might call it) sit within the IT estate?

This has, in fact, become harder to answer over time rather than easier – mainly because of two strong industry trends that have been around for while – but have grown very quickly in recent times.

The first trend is the oldest, and is where organisations see the benefit of a joined-up view of the organisation and try to bring data together into a central data store and business intelligence capability. Data warehouses have been trying to do that for many years, and this trend has accelerated and transformed into the many data lake projects that we see today. (I will probably come back to Data Warehouses and Data Lakes in a future post).

The second trend is that companies are increasingly looking to SaaS to deliver significant parts of their IT Estate, and this is especially challenging when the SaaS solution includes and masters a key data entity – such as finance, customer or product – and provides it’s own built-in reporting or BI capability.

As Enterprise Architects, the trap that we can fall into (and I certainly did for a while) is to be too focused on a single solution approach. Whether that single solution is an all-encompassing data warehouse or data lake, or whether it’s a company-wide diktat on the business intelligence toolset. While both those approaches can have their place, they risk becoming lowest common denominator solutions that deliver the least-worst solution for a broad community and a broad set of requirements.

We need to be able to recognise the necessary exceptions to the “rules” – which means effective architecture governance – governance that is not there just to say “No” but to genuinely engage with a specific requirement and look for the best balance between local needs and the the efficiencies of a single approach. We need to understand the correct scope for any “rules” that we put in place.

Let’s take the the case of a single strategic BI capability for an organisation. This clearly makes a lot of sense – one capability to learn, to support, with a single interface where all data relevant to your role is equally accessible. Fantastic.

But what about SaaS apps (or any packaged app solution, even on premises) – if they come with a different BI tool, what should we do? I have seen some companies (I did this once myself) deliberately disable the built-in reporting capabilities just because they are not on the “strategic” BI platform, and then attempt to extract the data from the app and built a new bespoke reporting capability in the “strategic” tool. This inevitably leads to a poor outcome; the ETL required is inevitably complex and brittle; every significant application upgrade needs an internal development and the cost is huge. As SaaS applications move to 6 monthly releases and deliver fully integrated BI, this approach has become increasingly infeasible.

So, the question is for an EA, how do we compromise between these contradictory drivers? I think we need to look more closely at how we can divide BI into different discrete areas, and then apply the rules sensibly across the organisation. To do this, I think we go all the way back to looking at the business functions that the BI is trying to support and the use of BI in an organisation.

Each business function will require BI to support that function, and the scope of the data will be mostly contained within that function. For example, to manage a callcentre team’s daily performance at the team level then all the data required will likely be in the CRM system, and the built-in BI within the CRM should be sufficient. The timeliness of the data also needs to be at least daily updated and even better hourly. However, the overall senior manager of the customer management function will likely need a much broader scope of data, but weekly will be sufficient, possibly even monthly. For the COO, then monthly or likely to be sufficient. As you move up the organisation, the breadth of the data requires expands, but the latency of the data becomes less critical.

We can use this to our advantage – if we satisfy the BI requirements up to team or possibly middle management level with the dedicated BI embedded in the application, then we can easily achieve true real-time reporting. For the higher levels of management, we have more time to extract the data, load it and consolidate into a broader view of the enterprise to satisfy an executive dashboard requirement. What we should guard against is the desire to use the same BI capability for all levels of the organisation which then presents us with the issues described above.

This leads us to the recognition that BI needs to exist in multiple places within the IT estate – embedded within applications to real-time operational reporting with a narrow data scope, and also a lower latency broad view of the enterprise to deliver the joined-up view of the organisation for long-term strategic decision-making.