How Does Data Lineage Assist Your Intelligence Solution?
- 17th May 2019
- Big Data & Advanced Analytics
- Luke Sharma
I like to think of Data Lineage as the “supply chain” for BI Solutions. If the logistics in a supply chain break down at any point, there is a ripple effect – causing more disruption further up the supply chain. It’s the same with data lineage!
One of the wonderful things about being a BI Consultant with one of Europe’s top BI Consultancies is the variety of work that I get to do. I usually do work on the front-end of tools such as Birst, OBIEE or BI Publisher, so one of the main issues that I have faced across my projects is when someone, somewhere in the world makes a small change in the data lineage of the solution. Anyone who works in BI knows that small changes can have HUGE impacts.
With data bringing the world closer together, we can have people in different timezones working on our ETL or RPDs when the front-end teams aren’t even awake. This leads to a lack of communication, that can subsequently lead to some big problems. For example, a simple changing of column aliases mean that your visualizations can’t find the column. You can spend hours trying to diagnose this problem, only to find you just need to change SELECT STAFF NO to SELECT STAFF NUMBER in your SQL.
So how do we eliminate this problem?
Data Lineage! Keep a track on everything that your data is doing, from source to front-end report. For my first few weeks as a consultant, myself and a colleague spent a lot of quality time with whiteboards and PowerPoint putting together a data lineage for one of our Birst clients. By doing this, it helped the back end and front-end developers know the ramifications of the changes that they make without the need to communicate directly with one another. It was time consuming, but once it was all put together it saved hours of diagnostic time for the front-end developers, and saved hours of time for back-end developers who didn’t need to wait for the OK from the front-end team to enact change.
Don’t have time to put together a Data Lineage? Fortunately, Peak Indicators has partnered with a data lineage company named Solidatus that provides a quick, clean, easy to use solution!
Solidatus is a data lineage tool, that is capable of visualising the links across the tables in your databases. It captures everything, transformations, combinations of columns, where the columns end up, what reports they are used in – by doing this, developers can quickly verify what changes they can make and the effect it will have on the work further down the lineage.
It’s a brilliant tool that we now use at Peak to do all of our lineage – even for companies that use hundreds/thousands of tables and columns. It’s got an extremely clean interface that is easy to understand, and it’s even easier to use. If I knew Solidatus existed when my colleague and I were spending hours writing up source, to scripted source, to secondary scripted source (you get the picture) – then we could have saved those hours for more development work.
Lineage is just something that data professionals don’t always think about, so having a tool like Solidatus available takes away the concern of how making changes could affect the rest of the project because you can see it right in front of your very eyes.
So if you would like to explore your Data Lineage Contact our team to find a quick, clean, easy to use solution!