Data Lineage - How We Use It & Why You Should
- 26th August 2019
- Big Data & Advanced Analytics
- Joe McHale
During a project one member of the development team would make a change which would Block another preventing them from being able to work, whilst they figured out what the change was and the impact it caused...
So what's the solution? We've partnered with a company called Solidatus - a data lineage platform that allows you to map your company flow of data, business processes, responsibilities (or almost any other use case you can think of!) quickly and easily. It is totally interactive, can be shared, worked on as a team and has multiple APIs to automate the mapping of databases (imagine 1000s of tables, all mapped with the flow of data right up to reports/dashboards). Having a visually interactive data-lineage tool makes life easier because a dev team making a change to a source object can instantly see the impact their change will have both upstream & downstream. Give all teams access to the Solidatus model and suddenly dev teams being blocked figuring out what has changed will be a thing of the past. Saving valuable time and frustration.
The image below is one we created with anonymous metadata from a utility company - showing how a company can map all their statutory reports in one visual model. It was quite an eye-opener to visually see the complexity of the process/data flow to produce statutory reports that appeared relatively simple. On the left we have directorates and as we move to the right we eventually have the final regulatory Ofwat reports. Each object represents a change, person or system that collates the reporting data.
Focusing in further we can see all the data flows that the customer service directorate are responsible for:
All clutter is removed - when we sent this view to the individual within the organisation responsible to get their buy-in and agreement that we had everything mapped correctly. By having this information they realised that there were some governance issues that needed to be addressed which in turn led to positive changes being made.
The reality of the situation above allowed them to adopt governance improvements, preventing future problems. You can only do this with this level of visibility.
So how to get started?
We offer a 2-week quick-start pilot, talking to individuals responsible for data and reporting we can rapidly build out a model which is the data flow through people, spreadsheets and systems. Then we automatically ingest the data flow using the Solidatus API - with an end goal of spotting system changes and flagging actions to either accept or reject them.
This allows us to leave you with some valuable insight and a means of getting buy-in internally with a working model.
Take a look at Our GDPR Compliance demo below to see for yourself...
If you think you could benefit from this kind of clarity within your organisation Contact our team to schedule a 30-minute call to show you a live demo and answer any questions you may have.