Imagine your data warehouse has been hit by invalid input which has messed up your reports. The business users are angry, and the business intelligence department is getting ready for a couple weeks of emergency operations.
One day a financial manager came to work and found out that according to reports he had gotten from the BI department that morning, the bank had sent more than 2 trillion euros somewhere. (That’s this much money: 2,000,000,000,000, by the way.) Instead of having a heart attack, he called the BI department to ask what had happened.
They easily figured out that one of the source systems had sent a wrong amount for one of its transactions. So the financial manager could stay calm, but he insisted that he needed a corrected report as soon as possible and that there were also other reports that could be corrupted.
Finding the rotten apple in the basket
How could they find out which reports could potentially be corrupted and therefore need to be recalculated? And not only reports, they should also find all workflows and scripts transforming the source data and launch them again in the right order. Doing this manually would take a lot of time and effort – they would either rely on the potentially obsolete or inaccurate documentation or browse code in a lot of scripts – while running a whole day‘s load would stop the whole BI department.
The original Manta to the rescue
Fortunately, they had one of the first versions of Manta. (The product name Flow didn’t exist back then.) So, how to fix this problem, then? They had the name of the faulty extract from the source system, and they also knew the precise column with the erroneous value. But they needed to find all scripts and workflows that use or transform that value and determine what order they should be executed in, so every value in the data warehouse that depended on the erroneous value would be recalculated. Using the right settings they were able to investigate how the data from the erroneous column was propagated throughout the data warehouse and visualize it.
With Manta Flow they reduced the time spent controlling data warehouse damage caused by incorrect data by more than 90%.