Manta Business

MANTA 4 Healthcare

April 12, 2019 by

After introducing “MANTA 4 Finance” as the pilot segment of our new “MANTA 4 Industries” series, we are moving on to another industry that MANTA is very familiar with. Read about what issues MANTA helps its healthcare customers solve in the article below.

After introducing “MANTA 4 Finance” as the pilot segment of our new “MANTA 4 Industries” series, we are moving on to another industry that MANTA is very familiar with. Read about what issues MANTA helps its healthcare customers solve in the article below.

Regulatory Compliance and GDPR

It can be said that the regulatory requirements in terms of data protection for this industry are even more strict than in other industries. The companies’ systems contain large amounts of sensitive personal data that, as we all know now, has to be safeguarded with extra care. The General Data Protection Regulation (GDPR) and Health Insurance Portability and Accountability Act (HIPAA) threaten astronomical penalties for noncompliance.

A big problem may also arise from an inability to accommodate the fundamental rights of data subjects (right to access, right to erasure, right to restrict data processing, etc.).  In these cases, the company has to show the supervisory authority exactly how it secures its data. 

In the worst-case scenario, if there would happen to be a data breach, the healthcare provider would have to prove that it did everything humanly possible to stop it! Data lineage is a way of showing that, by drawing out an end-to-end map of the data flows and all their movements within the BI and analytics environment.

Data Anonymization

Some of the consultancy companies that MANTA works with have recognized that the biggest struggle regarding GDPR compliance is most likely tracking customer data across multiple databases. The subject of data anonymization, personal identifiable information (PII), consists of data elements that alone or in combination can directly or indirectly lead to the identification of a specific individual.

Companies must identify the various locations where sensitive or noncompliant data is being stored as well as discover the relationships between this data. Not all records are equally sensitive; not all need to be anonymized. Sometimes, only parts of the data need to be re-written (e.g., a name and a country code in the same table will most likely not lead to the identification of a customer, but adding a city name could end up leading to quite precise identification).

Using MANTA you can construct and analyze metadata models that will identify PII in any component of your BI and analytics solution.

IoT and Healthcare

With all of the abovementioned regulations, healthcare is one of the most monitored industries in the world, and for good reason! It will only get harder to comply with these regulations, especially with the desire to digitalize all customer records and add IoT concepts that integrate data from wearable sensors. These wearables usually sync data in real time as well, giving doctors the ability to remotely monitor patients. This makes them a real-time security threat for medical experts, doctors, insurance providers, you name it! (Hence the possible data breach threats mentioned above.)

Conclusion?

The complexity of today’s BI and analytics environments makes it almost impossible to search for these relations manually. Luckily, MANTA can automatically analyze all database objects and data processing logic within your database, and if an additional description about the level of sensitivity of each record is provided, it can identify the locations of sensitive records. Then you can eliminate all potential threats to compliance as well as make sure you didn’t fail to find a complete record of customer data.

Do you have any questions for us, or do you want to read one of our case studies regarding this industry? Then hit us up at manta@getmanta.com and we will gladly reply! 

MANTA 4 Finance

MANTA is a solution for companies that have loads of data: huge, complicated data warehouses. But are you wondering how exactly MANTA fits into the data warehouses of big financial institutions? Welcome to our series called MANTA for Industries: Finance.

MANTA is a solution for companies that have loads of data: huge, complicated data warehouses. But are you wondering how exactly MANTA fits into the data warehouses of big financial institutions? Welcome to our series called MANTA for Industries: Finance.

Credit Risk Scoring

Financial institutions that offer consumer loans invest a lot of money, effort, and know-how from years of experience into the development of advanced credit scoring models. In most cases, only a few people in the organization know and have access to these algorithms.

One way to decode a credit scoring model is to collect a significant amount of customer data where credit scores are combined with variable credit factors and then use statistical methods to understand the model. These “dangerous” combinations of data are often present in BI solutions and data warehouses. Every BI solution should allow the separation of access to such data by properly categorizing data sensitivity and by enabling user entitlement setup. This is not easy to achieve, and it is especially difficult to verify if the setup is correct.

With MANTA, you can easily identify and visualize the components of BI solutions (for example, data marts) where unwanted combinations of such sensitive data are present, or you can analyze user data-access setup to see if there are direct or hidden and indirect ways to retrieve those data sets.

You can export lineage from MANTA that can then be used for BI security improvements. And you can even restrict the use of MANTA’s data lineage right in our native visualization. When you find such relations, you can then take them into account when setting up user access for different user groups and teams who have access to MANTA and monitor who has access to different parts of the lineage from different data marts.

Regulatory Compliance and GDPR

Another popular way to use MANTA in big banking institutions is to produce proof for internal auditors that your credit scoring models are well protected. With the enormous number of banking regulations as well as regulations such as GDPR, it is twice as important to have decent data lineage to show the auditor when he arrives. To learn more, read our GDPR article. (link)

GDPR has introduced many more threats pertaining to corporate internal data. For example, a customer may come and request that you honor one of his rights such as the right to be forgotten. In this case, you may need to use MANTA to find every single place in your company data marts where the customers data is being stored to make sure you delete every single one of those instances.

You may also need to anonymize data. And after using MANTA to find all the places where your data needs to be anonymized, you might want to use MANTA again to double check that there is really no way to identify that person.

Legal Threats

Being able to keep track of your environment and changes in your credit scoring algorithms is beneficial for many different reasons. One of our favorite client stories is about a customer who attempted to sue our client for not approving his loan the first time he applied, only to be approved a year later.

Using MANTA, the client was able to automatically find the changes made to the scoring algorithms over the last couple of years and identify the change in the algorithm for calculating credit scores for loans. The ability to show exactly what had changed and when allowed the financial institution to win the court case, saving them billions of dollars.

And how could you use MANTA in your financial institution?

Any comments or questions? Let us know at manta@getmanta.com, or go ahead and schedule a meeting using the form on the right.

 

MANTA Cases #3: (Not Always) Agile Development

Many companies seek to achieve an agile development strategy. Sometimes it is not exactly needed and might bring some difficulties (as our CEO Tomas Kratky already mentioned in one of his older articles here).  But sometimes an agile-type strategy naturally evolves in the development department. In the third part of our MANTA Cases series, we will take a look at how one of our customers uses MANTA in a (not agile) development team. 

Many companies seek to achieve an agile development strategy. Sometimes it is not exactly needed and might bring some difficulties (as our CEO Tomas Kratky already mentioned in one of his older articles here).  But sometimes an agile-type strategy naturally evolves in the development department. In the third part of our MANTA Cases series, we will take a look at how one of our customers uses MANTA in a (not agile) development team. 

One Too Many Patches

One of our customers, an international bank, was using a workshop-type software to release internal production software. Because this internal database software is so complex, they have 10 to 15 patches of code each day between releases. To make sure that the environment doesn’t crash, they have to deploy these patches of code all at once. Each patch of code is created by a different development team, and the individual teams are not aware of the dependencies between their parts of the code. Because of this, the patches were often deployed in the wrong order causing the entire environment to crash.

These crashes often made the deployment process last more than one day, with numerous other patches needing to be developed to fix the problems created while deploying the previous patches.

Understanding the Environment

The patches of code the development team wrote were all improvements to their internal database system (e.g., changing database logic, repairing certain objects, even adding new columns or creating more tables and views). Such patches resulted in changes in paths or even in values within the environment. And given the size of our customer’s database – which contained multiple banking systems, each having an average of 180 thousand scripts – and the amount of data being added to the database each day, it was necessary to perform the 10-15 patches daily.

The only way the customer could keep up with the (not) agile development pace and effectively find the dependencies between the patches was to apply automated software like MANTA.

Post-MANTA

Now that the customer has MANTA on their development team, they take all the patches that need to be deployed that day and let MANTA run them. MANTA draws out all the dependencies between the individual objects that the patches are related to. Then the customer exports them using MANTA’s API and has an automatically sorted order that it needs to deploy the patches in so that the environment will survive. This has cut the entire deployment down to a couple of minutes and has also made it a one-person job.

This one person now only needs to:

  • Gather the patches
  • Upload them into MANTA
  • Use MANTA to determine the order of deployment
  • Order the scripts accordingly
  • Release them in that order
  • Done!

Conclusion

Now, the customer can be sure that the patches will be deployed correctly and that it will be done in just a couple of minutes, making the daily-release strategy actually work out every day.

MANTA has been an enormous help to their situation. And well, what can we say: this case is proof that not every development team chooses to be agile. Some just have to adjust their development pace to the complexity of their environment.

Be sure to read more of our MANTA Cases! Any requests? Any questions? Send your comments to manta@getmanta.com.

MANTA x Record Level Lineage: Why we don’t have it

You may or may not have heard about record level lineage. This is a topic that our customers ask about quite frequently, so our vice president of development, Lukas Hermann, decided to write an article where he answers some of the FAQs. Continue reading to find out more about record level lineage and why we don’t have it.

You may or may not have heard about record level lineage. This is a topic that our customers ask about quite frequently, so our vice president of development, Lukas Hermann, decided to write an article where he answers some of the FAQs. Continue reading to find out more about record level lineage and why we don’t have it.

What is record level lineage?

Record level lineage is an approach to data lineage that is similar to data tagging. The idea behind data tracking is that each piece of data that is being moved or transformed is tagged/labeled by a transformation engine which then tracks that label all along its way from start to finish. This approach seems great, but it only works well when a transformation engine controls the data’s every move. Some good examples are controlled environments like Cloudera or Dremeo that focus only on the origin of one specific record.

Record level lineage vs. column level lineage

A feature that MANTA does have, that in a way is similar to record level lineage, is column level lineage. What exactly is the difference? Let’s look at an example.

Let’s say you have the column full name in your table. In this table, the full name is created by combining the first name and the last name. Imagine that in the full name column you have names like John Snow and Jack Snow. Now, let’s say that the name John Snow came to this table from your own CRM database, but Jack Snow came from a contact database acquired from a third party.

Record level lineage is able to tell you exactly that John Snow came from CRM and Jack Snow came from your contact database. Column level lineage, like in MANTA, is able to tell you that the column full name consists of data from these two databases—your CRM database and your contact database.

Why we don’t have it

The reason why MANTA does not have record level lineage is that MANTA doesn’t “see” your data; it doesn’t even “see” that you have a John Snow and a Jack Snow in your full name column. It only reads your metadata. That is why MANTA only sees a table that contains data from these databases and which databases they are.

Now, you might be thinking that the overall idea of the record level lineage approach might not be so bad after all. But keep in mind that if anything happens outside its walls, the lineage is broken. It is also important to realize that the lineage is only there if the transformation logic has been executed. But think about all the exceptions and rules that apply only once every couple of years. You will not see them in your lineage until they are executed, which is not exactly healthy for your data governance, especially if some of those pieces are critical to your organization.

Also, tags are formed by assigning additional metadata to the records. If you lose this metadata, you will never be able to form the lineage again. And without actually running the transformation engine, you don’t know how the given record was put together, and therefore don’t know the lineage behind it.

In conclusion

If MANTA wanted to have record level lineage, it would have to start reading your data instead of your metadata, and it would have to have much more information about your environment. This would make the entire process of getting data lineage far more complicated and time-consuming.

We can safely say that we are not planning on having record level lineage as a feature any time soon. On the other hand, we plan on putting more effort into understanding your data transformations. The fact that MANTA only reads your metadata and is only interested in your data transformations, not your actual data, is the reason why MANTA can be automated so well and get data lineage so fast.

And what about Conditional Lineage? 

MANTA also has conditional lineage as a feature, and you do look into the actual data when you are creating conditional lineage. Well, not quite. We only use the data that is specifically mentioned in the scripts. You can learn more about conditional lineage in the article: How to Handle Impact Analyses in Complex DWHs with Predicates.

So what MANTA does give you is a list of the exact databases that supply data to the given column in your table. For compliance with regulations such as GDPR and other financial or banking regulations, it is completely sufficient. And typically, there are no more than a few databases that supply each column, so then the question is: If you really need to have the specific database for each record in your table THAT BAD and you can have the databases narrowed down to a few for each column in a couple of hours, wouldn’t it be more efficient to just check those two databases manually for the specific record yourself?

Do you have any development-related questions for Lukas, or would you like to learn more about how MANTA can solve a specific issue in your company? Don’t hesitate to contact us at manta@getmanta.com.

MANTA Cases #2: Pure Development Efficiency

In the second part of the “MANTA Cases” series, we will take a look at a US investment management company that deployed MANTA, primarily for their development team to perform impact analyses and to boost the overall efficiency and agile development processes of their internal software. 

In the second part of the “MANTA Cases” series, we will take a look at a US investment management company that deployed MANTA, primarily for their development team to perform impact analyses and to boost the overall efficiency and agile development processes of their internal software. 

The Customer had a fairly small yet, for its size, very complex data environment. The environment consisted of three databases:

  1. One was used as the main data warehouse that collected data for the core business
  2. The second database was a special investment data mart database (this database had specific requirements, making it even more challenging for it to co-op with new releases)
  3. The third database contained resources for the sales team

The entire environment contained about 15 hundred scripts, of which 20 to 30 percent were legacy code. The other 70 to 80 percent were newer code that was evolving at a fast pace due to the customer’s agile development standards. The customer used Microsoft SQL, most of the ETL code was in T-SQL, and some scripts were in SSAS.

Given the user requirements for each of the separate databases, the environment was growing and its expansion was necessary – otherwise, it would have become a burden. And, as is always the case in development teams, that familiar question was floating around in everyone’s heads: What’s gonna break if we make changes? The customer had developed an automated testing process for new code involving Unitask software, but for the legacy code, there was nothing like that. This made it necessary to deploy a tool that would define the dependencies between various parts of the code.

Now that MANTA has become part of the customer’s development process, MANTA can show the development team the impact a new release will have on existing code, making it easier to catch most of the larger issues during the testing phase. The issues the customer faces after a release have been, in their words, reduced by half and are in general much smaller.

Another benefit of MANTA for the development team is that the overall testing time before a new software release has been cut by 10 to 20 percent, and the number of developers needed to perform a release is now 2 people!

In this case, MANTA helped the customer maintain an agile development strategy for their internal software and improve the overall performance of impact analyses. However, due to the benefits that came with it, such as the reduction in overall testing time and the number of people needed to perform a release, MANTA could easily be beneficial for companies having trouble with staffing as well as for other small development and business analytics teams.

Do you have any questions about this specific case, or would you like to learn more about how to deploy MANTA in your company? Write us at manta@getmanta.com.

We cherish your privacy.

And we need to tell you that this site uses cookies. Learn more in our Privacy Policy.