Use Cases & Case Studies

MANTA + Informatica: The Ultimate Compliance Strategy

The financial industry is known for being stitched through with tons of regulations, and it is indeed challenging to comply with them. Banks and financial institutions reach out for tools to automate the process as much as possible. Read how MANTA and Informatica’s tech bond creates the ideal compliance solution.

The financial industry is known for being stitched through with tons of regulations, and it is indeed challenging to comply with them. Banks and financial institutions reach out for tools to automate the process as much as possible. Read how MANTA and Informatica’s tech bond creates the ideal compliance solution.

Most of the regulations that are used today in banking were invented to protect the industry after the global financial crisis in the years 2007-2008. One of the most well known regulations is BCBS 239. But all of these regulations define a huge number of principles that need to be complied with, sometimes at very high costs.

BCBS 239 & more

We can take the above mentioned regulation BCBS 239 as an example. It contains 14 principles that need to be fully adopted, for the company to achieve  the status of regulatory compliance. Complying with such a complex regulation is a challenging, company-wide issue that requires processing information across a broad range of tasks, departments and entities within the financial institution.

But for all of the regulations in the industry, there is one thing that all implementation strategies have in common – the need for transparency. For example, when banks are asked to report aggregated risk metrics for BCBS 239, they are also asked to prove to the regulators that the risk scores have arrived correctly by:

  1. Showing end-to-end data lineage from the source of the data in the data warehouse to the end report in the BI environment. This includes where the data first came from – its origin, what transformations where performed on it and what processes and decisions affected its life-cycle.
  2. Demonstrating Data Quality Controls: identification, assessment and management of data quality.
  3. Showing an enterprise wide understanding of business concepts: by tracking which data is available to which appropriate teams within the organization.

 

 

Accuracy is excellency

One of our customers has told us that when it comes to regulatory compliance, they refuse to leave responsibility on their employees, but rather seek trust in their data governance solution. We can not blame them, because before choosing to implement a MANTA + Informatica regulatory compliance solution, the customer had a team of more than 20 people that manually drew data lineage diagrams for more than 200 risk metrics within their institution.

We know based on our experience with data lineage across various industries, that it is almost impossible for humans to create accurate data lineage visualization, because there is an endless amount and data and multiple dimensions that need to be incorporated – among others the quality of the data, the people that are allowed to handle it, the processes that are done with it, etc. It requires the team to talk to multiple departments to get the insight on a company-wide level, as well as that it is difficult to maintain because it requires the team to keep track of even the smallest changes within the systems as well as the data warehouse.

What MANTA together with Informatica does is purely automates the data lineage gathering process. Informatica’s Enterprise Data Catalog, with strong machine learning abilities can provide automatic mechanisms for data discovery across multiple internal systems in the company. What MANTA can then do is automatically scan the metadata within the systems and read even the toughest parts of custom made programming code that the EDC usually has a hard time dealing with.

Then the customer can either visualize data lineage in MANTA’s native visualization, or see the data lineage within the EDC, only this time it is with data lineage all the way down to the column level. This way this was the customer can achieve the highest level of detail possible, that would have otherwise been almost impossible to gain only with one solution. Now, whether it is an older data lineage diagram remodeled calculating with new dependencies, or if it is creation of a completely new one, the customer can be sure that the diagram is accurate and consists of all details.

Visit getmanta.com/informatica to learn even more details about our tech bond.

Would you like more similar use cases in the future? Or details about our scanners and integrations? Let us know on manta@getmanta.com, we are always here for you.

MANTA + Informatica EDC Tech Bond

MANTA can complete the Enterprise Data Catalog data governance solution from Informatica with some really tough programming code. In the following article we will let you zoom in on all the details of this technical bond. (INCLUDING A NEW VIDEO!)

MANTA can complete the Enterprise Data Catalog data governance solution from Informatica with some really tough programming code. In the following article we will let you zoom in on all the details of this technical bond. (INCLUDING A NEW VIDEO!)

MANTA connects to Informatica EDC and enhances all of its data lineag:

  1. MANTA connects to the same sources as EDC, and scans all the scripts.
  2. Then, it automatically provides data lineage from those scripts and integrates it with EDC’s native resources.
  3. Finally, MANTA analyzes programming code that is out of EDC’s native scope, e.g. stored procedures, views, triggers and other scripts.

With MANTA, every Informatica customer can see how every procedure works, find out how the data is transformed between tables, and get complete end-to-end data lineage down to the column level.

Here you can watch our brand new video where Lukas explains how the technical bond works:

Like our videos? There are more where that came from! Check out our video section right here. And don’t forget to subscribe to our YouTube channel! 

As you have heard in the video above, MANTA can enrich Informatica’s Enterprise Data Catalog with a number of SQL scripts, that allow EDC to show much more depth and detail than it normally could. Here is a list of scripts that MANTA currently pushes into Informatica.

What MANTA currently pushes into Informatica’s Enterprise Data Catalog:

  • BTEQ scripts, stored procedures, views, and macros from Teradata
  • PL/SQL scripts, stored procedures, packages, and more, including DB links, from Oracle DB & Exadata
  • T-SQL scripts, stored procedures, and more, including linked servers, from Microsoft SQL Server, Sybase (now SAP ASE), and PDW
  • NZPLSQL scripts, stored procedures, and more from IBM Netezza
  • DB2 scripts, stored procedures, and more from IBM DB2
  • PostgreSQL scripts, views and more from PostgreSQL, Amazon Redshift and Greenplum

It’s all about the details

MANTA’s key feature is its understanding superpower. The ability to read even the most complex custom code is crucial for obtaining detailed and complete end-to-end data lineage. This can also be used for real data protection analysis, automated business lineage extraction, migration of your DWH to a different platform or the cloud, and to comply with regulations such as GDPR, Basil II/III, and many more.

For customers who have EDC’s older brother, Informatica Metadata Manager, we have a connector for IMM as well. And for customers who have neither, but enjoy the advanced ETL capabilities of Informatica PowerCenter, we are also able to provide data lineage in our own visualization.

All our supported technologies, both scanners and integrations can be found here. If there is anything else you would like to talk to us about, don’t hesitate to contact us at manta@getmanta.com

Key Features of MANTA: Its Understanding Superpower

Welcome to another episode of “Key Features of MANTA”. This time, we are focusing on a feature so mighty, we don’t hesitate to call it a superpower. An understanding superpower, that is!

Welcome to another episode of “Key Features of MANTA”. This time, we are focusing on a feature so mighty, we don’t hesitate to call it a superpower. An understanding superpower, that is!

Many, many vendors sell solutions for metadata management solutions that have many great features that MANTA can’t match. But luckily, we do have a superpower that we lend to other solutions and partners and make use of ourselves, or all of the above. It is the ability to parse tough, scary, custom programming code.

MANTA made itself in SQL, because it can read the most complex segments of SQL, including but not limited to: PL/SQL scripts, BTEQ Scripts, T-SQL scripts, TPT scripts, NZPLSQL scripts,  stored procedures, SQL overrides, Informatica PowerCenter Workflows… The list goes on and on, depending on which technology you use.

But since over time MANTA has been supporting more and more scanners and integrations, we’ve needed to adjust over time to other just as complex programming languages. With MANTA 3.22, we will release support for PostgreSQL (which means Amazon Redshift and Greenplum as well, yay!).

This not the only new language we will add to our family of supported technologies. In 2019, as you have read in the roadmap article we released, MANTA will add Java support, which will open up a whole new world of MANTA + Cloud integrations.

Besides being able to read the actual language, MANTA can differentiate between languages and parse multi-language environments automatically. Automation in combination with our code-parsing, understanding superpower automation and our code parsing, understanding superpower is what makes MANTA unlike any other solution on the market.

Is there something you would like to ask us? We are available at manta@getmanta.com

How to Solve Impact Analysis with MANTA

Our customers use MANTA for all kinds of projects, impact analysis being one of them. When you have a really complex BI environment and still want to perform a reliable impact analysis, using predicates (and MANTA) is one way! Keep on reading to learn how.

Our customers use MANTA for all kinds of projects, impact analysis being one of them. When you have a really complex BI environment and still want to perform a reliable impact analysis, using predicates (and MANTA) is one way! Keep on reading to learn how.

During our pilots and deployments, we often find data warehouse environments that use very general physical models including several big tables like PARTY, BALANCE, ORDER, and others.

These tables contain data obtained from various source systems, and there are a lot of data marts and reports built on top of them. These tables make things difficult during impact analysis because data lineage from almost every report goes through them and into all the sources, making the results hard to use, or even worthless.

Impact Analysis Is Easier Than Ever Before

Let’s take a closer look at an example to understand exactly what happens when you use MANTA for your impact analysis. The table PARTY contains all information about individuals and companies that are somehow related to the organization. Thus, in one table, it is possible to have records for clients, employees, suppliers, and the organization’s branch network. Each type of entity is identified by the unique attribute or source system from which the data is obtained – for example, clients are managed in a different system than employees.

Now, let’s assume that the data from the PARTY table goes into two separate reports – a report EMPL_REPORT that displays information about employees and another report BRANCH_REPORT that displays information about the branch network. If we use the standard data lineage analysis, we can get this picture:

Although only data from the EMPLOYEE source table is relevant for the report EMPL_REPORT, the impact analysis from that report also includes the CLIENT, BRANCH, and SUPPLIER source tables due to the PARTY table. The problem is the same for the report BRANCH_REPORT. From the other side, the impact analysis from the EMPLOYEE source table includes both the EMPL_REPORT and BRANCH_REPORT which is confusing.

The Advantage of Using Data Lineage

Luckily, there is a solution. When data is inserted into the PARTY table from different source systems, there is often a column like PARTY.source_system_id where the identification of the source system is stored as a constant. Similarly, when a report is created that consumes data only from specific source systems, there is a condition in the statement filtering data based on the PARTY.source_system_id column. Thus, it is possible to automatically analyze both the insertion and selection to/from the PARTY table and create predicates such as PARTY.source_system_id = 20 that are then stored together with data lineage in the metadata repository. Therefore, it is possible to include them in the computation during the impact analysis.

Thanks to that, if we perform an impact analysis from the report EMPL_REPORT, the predicate PARTY.source_system_id = 20 is gathered before the table PARTY. When the analysis continues towards source tables, the predicate for each path is selected and compared to what has already been gathered. Therefore, when the path to the source table CLIENT with the predicate PARTY.source_system_id = 10 is tested, the result is that both predicates cannot hold at once, so data for this report cannot come from this source table. Conversely, when the path to the source table EMPLOYEE with the predicate PARTY.souce_system_id = 20 is tested, the result is that data for this report can come from this source table, so it is included in the results of the impact analysis. We can get similar results if we perform an impact analysis for the BRANCH_REPORT and also from sources like the EMPLOYEE table.

The results of the advanced data lineage analysis can look like this (in reality, if we perform the impact analysis from the EMPL_REPORT, we will only see the EMPLOYEE and PARTY tables):

Surely, the situation can be far more complex. For example, the data from the PARTY table can be pre-computed for more source systems first, and then several reports can be created on top of them for only a specific source system, like in this picture:

If you have any questions or comments, feel free to contact Lukas at manta@mantatools.com. You can try these predicate-based impact analyses in our free trial!

 

 

TopQuadrant and MANTA Partner to Further Automate the Discovery of Data

TopQuadrant™, a leading provider of knowledge graph-based data governance solutions, today announced that it has partnered with MANTA to help its customers automate the discovery of data lineage information.

TopQuadrant™, a leading provider of knowledge graph-based data governance solutions, today announced that it has partnered with MANTA to help its customers automate the discovery of data lineage information.

TopQuadrant’s flagship product, TopBraid Enterprise Data Governance (EDG), helps organizations succeed in data governance. It delivers easy and meaningful access for all data stakeholders to enterprise metadata, business terms, reference data, data and application catalogs, data lineage, data exchanges and pipelines, requirements, policies, and processes. MANTA helps enterprises all around the world to automate their data lineage across many different technologies. On top of its unique data lineage extraction capabilities, MANTA offers a broad variety of connectors to different data governance solutions, as well as API.

The technology partnership between TopQuadrant and MANTA delivers an integration between TopBraid EDG and MANTA. Customers will be able to take advantage of the combined capabilities for data governance and lineage insight. After the initial setup, MANTA automatically reads the customer’s metadata and provides the metadata to TopBraid EDG.

Our customers are looking to connect the business context and meaning of their diverse data to its technical metadata in order to understand and manage it as an enterprise asset. The integration of TopBraid EDG with MANTA lets them automate the discovery of technical data lineage embedded in the SQL code and seamlessly bring it into TopBraid EDG where it gets put in context and connected to other relevant information,” said Irene Polikoff, CEO of TopQuadrant.

The end result is faster delivery of comprehensive knowledge graphs that provide businesses with meaningful information inferred from connected catalogs of terms, data, technical and business assets. This enables our customers to comply with regulations, produce accurate reports, understand the impact of changes and meet other demands common in today’s data rich environments,” said Ralph Hodgson, CTO of TopQuadrant.

Tomas Kratky, CEO of MANTA, agrees that this technical partnership gives the customer outstanding features, that it would be hard to seek in any other one solution alone: “Building custom integrations for state-of-the-art data governance solutions such as TopBraid EDG gives MANTA the ability to serve a broad range of customers, allowing them to have a data governance solution with flawless data lineage extraction and visualization capabilities, which allows them to strengthen their trust in data.

Both MANTA and TopQuadrant have a very customer-centric approach and are aiming to target very specific customer needs as well as to become a one-stop-shop. Even though the partnership has formed only recently, MANTA’s and TopQuadrant’s technology partnership already hosts its first customer, a large financial services provider. The joint solution allows them to have complete trust in their data as well as be compliant with all global industry regulations.


About MANTA

MANTA automates its customers’ data lineage and enhances capabilities of data governance solutions. The essentiality of MANTA lies within its ability to process and understand custom programming code and the ability to describe its logic. The software uses metadata harvested from the customer’s systems to visualize data lineage throughout the Business Intelligence environment. The data lineage can be further used for data warehouse optimization, lowering development costs, impact analyses, data migration, data discovery, data security projects and for meeting regulatory compliance standards for CCAR, HIPAA, BASEL II/III, GDPR and other regulations. Some of MANTA’s customers include PayPal, Comcast, Vodafone and OBI. Visit getmanta.com for more.

About TopQuadrant

TopQuadrant helps organizations succeed in data governance with TopBraid Enterprise Data Governance (TopBraid EDG).  EDG is built on standards-based knowledge graph technology to seamlessly bring together enterprise information silos, connect enterprise metadata, ensure its quality and deliver easy and meaningful access for all data stakeholders. TopBraid EDG lets enterprises govern all relevant assets including business terms, reference data, enterprise metadata, data and application catalogs, data lineage, data exchanges and pipelines, requirements, policies, and processes. TopBraid EDG is the only solution built to support integrated governance across all types of assets and governance needs while, at the same time, offering a staged approach to data governance. TopQuadrant’s customer list includes organizations in financial services, pharma, healthcare, digital media, government and other sectors. www.topquadrant.com.

Any questions or comments about the partnership and/or its fruits? Ask MANTA here or TopQuadrant here.

We cherish your privacy.

And we need to tell you that this site uses cookies. Learn more in our Privacy Policy.