Use Cases & Case Studies

MANTA + Informatica EDC Tech Bond

MANTA can complete the Enterprise Data Catalog data governance solution from Informatica with some really tough programming code. In the following article we will let you zoom in on all the details of this technical bond. (INCLUDING A NEW VIDEO!)

MANTA can complete the Enterprise Data Catalog data governance solution from Informatica with some really tough programming code. In the following article we will let you zoom in on all the details of this technical bond. (INCLUDING A NEW VIDEO!)

MANTA connects to Informatica EDC and enhances all of its data lineag:

  1. MANTA connects to the same sources as EDC, and scans all the scripts.
  2. Then, it automatically provides data lineage from those scripts and integrates it with EDC’s native resources.
  3. Finally, MANTA analyzes programming code that is out of EDC’s native scope, e.g. stored procedures, views, triggers and other scripts.

With MANTA, every Informatica customer can see how every procedure works, find out how the data is transformed between tables, and get complete end-to-end data lineage down to the column level.

Here you can watch our brand new video where Lukas explains how the technical bond works:

Like our videos? There are more where that came from! Check out our video section right here. And don’t forget to subscribe to our YouTube channel! 

As you have heard in the video above, MANTA can enrich Informatica’s Enterprise Data Catalog with a number of SQL scripts, that allow EDC to show much more depth and detail than it normally could. Here is a list of scripts that MANTA currently pushes into Informatica.

What MANTA currently pushes into Informatica’s Enterprise Data Catalog:

  • BTEQ scripts, stored procedures, views, and macros from Teradata
  • PL/SQL scripts, stored procedures, packages, and more, including DB links, from Oracle DB & Exadata
  • T-SQL scripts, stored procedures, and more, including linked servers, from Microsoft SQL Server, Sybase (now SAP ASE), and PDW
  • NZPLSQL scripts, stored procedures, and more from IBM Netezza
  • DB2 scripts, stored procedures, and more from IBM DB2
  • PostgreSQL scripts, views and more from PostgreSQL, Amazon Redshift and Greenplum

It’s all about the details

MANTA’s key feature is its understanding superpower. The ability to read even the most complex custom code is crucial for obtaining detailed and complete end-to-end data lineage. This can also be used for real data protection analysis, automated business lineage extraction, migration of your DWH to a different platform or the cloud, and to comply with regulations such as GDPR, Basil II/III, and many more.

For customers who have EDC’s older brother, Informatica Metadata Manager, we have a connector for IMM as well. And for customers who have neither, but enjoy the advanced ETL capabilities of Informatica PowerCenter, we are also able to provide data lineage in our own visualization.

All our supported technologies, both scanners and integrations can be found here. If there is anything else you would like to talk to us about, don’t hesitate to contact us at manta@getmanta.com

Key Features of MANTA: Its Understanding Superpower

Welcome to another episode of “Key Features of MANTA”. This time, we are focusing on a feature so mighty, we don’t hesitate to call it a superpower. An understanding superpower, that is!

Welcome to another episode of “Key Features of MANTA”. This time, we are focusing on a feature so mighty, we don’t hesitate to call it a superpower. An understanding superpower, that is!

Many, many vendors sell solutions for metadata management solutions that have many great features that MANTA can’t match. But luckily, we do have a superpower that we lend to other solutions and partners and make use of ourselves, or all of the above. It is the ability to parse tough, scary, custom programming code.

MANTA made itself in SQL, because it can read the most complex segments of SQL, including but not limited to: PL/SQL scripts, BTEQ Scripts, T-SQL scripts, TPT scripts, NZPLSQL scripts,  stored procedures, SQL overrides, Informatica PowerCenter Workflows… The list goes on and on, depending on which technology you use.

But since over time MANTA has been supporting more and more scanners and integrations, we’ve needed to adjust over time to other just as complex programming languages. With MANTA 3.22, we will release support for PostgreSQL (which means Amazon Redshift and Greenplum as well, yay!).

This not the only new language we will add to our family of supported technologies. In 2019, as you have read in the roadmap article we released, MANTA will add Java support, which will open up a whole new world of MANTA + Cloud integrations.

Besides being able to read the actual language, MANTA can differentiate between languages and parse multi-language environments automatically. Automation in combination with our code-parsing, understanding superpower automation and our code parsing, understanding superpower is what makes MANTA unlike any other solution on the market.

Is there something you would like to ask us? We are available at manta@getmanta.com

How to Solve Impact Analysis with MANTA

Our customers use MANTA for all kinds of projects, impact analysis being one of them. When you have a really complex BI environment and still want to perform a reliable impact analysis, using predicates (and MANTA) is one way! Keep on reading to learn how.

Our customers use MANTA for all kinds of projects, impact analysis being one of them. When you have a really complex BI environment and still want to perform a reliable impact analysis, using predicates (and MANTA) is one way! Keep on reading to learn how.

During our pilots and deployments, we often find data warehouse environments that use very general physical models including several big tables like PARTY, BALANCE, ORDER, and others.

These tables contain data obtained from various source systems, and there are a lot of data marts and reports built on top of them. These tables make things difficult during impact analysis because data lineage from almost every report goes through them and into all the sources, making the results hard to use, or even worthless.

Impact Analysis Is Easier Than Ever Before

Let’s take a closer look at an example to understand exactly what happens when you use MANTA for your impact analysis. The table PARTY contains all information about individuals and companies that are somehow related to the organization. Thus, in one table, it is possible to have records for clients, employees, suppliers, and the organization’s branch network. Each type of entity is identified by the unique attribute or source system from which the data is obtained – for example, clients are managed in a different system than employees.

Now, let’s assume that the data from the PARTY table goes into two separate reports – a report EMPL_REPORT that displays information about employees and another report BRANCH_REPORT that displays information about the branch network. If we use the standard data lineage analysis, we can get this picture:

Although only data from the EMPLOYEE source table is relevant for the report EMPL_REPORT, the impact analysis from that report also includes the CLIENT, BRANCH, and SUPPLIER source tables due to the PARTY table. The problem is the same for the report BRANCH_REPORT. From the other side, the impact analysis from the EMPLOYEE source table includes both the EMPL_REPORT and BRANCH_REPORT which is confusing.

The Advantage of Using Data Lineage

Luckily, there is a solution. When data is inserted into the PARTY table from different source systems, there is often a column like PARTY.source_system_id where the identification of the source system is stored as a constant. Similarly, when a report is created that consumes data only from specific source systems, there is a condition in the statement filtering data based on the PARTY.source_system_id column. Thus, it is possible to automatically analyze both the insertion and selection to/from the PARTY table and create predicates such as PARTY.source_system_id = 20 that are then stored together with data lineage in the metadata repository. Therefore, it is possible to include them in the computation during the impact analysis.

Thanks to that, if we perform an impact analysis from the report EMPL_REPORT, the predicate PARTY.source_system_id = 20 is gathered before the table PARTY. When the analysis continues towards source tables, the predicate for each path is selected and compared to what has already been gathered. Therefore, when the path to the source table CLIENT with the predicate PARTY.source_system_id = 10 is tested, the result is that both predicates cannot hold at once, so data for this report cannot come from this source table. Conversely, when the path to the source table EMPLOYEE with the predicate PARTY.souce_system_id = 20 is tested, the result is that data for this report can come from this source table, so it is included in the results of the impact analysis. We can get similar results if we perform an impact analysis for the BRANCH_REPORT and also from sources like the EMPLOYEE table.

The results of the advanced data lineage analysis can look like this (in reality, if we perform the impact analysis from the EMPL_REPORT, we will only see the EMPLOYEE and PARTY tables):

Surely, the situation can be far more complex. For example, the data from the PARTY table can be pre-computed for more source systems first, and then several reports can be created on top of them for only a specific source system, like in this picture:

If you have any questions or comments, feel free to contact Lukas at manta@mantatools.com. You can try these predicate-based impact analyses in our free trial!

 

 

TopQuadrant and MANTA Partner to Further Automate the Discovery of Data

TopQuadrant™, a leading provider of knowledge graph-based data governance solutions, today announced that it has partnered with MANTA to help its customers automate the discovery of data lineage information.

TopQuadrant™, a leading provider of knowledge graph-based data governance solutions, today announced that it has partnered with MANTA to help its customers automate the discovery of data lineage information.

TopQuadrant’s flagship product, TopBraid Enterprise Data Governance (EDG), helps organizations succeed in data governance. It delivers easy and meaningful access for all data stakeholders to enterprise metadata, business terms, reference data, data and application catalogs, data lineage, data exchanges and pipelines, requirements, policies, and processes. MANTA helps enterprises all around the world to automate their data lineage across many different technologies. On top of its unique data lineage extraction capabilities, MANTA offers a broad variety of connectors to different data governance solutions, as well as API.

The technology partnership between TopQuadrant and MANTA delivers an integration between TopBraid EDG and MANTA. Customers will be able to take advantage of the combined capabilities for data governance and lineage insight. After the initial setup, MANTA automatically reads the customer’s metadata and provides the metadata to TopBraid EDG.

Our customers are looking to connect the business context and meaning of their diverse data to its technical metadata in order to understand and manage it as an enterprise asset. The integration of TopBraid EDG with MANTA lets them automate the discovery of technical data lineage embedded in the SQL code and seamlessly bring it into TopBraid EDG where it gets put in context and connected to other relevant information,” said Irene Polikoff, CEO of TopQuadrant.

The end result is faster delivery of comprehensive knowledge graphs that provide businesses with meaningful information inferred from connected catalogs of terms, data, technical and business assets. This enables our customers to comply with regulations, produce accurate reports, understand the impact of changes and meet other demands common in today’s data rich environments,” said Ralph Hodgson, CTO of TopQuadrant.

Tomas Kratky, CEO of MANTA, agrees that this technical partnership gives the customer outstanding features, that it would be hard to seek in any other one solution alone: “Building custom integrations for state-of-the-art data governance solutions such as TopBraid EDG gives MANTA the ability to serve a broad range of customers, allowing them to have a data governance solution with flawless data lineage extraction and visualization capabilities, which allows them to strengthen their trust in data.

Both MANTA and TopQuadrant have a very customer-centric approach and are aiming to target very specific customer needs as well as to become a one-stop-shop. Even though the partnership has formed only recently, MANTA’s and TopQuadrant’s technology partnership already hosts its first customer, a large financial services provider. The joint solution allows them to have complete trust in their data as well as be compliant with all global industry regulations.


About MANTA

MANTA automates its customers’ data lineage and enhances capabilities of data governance solutions. The essentiality of MANTA lies within its ability to process and understand custom programming code and the ability to describe its logic. The software uses metadata harvested from the customer’s systems to visualize data lineage throughout the Business Intelligence environment. The data lineage can be further used for data warehouse optimization, lowering development costs, impact analyses, data migration, data discovery, data security projects and for meeting regulatory compliance standards for CCAR, HIPAA, BASEL II/III, GDPR and other regulations. Some of MANTA’s customers include PayPal, Comcast, Vodafone and OBI. Visit getmanta.com for more.

About TopQuadrant

TopQuadrant helps organizations succeed in data governance with TopBraid Enterprise Data Governance (TopBraid EDG).  EDG is built on standards-based knowledge graph technology to seamlessly bring together enterprise information silos, connect enterprise metadata, ensure its quality and deliver easy and meaningful access for all data stakeholders. TopBraid EDG lets enterprises govern all relevant assets including business terms, reference data, enterprise metadata, data and application catalogs, data lineage, data exchanges and pipelines, requirements, policies, and processes. TopBraid EDG is the only solution built to support integrated governance across all types of assets and governance needs while, at the same time, offering a staged approach to data governance. TopQuadrant’s customer list includes organizations in financial services, pharma, healthcare, digital media, government and other sectors. www.topquadrant.com.

Any questions or comments about the partnership and/or its fruits? Ask MANTA here or TopQuadrant here.

To Migrate, or Not to Migrate, the Report?

When migrating data, you can get into some pretty tricky situations. It is always tough to decide what data to migrate and what not to. MANTA can help you decide as well as reveal some crucial relations within your environment. Read on to learn about the surprises you may find in your DWH.

When migrating data, you can get into some pretty tricky situations. It is always tough to decide what data to migrate and what not to. MANTA can help you decide as well as reveal some crucial relations within your environment. Read on to learn about the surprises you may find in your DWH.

The Case

While one of our customers was dealing with their data migration project, something unexpected came up. When MANTA helped the customer see the relations and detailed dependencies between all the reports, the customer came to realize that multiple reports in the environment were sharing tables with each other. But the reports were of different levels of importance.

MANTA’s Solution?

MANTA’s ability to show all the dependencies within the environment prior to migration helped the customer approach the problem proactively as a complex matter. This saved the customer a lot of time and allowed them to avoid future complications. Otherwise, the customer may have found these relations much later, such as after the migration when one of the reports stopped working properly. Then they would have needed to go through thousands of tables manually to see where the problem was. MANTA enabled the customer to be prepared and gave them the opportunity to make the necessary decisions ahead of time, before any problems occurred.

Difficult Decisions

Multiple reports in the environment were sharing tables with each other. But the reports were of different levels of importance. In one case, the customer wanted to migrate a report to the newly established cloud because it was a frequently used report that was worth keeping. But the report shared some tables with another report that had much lower priority for the customer’s company. However, regardless of its lower priority, it wasn’t exactly a report that could be deleted. It was still used occasionally.

The customer could migrate all the reports and tables involved, but that would inconveniently inflate the amount of data being migrated. Alternatively, the customer could skip these reports and tables and not include them in the migration, but then they would not be able to use the reports in the newly established cloud environment. The customer could duplicate all the data, but what if something changed? Having the same data in multiple places always requires effective synchronization to make sure the data is identical and up-to-date everywhere, which is usually a really complex problem.

So what should the customer do? Have you experienced a similar situation, or are you afraid that this is your case too and you need MANTA’s help to find out for yourself? Contact us at manta@getmanta.com or join the discussion under our post on social media!

We cherish your privacy.

And we need to tell you that this site uses cookies. Learn more in our Privacy Policy.