Use Cases & Case Studies

How to Solve Impact Analysis with MANTA

Our customers use MANTA for all kinds of projects, impact analysis being one of them. When you have a really complex BI environment and still want to perform a reliable impact analysis, using predicates (and MANTA) is one way! Keep on reading to learn how.

Our customers use MANTA for all kinds of projects, impact analysis being one of them. When you have a really complex BI environment and still want to perform a reliable impact analysis, using predicates (and MANTA) is one way! Keep on reading to learn how.

During our pilots and deployments, we often find data warehouse environments that use very general physical models including several big tables like PARTY, BALANCE, ORDER, and others.

These tables contain data obtained from various source systems, and there are a lot of data marts and reports built on top of them. These tables make things difficult during impact analysis because data lineage from almost every report goes through them and into all the sources, making the results hard to use, or even worthless.

Impact Analysis Is Easier Than Ever Before

Let’s take a closer look at an example to understand exactly what happens when you use MANTA for your impact analysis. The table PARTY contains all information about individuals and companies that are somehow related to the organization. Thus, in one table, it is possible to have records for clients, employees, suppliers, and the organization’s branch network. Each type of entity is identified by the unique attribute or source system from which the data is obtained – for example, clients are managed in a different system than employees.

Now, let’s assume that the data from the PARTY table goes into two separate reports – a report EMPL_REPORT that displays information about employees and another report BRANCH_REPORT that displays information about the branch network. If we use the standard data lineage analysis, we can get this picture:

Although only data from the EMPLOYEE source table is relevant for the report EMPL_REPORT, the impact analysis from that report also includes the CLIENT, BRANCH, and SUPPLIER source tables due to the PARTY table. The problem is the same for the report BRANCH_REPORT. From the other side, the impact analysis from the EMPLOYEE source table includes both the EMPL_REPORT and BRANCH_REPORT which is confusing.

The Advantage of Using Data Lineage

Luckily, there is a solution. When data is inserted into the PARTY table from different source systems, there is often a column like PARTY.source_system_id where the identification of the source system is stored as a constant. Similarly, when a report is created that consumes data only from specific source systems, there is a condition in the statement filtering data based on the PARTY.source_system_id column. Thus, it is possible to automatically analyze both the insertion and selection to/from the PARTY table and create predicates such as PARTY.source_system_id = 20 that are then stored together with data lineage in the metadata repository. Therefore, it is possible to include them in the computation during the impact analysis.

Thanks to that, if we perform an impact analysis from the report EMPL_REPORT, the predicate PARTY.source_system_id = 20 is gathered before the table PARTY. When the analysis continues towards source tables, the predicate for each path is selected and compared to what has already been gathered. Therefore, when the path to the source table CLIENT with the predicate PARTY.source_system_id = 10 is tested, the result is that both predicates cannot hold at once, so data for this report cannot come from this source table. Conversely, when the path to the source table EMPLOYEE with the predicate PARTY.souce_system_id = 20 is tested, the result is that data for this report can come from this source table, so it is included in the results of the impact analysis. We can get similar results if we perform an impact analysis for the BRANCH_REPORT and also from sources like the EMPLOYEE table.

The results of the advanced data lineage analysis can look like this (in reality, if we perform the impact analysis from the EMPL_REPORT, we will only see the EMPLOYEE and PARTY tables):

Surely, the situation can be far more complex. For example, the data from the PARTY table can be pre-computed for more source systems first, and then several reports can be created on top of them for only a specific source system, like in this picture:

If you have any questions or comments, feel free to contact Lukas at manta@mantatools.com. You can try these predicate-based impact analyses in our free trial!

 

 

TopQuadrant and MANTA Partner to Further Automate the Discovery of Data

TopQuadrant™, a leading provider of knowledge graph-based data governance solutions, today announced that it has partnered with MANTA to help its customers automate the discovery of data lineage information.

TopQuadrant™, a leading provider of knowledge graph-based data governance solutions, today announced that it has partnered with MANTA to help its customers automate the discovery of data lineage information.

TopQuadrant’s flagship product, TopBraid Enterprise Data Governance (EDG), helps organizations succeed in data governance. It delivers easy and meaningful access for all data stakeholders to enterprise metadata, business terms, reference data, data and application catalogs, data lineage, data exchanges and pipelines, requirements, policies, and processes. MANTA helps enterprises all around the world to automate their data lineage across many different technologies. On top of its unique data lineage extraction capabilities, MANTA offers a broad variety of connectors to different data governance solutions, as well as API.

The technology partnership between TopQuadrant and MANTA delivers an integration between TopBraid EDG and MANTA. Customers will be able to take advantage of the combined capabilities for data governance and lineage insight. After the initial setup, MANTA automatically reads the customer’s metadata and provides the metadata to TopBraid EDG.

Our customers are looking to connect the business context and meaning of their diverse data to its technical metadata in order to understand and manage it as an enterprise asset. The integration of TopBraid EDG with MANTA lets them automate the discovery of technical data lineage embedded in the SQL code and seamlessly bring it into TopBraid EDG where it gets put in context and connected to other relevant information,” said Irene Polikoff, CEO of TopQuadrant.

The end result is faster delivery of comprehensive knowledge graphs that provide businesses with meaningful information inferred from connected catalogs of terms, data, technical and business assets. This enables our customers to comply with regulations, produce accurate reports, understand the impact of changes and meet other demands common in today’s data rich environments,” said Ralph Hodgson, CTO of TopQuadrant.

Tomas Kratky, CEO of MANTA, agrees that this technical partnership gives the customer outstanding features, that it would be hard to seek in any other one solution alone: “Building custom integrations for state-of-the-art data governance solutions such as TopBraid EDG gives MANTA the ability to serve a broad range of customers, allowing them to have a data governance solution with flawless data lineage extraction and visualization capabilities, which allows them to strengthen their trust in data.

Both MANTA and TopQuadrant have a very customer-centric approach and are aiming to target very specific customer needs as well as to become a one-stop-shop. Even though the partnership has formed only recently, MANTA’s and TopQuadrant’s technology partnership already hosts its first customer, a large financial services provider. The joint solution allows them to have complete trust in their data as well as be compliant with all global industry regulations.


About MANTA

MANTA automates its customers’ data lineage and enhances capabilities of data governance solutions. The essentiality of MANTA lies within its ability to process and understand custom programming code and the ability to describe its logic. The software uses metadata harvested from the customer’s systems to visualize data lineage throughout the Business Intelligence environment. The data lineage can be further used for data warehouse optimization, lowering development costs, impact analyses, data migration, data discovery, data security projects and for meeting regulatory compliance standards for CCAR, HIPAA, BASEL II/III, GDPR and other regulations. Some of MANTA’s customers include PayPal, Comcast, Vodafone and OBI. Visit getmanta.com for more.

About TopQuadrant

TopQuadrant helps organizations succeed in data governance with TopBraid Enterprise Data Governance (TopBraid EDG).  EDG is built on standards-based knowledge graph technology to seamlessly bring together enterprise information silos, connect enterprise metadata, ensure its quality and deliver easy and meaningful access for all data stakeholders. TopBraid EDG lets enterprises govern all relevant assets including business terms, reference data, enterprise metadata, data and application catalogs, data lineage, data exchanges and pipelines, requirements, policies, and processes. TopBraid EDG is the only solution built to support integrated governance across all types of assets and governance needs while, at the same time, offering a staged approach to data governance. TopQuadrant’s customer list includes organizations in financial services, pharma, healthcare, digital media, government and other sectors. www.topquadrant.com.

Any questions or comments about the partnership and/or its fruits? Ask MANTA here or TopQuadrant here.

To Migrate, or Not to Migrate, the Report?

When migrating data, you can get into some pretty tricky situations. It is always tough to decide what data to migrate and what not to. MANTA can help you decide as well as reveal some crucial relations within your environment. Read on to learn about the surprises you may find in your DWH.

When migrating data, you can get into some pretty tricky situations. It is always tough to decide what data to migrate and what not to. MANTA can help you decide as well as reveal some crucial relations within your environment. Read on to learn about the surprises you may find in your DWH.

The Case

While one of our customers was dealing with their data migration project, something unexpected came up. When MANTA helped the customer see the relations and detailed dependencies between all the reports, the customer came to realize that multiple reports in the environment were sharing tables with each other. But the reports were of different levels of importance.

MANTA’s Solution?

MANTA’s ability to show all the dependencies within the environment prior to migration helped the customer approach the problem proactively as a complex matter. This saved the customer a lot of time and allowed them to avoid future complications. Otherwise, the customer may have found these relations much later, such as after the migration when one of the reports stopped working properly. Then they would have needed to go through thousands of tables manually to see where the problem was. MANTA enabled the customer to be prepared and gave them the opportunity to make the necessary decisions ahead of time, before any problems occurred.

Difficult Decisions

Multiple reports in the environment were sharing tables with each other. But the reports were of different levels of importance. In one case, the customer wanted to migrate a report to the newly established cloud because it was a frequently used report that was worth keeping. But the report shared some tables with another report that had much lower priority for the customer’s company. However, regardless of its lower priority, it wasn’t exactly a report that could be deleted. It was still used occasionally.

The customer could migrate all the reports and tables involved, but that would inconveniently inflate the amount of data being migrated. Alternatively, the customer could skip these reports and tables and not include them in the migration, but then they would not be able to use the reports in the newly established cloud environment. The customer could duplicate all the data, but what if something changed? Having the same data in multiple places always requires effective synchronization to make sure the data is identical and up-to-date everywhere, which is usually a really complex problem.

So what should the customer do? Have you experienced a similar situation, or are you afraid that this is your case too and you need MANTA’s help to find out for yourself? Contact us at manta@getmanta.com or join the discussion under our post on social media!

Automated, Painless Proof-of-Concept? Learn How MANTA Does It

The PoC (Proof of Concept) is a standard process intended to show potential MANTA customers that our solution works in their environments. To give you some insight into the process itself, we will walk you through it. Come right this way!

The PoC (Proof of Concept) is a standard process intended to show potential MANTA customers that our solution works in their environments. To give you some insight into the process itself, we will walk you through it. Come right this way!

Well, here’s just four things you should know:

1. It’s Automated

Automation has always been at the core of MANTA’s design. And we are very proud to say that our PoCs are easy, fast, and painless because of the amount of automation we can offer. There is one catch, though. MANTA needs to be installed (usually locally, but a remote option is also available) and connected to your environment (obviously non-production). In many organizations, that requires some effort – connections need to be approved by security officers and that might take a while. But how can you test anything without letting it – you know – do its thing?

2. It’s Free

Have you read number one? To put it simply: MANTA’s standard proofs-of-concept is free of charge because it’s effortless on both sides, and takes about 30 days (but most of it is within one week!). We know that many companies charge anything from $1000 to approximately a gazillion dollars to do a PoC. Our usual PoC is made to be quick and painless.

However, if the environment is complex, with a lot of proprietary steps and running on multiple platforms, we might not be able to run our standard PoC. Then there would be a made-to-measure PoC, with a fee depending on the amount of custom work required. It’s not a full integration, so we always agree on the scope, clear terms and conditions with each customer at the beginning so nobody is left in the wind.

3. It’s Properly Tracked & Documented

And that’s why we have everything logged via a simple, yet powerful documentation process. And how does that work?

There are usually five people who participate in the PoC process:

  • Sales and Technical Support experts from MANTA (That’s two and only two, we promise. No need to get overwhelmed.)
  • And on the customer’s side: PoC Manager for the process (let’s call him the champion, shall we?), Technical Participant who knows the environment, and Evaluation Analyst capable of evaluating results and the added value of MANTA in the overall solution implementation process (that’s three, obviously not all of them are needed at any given moment)

Set-Up

The process itself usually kicks off with a pre-installation phase, where the customer and MANTA check-off a list of prerequisites prior to the installation phase. These vary based on the technologies selected for the PoC, but we do have a list of requirements ready. After that comes the installation phase, where the future administrator/manager of MANTA is trained on the application and the software is installed into the customer’s system, configured and, finally, tested!

Live Testing

Then comes an inspection of the KPIs (Key Performance Indicators) that serves to verify how the solutions work. After the future MANTA users are trained on the software, they summarize how satisfied they were with the test.

The KPIs usually include:

  • the amount of automatically visualized data lineage and if it is visualized correctly
  • the amount of labor saved on a selected task, for example impact analysis, regulatory compliance, self-service or pretty much anything else (above 80% is what we aim to achieve)

We track this process in our system, MANTA Help Desk & Knowledge Base. The accounts are free of charge for any number of people necessary for the PoC, and later, for the actual use of MANTA.

4. It’s Not Demanding for the Customer Either

It’s completed in 30 days, but most of the work is done within a week or so. Yes, you can install-test-uninstall MANTA way faster (within one day or less), but these are the best practices that help our customers to achieve the most successful PoC possible (and hey, it’s still faster and cheaper than the other guys, right?).

For the technical part of the process, a physical or virtual machine with these specs is needed:

  • CPU minimum: 4 cores at 2.5 GHz
  • RAM minimum: 8 GB
  • HDD: 100 MB for Manta + space for metadata up to several dozens of GBs

There’s no need to have a dedicated machine, a virtual machine with these resources is also ok. You can use existing machine to start the PoC as soon as possible. Check out our supported technologies list for stuff that we can parse, and let’s just say that machine-wise, anything you can run Java on should be just fine. Or just ask us at manta@getmanta.com.

What the API*?#! – A Guide to MANTA’s API

If you are a regular reader, you might have noticed that the second half of 2017 was filled with all kinds of API updates. But what are they for, how do they work, and why do we have them? Here is a much-needed guide to our APIs!

If you are a regular reader, you might have noticed that the second half of 2017 was filled with all kinds of API updates. But what are they for, how do they work, and why do we have them? Here is a much-needed guide to our APIs!

The Repository API

MANTA made this API to allow customers to have a metadata management solution that is tailored to fit their needs. It is often tricky to find a solution with the exact functions you are looking for. This can be solved by requesting a custom-made tool from a professional developer or a consultancy company, but these tools will always lack MANTA’s critical metadata-reading capabilities. Or, you can get MANTA’s API!

The Repository API works with metadata that have already been processed by MANTA. It allows customers to take the data from MANTA’s native repository (reusing all of MANTA’s capabilities, like filtering) and inject them into any piece of software.

For example, let’s say your company is interested in only one table or column and has a program for watching this particular data. It can have an employee check MANTA on a daily basis and report the changes – or it can connect MANTA to a reporting tool through the Repository API and let it scan the area automatically, saving the company hours of time and a fair amount of money.

The Service API

The Service API was created for partners that want to use MANTA’s SQL scanning capabilities together with their own solutions. MANTA can be connected to their software and help them understand the parts of logic written in SQL (SQL overrides, Informatica Powercenter, Cognos, etc.) in the system they are analyzing.

If the partner’s solution isn’t able to analyze some of the custom SQL code in the data warehouse being analyzed, it can lead to incomplete data lineage because those parts are just playing hide-and-go-seek with them.

By integrating with MANTA or connecting the Service API to their own connectors (i.e., ETL scanners, reporting platforms), partners can borrow MANTA’s super-power and use it whenever they need to! The opportunities with the Service API in large-scale projects are limitless.

The Summary?

MANTA can help you get end-to-end data lineage no matter what metadata management or data governance solution you have.  We want partners and customers to be able to trust their own data and for that to be achievable for everyone. And we believe these APIs can do just that.

Need some Lineage yourself? manta@getmanta.com is the address. Shoot us an e-mail and we’ll set up a demo no matter what metadata management or data governance solution you have.

A Tech Support Love Letter: Be Our Valentine!

Valentine’s Day is heading our way and we know who our Valentine will be, but do you? We’ve made you this card so you can see how our customers get nothing but love and affection from MANTA’s support and development teams. It’s a love letter in numbers, enjoy!

Valentine’s Day is heading our way and we know who our Valentine will be, but do you? We’ve made you this card so you can see how our customers get nothing but love and affection from MANTA’s support and development teams. It’s a love letter in numbers, enjoy!

With love, MANTA.

A LOVE LETTER IN NUMBERS

MANTA 3.20: SSIS is in The Building, & More…

Well, well, well, another year is coming to an end! MANTA 3.20 is our last release of 2017, and we have made sure it will make it a year to remember. If you read this break-down, you will see just how much of a powerful finish it really is.

Well, well, well, another year is coming to an end! MANTA 3.20 is our last release of 2017, and we have made sure it will make it a year to remember. If you read this break-down, you will see just how much of a powerful finish it really is.

Microsoft SSIS

It has been coming to this moment for quite some time now. And here it is: MANTA now officially supports Microsoft SSIS. We support SSIS 2012 and newer. But we probably know what you are thinking: SSIS is an ETL tool, but MANTA doesn’t support ETL tools, so what’s going on?

SSIS is an ETL tool, but with plenty of SQL woven into it like it’s your granny’s holiday sweater. Since MANTA gets along so well with SQL, it comes as no surprise that it can effectively parse the code in the environment and provide you with the good old data lineage you’ve been waiting for.

Export

It’s always nice to get some of those foreign goods – that’s what TopQuadrant users will think from now on! MANTA 3.20 has fully automated the export of physical metadata to TopQuadrant TopBraid Enterprise Data Governance 5.4. Now you can have automatic, high-quality data lineage in EDG in just a matter of clicks!

What’s up?

That’s kind of what we tend to ask ourselves when we open our IBM InfoSphere Information Governance Catalogs… so we added automatic reporting of differences between the current and last revisions. Automatic reports of everything that changed since you were away – Someone added a new relation? Someone deleted a file that led to a change in the course of data lineage? Now you will know. MANTA will let you know what has changed since it last exported data to IGC.

And just in case…

Is there anything you need help with? Do you have any questions about MANTA 3.20? Then don’t hesitate to write to us at manta@getmanta.com. We are here for you. We’ve always been.

Happy New Year!

MANTA 3.19: Dynamic SQL Support, Fully-Automated Collibra, Service API & More…

The new MANTA 3.19 is here and it will leave you with exactly the same great feeling as the first sip of pumpkin spice latte on a rainy fall afternoon.* 

The new MANTA 3.19 is here and it will leave you with exactly the same great feeling as the first sip of pumpkin spice latte on a rainy fall afternoon.* 

In this version, we took a close look at our integration with Collibra and fully automated the whole process. Before, when we integrated MANTA with Data Governance Center, it required an initial setup that was tailored to fit each customer. But now it’s all part of the product, automatically ready to connect to your DGC!

This next new feature is a big deal for our partners who work with systems that MANTA doesn’t support, usually ETL tools, etc. These tools can contain SQL which our partners need to parse in order to understand their customers’ BI environments. With MANTA’s new „MANTA Service API“, our partners can now connect MANTA to their own solutions, make it crunch all the code in the customer databases that they can’t read, and then pull back all the information to provide their customers with detailed and accurate data-lineage.

So with the new “MANTA Service API” and Public API we introduced in our last release, you can now use MANTA’s SQL-analyzing superpower ANYWHERE. You’re welcome.

We boosted all the analyzation processes as well, especially the DB2 connector. So now when you are exporting to IBM InfoSphere Information Governance Catalog, you can see the SQL source code right in the window.

MANTA does static code analysis, and one of its handicaps was dynamic SQL analysis. In 3.19, we have made steps toward speeding up the process of analyzing dynamic SQL. MANTA is able to recognize and read your dynamic SQL patterns, although some specification is needed from time to time.

Last but not least there have been a few improvements affecting Informatica PowerCenter integrations. For example, MANTA can now easily read what database IFPC is connected to, which significantly decreases the amount of manual work required in the initial setup, saving many hours of valuable time on MANTA x Informatica integrations.

*We have not verified this claim; it’s just based on my personal experience. Please, don’t sue us.

Also, if you have any questions, just let us know!

Manta Goes Public with Its API!

Nowadays, every app, tool and solution needs to be connected to everything else. And MANTA is ready to join the club. 

Nowadays, every app, tool and solution needs to be connected to everything else. And MANTA is ready to join the club. 

You Asked for It

Here at MANTA HQ, we’ve been literally buried with customer requests to add various integration possibilities for Manta Flow. You asked for it! As of version 3.18, MANTA has a public REST API. This new feature, together with multi-level data lineage gives users the option to use MANTA with all kinds of technologies.

Through the public API you can connect MANTA to any custom tool or app and allow it to work with its data. How exactly? Take a look at this example:

Let’s say you have your own quality monitoring tool that monitors critical elements of data lineage for you. You can let MANTA export an excel file and then manually go through all the values, find out what their sources are, and manually look for changes. But now, thanks to public API, you can do all this automatically using your own tool!

Put an End to Boring Manual Reports

The tool can call MANTA’s API, automatically pull out all the critical elements of data lineage, and report the changes found. Now, you can automatically monitor all changes that occur to your data during a given time period, saving you and your company hours of manual labor spent pouring data from MANTA into your own tool.

And there are many, many other ways you can use our new API!

To learn more about capabilities of our solution, try a live demo, ask for trial or drop us a line on manta@getmanta.com.

How Manta Flow Now Works with IBM Information Governance Catalog

November 22, 2016 by

We are expanding our support for 3rd party metadata managers – to help our customers get the most out of their existing data governance solutions.

We are expanding our support for 3rd party metadata managers – to help our customers get the most out of their existing data governance solutions.

Our key product, Manta Flow, already complements Informatica Metadata Manager very well. With the addition of IBM InfoSphere Information Governance Catalog, we are able to deliver the same level of highly specialized code crunching to folks who use IBM’s tools as well. And how does that work? Well, it’s simple:

1. Manta Flow crunches programming code based on our supported technologies (Teradata, Oracle, Microsoft SQL and others).

2. After connecting to IGC, Manta Flow will create a new metamodel and perfectly integrate with the existing structure within IGC.

3. The customer can browse IGC as he or she is used to – it’s just going to have way more accurate data lineage ready to use.

Seamless integration into IGC is the key to success. We’ve created a short video to explain a little bit more how Manta Flow is integrated into Information Governance Catalog. And what’s inside?

1. A Brief Explanation of a New Metamodel in IGC: 0:10
2. How It Works with Queries: 1:40
3. Integration of Data Lineage Visualization: 2:13

A frinedly suggestion: Run the video on fullscreen.

And if you are not ready for IGC, stay tuned, we will soon present you our newest video about our oldest love – IMM. In the meantime, read the introductory article right here.

Any thoughts? Comments? Or do you simply want to try it out for yourself? Just let us know at manta@mantatools.com or use the form on the right.

Subscribe to our newsletter

We cherish your privacy.

And we need to tell you that this site uses cookies. Learn more in our Privacy Policy.