Engineer’s Notes

Manta Goes Public with Its API!

Nowadays, every app, tool and solution needs to be connected to everything else. And MANTA is ready to join the club. 

Nowadays, every app, tool and solution needs to be connected to everything else. And MANTA is ready to join the club. 

You Asked for It

Here at MANTA HQ, we’ve been literally buried with customer requests to add various integration possibilities for Manta Flow. You asked for it! As of version 3.18, MANTA has a public REST API. This new feature, together with multi-level data lineage gives users the option to use MANTA with all kinds of technologies.

Through the public API you can connect MANTA to any custom tool or app and allow it to work with its data. How exactly? Take a look at this example:

Let’s say you have your own quality monitoring tool that monitors critical elements of data lineage for you. You can let MANTA export an excel file and then manually go through all the values, find out what their sources are, and manually look for changes. But now, thanks to public API, you can do all this automatically using your own tool!

Put an End to Boring Manual Reports

The tool can call MANTA’s API, automatically pull out all the critical elements of data lineage, and report the changes found. Now, you can automatically monitor all changes that occur to your data during a given time period, saving you and your company hours of manual labor spent pouring data from MANTA into your own tool.

And there are many, many other ways you can use our new API!

To learn more about capabilities of our solution, try a live demo, ask for trial or drop us a line on manta@getmanta.com.

MANTA 3.18: We Are Going Public… With Our API! (And More!)

Manta Flow introduces new API, complete DB2 and Netezza in IMM, and detailed business lineage transformations.

Manta Flow introduces new API, complete DB2 and Netezza in IMM, and detailed business lineage transformations.

This month we went all out. We sat down and worked hard to bring you MANTA 3.18 as soon as possible, because it wouldn’t have been fair of us to have kept these amazing features to ourselves. Come and join the ride!

Growing Integration Capital

Up until now MANTA has had a standard API, but from now on we will have a public REST API, which gives users many more options. Through the public API you can connect MANTA to any app and let it run impact analyses to get data lineage information in CSV or JSON to use in custom analyses.

Speaking of connecting, MANTA can now read both of the previously mentioned IBM databases in IMM and IGC. DB2 and Netezza users, now you can enjoy data lineage at its finest in your own data management solution. And while we were at it, we also improved our Oracle, MSSQL, and Teradata connectors.

Deeper into Business Lineage

Another life-changing new feature that we already introduced in our last release is business lineage. But, this time we went back to it and added business lineage transformations. From now on, your business team will not only see where the data is coming from, but what happens to it along the way. This makes MANTA’s business lineage as detailed as physical lineage, but in more businessperson friendly language.

Last but not least, we have made a few tweaks and fixes to our native visualization and improved export for IBM InfoSphere Information Governance Catalog 11.5. We’ll be more than happy if you let us know what you think!

Trust Your Lineage, All of It

Can you trust your physical, business and logical lineage? Manta introduces support for many different levels of data lineage abstraction. 

Can you trust your physical, business and logical lineage? Manta introduces support for many different levels of data lineage abstraction. 

When it comes to data, trust is always the key. And getting a complete overview of data flows in your system is necessary to get that trust back. Now, different types of lineage won’t mess things up anymore. It’s almost impossible to map complex BI systems on more levels of abstraction. Many different tools provide physical (technical), business and logical lineage, but this lineage is only good when it is complete. Like, totally.

Physical Lineage Is the Key

At Manta, we’re good at getting detailed and accurate physical data lineage from your logic hidden in programming code. Never mind SQL overrides, manually defined procedures, stored procedures – MANTA will just map it all. On top of that, we are now able to include different levels of lineage, but backed up by original physical lineage so it’s 100% accurate. And it’s all fully automatic, so there’s no manual labor necessary… Here is how it works:

First, MANTA will do what it is the best in the world at – map detailed and accurate physical data lineage from the logic hidden in your programming code.

Second, it loads external mapping between physical and logical or business objects (like business name to specific table / column mapping) from available sources and ties that with rendered physical lineage.

Third, it uses mapped objects (business, logical) to transform existing physical lineage so it is better aligned with the objects provided (i.e. no detailed technical transformations for business objects but rather simplified descriptions). The result is accurate, trustworthy lineage of any kind – based in reality and yet useful for everyone who needs to understand the specific level of abstraction.

Any Lineage, Any Source

It does not matter which technology your physical lineage is from (check out our list of supported technologies!). It also does not matter how you provide the initial business/logical to physical object mapping – it could be:

  • your favorite business glossary
  • a data modelling platform
  • plain ol’ Excel spreadsheets
  • virtually any other structured data format

From an Excel Sheet to a Data Governance Solution

Our main goal is to connect and share information with other tools and solutions in our customers’ BI. That’s why we are not only able to pull metadata from other tools, but MANTA can easily push everything back into 3rd party solutions. Want an example? At one of our successful implementations, MANTA:

  1. Loaded business lineage mapping from Collibra’s Data Governance Center
  2. Combined it with complete physical lineage from actual code
  3. Pushed lineage back to Collibra, ensuring that the lineage was complete and functional

Additionally, MANTA is always capable of visualizing everything in our visualization – feel free to take a look at our introductory video:

To learn more about MANTA, simply get in touch and ask for 30-day free trial!

MANTA + Informatica

MANTA completes Informatica to form a comprehensive metadata management platform. But how precisely does the bond work? [VIDEO BELOW ↓ ]

MANTA completes Informatica to form a comprehensive metadata management platform. But how precisely does the bond work? [VIDEO BELOW ↓ ]

Trust Through Understanding

Our product allows you to trust the data in your BI environment, because it is specialized in cracking SQL code and helps to fill gaps in metadata management solutions. It does not matter whether they were deployed to fulfill compliance regulations or create the backbone of your data governance efforts. The Informatica Metadata Management solution has a rudimentary capability to parse SQL, but, in our experience with our customers over the years, there are still blind spots here and there.

MANTA connects to IMM through XConnect (native API plugin), and enriches the metadata model of IMM with missing pieces of data lineage:

  • BTEQ scripts, stored procedures, views, and macros from Teradata
  • PL/SQL scripts, stored procedures, packages, and more, including DB links, from Oracle DB & Exadata
  • T-SQL scripts, stored procedures and more, including linked servers, from Microsoft SQL Server, Sybase (now SAP ASE), and PDW
  • NZPLSQL scripts, stored procedures, and more from IBM Netezza
  • DB2 scripts, stored procedures, and more from IBM DB2
  • SQL overrides from Informatica PowerCenter, Cognos and Microsoft SQL Server Reporting Services

It fills in the gaps in data flows and allows our customers to get end-to-end data lineage (including those pesky indirect data flows).

Here is a live shot of MANTA + IMM,  fullscreen is recommended.

Beyond Lineage

MANTA’s ultimate goal is to understand the semantics of the code being analyzed. If you have ever thought about advanced performance tuning, real data protection analysis, automated business lineage extraction,or migration of your code base to a different platform, MANTA is exactly what you need.

For customers who do not have Informatica Metadata Manager, but do enjoy the advanced ETL capabilities of  Informatica PowerCenter, we are also able to provide data lineage in our own visualization or IBM IGC. MANTA’s open API allows customers to push metadata to 3rd-party tools such as Collibra, Adaptive, Alation, Axon, and others.

Faster and Cheaper

The impressive part is how fast MANTA works. It can map a BI environment at a speed that is incomparable with any  human workforce, which means you save quite a lot of money on man-hours.

If you would like a more detailed explanation of MANTA + Informatica, and how it can help with your current situation, be sure to contact one of our specialists! They can discuss your situation in detail and give you a better idea of how MANTA can help with your data governance efforts.

MANTA 3.17: Say Hello to Netezza, Business Data Lineage, and Much More

It’s a bird! It’s a plane! It’s MANTA Flow 3.17 coming your way!

It’s a bird! It’s a plane! It’s MANTA Flow 3.17 coming your way!

When we said (and we always say this) that we would expand the list of our supported technologies, we weren’t lying. MANTA 3.17 comes with support for IBM Netezza and IBM DB2, provides business data lineage, and much more!

The biggest news among big news is that MANTA now supports Netezza. As always, pioneers are wanted and needed (ask for your trial).  It can do everything we taught it to, so if there is anyone out there who has a Netezza solution and wants to have the data lineage all nice and clean, we’ve got you covered.

This release is also the debut for a DB2 connector, now (almost) in stock. It’s no secret that we still have some plans for that one, so stay tuned. Because the Parser Team rocks, we worked on all of our parsers and we can proudly say that every new version of MANTA brings more and more improvements.

Did you know that complete business data lineage is now a thing? Yes, it is! And MANTA can provide that for you!

What we do is we can put together the physical metadata from the existing MANTA connectors with business terms provided by the user. From that, MANTA Flow can give you end-to-end business data lineage. We are the first ones who can provide you with data lineage that suits the needs of your business users as well as your BI team from information that your company already owns.

And as usual, that’s not it. We have added some cool features to MANTA, e.g. searching in source code, and much more.

How to Handle Impact Analyses in Complex DWHs with Predicates

“How to get full data lineage in complex BI environments and perform reliable impact analyses?” Predicates (with the help of Manta Flow!) might be the answer. 

“How to get full data lineage in complex BI environments and perform reliable impact analyses?” Predicates (with the help of Manta Flow!) might be the answer. 

During our pilots and deployments, we often find data warehouse environments that use very general physical models including several big tables like PARTY, BALANCE, ORDER and others. These tables contain data obtained from various source systems, and there are a lot of data marts and reports built on top of them. These tables make things difficult during the impact analysis because data lineage from almost every report goes through them to all sources making the result worthless.

Impact Analyses Do Not Have to Be THIS BIG 

Let’s take a look at an example to understand exactly what happens. The table PARTY contains all individuals and companies that are somehow related to the organization. Thus, in one table, it is possible to have records for clients, employees, suppliers and its branch network. Each type of entity is identified by a unique attribute or source system from which data is obtained – for example, clients are managed in a different system than employees.

Now, let’s assume we have two reports based on data from the PARTY table – a report EMPL_REPORT that displays information about employees and another report BRANCH_REPORT that displays information about the branch network. If we use the standard data lineage analysis, we can get this picture:

predicates1

Although only data from the EMPLOYEE source table is relevant for the report EMPL_REPORT, the impact analysis from that report also includes the CLIENT, BRANCH and SUPPLIER source tables due to the PARTY table. The problem is the same for the report BRANCH_REPORT. From the other side, the impact analysis from the EMPLOYEE source table includes both the EMPL_REPORT and BRANCH_REPORT which is confusing.
In the real environment, there are dozens of source systems and hundreds of reports, which makes the standard data lineage analysis worthless.

The Advanced Data Lineage Analysis 

Fortunately, there is a solution. When data is inserted into the PARTY table from different source systems, there is often a column like PARTY.source_system_id where the identification of the source system is stored as a constant. Similarly, when a report is created that consumes data only from specific source systems, there is a condition in the statement filtering data based on the PARTY.source_system_id column. Thus, it is possible to automatically analyze both the insertion and selection to/from the PARTY table and create predicates such as PARTY.source_system_id = 20 that are then stored together with data lineage in the metadata repository. Therefore, it is possible to include them in the computation during the impact analysis.

Thanks to that, if we perform an impact analysis from the report EMPL_REPORT, the predicate PARTY.source_system_id = 20 is gathered before the table PARTY. When the analysis continues towards source tables, the predicate for each path is selected and compared to what has already been gathered. Therefore, when the path to the source table CLIENT with the predicate PARTY.source_system_id = 10 is tested, the result is that both predicates cannot hold at once, so data for this report cannot come from this source table. Conversely, when the path to source table EMPLOYEE with the predicate PARTY.souce_system_id = 20 is tested, the result is that data for this report can come from this source table, so it is included in the result of the impact analysis. We can get similar results if we perform an impact analysis for the BRANCH_REPORT and also from sources like the EMPLOYEE table.

The result of the advanced data lineage analysis can look like this (in reality, if we perform the impact analysis from the EMPL_REPORT, we will only see the EMPLOYEE and PARTY tables):

predicates2

Surely, the situation can be far more complex. For example, the data from the PARTY table can be pre-computed for more source systems first, and then several reports can be created on top of them for only a specific source system, like in this picture:

predicates3

This is also something that can be handled and, as you may have expected, even this is a part of the Manta Flow product analysis.

If you have any questions or comments, feel free to contact Lukas at manta@mantatools.com. You can try these predicate-based impact analyses in our free trial – just request it using the form on the right. 

 

How To Inspect Raw Data Lineage With Manta Flow

Risk departments have a lot of complex SQL queries in their data warehouses and data marts. But sometimes it’s really difficult to find the right level of detail. Manta Flow can help.

Risk departments have a lot of complex SQL queries in their data warehouses and data marts. But sometimes it’s really difficult to find the right level of detail. Manta Flow can help.

“When we present Manta Flow to potential customers, most of them are happy that we can reduce very complex SQL statements to a few simple rectangles connected by arrows”, explains Lukas Hermann, our Director of Engineering. “They need to be able to quickly understand what source tables their SQL queries read, what target tables they fill, what columns are involved in computing a particular column and how.”

The Usual

For example, let’s look at just two ordinary insert statements moving data from a stage to a datamart and to a report:

raw_code

It could take you quite a while to analyze which columns are involved in the computation. But with Manta Flow it is really easy to see, including all the statements involved:

Direct

(click to show the large version in a new tab)

This is perfectly sufficient for all business analytics in data warehouse environments. All the unnecessary details like exactly how data is computed, filtered, aggregated, or ordered are hidden. And if you want to go deeper, Manta Flow can easily show the SQL code of the statements where you find the full detail.

The Raw

However, some analysts (particulary from the aforementioned risk departments) say that their SQL statements are really huge, including many subselects, complex expressions, etc., so the jump between the clear picture and the SQL code is too big. Therefore, they would like to see all the computation steps in a similar simplified format, and they ask if Manta Flow can handle it, if it has all the information necessary to show it.

The answer is that Manta Flow has the most detailed information possible about each part of the statement, but so as not to disturb you with what are in most cases useless details, it filters the information to the best level of detail. If you want to see everything including expressions, conditions, aggregations, etc., it’s possible to configure or completely turn off the filtering. Manta Flow is able to show you unfiltered information, but still keep you in the loop and oriented within your own systems.

click_to_see_the_big_picture

(image will open in a new tab)

See? It’s possible to show the SQL code in the precise position of each part of the statement shown.

If you’d like to try something like that yourself, just let us know in the form on the right. Also, do not forget to follow us on Twitter.

Manta Tools 3.12: The New Versions of Oracle & Informatica, a New Demo, and More!

Surprise! Another Manta Tools release here a bit early. And what’s in the box?

Surprise! Another Manta Tools release here a bit early. And what’s in the box?

Our customer base is growing quickly, and feature requests are piling up. That’s why Lukas (our Codehead Prime) decided to squeeze one more major release into the schedule. So, what’s new?

We need to catch up on the development of technologies we support, that’s one of the keys to our success. We generally support the latest versions, but the guys did a thorough review and declared Oracle 12c and Informatica 10 (namely Informatica PowerCenter and Informatica Metadata Manager) fully supported. Although most of the market still has older versions, both Oracle and Informatica are pushing people really hard to switch to newest releases. Informatica even announced the end of support for 9-something versions next year.

Manta Flow, in particular, received a few major updates (minor updates are not worth mentioning due to the very fact that they are minor). REST API is now available in Manta Flow. Our native visualization can now easily show SQL overrides, and individual nodes (and their children) are now easy to contract with just one click. What does it mean to “contract nodes,” you ask? I thought you would ask, so I made an animated GIF about it! Check it out:

node_contraction

See? Big fat script in the middle got out of the way and you can see just tables you wanted. Notice the line from one table to another is thicker, so you can click on it and show the script again. 

And last, but not least: Manta Flow online demo was updated, and it now corresponds with the latest version of Manta Flow you can get off the shelf. There are some limits to it though – as with any cloud product, it does not have access to your system and it cannot show full data lineage (only the part you push in). But try it anyway, it’s awesome. (Or ask for a free trial right away, and test Manta Flow at home.)

Did you know we are on Twitter? Yes, it’s true. Follow us, please. Also, do not hesitate to ask any questions and submit any comments via email to manta@mantatools.com. 

 

 

 

Manta Tools 3.11: Increased Compatibility, Improved Visualization & More

Compatible with yet another data source & offering more user-friendly visualizations, that’s Manta Tools 3.11.  

Compatible with yet another data source & offering more user-friendly visualizations, that’s Manta Tools 3.11.  

So, what’s new? Manta Flow’s visualization export functions have been enhanced significantly – you can now filter resources and change the level of detail on which the lineage is extracted. Also, native Manta Flow visualization now shows SQL override queries in Informatica PowerCenter as a parameter.

For the even faster automated visualization settings, Manta Flow now supports selection via text input (such as CSVs). This is especially useful when a customer with a huge environment would like to visualize a very specific part of it. And, as always, Teradata & Oracle parsers are now better than ever.

And last, but definitely not least, is our biggest new “feature” – compatibility with Microsoft SQL Server. But this topic is waay to big for a humble release blog post like this one. We will announce it in separate article tomorrow.

Any questions, comments or concerns? Just send them to manta@mantatools.com. And do not forget to follow us on Twitter!

Manta Tools 3.10: Unlimited Time Machine, Automated Backup & More

The new version is out – what is new with Manta Tools 3.10?

The new version is out – what is new with Manta Tools 3.10?

One of the all time most requested Manta Flow features was the unlimited time machine function. It is now possible to see every visualization you’ve ever performed on your system and freely explore them like you would with current ones. Manta Tools’ metadata repository is also automatically backed-up whenever you like, so even your most paranoid teammates can sleep at night.

When it comes to technology, we’ve improved metadata extraction from Teradata databases and the predicate system for Oracle and Informatica. The Manta Flow visualization itself has also been updated. We’ve added extended highlighting (you can try this in our online demo!) and also a direct export option on the customer’s catalog page.

And what’s coming in our next version? We will finally release support for a whole new technology – Microsoft SQL Server, including a connection with Informatica Metadata Manager.

Do you have any questions or suggestions? Please let us know at manta@mantatools.com, and do not forget to follow us on Twitter.

 

 

Subscribe to our newsletter

We cherish your privacy.

By using this site, you agree with using our cookies.