Engineer’s Notes

MANTA 3.24: New scanners for ODI, SSRS, ER/Studio and More!

March 27, 2019 by

Here in Prague, where MANTA’s engineering office is located, the snow has melted and sunny spring has arrived. As the first baby otters are born, we are delivering a new little baby of our own: MANTA 3.24. Read about it in our blog post below or check out the two-minute video where Jan Ulrych summarizes all the changes and updates.

Here in Prague, where MANTA’s engineering office is located, the snow has melted and sunny spring has arrived. As the first baby otters are born, we are delivering a new little baby of our own: MANTA 3.24. Read about it in our blog post below or check out the two-minute video where Jan Ulrych summarizes all the changes and updates.

What’s new this time? After finalizing our Microsoft SSRS connector, we have added two more new connectors. The first one is a scanner for Oracle Data Integrator (ODI); the second is for ER/Studio, which expands our influence in the realm of data modelling tools so that we can now create logical lineage automatically, making data lineage from MANTA more accessible for users who aren’t database tech pros.

However, the biggest success in this release is the direct integration with Collibra via API. We have been partnering with Collibra on the development of this synchronization API for quite some time now. So, we are pleased to inform you that we can now introduce the final version.

How is it different from the old integration?

  1. Direct integration. We are so integrated that we are basically part of Collibra DGC. This makes your work with Collibra and MANTA so much faster and easier.
  2. Automatic metadata update. Collibra can fully use this MANTA feature now.
  3. Table synchronization. We are the first ones on the planet able to update your Collibra DGC with your database 1:1, meaning you can now get rid of non-existent tables in DGC and make room for new ones.
  4. All in one. We are able to export all MANTA data lineage to Collibra, including the newly supported Microsoft SSRS.
  5. Logical Lineage. Since we support metadata extraction from E/R models and mappings between physical and logical layers, we can provide this information to Collibra so it can provide logical data lineage.
  6. Installation. It is just so much easier now.

Besides the hot stuff mentioned above, MANTA 3.24 finally offers transformation logic in Teradata and a long-awaited experimental Java version. We are currently doing closed beta testing with some of our customers, and from the next software release onward, we will be doing open testing.

Interested? Got questions? We are here for you. Throw a message into our trusty mailbox at manta@getmanta.com. We will reply!

MANTA x Record Level Lineage: Why we don’t have it

You may or may not have heard about record level lineage. This is a topic that our customers ask about quite frequently, so our vice president of development, Lukas Hermann, decided to write an article where he answers some of the FAQs. Continue reading to find out more about record level lineage and why we don’t have it.

You may or may not have heard about record level lineage. This is a topic that our customers ask about quite frequently, so our vice president of development, Lukas Hermann, decided to write an article where he answers some of the FAQs. Continue reading to find out more about record level lineage and why we don’t have it.

What is record level lineage?

Record level lineage is an approach to data lineage that is similar to data tagging. The idea behind data tracking is that each piece of data that is being moved or transformed is tagged/labeled by a transformation engine which then tracks that label all along its way from start to finish. This approach seems great, but it only works well when a transformation engine controls the data’s every move. Some good examples are controlled environments like Cloudera or Dremeo that focus only on the origin of one specific record.

Record level lineage vs. column level lineage

A feature that MANTA does have, that in a way is similar to record level lineage, is column level lineage. What exactly is the difference? Let’s look at an example.

Let’s say you have the column full name in your table. In this table, the full name is created by combining the first name and the last name. Imagine that in the full name column you have names like John Snow and Jack Snow. Now, let’s say that the name John Snow came to this table from your own CRM database, but Jack Snow came from a contact database acquired from a third party.

Record level lineage is able to tell you exactly that John Snow came from CRM and Jack Snow came from your contact database. Column level lineage, like in MANTA, is able to tell you that the column full name consists of data from these two databases—your CRM database and your contact database.

Why we don’t have it

The reason why MANTA does not have record level lineage is that MANTA doesn’t “see” your data; it doesn’t even “see” that you have a John Snow and a Jack Snow in your full name column. It only reads your metadata. That is why MANTA only sees a table that contains data from these databases and which databases they are.

Now, you might be thinking that the overall idea of the record level lineage approach might not be so bad after all. But keep in mind that if anything happens outside its walls, the lineage is broken. It is also important to realize that the lineage is only there if the transformation logic has been executed. But think about all the exceptions and rules that apply only once every couple of years. You will not see them in your lineage until they are executed, which is not exactly healthy for your data governance, especially if some of those pieces are critical to your organization.

Also, tags are formed by assigning additional metadata to the records. If you lose this metadata, you will never be able to form the lineage again. And without actually running the transformation engine, you don’t know how the given record was put together, and therefore don’t know the lineage behind it.

In conclusion

If MANTA wanted to have record level lineage, it would have to start reading your data instead of your metadata, and it would have to have much more information about your environment. This would make the entire process of getting data lineage far more complicated and time-consuming.

We can safely say that we are not planning on having record level lineage as a feature any time soon. On the other hand, we plan on putting more effort into understanding your data transformations. The fact that MANTA only reads your metadata and is only interested in your data transformations, not your actual data, is the reason why MANTA can be automated so well and get data lineage so fast.

And what about Conditional Lineage? 

MANTA also has conditional lineage as a feature, and you do look into the actual data when you are creating conditional lineage. Well, not quite. We only use the data that is specifically mentioned in the scripts. You can learn more about conditional lineage in the article: How to Handle Impact Analyses in Complex DWHs with Predicates.

So what MANTA does give you is a list of the exact databases that supply data to the given column in your table. For compliance with regulations such as GDPR and other financial or banking regulations, it is completely sufficient. And typically, there are no more than a few databases that supply each column, so then the question is: If you really need to have the specific database for each record in your table THAT BAD and you can have the databases narrowed down to a few for each column in a couple of hours, wouldn’t it be more efficient to just check those two databases manually for the specific record yourself?

Do you have any development-related questions for Lukas, or would you like to learn more about how MANTA can solve a specific issue in your company? Don’t hesitate to contact us at manta@getmanta.com.

MANTA 3.23: Talend, Sqoop, Microsoft SSAS, and More!

December 18, 2018 by

Depending on where you are, either snow is slowly covering the ground and rooftops or, well, not. Here in Prague where MANTA’s development center is, it’s freezing outside. Let’s grab a hot chocolate and dive into our last software release of 2018.

Depending on where you are, either snow is slowly covering the ground and rooftops or, well, not. Here in Prague where MANTA’s development center is, it’s freezing outside. Let’s grab a hot chocolate and dive into our last software release of 2018.

Each new version of MANTA supports a new technology. In the last release of this year, we are bringing you not one, but two! They are connectors for Talend Data Integration and Sqoop, and from Microsoft’s SQL Server toolbox, we now fully support SSAS. If your BI environment includes Talend projects, jobs, and subjobs with SQL code, you can now include them in your data lineage in MANTA’s native visualization, IBM IGC, Collibra DGC, and TopQuadrant.

For now, SSAS is only supported in MANTA’s native visualization. In the previous release, we added support for tabular models. Now, we support all SSAS models, including multidimensional models with cubes. MANTA now has the ability to automatically read SSAS dimensions and measure groups with all their hierarchies and sets.

The second big hit in this release is a Sqoop connector. If you have Sqoop scripts written in shell, MANTA can now add them to the data lineage in its native visualization, IBM IGC, Collibra DGC, and TopQuadrant. Before the end of the year, we will add a detailed Sqoop page to the Scanners & Integrations section on our website, so keep an eye open for that!

Our development teams have also improved existing connectors. The Apache Pig connector can now read macros, and the connectors for PostgreSQL, Amazon Redshift and Greenplum can now read stored procedures. The biggest improvements, however, have been made to our Informatica EDC connector. MANTA can now upload data into EDC fully automatically. This is possible mostly thanks to Informatica’s new API. Besides automated upload, we are now also able to load metadata from Teradata Parallel Transporter (TPT) scripts.

Last but not least, our installer, configurator, and updater now work for all integrations in our tech hub. You can now integrate each and every one of your connectors from the comfort of your own user interface – preferably from your own couch with cookies and hot chocolate. Happy Holidays!

Don’t forget that MANTA does a more detailed technological breakdown for every release together with screenshots and more, made especially for our partners & customers. If you haven’t gotten yours, contact us at manta@getmanta.com and we will make sure it arrives in your inbox.

MANTA is Built to Last

Did you know that MANTA is built to last a company’s lifetime? If you update it periodically, you only need to adjust your monthly subscription plan based on how your BI and environment grow in capacity. Other than that, MANTA is a keeper.

Did you know that MANTA is built to last a company’s lifetime? If you update it periodically, you only need to adjust your monthly subscription plan based on how your BI and environment grow in capacity. Other than that, MANTA is a keeper.

But like with everything, there are a few simple rules you have to follow to make sure MANTA is in the best possible condition. Here are the two (yes, only two!) secrets to ensuring your MANTA will last till you retire:

1. Update

2. Upgrade

So, without further delays, lets get to it.

Update

If you are following us on all corners of the internet, you may have noticed that MANTA is constantly coming up with new connectors for different metadata management solutions and platforms. It’s not always because we didn’t connect with them before—it’s because they didn’t exist! That’s right, other companies are continuously releasing new solutions and developing new environments as well. And MANTA constantly develops connectors to support them.

Developing new connectors allows MANTA to be seamlessly reborn into new BI environments. This can save companies thousands of dollars! If you have already bought MANTA and you decide to upgrade the technologies we already support in your environment, you can be sure that MANTA’s newest version will support your newest version. Many of the companies that develop the solutions we support, such as IBM, Informatica, Teradata, and Collibra, are even MANTA’s partners. This means we often develop our connectors using and doing continuous testing on their own software, usually months before they release it. So, when something like EDC comes out, you can be sure we already integrate with it.

If you pay your subscription fees, you just need to check and make sure you have the newest version of the software installed. We make this easy for you by releasing new updates quarterly so you know when to watch out for an upgrade. Currently, we are also testing a new MANTA updater in beta, to make MANTA software updates even more user friendly.

Upgrade

As you know from our pricing, MANTA is priced according to the number of scripts in your data warehouse environment. The fact that you have upgraded to a new technology doesn’t necessarily mean that you have more scripts in your environment than you did before.

If you can fit into the “script budget” that you are currently in, then you might not need to pay even a dollar more, even after changing the software in your BI environment! So, if you change your technology but your data environment stays the same size, MANTA will adjust to your needs and simply be reborn in the new technological environment.

If you have any questions about how to get the most out of MANTA, feel free to contact us at manta@getmanta.com.

MANTA + Informatica EDC Tech Bond

MANTA can complete the Enterprise Data Catalog data governance solution from Informatica with some really tough programming code. In the following article we will let you zoom in on all the details of this technical bond. (INCLUDING A NEW VIDEO!)

MANTA can complete the Enterprise Data Catalog data governance solution from Informatica with some really tough programming code. In the following article we will let you zoom in on all the details of this technical bond. (INCLUDING A NEW VIDEO!)

MANTA connects to Informatica EDC and enhances all of its data lineage:

  1. MANTA connects to the same sources as EDC, and scans all the scripts.
  2. Then, it automatically provides data lineage from those scripts and integrates it with EDC’s native resources.
  3. Finally, MANTA analyzes programming code that is out of EDC’s native scope, e.g. stored procedures, views, triggers and other scripts.

With MANTA, every Informatica customer can see how every procedure works, find out how the data is transformed between tables, and get complete end-to-end data lineage down to the column level.

Here you can watch our brand new video where Lukas explains how the technical bond works:

Like our videos? There are more where that came from! Check out our video section right here. And don’t forget to subscribe to our YouTube channel! 

As you have heard in the video above, MANTA can enrich Informatica’s Enterprise Data Catalog with a number of SQL scripts, that allow EDC to show much more depth and detail than it normally could. Here is a list of scripts that MANTA currently pushes into Informatica.

What MANTA currently pushes into Informatica’s Enterprise Data Catalog:

  • BTEQ scripts, stored procedures, views, and macros from Teradata
  • PL/SQL scripts, stored procedures, packages, and more, including DB links, from Oracle DB & Exadata
  • T-SQL scripts, stored procedures, and more, including linked servers, from Microsoft SQL Server, Sybase (now SAP ASE), and PDW
  • NZPLSQL scripts, stored procedures, and more from IBM Netezza
  • DB2 scripts, stored procedures, and more from IBM DB2
  • PostgreSQL scripts, views and more from PostgreSQL, Amazon Redshift and Greenplum

It’s all about the details

MANTA’s key feature is its understanding superpower. The ability to read even the most complex custom code is crucial for obtaining detailed and complete end-to-end data lineage. This can also be used for real data protection analysis, automated business lineage extraction, migration of your DWH to a different platform or the cloud, and to comply with regulations such as GDPR, Basil II/III, and many more.

For customers who have EDC’s older brother, Informatica Metadata Manager, we have a connector for IMM as well. And for customers who have neither, but enjoy the advanced ETL capabilities of Informatica PowerCenter, we are also able to provide data lineage in our own visualization.

All our supported technologies, both scanners and integrations can be found here. If there is anything else you would like to talk to us about, don’t hesitate to contact us at manta@getmanta.com

We cherish your privacy.

And we need to tell you that this site uses cookies. Learn more in our Privacy Policy.