Engineer’s Notes

MANTA 3.26: IBM DataStage, erwin, and Detailed Lineage in EDC!

Leaves are gently falling to the ground in front of MANTA’s engineering office here in Prague as autumn arrives together with Software Release 3.26. What masterpiece have our developers put together this time? Read all about it in our release blog post.

Leaves are gently falling to the ground in front of MANTA’s engineering office here in Prague as autumn arrives together with Software Release 3.26. What masterpiece have our developers put together this time? Read all about it in our release blog post.

Not Just Any Ol’ Scanners

As always, we bring you the new and greatly improved as well as the old BUT tuned and tweaked. In this release, we have added two new scanners. The first one is an experimental version of the ETL tool from the IBM InfoSphere family, IBM InfoSphere DataStage. Experimental means that we are looking for pioneers for testing. But you had better be quick because the waiting list for this scanner was quite long, and so is the list of prospective testers!

MANTA analyzes DataStage parallel jobs provided by users, then creates a detailed visualization of the data lineage that can be pushed into any third-party metadata management solution or viewed in MANTA’s native visualization. This allows you to plug information such as data validation rules and other data from separate databases into DataStage, all visualized in the lineage showing the complete end-to-end journey of your data.

The second scanner we’ve added is erwin Data Modeler. MANTA can scan erwin and automatically pull physical and logical models that can then be added to your data lineage. This can come in extremely handy when it comes to your database architecture projects, as you can add complex data models from erwin to MANTA and use them to accurately project a situation—an enormously valuable asset for your database engineers.

Up a Notch

We have worked hard to perfect our Cognos scanner. MANTA’s scanner now processes Framework Manager report specifications retrieved from Cognos’ report storage (Content Store). Using the extracted information and other scanners, MANTA can create complete end-to-end data lineage (from database data sources through analytical models to reports in Cognos Analytics).

We are also happy to inform you that we have stepped our Informatica EDC integration up a notch. We have added more filtering options to detailed lineage in EDC, allowing you to see both the big picture as well as the finest details all the way down to the column level, and you can filter dataflows based on what information you want in your final lineage map. Now, MANTA helps you see not only complex SQL code in your EDC Catalog but also information from all the other scanners and integrations that MANTA supports—all ETL, analytics, and reporting tools. This causes user productivity and workflow efficiency to skyrocket!

Last but Not Least

Last but not least, we have made some improvements to the install and update process. Now, the magic happens automatically. When you download a new update packet and the installer notices that you need to update MANTA, it can automatically run your updater when needed, saving you a wrinkle or two in the process!

That’s all for MANTA 3.26. Any questions? Want to become a pioneer? Write to us at manta@getmanta.com.

Data Lineage from… Microsoft Excel?!?

Yes, you heard correctly. Excel is one of the new connectors we have introduced in MANTA 3.25. The reason for this is that many of our customers still have disturbing amounts of data in Excel databases, which was more challenging to include in their data lineage. Now, everything is possible.

Yes, you heard correctly. Excel is one of the new connectors we have introduced in MANTA 3.25. The reason for this is that many of our customers still have disturbing amounts of data in Excel databases, which was more challenging to include in their data lineage. Now, everything is possible.

MANTA now tracks data lineage in Excel, being able to read multiple sheets and slides. After connecting to MS Excel, MANTA can process XLSM and XLSX files as well as look up Excel objects such as graphs and pivot tables. All the mapped objects are then connected to their source objects by analyzing queries with MANTA’s database connectors.

We also have a detection algorithm with which MANTA can automatically reveal aggregated lists and tables that have formula relations among each other. MANTA is able to track data lineage from database tables and CSV files through tables and graphs among multiple Excel workbooks and push the whole picture into MANTA’s native visualization as well as into third-party solutions.

Here you can take a look at what Data Lineage with Excel looks like:

Visit our tech hub to learn more.

Interested in a MANTA Demo? Contact us at manta@getmanta.com or schedule a call with our bot! 

Meet Us at the MIT CDOIQ Symposium!

We often tell you about the technical conferences and meet-ups we visit and how we present our solution there. But this time, we are showing up at a completely different kind of event. It’s the Chief Data Officer and Information Quality (CDOIQ) Symposium hosted annually by the Massachusetts Institute of Technology (MIT).

We often tell you about the technical conferences and meet-ups we visit and how we present our solution there. But this time, we are showing up at a completely different kind of event. It’s the Chief Data Officer and Information Quality (CDOIQ) Symposium hosted annually by the Massachusetts Institute of Technology (MIT).

What’s This Event About?

As mentioned by MIT, this symposium is one of the key events for sharing and exchanging cutting edge ideas, content, and discussions. And, as data is a critical aspect of every organization, the symposium focuses on the management and leadership of this critical element in the 21st century that will benefit every organization. Since data is a big part of MANTA’s own mission, we will be talking our hearts out.

What Do We Want to Share?

We are looking forward to sharing our own take on the future of data with you, and that will, as usual, revolve around data lineage. But this time, we will let you take a peek behind the curtains at MANTA’s very own research. It’s no secret that we closely cooperate with universities, that our CEO Tomas Kratky has been a teaching associate at the Czech Technical University in Prague for many years, and that our amazing developers are mostly selected from students who did their Master’s or Bachelor’s thesis research with MANTA’s Engineering team. So what exactly do our current research projects include?

Database Language Research

MANTA as a company always tries to support all our customers’ needs by adding more and more scanners and connectors for the technologies that they use in their data environments. MANTA as a research institution likes exploring new database languages and examining ways to automate their reading and processing for the purpose of creating data lineage. An exciting challenge for us at the moment is Java. Java, as well as the growing family of database languages that we support for our customers, produces a significant amount of metadata that needs to be stored in MANTA in order to perform various analyses on the data lineage graph. Due to the nature of data lineage visualization, MANTA has been storing its own program data in a graph database for quite some time now—which means we have been continuously doing research in this field as well.

Graph Databases

A graph database is a database that uses graph structures for semantic queries with nodes, edges, and properties to represent and store data. A key concept of the system is the graph. The graph represents the data items and their relations as a collection of nodes and edges, with the edges representing the relationships between the nodes. The relationships allow stored data to be linked together directly and, in many cases, retrieved with one operation. Graph databases hold the relationships between data as a priority. Querying relationships within a graph database is fast because they are perpetually stored within the database itself. Relationships can be intuitively visualized using graph databases, making them useful for heavily interconnected data—like data lineage.

In a future where network structures are frequently used (social networks, maps, car sharing, other sharing apps, etc.), graph databases might just be the most efficient and fastest way to document data. MANTA uses graph databases to store data lineage graphs, which is a pretty innovative way to use these databases.

But our research lies in being even better. Finding more efficient ways to store data as well as quickly finding dependencies between various versions of data lineage graphs can greatly boost the overall speed of creating data lineage and make our software significantly faster than all the other data lineage solutions out there.

If we can find more fast and efficient ways to store data in graph databases, we might be ensuring the painless and efficient creation of end-to-end data lineage in the data world of tomorrow. And that is a challenge we will gladly take on today.

Want to join us at the MIT CDOIQ Symposium from July 31 to August 2 and listen to our talk about the future of data lineage? With the code MANTA25, you’ll get a 25% discount on tickets. And that’s worth it. 

Anything else you want to tell us? We are here for you at manta@getmanta.com

 

 

MANTA 3.25: Microsoft Excel, IBM Cognos, SAP PowerDesigner, and more!

It has been a long time since we swore our 3-new-supported-technologies-every-release oath, and we have kept it every new software release since. And this version of MANTA is no different. In Release 3.25, we bring you support for Microsoft Excel (yes, Excel!), IBM Cognos, and SAP (Sybase) PowerDesigner. Read more in our regular blog post.

It has been a long time since we swore our 3-new-supported-technologies-every-release oath, and we have kept it every new software release since. And this version of MANTA is no different. In Release 3.25, we bring you support for Microsoft Excel (yes, Excel!), IBM Cognos, and SAP (Sybase) PowerDesigner. Read more in our regular blog post.

No time to read? Watch a video or dip into the details below:

Data Lineage for Excel? From Excel? With Excel? YES

No, this is not a drill. We have just released support for Microsoft Office’s Excel. The reason for this, which may seem like a weird flex to some, is that many of our customers still have disturbing amounts of data in Excel databases, which was more challenging for our customers to include in their data lineage.

MANTA now tracks data lineage in Excel, being able to read multiple sheets, pivot tables, slides, and graphs. We also have a detection algorithm with which MANTA can automatically reveal aggregated lists and tables that have formula relations among each other. MANTA is able to track data lineage from database tables and CSV files through tables and graphs among multiple Excel workbooks and push the whole picture into MANTA’s native visualization, as well as into third-party solutions.

Reporting Tools Are Our Friends…

The next connector that MANTA added to its technology portfolio is Cognos, a reporting tool from IBM. MANTA is able to create complete end-to-end data lineage from the database data sources, analytical models, and reports in Cognos Analytics by scanning:

• Reports (including queries)
• Interactive reports
• Framework Manager models
• Database connections

MANTA then pushes them into its own data visualization as well as into third-party solutions. (Visit our Tech Hub to learn more.)

Modeling Tools Too!

The third connector we have added is the modeling tool SAP (Sybase) PowerDesigner, our second after adding ER/Studio in Release 3.24. MANTA can scan PowerDesigner and automatically pull physical, logical, and conceptual models that can then be added to your data lineage to create end-to-end logical data lineage.

Behind the Scenes

Besides broadening MANTA’s Tech Hub, we have also made some improvements to existing supported technologies. The most work was probably done on SSAS, where we have added support to the newer version (level 1200) of the tabular models.

We also detect a connection between SSAS and SSRS from our Microsoft family, and these technologies are currently sharp crisp alfa versions. MANTA is also able to export more and more technologies into third-party solutions, including transformation logic.

And, as of MANTA 3.25, ETL Tools (SSIS, ODI, Talend) and reporting tools (SSAS, SSRS) from our tech hub are now exported into IBM IGC, Informatica EDC. and other third-party solutions.

Anything you want to ask us? Go ahead and write to us at manta@getmanta.com or chat with our friendly MANTA Bot. 

MANTA 3.24: New scanners for ODI, SSRS, ER/Studio and More!

March 27, 2019 by

Here in Prague, where MANTA’s engineering office is located, the snow has melted and sunny spring has arrived. As the first baby otters are born, we are delivering a new little baby of our own: MANTA 3.24. Read about it in our blog post below or check out the two-minute video where Jan Ulrych summarizes all the changes and updates.

Here in Prague, where MANTA’s engineering office is located, the snow has melted and sunny spring has arrived. As the first baby otters are born, we are delivering a new little baby of our own: MANTA 3.24. Read about it in our blog post below or check out the two-minute video where Jan Ulrych summarizes all the changes and updates.

What’s new this time? After finalizing our Microsoft SSRS connector, we have added two more new connectors. The first one is a scanner for Oracle Data Integrator (ODI); the second is for ER/Studio, which expands our influence in the realm of data modelling tools so that we can now create logical lineage automatically, making data lineage from MANTA more accessible for users who aren’t database tech pros.

However, the biggest success in this release is the direct integration with Collibra via API. We have been partnering with Collibra on the development of this synchronization API for quite some time now. So, we are pleased to inform you that we can now introduce the final version.

How is it different from the old integration?

  1. Direct integration. We are so integrated that we are basically part of Collibra DGC. This makes your work with Collibra and MANTA so much faster and easier.
  2. Automatic metadata update. Collibra can fully use this MANTA feature now.
  3. Table synchronization. We are the first ones on the planet able to update your Collibra DGC with your database 1:1, meaning you can now get rid of non-existent tables in DGC and make room for new ones.
  4. All in one. We are able to export all MANTA data lineage to Collibra, including the newly supported Microsoft SSRS.
  5. Logical Lineage. Since we support metadata extraction from E/R models and mappings between physical and logical layers, we can provide this information to Collibra so it can provide logical data lineage.
  6. Installation. It is just so much easier now.

Besides the hot stuff mentioned above, MANTA 3.24 finally offers transformation logic in Teradata and a long-awaited experimental Java version. We are currently doing closed beta testing with some of our customers, and from the next software release onward, we will be doing open testing.

Interested? Got questions? We are here for you. Throw a message into our trusty mailbox at manta@getmanta.com. We will reply!

We cherish your privacy.

And we need to tell you that this site uses cookies. Learn more in our Privacy Policy.