Manta Business

Pioneers Wanted: MANTA Connectors for Pig and Talend

November 9, 2018 by

We are upgrading our technology stack again and need you to try out our connectors! Since the last release, MANTA has supported Pig, and before Christmas, you can look forward to a gift from us: software release 3.23 with support for Talend.

We are upgrading our technology stack again and need you to try out our connectors! Since the last release, MANTA has supported Pig, and before Christmas, you can look forward to a gift from us: software release 3.23 with support for Talend.

Every time we begin supporting another technology, it is a big step for us. We always look for pioneers who have this technology in their metadata management environment and are not afraid to take the challenge of testing it in the field, giving us feedback, and allowing us to tailor the technological connector to fit the actual solution.

The Pig Connector

When we came out with MANTA 3.22, we provided our customers with the Apache Pig connector.

Apache Pig is a platform for analyzing large sets of data in Apache Hadoop. Pig Latin programming language is a high-level, procedural language for expressing data analysis programs. Pig’s structure enables substantial parallelization.

MANTA understands Pig Latin and is able to analyze and visualize it. MANTA is able to scan:

  • Pig Latin statements
  • Relations
  • Bags
  • Tuples
  • Fields

The Talend Connector

In our last release of the year, we are going to introduce our Talend connector. Talend Data Integration is an open-source-based ETL tool. The software is driven by the commercial open-source vendor Talend and is used for the visual design of data transformations with a number of ready-to-use components and connectors.

MANTA understands both Talend Open Studio for Data Integration as well as the commercial edition. MANTA will be able to export and analyze:

  • Projects, jobs, and subjobs
  • Connectors
  • Components
  • Expressions
  • SQL overrides

 

Did we get you interested in becoming a MANTA Pioneer for Pig or Talend? Or is there another technology on our scanners & integrations list that you are interested in testing? Drop us a line at manta@getmanta.com. We are looking forward to hearing from you!

MANTA and “The Quest To Automate The Ancient Database”

The title sounds like that of an opening weekend success action movie, doesn’t it? But it’s just a case that MANTA solves daily: Automating Data Lineage Visualization from old SQL based Databases. Let’s look at one of our customer’s cases with Teradata database.

The title sounds like that of an opening weekend success action movie, doesn’t it? But it’s just a case that MANTA solves daily: Automating Data Lineage Visualization from old SQL based Databases. Let’s look at one of our customer’s cases with Teradata database.

1. Problem

The customer, an international Healthcare provider, had an ancient database system. They had been using Teradata for over 30 years, within which they had accumulated over 21,000 scripts of code. Their metadata management solution was also integrated with the application Workday. This solution was regularly making changes to the environment, notifying them 6 weeks before performing the changes. This included changes to the data scheme and interferences with data publishing.

When the customer wanted to perform an impact analysis, the time needed to analyze 8 tables was 800 hours (100 man-days of work) and that was nearly impossible to achieve in the 6-week time-frame.

2. Solution

In order for the customer’s metadata management solution to work, they needed to automate the impact analysis process and the implementation of changes to the data warehouse structure so that it would be possible to perform the structural changes across the entire 21,000 script Teradata warehouse within the 6-week time frame. MANTA was pretty much their only hope.

3. Result

After deploying MANTA, the customer was able to successfully automate the impact analysis processes as well as the search for and implementation of changes to the data warehouse structure from the integrated Workday application.

MANTA gave the customer a comfortable user interface, seamlessly integrating all their metadata management applications, including EDC, and improved the overall state of the customer’s data governance projects.

This case study is also available here to share and download in PDF.

Any questions? We’re here for you at manta@getmanta.com, as always.

Solve Data Provenance Issues with MANTA

October 30, 2018 by

Data provenance is a term we often hear in combination with big data solutions and various other corporate data-related projects. But how do I achieve data provenance and how is it related to data lineage? Let’s take a look at the MANTA x Data Provenance breakdown.

Data provenance is a term we often hear in combination with big data solutions and various other corporate data-related projects. But how do I achieve data provenance and how is it related to data lineage? Let’s take a look at the MANTA x Data Provenance breakdown.

You may have heard the term data provenance before. It is related to data lineage more than you would guess. To be precise, it can be taken as a part of data lineage, a subset of it, or an additional supplement within the company’s data governance strategy.

We can pretty much say that data provenance examines the data’s point of origin. It includes a high-level view of the system for business users, so they can roughly navigate where their data comes from. Data provenance can be provided by a simple custom table and a few charts. Because it is uncomplicated to obtain, it is often used to map the origin of data sets in big data solutions.

Data lineage, on the other hand, pictures the complete data transformation journey from the data’s point of origin to any current observation point or end report within the system. It is based on reading technical metadata and therefore tracks data flows down to the lowest level – the actual scripts and statements.

So, whether you need to quickly find the origin of certain data and where its roots go for your data provenance project or you need complete end-to-end data lineage for more complex reports, regulatory compliance, data migration, and so on, MANTA has you covered.

Want to find out more about the holy grail of data lineage automation, a.k.a. MANTA? Check out our website or drop us a line at manta@getmanta.com. We don’t bite. :-)

MANTA + Informatica: The Ultimate Compliance Strategy

The financial industry is known for being stitched through with tons of regulations, and it is indeed challenging to comply with them. Banks and financial institutions reach out for tools to automate the process as much as possible. Read how MANTA and Informatica’s tech bond creates the ideal compliance solution.

The financial industry is known for being stitched through with tons of regulations, and it is indeed challenging to comply with them. Banks and financial institutions reach out for tools to automate the process as much as possible. Read how MANTA and Informatica’s tech bond creates the ideal compliance solution.

Most of the regulations that are currently used in banking were invented to protect the industry after the global financial crisis of 2007-2008. One of the most well-known regulations is BCBS 239. But all of these regulations define a huge number of principles that need to be complied with, sometimes at very high costs.

BCBS 239 and more

We can use the above-mentioned regulation BCBS 239 as an example. It consists of 14 principles that need to be fully adopted for the company to achieve the status of regulatory compliance. Complying with such a complex regulation is a challenging, company-wide issue that requires processing information across a broad range of tasks, departments, and entities within the financial institution.

But for all regulations in the industry, there is one thing that all implementation strategies have in common – the need for transparency. For example, when banks are asked to report aggregated risk metrics for BCBS 239, they are also asked to prove to regulators that risk scores are arrived at correctly by:

  1. Showing end-to-end data lineage from the source of the data in the data warehouse to the end report in the BI environment. This includes where the data first came from (its origin), what transformations were performed on it, and what processes and decisions affected its life cycle.
  2. Demonstrating Data Quality Controls: the identification, assessment, and management of data quality.
  3. Showing an enterprise-wide understanding of business concepts by tracking which data is available to which teams within the organization.

 

 

Accuracy is a top priority

One of our customers told us that when it comes to regulatory compliance, they refuse to leave the responsibility to their employees, but rather seek trust in their data governance solution. We cannot blame them, because before choosing to implement a MANTA + Informatica regulatory compliance solution, the customer had a team of more than 20 people that manually drew data lineage diagrams for more than 200 risk metrics within their institution.

Based on our experience with data lineage across various industries, we know that it is almost impossible for humans to create accurate data lineage visualization, because there is an endless amount of data and multiple dimensions that need to be incorporated, including the quality of the data, the people who are allowed to handle it, the processes that are done with it, etc. It requires the team to talk to multiple departments to get insight on a company-wide level, and it is difficult to maintain because it requires the team to keep track of even the smallest changes within the systems as well as the data warehouse.

MANTA together with Informatica purely automates the data-lineage gathering process. Informatica’s Enterprise Data Catalog, with strong machine learning abilities, can provide automatic mechanisms for data discovery across multiple internal company systems. What MANTA can then do, is automatically scan the metadata within the systems and read even the toughest segments of custom-made programming code that EDC usually has a hard time dealing with.

Then the customer can either visualize the data lineage in MANTA’s native visualization or see the data lineage within EDC, only this time, with data lineage all the way down to the column level. This way, the customer can achieve the highest level of detail possible, which otherwise would have been almost impossible to gain using only one solution. Now, whether it is an older data-lineage diagram that has been remodeled by calculating new dependencies or if it is the creation of a completely new one, the customer can be sure that the diagram is accurate and contains all the details.

Visit getmanta.com/informatica to learn even more about our tech bond.

Would you like to read more use cases like this in the future? Or details about our scanners and integrations? Let us know at manta@getmanta.com. We are always here for you.

MANTA is Built to Last

Did you know that MANTA is built to last a company’s lifetime? If you update it periodically, you only need to adjust your monthly subscription plan based on how your BI and environment grow in capacity. Other than that, MANTA is a keeper.

Did you know that MANTA is built to last a company’s lifetime? If you update it periodically, you only need to adjust your monthly subscription plan based on how your BI and environment grow in capacity. Other than that, MANTA is a keeper.

But like with everything, there are a few simple rules you have to follow to make sure MANTA is in the best possible condition. Here are the two (yes, only two!) secrets to ensuring your MANTA will last till you retire:

1. Update

2. Upgrade

So, without further delays, lets get to it.

Update

If you are following us on all corners of the internet, you may have noticed that MANTA is constantly coming up with new connectors for different metadata management solutions and platforms. It’s not always because we didn’t connect with them before—it’s because they didn’t exist! That’s right, other companies are continuously releasing new solutions and developing new environments as well. And MANTA constantly develops connectors to support them.

Developing new connectors allows MANTA to be seamlessly reborn into new BI environments. This can save companies thousands of dollars! If you have already bought MANTA and you decide to upgrade the technologies we already support in your environment, you can be sure that MANTA’s newest version will support your newest version. Many of the companies that develop the solutions we support, such as IBM, Informatica, Teradata, and Collibra, are even MANTA’s partners. This means we often develop our connectors using and doing continuous testing on their own software, usually months before they release it. So, when something like EDC comes out, you can be sure we already integrate with it.

If you pay your subscription fees, you just need to check and make sure you have the newest version of the software installed. We make this easy for you by releasing new updates quarterly so you know when to watch out for an upgrade. Currently, we are also testing a new MANTA updater in beta, to make MANTA software updates even more user friendly.

Upgrade

As you know from our pricing, MANTA is priced according to the number of scripts in your data warehouse environment. The fact that you have upgraded to a new technology doesn’t necessarily mean that you have more scripts in your environment than you did before.

If you can fit into the “script budget” that you are currently in, then you might not need to pay even a dollar more, even after changing the software in your BI environment! So, if you change your technology but your data environment stays the same size, MANTA will adjust to your needs and simply be reborn in the new technological environment.

If you have any questions about how to get the most out of MANTA, feel free to contact us at manta@getmanta.com.

We cherish your privacy.

And we need to tell you that this site uses cookies. Learn more in our Privacy Policy.