All Topics

Trust Your Lineage, All of It

Can you trust your physical, business and logical lineage? Manta introduces support for many different levels of data lineage abstraction. 

Can you trust your physical, business and logical lineage? Manta introduces support for many different levels of data lineage abstraction. 

When it comes to data, trust is always the key. And getting a complete overview of data flows in your system is necessary to get that trust back. Now, different types of lineage won’t mess things up anymore. It’s almost impossible to map complex BI systems on more levels of abstraction. Many different tools provide physical (technical), business and logical lineage, but this lineage is only good when it is complete. Like, totally.

Physical Lineage Is the Key

At Manta, we’re good at getting detailed and accurate physical data lineage from your logic hidden in programming code. Never mind SQL overrides, manually defined procedures, stored procedures – MANTA will just map it all. On top of that, we are now able to include different levels of lineage, but backed up by original physical lineage so it’s 100% accurate. And it’s all fully automatic, so there’s no manual labor necessary… Here is how it works:

First, MANTA will do what it is the best in the world at – map detailed and accurate physical data lineage from the logic hidden in your programming code.

Second, it loads external mapping between physical and logical or business objects (like business name to specific table / column mapping) from available sources and ties that with rendered physical lineage.

Third, it uses mapped objects (business, logical) to transform existing physical lineage so it is better aligned with the objects provided (i.e. no detailed technical transformations for business objects but rather simplified descriptions). The result is accurate, trustworthy lineage of any kind – based in reality and yet useful for everyone who needs to understand the specific level of abstraction.

Any Lineage, Any Source

It does not matter which technology your physical lineage is from (check out our list of supported technologies!). It also does not matter how you provide the initial business/logical to physical object mapping – it could be:

  • your favorite business glossary
  • a data modelling platform
  • plain ol’ Excel spreadsheets
  • virtually any other structured data format

 

From an Excel Sheet to a Data Governance Solution

Our main goal is to connect and share information with other tools and solutions in our customers’ BI. That’s why we are not only able to pull metadata from other tools, but MANTA can easily push everything back into 3rd party solutions. Want an example? At one of our successful implementations, MANTA:

  1. Loaded business lineage mapping from Collibra’s Data Governance Center
  2. Combined it with complete physical lineage from actual code
  3. Pushed lineage back to Collibra, ensuring that the lineage was complete and functional

Additionally, MANTA is always capable of visualizing everything in our visualization – feel free to take a look at our introductory video:

To learn more about MANTA, simply get in touch and ask for 30-day free trial!

The Dark Side of the Metadata & Data Lineage World

June 10, 2017 by

You wouldn’t believe it, but there is a dark side to the metadata & data lineage world as well. Tomas Kratky digs deep and explains how you can get into trouble. 

You wouldn’t believe it, but there is a dark side to the metadata & data lineage world as well. Tomas Kratky digs deep and explains how you can get into trouble. 

It has been a wonderful spring this year, hasn’t it? The first months of 2017 were hot for us. Data governance, metadata, and data lineage are everywhere. Everyone is talking about them, everyone is looking for a solution. It’s an amazing time. But there is also the other side, the dark side.

The Reality of Metadata Solutions

As we meet more and more large companies, industry experts & analysts, investors and just data professionals, we see a huge gap between their perception of reality and reality itself. What am I talking about? About the end-to-end data lineage ghost. With data being used to make decisions every single day, with regulators like FINRA, Fed, SEC, FCC, and ECB requesting reports, with initiatives like BCBS 239 or GDPR (a new European Data Protection Directive), proper governance and a detailed understanding of the data environment is a must for every enterprise. And E2E (end-to-end) data lineage has become a great symbol of this need. Every metadata/data governance player on the market is talking about it and their marketing is full of wonderful promises (in the end, that is the main purpose of every marketing leaflet, isn’t it?). But what’s the reality?

The Automated Baby Beast

The truth is, that E2E data lineage is a very tough beast to tame. Just imagine how many systems and data sources you have in your organization, how much data processing logic, how many ETL jobs, how many stored procedures, how many lines of programming code, how many reports, how many ad-hoc excel sheets, etc. It is huge. Overwhelming!

If your goal is to track every single piece of data and to record every single processing step, every “hop” of data flow through your organization, you have a lot of work to do. And even if you split your big task into smaller ones and start with selected data sets (so-called “critical data elements”) one by one, it can still be so exhausting that you will never finish or even really start. And now data governance players have come in with gorgeous promises packaged in one single word – AUTOMATION.

The promise itself is quite simple to explain – their solutions will analyze all data sources and systems, every single piece of logic, extract metadata from them (so-called metadata harvesting), link them up (so-called metadata stitching), store them and make them accessible to analysts, architects, and other users through best-in-class, award-winning user interfaces. And all of this through automation. No manual work necessary, or just a little bit. Is it so tempting that you are open to it, you want to believe. And so you buy the tool. And then the fun part starts.

The Machine Built to Fail

Almost nothing works as expected. But somehow you progress with the help of hired (and usually overpriced) experienced consultants. Databases (tables, columns) are there, your nice graphically created ETL jobs are there, your first simple reports also, but hey! There is something missing! Why? Simply because you used a nasty complex SQL statement in your beautiful Cognos report. And you used another one when you were not satisfied with the performance of one Informatica PowerCenter job. And hey! Here, lineage is completely broken? Why is THAT? Hmmm, it seems that you decided to write some logic inside stored procedures and not to draw a terrifying ETL workflow, simply because it was so much easier with all those Oracle advanced features. Ok, I believe you have got it. Different kinds of SQL code (and not just SQL but also Java, C, Python and many others) are everywhere in your BI environment. Usually, there are millions and millions of lines of code everywhere. And unfortunately (at least for all metadata vendors) programming code is super tough to analyze and extract the necessary metadata from. But without it, there is no E2E data lineage.

At this moment, marketing leaflets hit the wall of reality. As of today, we have met a lot of enterprises but only very few solutions capable of automated metadata extraction from SQL programming code. So what do most big vendors usually do in this situation (or big system integrators implementing their solutions)? Simply finish the rest of the work manually. Yes, you heard me! No automation anymore. Just good old manual labor. But you know what – it can be quite expensive. For example, a year ago we helped one of our customers reduce the time needed to “finish” their metadata project from four months to just one week! They were ready to invest the time of five smart guys, four months per person, to manually analyze hundreds and hundreds of BTEQ scripts, extract metadata from them and store them on the existing metadata tool. In the United States, we typically meet clients with several hundreds of thousands of database scripts and stored procedures. That’s sooo many! Who is going to pay for that? The vendor? The system integrator? No, you know the answer. In most cases, the customer is the one who has to pay for it.

Know Your Limits

I have been traveling a lot the last few weeks and have met a lot of people, mostly investors and industry analysts, but also a few professionals. And I was amazed by how little they know about the real capabilities and limitations of existing solutions. Don’t get me wrong, I think those big guys do a great job. You can’t imagine how hard it is to provide a really easy-to-use metadata or data governance solution. There are so many different stakeholders, needs and requirements. I admire those big ones. But it should not mean that we close our eyes and pretend that those solutions have no limitations. They have limitations and fortunately, the big guys, or at least some of them, have finally realized, that it is much better to provide open API and to allow third parties like Manta to integrate and fill the existing gaps. I love the way IBM and Collibra have opened their platforms and I feel that others will soon follow.

How can you protect yourself as a customer? Simply conduct proper testing before you buy. Programming code in BI is ignored so often, maybe because it is very low-level and it is typically not the main topic of discussion among C-level guys. (But there are exceptions – just recently I met a wonderful CDO of a huge US bank who knows everything about the scripts and code they have inside their BI. It was so enlightening, after quite a long time.) It is also very hard to choose a reasonable subset of the data environment for testing. But you must do it properly if you want to be sure about the software you are going to buy. With proper testing, you will realize much sooner that there are some limitations and you will start looking for a solution to solve them up-front, not in the middle of your project, already behind schedule and with a C-level guy sitting on your neck.

It is time to admit that marketing leaflets lie in most cases (oh, sorry, they just add a little bit of color to the truth), and you must properly test every piece of software you want to buy. Your life will be much easier and success stories of nicely implemented metadata projects won’t be so scarce.

Originally published on Tomas Kratky’s LinkedIn profile.

MANTA + Informatica

MANTA completes Informatica to form a comprehensive metadata management platform. But how precisely does the bond work? [VIDEO BELOW ↓ ]

MANTA completes Informatica to form a comprehensive metadata management platform. But how precisely does the bond work? [VIDEO BELOW ↓ ]

Trust Through Understanding

Our product allows you to trust the data in your BI environment, because it is specialized in cracking SQL code and helps to fill gaps in metadata management solutions. It does not matter whether they were deployed to fulfill compliance regulations or create the backbone of your data governance efforts. The Informatica Metadata Management solution has a rudimentary capability to parse SQL, but, in our experience with our customers over the years, there are still blind spots here and there.

MANTA connects to IMM through XConnect (native API plugin), and enriches the metadata model of IMM with missing pieces of data lineage:

  • BTEQ scripts, stored procedures, views, and macros from Teradata
  • PL/SQL scripts, stored procedures, packages, and more, including DB links, from Oracle DB & Exadata
  • T-SQL scripts, stored procedures and more, including linked servers, from Microsoft SQL Server, Sybase (now SAP ASE), and PDW
  • NZPLSQL scripts, stored procedures, and more from IBM Netezza
  • DB2 scripts, stored procedures, and more from IBM DB2
  • SQL overrides from Informatica PowerCenter, Cognos and Microsoft SQL Server Reporting Services

It fills in the gaps in data flows and allows our customers to get end-to-end data lineage (including those pesky indirect data flows).

Here is a live shot of MANTA + IMM,  fullscreen is recommended.

Beyond Lineage

MANTA’s ultimate goal is to understand the semantics of the code being analyzed. If you have ever thought about advanced performance tuning, real data protection analysis, automated business lineage extraction,or migration of your code base to a different platform, MANTA is exactly what you need.

For customers who do not have Informatica Metadata Manager, but do enjoy the advanced ETL capabilities of  Informatica PowerCenter, we are also able to provide data lineage in our own visualization or IBM IGC. MANTA’s open API allows customers to push metadata to 3rd-party tools such as Collibra, Adaptive, Alation, Axon, and others.

Faster and Cheaper

The impressive part is how fast MANTA works. It can map a BI environment at a speed that is incomparable with any  human workforce, which means you save quite a lot of money on man-hours.

If you would like a more detailed explanation of MANTA + Informatica, and how it can help with your current situation, be sure to contact one of our specialists! They can discuss your situation in detail and give you a better idea of how MANTA can help with your data governance efforts.

MANTA 3.17: Say Hello to Netezza, Business Data Lineage, and Much More

It’s a bird! It’s a plane! It’s MANTA Flow 3.17 coming your way!

It’s a bird! It’s a plane! It’s MANTA Flow 3.17 coming your way!

When we said (and we always say this) that we would expand the list of our supported technologies, we weren’t lying. MANTA 3.17 comes with support for IBM Netezza and IBM DB2, provides business data lineage, and much more!

The biggest news among big news is that MANTA now supports Netezza. As always, pioneers are wanted and needed (ask for your trial).  It can do everything we taught it to, so if there is anyone out there who has a Netezza solution and wants to have the data lineage all nice and clean, we’ve got you covered.

This release is also the debut for a DB2 connector, now (almost) in stock. It’s no secret that we still have some plans for that one, so stay tuned. Because the Parser Team rocks, we worked on all of our parsers and we can proudly say that every new version of MANTA brings more and more improvements.

Did you know that complete business data lineage is now a thing? Yes, it is! And MANTA can provide that for you!

What we do is we can put together the physical metadata from the existing MANTA connectors with business terms provided by the user. From that, MANTA Flow can give you end-to-end business data lineage. We are the first ones who can provide you with data lineage that suits the needs of your business users as well as your BI team from information that your company already owns.

And as usual, that’s not it. We have added some cool features to MANTA, e.g. searching in source code, and much more.

We are on Instagram!

As you all know, Manta has already expanded from Prague to Germany and then all the way to the US. Now it’s time for us to step up our social media game!

As you all know, Manta has already expanded from Prague to Germany and then all the way to the US. Now it’s time for us to step up our social media game!

We have decided to create an Instagram profile. This profile will document life in our Prague office, the “behind the scenes” of our projects and the conferences we attend. It’s a chance to see what it’s like to work for Manta, to live the “Manta Life.”

If you would like to get some insight into the people behind Manta and what they do, then follow us on our Czech Instagram profile @getmanta.

 

Follow us on Imstagram!

See you there!

Manta & Collibra: How To Get Complete Data Lineage

May 2, 2017 by

MANTA improves accuracy, completeness, effectivity, and automation to your Collibra-based data governance solution.

MANTA improves accuracy, completeness, effectivity, and automation to your Collibra-based data governance solution.

With MANTA, there is no need to worry about end-to-end data lineage to support your BASEL, CCAR, or GDPR projects because MANTA extracts technical metadata from all the databases, scripts, stored procedures, and other kinds of data processing logic in your BI environment and feeds them into Collibra so you don’t have to analyze them manually.

Why do you need data lineage in Collibra?

Because accurately documented data linage is an important source of eff ectivity in BI. It is needed for daily work as it describes business and data processing logic in a company. Data lineage is also a must-have to comply with several regulations (like BCBS or GDPR).

But Collibra already supports data lineage. So what is the difference with MANTA?

Collibra provides great features here, but any data governance solution, including Collibra, is only as good as the metadata in it. And this is where MANTA helps – brings technical metadata to Collibra in an automated way.

Data lineage provided by Manta in Collibra’s native visualization (click here to open a larger image in a new tab)

Why should we care about automation when we can get our developers and analysts to document metadata manually?

Because the manual approach is labor intensive, time consuming, and error-prone. Your guys have better things to do than manually document every detail of the logic hidden inside the millions of lines of data processing code they develop. MANTA is a better way.

But Collibra reads metadata from our databases. How does MANTA differ?

There are two critical pieces to any data environment – structures (like files, database tables, columns, and associated business entities) and logic (programs, applications, ETL workfl ows). MANTA does what others can’t – brings metadata from your business and data processing logic to Collibra.

MANTA and Collibra are best when served together.

Increase Efficiency of Your BI by 30% in 2017

April 3, 2017 by

MANTA helps enterprises get full data lineage of their business intelligence systems and achieve three basic goals:

MANTA helps enterprises get full data lineage of their business intelligence systems and achieve three basic goals:

  1. Complete regulatory output and improve data governance
  2. Eliminate manual labor and increase effectivity
  3. Unlock the potential of existing data governance solutions

But they key question always remains – how do these goals translate into actual $$$? We’ve gathered data from our existing customers both in the US and Europe and put it together:

20% – 30% increase in overall efficiency of the BI/DWH team*
*including analysts, architects, developers, testers

For a team of 20 with an average salary of $100,000 per year, MANTA provides a $400,000 – $600,000 value per year (or $20,000 – $30,000 a year per team member).

1. Full compliance with existing toolsets

Data governance has always been about making sure that all your solutions work perfectly together. Many of our customers, especially in banking and finance, are also pressured by regulators to prove they completely understand data flows in their BI. There’s usually many different (and expensive) metadata and other tools involved, and MANTA is able to connect all of them and accelerate compliance projects by up to 45%.

2. More efficient onboarding of new team members

On average, a typical onboarding process takes from 5 to 10 months before a new team member is fully up-to-speed (depending on the complexity of the environment). MANTA reduces this time by 40% by providing newcomers with complete, transparent documentation of code, even the old/legacy code, without any extra manual work needed.

3. Agile and more efficient maintenance of BI/DWH

With the growing importance of data for running a company, business departments are exerting even greater pressure than ever before to obtain insights quickly and get new features implemented as soon as possible. Analyses of the current state and the anticipated impacts of any changes (as-is analyses, impact analyses) are critical and time consuming phases of a development life cycle. Industry practice says that analysis represents 20% to 40% of the effort needed for any change request. MANTA increases the efficiency of doing as-is and impact analyses by 90% by automating them and eliminating all the manual work.

4. Less money spent correcting errors

One of the most frequent causes of errors in programming systems is the combination of missing documentation and the limited ability to foresee the impact of the changes being implemented. Also, when fixing bugs there are typically more senior team members involved which further increases the costs associated with every issue. MANTA cuts the time spent by the whole team on bug fixing by 30%.

5. Always up-to-date documentation

Most teams struggle with the right approach to documentation. They usually either spend no time doing it, which leads to enormous problems and costs associated with long term maintenance, or they spend too much time on it, trying to cover all important aspects – starting with the bigger picture (business requirements, goals, KPIs) and then diving into the details (logical, physical models). But there are more static and more dynamic parts of documentation and trying to keep them up-to-date becomes very expensive and time consuming. MANTA reduces the time needed to create and maintain documentation of code by 80%.

The Next Steps

Join Us At FIMA 2017 in Boston in May

March 20, 2017 by

Another year, another FIMA. Come with us and save money. 

Another year, another FIMA. Come with us and save money. 

We went to FIMA last year and since we’ve really felt like home there, we are going this year again! In 2017, FIMA aims for three days of content and 12+ hours of networking with total over 300 attendees. You can also take a look at sessions and discussions lead by top reference data management professionals, all covering topics that are of fundamental importance to enterprise-wide data management initiatives.

And here’s the sweet bonus – if you decide to come, you can use our code to get 25% discount on your ticket!

When: May 8-10, 2017

Where: The Westin Copley Place, Boston, MA (map)

25% Discount Code (use during the registration): FIMA17MANTA

And do not forget to stop by our stand and take a look at a couple awesome new features, technologies and our Big Data integration. See you there!

Manta Meets Big Data: The Mirror between Two Worlds

March 14, 2017 by

MANTA is entering the world of Big Data. How? We now help enterprises with migration projects and automate lineage for Big Data technology Hive (with more to come soon). 

MANTA is entering the world of Big Data. How? We now help enterprises with migration projects and automate lineage for Big Data technology Hive (with more to come soon). 

The world of business intelligence has been rocked by Big Data over the last few years and MANTA has been positioned in a bit more conservative part of it. That’s why we started with more traditional SQL-based technologies and now support all the big players in the field (Teradata, Informatica, Oracle, and IBM, to name a few).

Our ability to create complete data lineage from different sources has many unique uses for our customers. One of them is especially useful – complete data lineage proves absolutely necessary when it comes to migrating between different technologies. And that’s why we’ve set foot on this tricky path to Big Data.

How to Migrate without Losing Your Mind

Our key data lineage product, Manta Flow, can serve as a mirror between the old and the new. But that statement is a bit vague (yet poetic), so let’s illustrate it using a classic example:

A company is thinking about migrating part of their BI into a fresh new Big Data technology (like Hive) and wants to keep some parts in a solid old database (like Teradata). 

Using an all-powerful database for a routine data transfer is obvious overkill, but those heavy-duty data marts are good where they are for now. So, how exactly can Manta be beneficial during the migration?

  • You must understand the old code to migrate the logic into a new system. Manta Flow can help you with that – we’re more than capable of documenting data lineage and helping your data professionals make everything work as it is supposed to.
  • Step-by-step approach. Everything should be under control, especially when your data is in question:
    1. Manta Flow analyzes the old part of the BI and shows you all data flows.
    2. After that, your BI pros can migrate it to Hive and test it.
    3. Manta Flow analyzes Hive as well and provides you with data lineage for Hive.
    4. You can easily see how everything works and if the old and new fits mirror each other.
  • The “discovery phase” is now manageable. Even if you are not so sure about the migration, a valuation alone could cost you a fortune. But you have to know what you are migrating before you do it, right? Manta Flow can help you with migration difficulty scoring and provide you with a detailed analysis of the code and its logic.
  • We are ready for more. So far, we’ve started with Hive, but we are working on other technologies – Amazon Redshift, Google BigQuery, Spark, Pig, Google Spanner, and more.

Are you ready to discuss your plans with us? Get in touch!

 

 

 

 

Take a Peek at What’s Coming in 2017

February 10, 2017 by

MANTA is sailing west (again)!

MANTA is sailing west (again)!

The new year is undeniably here and we want to give you a chance to meet and get to know MANTA even better.

Teradata Universe is the first European conference MANTA will participate in this year. We want to show you how to maximize the value of your Tedarata-based DWH. Our product, Manta Flow, deployed in Teradata-based environments helps improve data lineage, manage off-loading projects, and much more. So, pack your bags and meet us on the beautiful beaches of Nice.

As the year goes by, we will travel across the Atlantic to take part in FIMA US in “The City on a Hill.” Boston has one of the oldest public parks in the US where we can discuss how MANTA can save you millions by completing regulatory output and eliminating manual work.

Just a couple of days later, Informatica World will open its gates and, of course, we will be there. In sunny San Francisco you can learn more about our partnership with Informatica and how Manta Flow brings additional value to IMM (Informatica Metadata Manager) and enriches its metadata model with data lineage.

But those aren’t the only places where our ship will anchor. We are going to attend a lot more conferences where you will have the opportunity to meet MANTA and chat with us. Don’t worry, we will keep you updated.

Don’t hesitate to join us on this tremendous journey!

Subscribe to our newsletter

We cherish your privacy.

By using this site, you agree with using our cookies.