HomeBig DataAtlan + Airflow: Higher Pipeline Monitoring and Information Lineage with Our Latest...

Atlan + Airflow: Higher Pipeline Monitoring and Information Lineage with Our Latest Integration – Atlan


One morning at 8 am, I woke as much as the Cupboard Minister of India calling me. He mentioned, “Prukalpa, the quantity on this dashboard doesn’t appear proper.”

Frantic, I opened up my laptop computer and loaded the dashboard to appreciate the quantity was clearly off. And but, at that second, there was nothing I might do to elucidate it. I might really feel myself dropping the credibility and hard-earned belief that had taken months to construct.

I referred to as my Undertaking Supervisor, who was implausible at stakeholder administration however couldn’t perceive the nitty-gritties of knowledge. She referred to as our Information Analyst, who appeared on the dashboard and mentioned, “Looks like one thing broke down within the pipeline”. Our Analyst then referred to as our solely Information Engineer, who pulled out logs from Apache Airflow. However he couldn’t troubleshoot it as a result of he didn’t know what the variables meant and didn’t have the info context.

It took us 8 hours and 4 folks to determine what went flawed. We misplaced time that day.

However extra importantly, we misplaced belief. Belief with our buyer. Belief in our crew.

Belief is usually not about issues breaking. In years of working with information, I’ve realized that information will all the time be chaos. However when issues break and you discover out too late, or you possibly can’t clarify why one thing broke, that’s what breaks belief.

Think about if, at that second when the cupboard minister referred to as me, I might shortly open a dashboard and say, “Sure, looks like the pipeline didn’t run on time right now. We’ve acquired an alert and it has already been escalated to information engineering.” And even higher, think about if the dashboard had an alert on it, signaling to the minister that one thing was flawed and he shouldn’t use it.

As we speak we’re excited to announce that Atlan natively integrates with Apache Airflow. For information groups all over the place, this implies extra transparency and belief, and fewer time spent debugging pipelines after a damaged dashboard or mismatched metrics.

Atlan + Airflow: Constructing an ecosystem of belief and transparency

With this integration, information groups can construct higher information engineering experiences centered round constructing data and belief of their information.

First, Atlan’s integration with Airflow brings much-needed pipeline context to information property.

Now you possibly can share any sort of metadata from Airflow pipelines to Atlan information asset profiles, the place information analysts, scientists, and enterprise customers have entry to it. This opens up pipeline context and makes it totally clear in order that information groups and customers can all the time know the standing of the info pipeline related to every information asset.

Listed below are some nice context fields that we’ve seen folks convey from Airflow to Atlan:

  • Freshness: When was my desk final up to date?
  • Run schedule: Did the pipeline run as anticipated?
  • Pipeline standing: Was the final pipeline run profitable?
Customized Airflow metadata on an Atlan asset profile

Atlan already connects to information warehouses (e.g. Snowflake, Redshift) and BI instruments (e.g. Tableau and Looker). Bringing Airflow into this ecosystem additionally signifies that information groups can now map relationships throughout all of their information. Whether or not you’re loading in new information, revising a pipeline, or establishing a dashboard, now you can assemble and visualize information lineage from finish to finish.

Atlan: Tableau assets linked with source Snowflake tables
Tableau property linked with supply Snowflake tables

Much less time debugging, extra time constructing

Getting an pressing name about damaged information is among the worst experiences for an information crew. As a substitute of calling everybody who has ever touched the info, now you can diagnose the issue in seconds.

All it takes is opening an information asset profile and checking the pipeline standing and metrics. No extra hours of scrambling or damaged belief, Atlan and Airflow’s integration helps you to see your whole information and its context in a single place.


Able to get began with this integration? Try a demo of Atlan.

Listed below are two assets that can assist you get began with bringing Airflow and Atlan collectively:

RELATED ARTICLES

LEAVE A REPLY

Please enter your comment!
Please enter your name here

Most Popular

Recent Comments