The ultimate spin-off of the GREAT data debate & audience verdicts on where we're headed

✨Spotlight: Highlights from the GREAT data debate and verdicts from audience on where we're headed

Welcome to this week's edition of the ✨ Metadata Weekly ✨ newsletter. Every week I bring you my recommended reads and share my (meta?) thoughts on everything metadata!

And this week I'm with the data community in Austin! If you're around and want to hang out, DM me! (If you’re staying back on the 24th March evening, would LOVE to have you join us for a bat cruise Atlan is hosting with Great Expectations). 👋 Now back to this week's edition of the newsletter.

In the last month, the data world has been on fire with the recent debate on bundling vs unbundling the data stack. 🔥

In February, Gorkem Yurtseven kickstarted this debate with an article on The Unbundling of Airflow. Nick Schrock wrote a response blog titled the “Rebundling of the Data Platform”, which broke the data community (again). Then Ananth recently shared his take on this great data debate in his blogs here and here.

So we decided to bring these OGs of this discussion to one Zoom, for the first time, for an ultimate spin-off of this debate! I personally had so much fun hosting this, so I’m sharing some of the highlights from our discussion in this week’s edition of the newsletter. You can watch the full recording from the event here.

What does the audience say? Bundling vs unbundling polls! 

During the event, we polled our audience on whether they thought that different parts of the modern data stack should become bundled or unbundled. Here we are sharing the verdict from 80+ data people – data engineers, data leaders, data analysts, and more.

Bundling vs unbundling: reverse ETL with data ingestion?

The verdict from our audience on reverse ETL and data ingestion: unbundled (according to 59% of attendees).

We’re now talking more about how to use data in daily operations than how to warehouse it. That’s an incredible sign of how mature the fundamental building blocks of the data stack (warehousing, transformation, etc) have become!

Players like Hevo Data have already started offering both ingestion and reverse ETL services in the same product. But as Ananth pointed out during our chat, “Reverse ETL is about domain knowledge and the vertical systems are well equipped to provide more and more insights”. So while there will be attempts to bundle the two categories, the vendors in this space may also choose to sell them separately.

Bundling vs unbundling: metrics layer with transformation or BI tools?

The verdict from our audience: bundled (according to 57% of attendees).

Stand-alone metrics layers will struggle to encourage BI tools to adopt their frameworks and will be forced to build BI like Looker was forced to many years ago. dbt made their foray into the metrics layer. dbt already has enough distribution to encourage at least the modern BI tools (e.g. Preset, Mode, Thoughtspot) to integrate deeply into the dbt metrics API, which may create competitive pressure for the larger BI players.

Ananth provides an interesting “task execution” and “model” framework in his blog: “We need to merge both the model and task execution unit into one unit. Otherwise, any abstraction we build without the unification will further amplify the disorganization of the data.”

I personally think that existing BI tools aren’t really incentivized to integrate an external metrics layer into their tools, metrics layers are so deeply intertwined with the transformation process that intuitively the bundling makes sense.

Bundling vs unbundling: data observability with other categories?

The verdict from our audience: bundled (according to 61% of attendees).

Today data observability companies are doing both – data quality as well as observability because of the gap that existed in the market a few years ago.

I believe that in the past two years, data teams have realized that tooling to improve productivity is not a good-to-have but a must-have. After all, data professionals are one of the most sought-after hires you will ever make, so they shouldn’t be wasting their time on troubleshooting pipelines.

Observability is a diversity problem. Ideally, if you have all your metadata in one open platform, you should be able to leverage it for a variety of use cases (like data cataloging, observability, lineage, and more). I wrote about that idea last year in my article on the metadata lake.

That being said, today, there’s a ton of innovation that these spaces need independently. My sense is that we’ll continue to see fragmentation before we see consolidation in the years to come.

So... the real question: when do you bundle and when do you unbundle? The real answer: it depends.

In one of my previous editions, I wrote about why I believe that there will be more bundling from our current version of the data stack but we will never (and shouldn’t) have a fully bundled data stack. Read more here.

Check out the recording from the event here. Watch for some behind-the-scenes fun and some 🌶 takes on one of the greatest data debates of 2022, and tell me which side are you on. I’d love to hear!

❤️ Fave links from last week

Growing Data Teams from Reactive to Influential by Emily Thompson

“Seeing the landscape of requests is a good start, but in order to drive real change in the way things operate, it’s important to know how those analysis results, experiments, dashboards, and other data artifacts are being used to make decisions. It should never be the data team’s job (or the data team manager’s job) to prioritize all of the incoming requests by themselves. In an ROI equation, the data team only knows the ‘I’ with certainty, and can only speculate what the ‘R’ might be if they don’t know what the higher-level goals of their business partners are.”

My take: First of all, I highly recommend recent articles from Emily if you’re leading a data team or are an aspiring data leader. I often speak to founders about their data strategy, and when we speak about “ROI”, I encourage them to also think of “Data Advantages” — which data investments will help you build sustainable competitive advantages to outperform your competitors?

As Emily rightly points out, the data team doesn’t know what the higher-level goals of business partners are. Emily shares some great points in her blog on driving company-wide data maturity. I also believe that once the data leaders bring the “advantage” lens, it will open a way to go from a “reactive data team” to “a data team with influence” over company decisions.

For data leaders looking to make this happen, I share the concept of the Data Advantage Matrix with them, something that I believe can help create a strong data-driven culture. I wrote about this in one of my previous blogs here.

📘 More from my reading list

• A Different Way to Bundle Data Platforms by Petr Janda

• 7 Antifragile Principles for a Successful Data Warehouse by Iliana Iankoulova

• How ManyPets Implemented the Modern Data Stack by Mark MacArdle

• Leaders Show Their Work by Ben Balter

• Wander Well by Stephen Bailey

I’ve also added some more to my data stack reading list. If you haven’t checked out the list yet, you can find and bookmark it here.

🗓️ Upcoming events

Metrics Store Summit | Apr 26, 9:00AM - 4:00PM PST

Transform is hosting a one-day virtual summit on the future of the modern data stack powered by the metrics store. Registrations are now open.

Airflow Summit 2022 | May 23 to May 27, 2022

Registrations for this year’s Airflow Summit are now open! It’s a hybrid event this time. Check it out and register here.

😂 Lastly, no one says it better than Dilbert comics

The latest one is on data quality!

I'll see you next week with more interesting updates from the modern data stack! 👋 Meanwhile, you can subscribe to the newsletter on Substack and connect with me on LinkedIn here.