The Rise of DataOps 🚀

From Forrester's Wave report and Gartner's Hype Cycle to modern data team building entire strategies, functions, and even roles around DataOps . ✨

Let’s face it — traditional data management doesn’t work. Today, 75% of executives don’t trust their own data, and only 27% of data projects are successful. Those are dismal numbers in what has been called the “golden age of data”.

DataOps promises to solve this problem, and it’s on fire right now. In the last couple of months, Forrester and Gartner recently made major shifts toward recognizing the importance of DataOps with their Wave report and Hype Cycle.

But the rise of DataOps isn’t just coming from analysts. At Atlan, we work with modern data teams around the world. I’ve personally seen DataOps go from an unknown to a must-have, and some companies have even built entire strategies, functions, and even roles around DataOps. While the results vary, I’ve seen incredible improvements in data teams’ agility, speed, and outcomes.

Let’s dive into everything you should know about DataOps. Happy reading!

✨ Spotlight: The Rise of DataOps

What is DataOps?

The first, and perhaps most important, thing to know about DataOps is that it’s not a product. It’s not a tool. In fact, it’s not anything you can buy, and anyone trying to tell you otherwise is trying to trick you.

Instead, DataOps is a mindset or a culture — a way to help data teams and people work together better.

There’s no standard definition for DataOps. However, you’ll see that everyone talks about DataOps in terms of being beyond tech or tools. Instead, they focus on terms like communication, collaboration, integration, experience, and cooperation.

In our mind, DataOps is really about bringing together today’s increasingly diverse data teams and helping them work across equally diverse tools and processes. Its principles and processes help teams drive better data management, save time, and reduce wasted effort.

The four fundamental ideas behind DataOps

Some people like to say that data teams are just like software teams, and they try to apply software principles directly to data work. But the reality is that they couldn’t be more different.

In software, you have some level of control over the code you work with. After all, a human somewhere is writing it. But in a data team, you often can’t control your data, because it comes from diverse source systems in a variety of constantly changing formats.

The way we like to think about DataOps is, how can we take the best learnings from other teams and apply them to help data teams work together better? DataOps combines the best parts of Lean, Product Thinking, Agile, and DevOps, and applying them to the field of data management.

How do you actually implement DataOps?

Every other domain today has a focused enablement function. For example, SalesOps and Sales Enablement focus on improving productivity, ramp time, and success for a sales team. DevOps and Developer Productivity Engineering teams are focused on improving collaboration between software teams and productivity for developers.

Why don’t we have a similar function for data teams? DataOps is the answer. Here are the three key steps to implementing it:

• Identify the end consumers who will be affected by a DataOps strategy and function

• Create a dedicated DataOps function, based around two key personas: DataOps Enablement Lead, and DataOps Enablement Engineer

• Map out value streams, reduce waste, and improve collaboration with Agile and the JBTD framework

Learn more about the rise of DataOps in this blog.

📚 More from my reading list

• The unsung data heroes by Mikkel Dengsøe

• In search of new standards by Tristan Handy

• Data Products Aren't Exempt From Good Engineering Practices by Eric Weber

• Data Mesh — A Data Movement and Processing Platform @ Netflix by Bo Lei, Guilherme Pires, James Shao, Kasturi Chatterjee, Sujay Jain, Vlad Sydorenko

• Why Are We Still Struggling To Answer How Many Active Customers We Have? by Ben Rogojan

I’ve also added some more resources to my data stack reading list. If you haven’t checked out the list yet, you can find and bookmark it here.

💙 This Year @ Gartner Summit in Orlando

Next week I am in Florida for much awaited Gartner’s Data and Analytics Summit. I am also super excited to team up with Venkat Gopalan, Chief Data Office at Belcorp (one of the largest AWS accounts in the LATAM region) to share how he has led the charge of transforming the entire company into a data-first company. Some of the things on our agenda:

• What is DataOps, and why more data teams should care about it

• How, as a new CDO, Venkat created a DataOps strategy for the entire company

• How to get management and executive buy-in for a digital data transformation

• Choosing the right tools to enable the DataOps journey

• DataOps enablement initiatives and best practices, like Belcorp’s Data University, which had massive success in improving data literacy

If you’re around attending the summit, come join us for this session on 23rd August at 1:15 pm ET. More details here.

Cannot make it for this one? Atlan is hosting an invite-only masterclass on the essentials of building a DataOps culture. Ping me for an early invite. :)

See you next week!

P.S. Liked reading this edition of the newsletter? I would love it if you could take a moment and share it with your friends on social.

---