Stop trying to become “AI ready”
Stop trying to become “AI ready”
The real question you should be asking about your data governance
Let’s start this week with a hot take — even though we hear it all the time, “How can I get AI ready?” is the wrong question to be asking today.
But what should we be asking?
I recently came across an interesting mental model by Ray Dalio, the founder of the world’s largest hedge fund. He studied the top 10 most powerful empires in the world since 600 BC to understand why nations succeed and fail.
He found something interesting: any empire that rises inevitably falls. “Of course”, I hear you saying. But here’s the kicker — every empire follows the same lifecycle as it rises and inevitably falls.
Each empire has a rise, top, and decline with the same characteristics. You can’t change that. What you can change is the slope, or how fast an empire can adapt and thrive as the world changes around it.
So what changes the slope of this line? Good governance.
Across the past 2,600 years, we’ve seen that nations and empires that govern effectively can increase the slope upward and decrease the slope downward, maximizing their success over a longer period of time.
The problem is that society is complex, and that makes governing hard. There are so many personas to consider, from tradespeople and doctors to scientists and merchants. Empires must also develop a variety of resources, such as natural resources, new technologies, human skills, and economic trade. These then need to be strategically leveraged towards a diverse set of outcomes, such as infrastructure, healthcare, innovation, and wealth.
Data teams face the same challenge as societies — increasingly diverse personas, tools, and outcomes. So as we think about the future of data, we should be thinking about more than what’s immediately ahead — AI. We instead need to think about how to alter our slope.
In other words, our question should be “How can I build an enduring organization that can adapt to a world that’s changing faster than ever before?” The answer to this lies in great data governance.
✨ Cracking the data governance problem
Last week, my coauthor Sharif Karmally and I talked about why data governance fails. Today, we’ll talk about solutions to these problems, or what 2,600 years of history can teach us about better data governance. Here are the Principles of Active Governance that we’ve been crafting at Atlan based on what we’re hearing from the community, four ideas you should know to make data governance actually work.
1. Use data products as a common currency
Governance should translate diversity into a common language.
One of the key issues with data governance today is that companies lack a “common currency” for talking about data. Data governance relies on information flowing from person to person and team to team, but different groups of people talk about data differently. Governance relies on documentation to create a shared currency, but this is often too much of a burden, leaving data producers and consumers alike in the dark.
What’s the solution? Let’s look to history — specifically, the Persians. Around 515 BC, Darius I of the Achaemenid Empire (the first Persian Empire) introduced the daric, a gold coin that was arguably the first ever standardized currency. By unifying a vast, diverse population under a single currency, Darius was able to increase trade, streamline governance, and eventually create the largest empire of the time, spanning about 5.5 million kilometers.
When dealing with vast diversity, establishing a common medium of exchange is key. So a question I’ve been thinking about is, what will the common currency of data look like?
Since we pioneered a native data products experience in Atlan, I’ve seen data products become a connective tissue between technical and non-technical users. Rather than talking about individual components of data (e.g. a data asset or a pipeline or a dashboard), data products focus on end usability. This allows them to act as a common currency, creating a curated, reusable, reproducible, and trusted interface between diverse data personas, tools, and outcomes.
2. Don’t treat governance as a “one size fits all”
Governance should adapt to people. People shouldn’t adapt to governance.
I’ve yet to come across two governance practices that look exactly the same, and yet most people, from different teams and even different companies, are often expected to follow the same governance rules. That’s why one of the biggest challenges for data governance today is moving from prescriptive governance to something that can accommodate the diverse people, processes, and technology within a company.
One thing that distinguished the Roman Empire was its selective flexibility. Rather than copy-pasting its rules and culture on every region, the Roman Empire often adapted to key aspects of local context. For example, even though the Romans had well-defined laws like the 12 Tables, they allowed Greece to continue its own well-loved legal tradition. By choosing its battles, the Roman Empire was able to focus more on economic growth rather than stamping out diversity, allowing it to become one of the largest, most powerful empires in history.
This brings us to the third lesson from history — governance is not a “one size fits all”. Just like with the Romans, it needs to start with the people.
In Atlan’s early days, we had a saying: “Diversity is the only reality in a data team.” Every organization is different, and every data team and data person in it is different. Governance often fails by expecting an incredibly diverse set of people to adapt to technology, rather than technology adapting to people. Whether it’s centralized, decentralized, or federated, governance should be truly adaptive and flexible to a company’s specific structure. This may involve personalizing the data experience to each user persona, business domain, and data project, or even building custom experiences on top of an open data platform.
3. Embed governance in daily work
Governance should embed itself as code, not get lost in documentation.
In most companies, the policies governing data live apart from data work. This means that, no matter how well-intentioned a company and its employees are, data governance policies and rules are often not actually put into practice. People may miss important aspects of data governance while deep in daily work, or they may forget about them entirely. Is there a way to embed data governance into the way people work so it isn’t an afterthought?
The solution to this problem lies with the Egyptians, who built what became one of the Seven Wonders of the Ancient World — the Lighthouse of Alexandria. How did they pay for something so massive that it remained one of the tallest structures for centuries? Rather than taxing workers once or twice a year, the Egyptians introduced the concept of tax withholding. People no longer had to think about paying their taxes — it just happened automatically. As a result, the ancient Egyptian empire was able to increase revenue and make it more predictable, allowing them to fund wonders like this lighthouse.
One of the key ideas of active metadata, a concept I’ve been thinking about since 2021, is bringing data context (such as metadata and ownership) and data actions (such as approvals and collaboration) to the places where data people naturally work. The natural evolution of this vision is embedding data governance in daily work, through built-in code rather than distant PDFs, to ensure it’s never forgotten.
For example, we can shift governance “left” into the data producer workflow with things like automated documentation and “ELTP” architectures, or “right” by embedding context into the BI tools that data consumers use every day. We can shift governance “down” to make it invisible, such as making tags and security from the metadata layer available in every other tool in the data stack. We can even shift governance “up” to stewards, enabling them to proactively review policy violations and govern by exception.
4. Treat change as a feature, not a bug
Governance should change and evolve. It is never “done”.
In search of privacy and protection, enterprises often wall off their data in closed, legacy systems. The problem is that this approach to governance can hinder them from actually using their data. Without the ability to openly build on top of their data platform, companies end up locking themselves out of new initiatives like AI.
The fourth and final history lesson isn’t from that long ago — the 2020 U.S. Census, a hallmark of adapting to change. In 1890, the U.S. government introduced tabulating machines, reducing data capture and processing time by 59%. Then, in the midst of the pandemic, the 2020 Census introduced a fully online option for the first time, resulting in a higher response rate than in 2010. By actively adapting to change, the Census was able to persist and even thrive amidst challenging conditions.
Just like the real world, the data world is constantly changing — two years ago, we weren’t talking about vector databases and LLMs, and we guarantee that two years later we’ll be talking about something else entirely. More than just adapting governance to the latest technology (i.e. AI), governance should include change as a feature, not a bug. This requires a governance operating model that is fundamentally open by default, can incorporate shifts like AI, and provides a flexible foundation that works for both current knowns and future unknowns.
We spoke about these ideas last week at Re:Govern, the industry conference for a new era of data and AI governance. Start your journey towards better data governance with my keynote and recorded sessions from leading data teams like Dropbox, Patagonia, and General Motors.
📚 More from my reading list
Data about data from 1,000 conversations with data teams by Mikkel Dengsøe
What is the best advice you have ever received? by Olga Berezovsky
How to price a data asset by Abraham Thomas
The limits of data by C. Thi Nguyen
Unexpected tips for data managers by Emmanuel Martin Chave
How tech debt, Databricks, and Spark UDFs ruined my weekend by Daniel Beach
10 technical data newsletters you should know by Meri Nova
An ambitious San Francisco lawmaker is in the middle of a battle for AI’s future by Jeremy B. White
AI Roundup: GPT-4o and Google I/O by Charlie Guo