The missing “collaboration layer” in today’s data stack: an insider story from Postman
How to encourage data collaboration without losing our minds
It’s easy to say that everyone in a company should be able to use its data. In practice, though… pure chaos.
Too often, no one knows where the data is, which assets to use, or what it all means.
Data teams quickly get flooded with messages: What does metric X mean? How is this different than metric Y? Am I looking at the right table or metric for my analysis? The questions and requests can become endless.
Last year, after closing its $150 million Series C and becoming a unicorn, Postman was struggling with this chaos. But instead of giving up on the open data dream, they took on the challenge of democratizing company data before their Series D.
How can we encourage collaboration among the data team or even across other teams without losing our minds? Today’s Metadata Weekly is all about this collaboration challenge, featuring an insightful, evergreen story that our friends at Postman shared just over a year ago. 💙
✨ Spotlight: Behind-the-scenes look at how Postman took on data collaboration and chaos
“As Postman grew, it became difficult for everyone to understand and, more importantly, trust our data.
We had been creating dashboards and visualizations based on requests from across the company: whatever people needed, we designed. However, the metrics on those dashboards often overlapped, so we had inadvertently created different versions of the same metrics. When it wasn’t clear how each metric was different, we lost people’s trust…
The data team’s Slack channel was filling up with questions from other teams asking us things like ‘Where is this data?’ and ‘What data should I use?’ Our experienced data analysts spent hours each week fielding these questions from new hires. Of course, this was frustrating for all involved.”
When Prudhvi Vasa joined Postman as an Analytics Leader, their data was a mystery. It took him a while to learn how to smoothly navigate through the data stack — which tables had different versions of the same data, different filters, sync issues, etc.
When you have a small team, it may be fine for people to learn and save data knowledge in their heads. But when hundreds of team members across four continents are trying to use data, this just doesn’t cut it.
That’s why Postman embarked on a project to democratize their data, spread context, increase trust, and save the data team’s time. They started with small hacks on Confluence and Google Sheets, and worked their way up to bigger and better challenges as they learned what worked for them.
Democratizing a large-scale data system is a big challenge, and, as Postman said, they’re definitely not the only company trying to crack it. That’s why they decided to share their experience last year about what worked, what didn’t, and what they’ve learned so far.
A few quick takeaways:
🌍 Async context is key to remote work
Collaboration has always been an issue, but it’s even more important today. Working together with full context, while async and with minimal back-and-forth, is crucial for companies whose employees are remote and stretched across multiple time zones or even continents.
🔒 Context is the foundation of data trust
Collaboration and context aren’t particularly “fun”, but they’re the foundation of all data work. When data assets were duplicated or metrics were contradictory, people lost trust in data. As the saying goes: building trust is hard, but losing it is easy — it just takes one mistake.
🔀 Lineage is an important part of data collaboration
Having a single source of truth for data helps data teams and the company as a whole collaborate while growing. Every time you change something or add something new, it’s important to check how it will affect everything else in the data system. Instead of posting a question on Slack, data lineage can show where data is coming from and how it will be changed.
📈 Pay attention to scale
When designing for better collaboration, it’s important to think about the long term. As Postman said, their early solutions were solid ideas but failed to keep up over time. Consider how a solution or data product will scale as your company and data scale. Will it keep up if your data grows by 100x in a year? Can it handle lots of users, all with different needs and access levels?
Read Postman’s blog for the full story. ➡️
🤝 Metadata in action: Collaborate where you work
ICYMI: We recently introduced a new section to Metadata Weekly — metadata in action, highlighting real use cases of active metadata. Today we’re focusing on how it can help solve this collaboration challenge.
Harvard Business Review research showed that people switch between different apps and websites nearly 1,200 times each day. No one wants to remember another login and go into yet another tool, only to search through a ton of information for the answers they need. How can we reduce service requests while increasing the data and business teams’ productivity in the process?
At a leading esports platform, analysts were spending nearly half of their bandwidth addressing ad-hoc data requests. Now, with Atlan’s Chrome extension, they empowered business users to self-serve the context they need, saving each analyst 70 hours per month and increasing productivity by 43%.
Learn how Atlan embeds data and context in everyday applications like BI tools and communication platforms to increase context and save time.
📚 More from my reading list
What’s next for data engineering in 2023? 7 predictions by Barr Moses
Data systems tend towards production by Ian Macomber
Understanding your data: the people behind it by Emilie Schario
On measuring: company metrics, team KPIs, and OKRs by Erica Louie
Why your team needs a weekly metrics review by Julie Zhuo
How GoDaddy built a data mesh to decentralize data ownership by Ankit Jhalaria, Harsh Vardhan Singh Gaur, and Kyle Tedeschi
See you next week!
P.S. Liked reading this edition of the newsletter? I would love it if you could take a moment and share it with your friends on social.