Data mesh

Technology leaders understand that data has become the lifeblood of their company and that they have the primary responsibility to provide the business with easy, secure and reliable access to that data.

The challenge is that the data is typically stored in dozens or even hundreds of siloes scattered across the enterprise — in legacy on-premise solutions, in the cloud, across departments, business units, lines of business and so on. By some estimates, data scientists spend as much as 80 percent of their time looking for and obtaining access to the data they need, instead of spending that valuable time building models and delivering insights. IT, too, spends much of its time in one-off projects providing that ad-hoc access.

In recent years, many companies have implemented data lake technology to help solve the problem. The concept is simple and compelling: break down those siloes by pulling all that data, both structured and unstructured into a single, centralized platform.

A data lake also provides what many see as the holy grail of data: a single source of truth. Each division in a large enterprise, for example, might have a different definition of revenue or a customer’s worth to the business. Centralizing data eliminates such inconsistencies.

Data scientists spend as much as 80% of their time looking for and obtaining access to data¹

Problem solved?

While it may sound like an ideal solution, that isn’t always the case. The siloes may be gone and the data now centralized, but more often than not, data consumers still require IT’s help to access it. As with anything that attempts to centralize large and complex things, scalability and maintainability are significant challenges.

More importantly, data quality can suffer. The engineers responsible for it will be intimately familiar with the lake’s plumbing — the technologies and services used to centralize the data — but they have little or no experience with or functional understanding of the data itself. When data enters the lake, ownership of or responsibility for it is severed. It loses domain-specific knowledge about its origin — what it means or why it’s needed. Is the data the most up-to-date version? What does a certain field mean? Data provenance becomes more challenging.

Further, data scientists have come to realize that creating a “single source of truth” can often do more harm than good. What had been seen as undesirable inconsistencies across different data sources may actually contain valuable insights — there’s a reason different divisions define a customer’s worth differently, for example. Eliminating those differences can eliminate the insights.

As many have discovered, the result can be more of a messy “data swamp” than a lake, into which they pour increasing amounts of money without realizing any of the promised returns.

A new way of thinking

The disappointment with traditional data lake architecture has spurred technology leaders to seek alternate architectures to build their data lakes. The most promising of these comes from a lesson learned in our journey to the cloud. When the cloud first appeared, most companies treated it as a replacement for their expensive and difficult-to-maintain data centers. The “lift and shift” of on-premise applications to the cloud was the only goal.

Then technology leaders had an epiphany. They realized that the cloud held far more value. Unlocking that value would require a new way of thinking about software development and deployment. It would require a new operating model and new technologies designed to support it, such as Agile development, containerization and continuous integration/continuous deployment.

Instead of building big, complex, monolithic applications, we now break down those monoliths into much smaller, discreet microservices, with each providing a narrowly focused set of capabilities. These individual, stand-alone microservices can communicate with each other via standardized application programming interfaces (APIs), and so can be dynamically combined in multiple ways to form complete, robust solutions.

While those big monolithic applications required large teams of software developers to build, microservices use small, independent teams comprising both technology and business professionals with domain-specific expertise related to each microservice’s function. This enables each team to treat its microservice as their product, and so team members collaborate to continually improve it. Because each team can deploy new features and enhancements independent of all other teams, they’re able to innovate more rapidly with greater agility.

While this radical new approach has firmly taken hold in software development, not much had changed with how we approach data — that is, until 2019, when Zhamak Dehghani published her now-famous white paper arguing that the same approach should be applied to data — a concept she called data mesh.

In 2019, Zhamak Dehghani published her now-famous white paper arguing that the same approach should be applied to data — a concept she called data mesh