The explosion of data in recent years has led to the emergence of new approaches to data management, and putting Data Mesh into practice is one of the most promising responses organizations can have. As we already pointed out in our article What is a Data Mesh?, Data Mesh is a distributed data management architecture that enables organizations to integrate, access and share data across multiple systems, applications and platforms seamlessly. Data Mesh at its core, is founded on decentralization and distribution of data responsibility to people who are closest to data.
What impact does Data Mesh have on (agile) organizations?
The Data Mesh approach is an attempt to lift the isolated world of data into the rest of the organization, thereby changing the roles and responsibilities of the teams. This, in turn, means that people are faced with the importance of data who previously had no relation to it and maybe couldn’t even see the benefit. Organizationally, this gap needs to be closed so that the importance of data is firmly embedded in people and culture.
Data Mesh is an approach to designing data architectures that emphasizes data as a product and enables cross-functional teams to take ownership of specific domains. Agile Frameworks, like Scrum or Scaled Agile Framework (SAFe), are forcing a software development process that focuses on delivering working software in short iterations. But how does the approach of Data Mesh affect the agile team set up and what roles must be part for making Data Mesh to an efficient operating model?
This article focusses on the social and human aspects which are affected and compares traditional team set-ups with the new approach of Data Mesh. Organizational re-thinking and design decisions affect
- roles and team setups
- capabilities
- education and personal development
- recruiting.
With implementing Data Mesh, new roles, and responsibilities as well as new development pathways must be implemented to generate talent, skills and the necessary mindset. Change management includes training of employees and making them see the benefits and chances coming with the new approach as well as recruit the right team members.
How does a “traditional” agile team look like?
An agile team is a cross-functional group of people that is organized to work collaboratively in order to deliver a product increment iteratively. However, working in agile teams means mostly with the methods of Scrum or Kanban, which both have clearly defined roles for making those approaches as effective as possible. Let’s have a short look at the roles within an agile team:
Product Owner: The Product Owner represents the needs of stakeholders and is responsible for managing the product backlog. For realizing the maximum value for the organization, a product owner is a visioner who connects stakeholder requirements with developing strategies for making the product as valuable as possible.
Scrum Master: The Scrum Master is a key role in the Scrum framework and mainly responsible for the Scrum Team’s effectiveness. The Scrum Master, as a servant-leader, is responsible for ensuring that the Scrum team adheres to the principles and practices of Scrum, and for facilitating the team’s progress toward its goals.
Development Team: The Development Team is the central part of developing a product increment within each sprint. Beyond that, the team acts as accountable professionals who embrace their practices by mentoring, teaching, and coaching.
When implementing Data Mesh, data experts such as data engineers, data analysts or data scientists move from central data teams to the domain teams. If these domains are working in an agile framework such as Scrum, they become part of the development team.
What new roles come with Data Mesh in practice?
When fully embraced Data Mesh comes with the following roles as an example:
Data Product Owner
Data Mesh means treating data as a product, so like any other product it is a service to fulfill its customer’s needs. Data Product Owners work within a particular domain and are accountable for the domain’s data products. There can be multiple data product owners in one domain or just one – depending on the complexity of the domain. Data product owners facilitate making decisions around the vision and the roadmap for the data products which are constantly aligned with their stakeholder and data customers’ expectations. Data consumers will use data catalogues to find requested data.
Domain Data Product Developer
A Domain Data Product Developer creates, delivers, maintains and evolves domain-specific data that is modelled as a product. The Data Product Developer owns the code, logic and schema of their domain data and provides necessary APIs for consuming it. The Domain Data Product Developer needs additional skillset for mapping the context data within the domain to the data product context.
Data Governance Office
The principle of federated governance leads also to a shift of data governance roles from a central instance to integrated responsibilities within the domains itselves. This can be quite a shift for the respective team. When applying Data Mesh governance becomes part of the domain itself guided by overall standards. Those Data Governance Offices can be built out of the domains’ Data Product Owners together with governance experts.
Data Platform Product Owner
The Platform Owner is responsible for providing the data platform to its decentral organized users. The Platform owner can be supported by a team – the so-called platform team splitting specific tasks around the data platform. Their goal is to facilitate the work and user experience on the data platform and assure its internal customer satisfaction (e.g., by conducting surveys for platform feedback).
What impact does Data Mesh have on the Scrum framework?
Data Mesh is the organizational, sociotechnical approach towards data management, whereas the Scrum framework defines the agile project management. Data Mesh in practice can have impact on agile teams in a few ways:
- Prioritization and backlog management: As data becomes a product in a Data Mesh architecture, Scrum teams may need to prioritize data-related tasks alongside other product features. This may require adjusting the backlog and sprint planning process to accommodate data-related work.
- Data quality and governance: Data Mesh emphasizes the importance of data quality and governance, which can impact agile teams that work with data. These teams may need to spend more time ensuring that their domain data is accurate, complete, and consistent.
- Technical infrastructure: Data Mesh also requires a technical infrastructure that supports distributed data domains, which may require changes to the way agile teams work with data. Teams may need to adopt new tools, technologies, and processes to work effectively within a Data Mesh architecture.
- Data ownership and collaboration: In a Data Mesh architecture, ownership and responsibility for specific data domains are distributed across teams. This means that agile teams may need to collaborate more closely with other teams to ensure that data is shared effectively and that there are no conflicts between different data domains.
Impact of Data Mesh on cross-functional collaboration
Data Mesh represents another step in the evolution of an organizational design. We recommend considering a transformation program and careful change management. How this is created exactly depends on how your organization operates today.
A Data Mesh approach enables domain experts to take ownership of their data and become responsible for its quality, governance and delivery. This empowers them to make better decisions and develop more innovative solutions leading to improved job satisfaction and career growth. Domains teams can quickly adapt to new requirements, build and deploy new data products and make changes as needed without being hampered by traditional centralized governance structures.
By breaking down data silos and encouraging cross-functional collaboration, Data Mesh facilitates better communication and knowledge sharing between different job roles, leading to improved productivity, higher quality outcomes and better overall results. But let’s face it: When introducing Data Mesh in an organization this can be quite a change for employees. When HelloFresh switched from a central data team to a decentral organization, they introduced data chapters as a bracket for the data specialists that did not work together anymore. Employees with the same or similar capabilities can work together in so-called chapters. They promote team collaboration and innovation. At HelloFresh two Data Chapters work on topics like talent acquisition, career paths, tools and methods, best practices, and knowledge sharing.
Data Mesh in practice depends on characteristics and needs of each individual organization
There is no all-in-one solution when it comes to Data Mesh. It is still an emerging paradigm which is continuously evolving. The organizational set-up, architecture and roles must be tailored to the specific characteristics and needs of each organization and can be slowly implemented, step by step.
Overall, Data Mesh can affect agile teams in various ways, depending on how it is implemented. It may require adjustments to the way teams collaborate, prioritize work, manage backlogs, and ensure data quality and governance. However, with careful planning and communication, Scrum teams can work effectively within a Data Mesh architecture and deliver high-quality software products that incorporate data as first-class products.
Depending on the depth an organization plans to implement Data Mesh, new roles and responsibilities have to be clarified and defined. Management and employees of the whole company must understand the concept and their new functions. Data Mesh is an architecture that organizations should consider if they want to improve their data management capabilities and realize the full value of their data assets.