Digital Experience Mesh overview

To understand the limitations and challenges, it’s important to define the most common web system architectures. These architectures include monolithic systems, distributed monolithic systems (a cluster of monoliths), and distributed MACH architecture. Each architecture will be described by using example diagrams of CMS deployments.

The first and probably most common web system architecture is a single monolithic application shared by two types of users: contributors (for example content authors) and visitors. This is how most CMS systems are built; if you think about a monolithic application, this is likely the model example. It’s simple to set up, monitor, and maintain because it uses a single database and a single application running on one server. All the features are developed within a single codebase, so functions like authorization, authentication, digital asset management, full-text search, tagging, and publishing are already available. If integration is needed, it is done within the same codebase. However, the simplicity and completeness of this architecture become problematic because system complexity grows or when it must scale to handle a large number of publications or high traffic.

Over the past two decades, vendors have tried to address these issues with monolithic applications. The diagram above shows a common architecture used by an enterprise CMS (often called Web Content Management (WCM) or Digital Experience Platforms (DXP)). The key changes to the original CMS architecture are:

This architecture addressed some of the issues and was a step in the right direction. Enterprise CMS systems can handle more load due to the use of multiple rendering instances and are more secure because public traffic cannot directly access backend instances. However, many problems are inherited from traditional monolithic architectures, and additional operational and maintenance challenges make this option suitable only for enterprise customers.

A relatively new alternative to the monolithic web system is the MACH architecture. It limits the role of systems like CMS, PIM, or e-commerce to that of data providers. In most cases, frontend technologies are responsible for rendering and logic is moved to the microservices layer. All is controlled in a cloud-native environments.

By definition, MACH architecture solves the following issues:

With its benefits, MACH can introduce additional challenges:

There is a group of challenges common to most web systems. Not all of these issues occur simultaneously, but they are recurring problems. Modern architecture should address all of them.

One of the main challenges is that web systems do not operate in isolation. They rely on data and services from other systems, all evolving at different rates, with protocols and formats changing over time. These systems and services have varying performance and scaling characteristics, they might be deployed in different geographic locations and accessed through different networks. Each one can have distinct security policies, yet they must work together to produce a consistent, reliable, and high-performing web system.

To address the challenges unresolved by current solutions, we must start exploring alternative approaches. Instead of building yet another architecture around headless systems with a serverless computing layer and modern frontend frameworks, we should first ask: what’s wrong with the current approaches? The key observation is that we must decouple source systems from the processing and delivery layers. Each of these layers has different responsibilities and characteristics.

With an understanding of these three layers, it’s time to move on to the communication between them. It stands to reason that the highly available, distributed delivery layer must not make direct requests to source systems, which might be legacy, relatively slow, and not scalable. Furthermore, given the challenges engineers face with various caching mechanisms, the system should not be overly dependent on the caching layer. This could lead to a system that is unstable when the cache is flushed, or outdated when the cache contains stale data. To build a system that contains up-to-date data and does not depend on cache or backend systems' availability and performance, we must change the communication model. In that model the Digital Experience Mesh must be informed of any relevant changes occurring in source systems. These changes, such as page publications, product description updates, or user profile changes, must be sent in the form of events. It then executes a set of actions to reflect the state of the backend systems in the delivery services. With this model, the architecture always generates resources and updates the service state before a visitor makes a request.

Moreover, since we are defining a composable architecture, it naturally must be message-oriented.

Not all resources can be pre-generated and used unchanged by the delivery service. A good example is search - we can’t generate a response to a search query if we don’t know what the request will be. In such cases, the system should provide a search service at the delivery layer that uses pre-processed data to create local state and generate a response by using an optimized index. This allows the delivery services to respond fast and scale efficiently.

At this point it’s important to emphasize that the Digital Experience Mesh is:

These three principles enable us to decouple site performance and availability from source systems. They allow us to process large amounts of data in real time and deliver both static and dynamic resources globally through a high-performance, scalable system. Further details will be covered in the following sections.

The first group of characteristics relates to web system performance, scalability, and availability. These characteristics vary across different layers. For example, performance can be measured by using different metrics for a DAM source system, an image processing service, and the web servers hosting the images.

For a DAM system, performance is measured by how fast an image can be uploaded and stored in the database. In the processing layer, performance is evaluated by the end-to-end latency of handling conversion requests and distributing the converted images to web servers. For web servers, performance is measured by the latency in accessing publicly available files. Scalability follows a similar pattern. For a DAM system, it refers to the number of images that can be stored and managed. In the processing layer, it refers to the throughput, which includes the number of conversions and the amount of data transmitted. For web servers, scalability is measured by the number of visitors that can access the assets simultaneously.

The following are the first required characteristics for the system:

Another group of characteristics is derived from the Reactive Systems architectural principles. Although the Digital Experience Mesh implementation does not explicitly require being a Reactive System, applying its principles might be the best approach for achieving a well-designed system architecture.

Reactive Systems are:

The principles of reactive systems are important at the application level. They can be implemented through modern reactive programming libraries, non-blocking I/O, patterns like the event loop, and asynchronous processing. Additionally, these principles help define contracts between components. Asynchronous, message-driven communication ensures loose coupling between different parts of the system, improving both scalability and resilience, while also contributing to a clean and transparent architecture.

The final group of characteristics is not directly related to scalability, performance, or availability, nor are they derived from the principles of Reactive Systems. However, these issues have been recognized challenges in traditional monolithic web systems and remain unresolved - or have even worsened - in modern architectures like MACH.

To address these challenges, systems must provide a high level of:

To start building event-driven systems, in contrast to the more popular request-driven (polling) systems, a shift in mindset is required. For example, a common use of event-driven design is how a webpage is rendered in the browser. Each link on a page has onClick handlers that instruct the browser what to do when a visitor clicks an anchor. If we think about it, there’s no requirement to periodically query the DOM to check the status of components, such as those clicked and those not clicked. We’re simply interested in reacting to user actions.

The same approach applies when designing interactions between systems or components in a broader web system. For tasks like rendering a page or optimizing an image from a DAM system, querying the source system and running application logic based on its current state is unnecessary. Instead, the focus should be on events—such as a new page version being published or a new image rendition being activated. Rather than repeatedly checking the source system, we can respond to these events with a series of actions. These actions can trigger additional workflows until the final output, like a rendered page or optimized image, is stored at its destination. The result of each action can trigger parallel actions or workflows, for example indexing the page in a search engine, updating listings, or generating a sitemap.

The consequence of understanding the importance of event-driven architecture is that the system can:

There are two key aspects of a distributed web system:

Digital Experience Mesh distribution by layer:

The goal is to push data closer to the user geographically, minimizing the distance and time required for the client to pull data. This is achieved by composing the system of small, independent services rather than a single monolithic application. Each service only receives the data relevant to its needs. The service declares the types of changes it will track and is responsible for maintaining its own state. This ensures that the amount of data to replicate is relatively small and optimized by the receiver.

As mentioned earlier, to enable data distribution and event-driven use cases, the Digital Experience Mesh must be able to maintain state.

Each service is responsible for maintaining its own state. This means that the service must subscribe to events of a given type, convert the data, and store it locally for future use. New service instances must be able to recover their state based on historical events, which requires external data storage used only during service initialization.

The fact that each service maintains its own state makes the Digital Experience Mesh horizontally scalable with fine granularity. It is possible to scale the system on a per-service basis.

The combination of the fundamental design decisions mentioned in this section leads to several consequences. To provide more context, let’s explore some examples.

The consequences mentioned above are just examples and not an exhaustive list. However, they help illustrate how shifting from a traditional request-reply model to a modern push-based architecture fundamentally changes the system’s characteristics.

Source Systems represent the origin of data ingested into the Mesh. The Mesh definition must explicitly specify sources that are permitted to send messages to specific channels. Based on this definition, the Ingestion API will manage the authorization and authentication processes. In addition, the Ingestion API is responsible for schema validation, ensuring that only valid messages are accepted.

The example above shows the CMS that can publish events to the pages, assets, and templates channels, but it cannot send updates to the products or prices channels. The CMS must also ensure that the message payload is valid, because invalid messages will be rejected by the Ingestion API.

Other common source systems include PIM systems, e-commerce systems, DAM, headless CMS, legacy APIs, or bash scripts executed during CI/CD pipeline execution. The last example is particularly interesting because it allows changes to be deployed directly from the Git repository. Another relevant source in the context of this architecture is user actions. For example, we can track user activity to generate real-time recommendations that use machine learning algorithms.

Processing Services are services that read from and write to channels. These services contain functions that can be composed together to build a data pipeline. Multiple instances of a single service can operate at the same time, with messages being equally distributed between all instances. If a service is stateful, meaning it maintains a state built from historical events, the framework must enforce synchronization between service instances. An instance must ensure that all historical data has been read before it can start processing incoming messages. New instances of a service must automatically initialize their state before being enabled in the Mesh.

If a service cannot process a message within a given time, the framework can automatically reassign the message to a different instance, ensuring the message is eventually processed. Services must also reject outdated messages.

An example of a processing service might be a Sitemap Generator that listens for page publication events, generates a sitemap, and sends it to a web server through the resources channel. Another example is a Rendering Engine that listens for data and template publications and generates a page publication event in response. Note that the Rendering Engine might trigger the Sitemap Generator, resulting in the following pipeline:

In this example, the Rendering Engine uses multiple instances of the service, and each instance should maintain its own state built from historical Data and Template publications. This is necessary because rendering a page requires both data and templates, and each service instance can process only one event at a time. The Sitemap Generator also uses a local state, but it doesn’t require synchronization since there is only one instance of the service. Therefore, only the Rendering Engine requires synchronization.

Delivery Services are services that read from the final channel in the mesh, and cannot write to any channels. These services contain functions that send the final events processed by the previous layer to a sink. The sink is the terminal step of a pipeline and represents the final destination of the data, such as a search index, volume, or database. Multiple instances of a single service can operate simultaneously, with all messages delivered to each instance. This differs from the semantics of processing services, because in this case, the goal is to increase the throughput of the delivery layer. For example, when adding a new instance of a web server, it must consume all the pages, even if other web server instances have already processed them. New service instances must automatically initialize their state before being activated in the Mesh.

Each service instance is a complete copy of both data and functionality. This allows the system to scale horizontally in a linear fashion. Moreover, it increases redundancy of services and data, improving both availability and resilience. As a result, there is no single point of failure in the system.

Example delivery services include:

A Mesh is a complete set of sources definitions and pipelines deployed in an environment to orchestrate multiple sources of data and deliver the results to various channels. The Mesh contains business logic and executes a series of actions.

To simplify the design, building, testing, and maintenance of meshes, it is important to recognize that pipelines are often independent. Breaking the Mesh into smaller pipelines and composing them together later is an effective technique for managing complex setups.

In the example above, the image optimization pipeline is separate from the page rendering pipeline. As a result, it can be developed, tested, and deployed at a different pace, or even by a different team.

Microservices are a key architectural pattern that aligns well with the principles of the Digital Experience Mesh. By breaking the system into smaller, independent services, microservices enable flexibility, scalability, and maintainability, which are critical for modern web platforms. While it’s still theoretically possible to use alternative technologies like Serverless Functions or Modular Monoliths, microservices seem to offer the right balance between control and modularity.

Event streaming is another crucial technology that supports the event-driven nature of the Digital Experience Mesh. Event streaming platforms, like Apache Pulsar or Kafka, are designed to process real-time data streams with high throughput and low latency. Event streaming platforms are designed to be fault-tolerant, scalable, and provide strong message delivery guarantees such as at-least-once or exactly-once delivery, ensuring that important data is never lost.

If microservices together function like a brain, then event streaming plays the role of the spinal cord carrying the messages. Both enabling the system to be responsive, scalable, and efficient in processing real-time data pipelines.

While other messaging technologies can be used to build channel abstractions, only event streaming can achieve the scale and performance required for modern web systems.

A log-based architecture provides an abstraction for data storage. There are multiple database types, including relational, document, key-value, and object databases. However, for working with event streams, log-based structures and the key-value stores built on top of them are the most efficient.

In some scenarios, especially when data must be accessed by query or key, it is necessary to build additional abstractions on top of logs. However, logs remain the default primitive to consider when implementing the Digital Experience Mesh architecture.

A container orchestration engine is preferred for implementing the Digital Experience Mesh architecture because it automates the deployment, scaling, and management of containerized services. DXM relies on microservices and distributed components that must run reliably across multiple environments. Container orchestration engines, such as a Kubernetes resolves most of the challenges of these setups.

A Content Delivery Network (CDN) is widely used because it improves website performance by distributing content across various geographic locations, reducing latency for users. It enhances scalability, allowing websites to handle traffic spikes without performance degradation. Additionally, a CDN improves reliability and security by providing redundant server locations and protection against online threats like DDoS attacks.

However, there are certain limitations of a CDN:

The Digital Experience Mesh (DXM) can complement a CDN by distributing both services and data. It enables the deployment of services closer to users, allowing data to be served in the static and dynamic form. Unlike a request-reply cache, DXM utilizes a push-based architecture, offering complete control over the availability of data at each location. Resources are pushed to edge locations ahead of time, making them available before the user requests them. This results in performance gains from the first request, meaning there are no cold-cache or cache invalidation issues. Consequently, the site’s availability is no longer dependent on the origin server’s availability.

In traditional web systems, integrations — including data orchestration — were often handled through custom implementations within the system’s codebase. Most CMS and e-commerce platforms were able to communicate with external services and process data. However, with the increasing complexity of integrations, especially in enterprise organizations, more sophisticated solutions like Enterprise Service Buses (ESB) were employed. These solutions reduced the number of point-to-point integrations and offloaded systems like CMS from being the integration hubs.

The situation became more critical with the shift to headless and cloud technologies, because many headless systems lack built-in integration capabilities. The challenge of integrating multiple headless and legacy systems, including orchestrating data, became so widespread that a new software category emerged: Digital Experience Orchestration. This category focuses on managing the orchestration of data and services in relation to web systems. The Digital Experience Mesh, which is a middleware solution, is perfectly positioned to take on this role.

ESB products are not inherently designed to work with modern web systems. As a result, they are often monolithic applications, centrally deployed in an organization’s data center, which creates a single point of failure and limits both performance and scalability. ESB systems also lack the real-time performance found in event-streaming platforms, which can introduce additional processing and request latencies. Additionally, ESB systems come with complexity and overhead, whereas lightweight, decentralized approaches that favor agility and faster integration cycles are better suited for web systems.

The Digital Experience Mesh, a distributed, microservice-based system driven by events, is an excellent option for building integrations. Its powerful integration capabilities, along with near real-time processing and high throughput, can deliver significantly better results for the needs of modern web systems.

It is a common technique to pre-generate resources before making them available for consumption. A common example is Static Site Generators (SSG), which are used to build a static version of a site before deploying it online. Incremental Static Regeneration (ISR), a technique developed by the Next.js team, is more sophisticated. Instead of generating the entire site at once, it allows for per-page generation based on specific events. ISR retains the benefits of SSG but scales efficiently to millions of pages.

Moreover, similar techniques are used when working with data. Data pipelines are often built to perform data transformation and optimization while moving data from one place to another. Preprocessing data allows operations to be performed on an optimized, often smaller dataset, resulting in faster querying and processing.

SSG usage is often limited to relatively small sites due to the time required to generate the entire site. While ISR partially solves this problem, it remains a basic technique that cannot manage custom relationships or respond to events beyond data changes. Additionally, there is a lack of standardized methods for building data pipelines for web systems that can work with both SSG and ISR, without requiring time-consuming and resource-intensive development.

The Digital Experience Mesh has the capability to pre-generate resources such as pages, fragments, or data. Pipelines for generating these resources can be triggered by various events from multiple systems, such as template updates, data changes, or content updates. Data pipelines can be used to efficiently optimize event-driven data for feeding services, enabling the system to deliver more than just statically generated resources. By utilizing optimized data, the system maintains high performance and scalability.

Real-time data processing is becoming the standard for handling data. Unlike batch processing, real-time updates offer systems higher-quality and more consistent data. However, web systems have traditionally been built around monolithic applications, relying on a single database to store all system information. This created multiple problems, especially because databases used by systems like PIM, CMS, or e-commerce platforms were never optimized for handling large datasets. What’s more these systems were not designed to move and process data efficiently. It’s common for organizations to tolerate outdated information available on their websites, such as product details or availability, which can only be updated once a day. Batch price updates can lead to significant financial losses if the company is unable to adjust its pricing strategies in real time.

As a result, more companies are adopting streaming platforms like Apache Kafka and Apache Pulsar. Real-time data processing frameworks and engines, such as Apache Flink, are also becoming increasingly popular.

Designing and developing custom data platforms based on the mentioned technologies is expensive. There is a shortage of specialists in the market with the skills to effectively design, implement, and manage these systems. The time and resources required to implement and maintain such solutions often exceed the capabilities of medium-sized organizations.

The Digital Experience Mesh leverages event streaming to enable stateful computations over data streams and is designed specifically for web-related use cases. This approach achieves similar outcomes without the complexity of managing low-level event streaming platforms. The architecture abstracts data sources, processing, and delivery, while providing the necessary guarantees to meet most web-related requirements.

Monolithic web systems often tend to be neither high-performing nor scalable. As a result, teams frequently try to offload unnecessary data from their repositories. Search indexes like Elasticsearch have become a standard way to create an additional data layer. This solution becomes even more popular when multiple systems contribute to the repository. Search indexes are fast, easy to use, and can be directly queried by frontend technologies to build responsive interfaces for web systems.

However, when a web system grows, search indexes - never designed to serve the role of a unified data layer for web systems - begin to hit their limits. It might be easy to start with, but working with a storage system that holds unstructured documents, lacks relationships, and has no clear data ownership can become challenging. Search indexes often slow down when the incoming data is not optimized, leading to the need for replacing multiple point-to-point integrations with standardized pipelines. Such a solution becomes expensive to build and operate, while still failing to address issues like geo-replication. Additionally, search index APIs are not designed to handle different types of resources, such as XML, JSON, or HTML files, without additional processing.

The Digital Experience Mesh can address the issues associated with search indexes. The architecture is designed to connect multiple Source Systems and process data before sending it to the Delivery Layer, which could be a search engine, GraphQL, or a web server. The system includes built-in authorization and authentication for the Source Systems, ensuring clear ownership and control over the data. A schema, enforced and validated during ingestion, guarantees the accepted data is consistent and of high quality. It also includes built-in geo-replication capabilities and can serve any type of resources, not just searchable documents.

The Digital Experience Mesh addresses similar challenges to a search index. It can fulfill the role of an external data repository but offers a more comprehensive solution for building and managing the Unified Data Layer.