StreamX Mesh YAML file

The StreamX Mesh YAML file is used to configure various aspects of a StreamX Mesh, such as services, docker images, environment configurations. It’s an entry point for starting StreamX Mesh.

Usage

To run StreamX Mesh, using a Mesh YAML file as its configuration, you need to use StreamX CLI.

With StreamX CLI, executing streamx run command lets you run a StreamX Mesh locally:

streamx run -f mesh.yml

Technical reference

The StreamX Mesh YAML file consists of three main blocks:

  1. Docker Registry and Tenant configuration

  2. Configuration of Processing Services

  3. Configuration of Delivery Services

Here is the sample StreamX Mesh file:

defaultRegistry: sample-docker-registry/streamx-images
defaultImageTag: 1.0.0
tenant: streamx
processing:
  processing-service-name:
    image: processing-service-name
    incoming:
      incoming-channel-1-name:
        topic: inboxes/topic-for-incoming-channel-1
      incoming-channel-2-name:
        topic: inboxes/topic-for-incoming-channel-2
    outgoing:
      outgoing-channel-1-name:
        topic: outboxes/topic-for-outgoing-channel-1
      outgoing-channel-2-name:
        topic: outboxes/topic-for-outgoing-channel-2
    environment:
      PROCESSING_SERVICE_PROPERTY_1_NAME: value-1
      PROCESSING_SERVICE_PROPERTY_2_NAME: value-2
delivery:
  delivery-service-name:
    image: delivery-service-name
    incoming:
      delivery-channel-1-name:
        topic: outboxes/topic-for-outgoing-channel-1
      delivery-channel-2-name:
        topic: outboxes/topic-for-outgoing-channel-2
    environment:
      DELIVERY_SERVICE_PROPERTY_1_NAME: value-1
      DELIVERY_SERVICE_PROPERTY_2_NAME: value-2

Docker Registry configuration

The StreamX Mesh file declares a set of services. Each of them has a mandatory image field, that corresponds to a Docker image that delivers that service. It is possible to define a default Docker registry and a default image tag to be shared by all or some of the services.

Example:

defaultRegistry: sample-docker-registry/streamx-images
defaultImageTag: 1.0.0

You can provide a publicly available Docker registry, as well as a custom one (including your own).

These two fields are optional. If not specified, each image declaration in the services section must contain explicit Docker registry and image tag information.

Refer to canonical image declaration for example.

Tenant

tenant: streamx

The tenant setting is used to distinguish different tenants in StreamX.

A tenant is used in the underlying messaging system to set up dedicated channels and topics.

Processing Services

StreamX Processing Service is a category of a microservice that facilitates the processing of data pipelines. The Processing Service processes data originally published to StreamX by a Rest Ingestion Service. Note that Rest Ingestion Service is an internal service in StreamX, and therefore is not part of the StreamX Mesh configuration file.

You can find exhaustive information about Processing Services in StreamX Data Processing Service reference.

Processing Service Name

StreamX Mesh file can include configurations for one or more microservices. Each service is assigned a name, which can be any name you prefer. It is recommended to use names that are clear and descriptive, as StreamX will generate a Docker container for each service, using the service name as the container name.

In the example Mesh file above, processing-service-name is the name of the Processing Service. The sample Mesh contains only one Processing Service.

Docker Image for the Processing Service

The image setting provides information about the location of the service’s Docker image.

Short form of specifying the image name is:

image: processing-service-name

The image will be searched in defaultRegistry, with defaultImageTag, if those settings are specified at the top of the current StreamX Mesh file.

If no defaults are given, or if some images are to be pulled from different repositories, the canonical form should be used for the image setting, such as:

image: sample-docker-registry/streamx-images/processing-service-name:1.0.0

Channels

Every Processing Service must specify a list of incoming and outgoing channels. Both types of channels - incoming and outgoing - must be declared, because the Processing Services, connected by these channels, create a data pipeline (the flow of data events within the Mesh) that starts at inboxes topics populated by the REST Ingestion Service and ends at outboxes topics consumed by the Delivery Services. Note: if a Processing Service does not use both incoming and outgoing channels, it may cause invalid default configuration setups and other issues.

The channels and topics are configured using the syntax presented in the sample StreamX Mesh YAML file. Below you can find the reference of the configuration fields:

processing:
  processing-service-name:
    image: processing-service-name
    incoming: (1)
      incoming-channel-1-name: (2)
        topic: inboxes/topic-for-incoming-channel-1 (3)
      incoming-channel-2-name: (4)
        topic: inboxes/topic-for-incoming-channel-2
    outgoing: (5)
      outgoing-channel-1-name: (6)
        topic: outboxes/topic-for-outgoing-channel-1 (7)
      outgoing-channel-2-name: (8)
        topic: outboxes/topic-for-outgoing-channel-2
1 incoming is a fixed token, that allows StreamX to distinguish incoming channels from outgoing channels, while parsing the Mesh file.
2 incoming-channel-1-name is the actual name of the incoming channel.
3 inboxes/topic-for-incoming-channel-1 is the incoming channel topic, where the Service subscribes to incoming messages. By default, StreamX uses Apache Pulsar as a messaging system under the hood.
4 The example Processing Service declares two incoming channels, the second one is incoming-channel-2-name. There’s no explicit limit on the number of channels. It depends on the actual needs of a service.
5 outgoing is a fixed token, that allows StreamX to distinguish outgoing channels from incoming channels, while parsing the Mesh file.
6 outgoing-channel-1-name is the actual name of the outgoing channel.
7 outboxes/topic-for-outgoing-channel-1 is the outgoing channel topic, where the Service publishes its results. The results may be further processed by another Processing Service, or consumed by a Delivery Service, by configuring them to listen on the same channel and topic.
8 The example Processing Service declares two outgoing channels, the second one is outgoing-channel-2-name. As in the case of incoming channels - there’s no explicit limit on the number of channels. It depends on the actual needs of a service.

More detailed information about Channels and Topics can be found in the Processing Service reference.

Environment variables

Every Service can be extended with a list of environment variables, that will be applied to the Docker Container of a Service while running it.

This is done by providing environment data, which is a set of property name and value pairs. The environment field should be set at the same level as image / incoming / outgoing blocks in the Mesh YAML file.

The syntax for specifying environment properties is as follows:

    environment:
      PROPERTY_1_NAME: value-1
      PROPERTY_2_NAME: value-2

Delivery services

StreamX Delivery Service is a category of a microservice that facilitates the delivery of processed data. You can find exhaustive information about Delivery Services in StreamX Delivery Service reference.

Delivery Service Name

StreamX Mesh file can include configurations for one or more microservices. Each service is assigned a name, which can be any name you prefer. It is recommended to use names that are clear and descriptive, as StreamX will generate a Docker container for each service, using the service name as the container name.

In the example Mesh file above, delivery-service-name is the Delivery Service’s name. The sample Mesh contains only one Delivery Service.

Docker Image for the Delivery Service

This setting shares the concepts of Docker Image for the Processing Service.

Channels

Every Delivery Service specifies a list of incoming channels. Delivery Services are used to deliver processed data to the expected destinations. The StreamX Mesh file allows specifying the destination channel / topic of the data.

The channels and topics are configured using the syntax presented in the sample StreamX Mesh YAML file. Below you can find the reference of the configuration fields:

delivery:
  delivery-service-name:
    image: delivery-service-name
    incoming: (1)
      delivery-channel-1-name: (2)
        topic: outboxes/topic-for-outgoing-channel-1 (3)
      delivery-channel-2-name: (4)
        topic: outboxes/topic-for-outgoing-channel-2
1 incoming is a fixed token, that allows StreamX to distinguish incoming channels from outgoing channels, while parsing the Mesh file.
2 delivery-channel-1-name is the actual name of the incoming channel.
3 outboxes/topic-for-outgoing-channel-1 is the outgoing channel topic for the data producer (i.e. a Processing Service) - and the incoming channel topic for a Delivery Service, where the Service subscribes to incoming messages. By default, StreamX uses Apache Pulsar as a messaging system under the hood.
4 The example Delivery Service declares two incoming channels, the second one is delivery-channel-2-name. There’s no explicit limit on the number of channels. It depends on the actual needs of a service.

Note: In the above examples, outgoing channels of the Processing Service are connected to incoming channels of the Delivery Service by topic names. This approach allows creating the data flow as described in Processing Service channels reference.

More detailed information about Channels and Topics can be found in the Delivery Service reference.

Port

A Delivery Service can also be configured with a port:

    port: [numeric_port]

The port setting should be declared on the same level as image and incoming fields in the StreamX Mesh YAML file. In case when the implementation of your Delivery Service is an HTTP web server - the port allows specifying the port number to communicate with the web server.

Environment variables

This feature works identically for Delivery Services as described in the Environment variables section for Processing Services.

Composite Delivery

The Composite Delivery is a concept referring to the possibility of configuring nested Delivery components in the StreamX Mesh YAML file. This allows you to deliver incoming data to any destination, such as a Web Server, to make the data available as web pages with images. Below, you can see a sample Mesh YAML file for such configuration:

defaultRegistry: sample-docker-registry/streamx-images
defaultImageTag: 1.0.0
tenant: streamx
processing:
  processing-service-name:
    image: processing-service-name
    incoming:
      incoming-channel-name:
        topic: inboxes/topic-for-incoming-channel
    outgoing:
      outgoing-channel-name:
        topic: outboxes/topic-for-outgoing-channel
delivery:
  composite-delivery-service-name:
    image: composite-delivery-service-name
    incoming:
      composite-delivery-channel-name:
        topic: outboxes/topic-for-outgoing-channel
    port: 8088
    environment:
      REPOSITORY_RESOURCE_ROOT_DIRECTORY: /srv/www (1)
    repositoryVolume: /srv/www
    components:
      webserver:
        image: docker.io/library/nginx:1.26.0
        ports:
        - 8089:80 (3)
        repositoryVolume: /usr/share/nginx/html (2)
        volumes:
          - "classpath:overridden-nginx-conf.txt:/etc/nginx/conf.d/default.conf"

This Mesh file contains one Processing Service (processing-service-name) and one Delivery Service (composite-delivery-service-name). The Delivery service contains a nested component, which is a NGINX web server.

To allow sharing files between the Delivery Service and NGINX component, additional settings are applied to the Delivery Service:

1 The environment.REPOSITORY_RESOURCE_ROOT_DIRECTORY and repositoryVolume settings point to a shared directory inside the Docker container for the Composite Delivery Service. Note: the environment variable can have any name, and it will be available inside the Docker Container created for the Delivery Service.
2 Using the webserver.repositoryVolume setting, the shared directory is configured as a Docker Volume associated with a directory in the filesystem of the NGINX webserver.
3 In this example, the NGINX webserver is also configured to expose its 80 port to the outside as port 8089 from the Docker container.

Volumes

It is possible to configure additional files or directories to be available for the nested delivery component, by specifying volumes. In the above example composite StreamX Mesh YAML, overridden NGINX configuration file is configured to overwrite the default /etc/nginx/conf.d/default.conf file. The classpath: prefix tells StreamX to search for the file in the current java classpath that is available to StreamX CLI.

It’s also possible to pass such additional files using relative or absolute paths to files available in your filesystem. Here are the examples:

Example:

        volumes:
          - "/path/to/directory/overridden-nginx-conf.txt:/etc/nginx/conf.d/default.conf" (1)
          - "file1.txt:/file1.txt" (2)
          - "file2.txt" (3)
1 Path to the local file and the destination path in the Docker Container are specified using absolute paths.
2 Path to local file is specified using a relative path, path in the Docker Container must always be specified as an absolute path.
3 It’s also possible to specify only the local file path, as a shorter form of the volume directive. In this example - the file will be available in the Docker Container under /file2.txt path.