For the latest stable version, please use StreamX Guides 1.0.1!

StreamX Mesh YAML file

The StreamX Mesh YAML file is used to configure various aspects of a StreamX Mesh, such as services, container images, environment configurations. It’s an entry point for starting StreamX Mesh.

Usage

To run StreamX Mesh, using a Mesh YAML file for configuration, you must use StreamX CLI.

With StreamX CLI, executing streamx run command lets you run a StreamX Mesh locally:

streamx run -f mesh.yaml

Technical reference

The StreamX Mesh YAML file consists of several main blocks:

  1. Container registry

  2. Configuration of Sources

  3. Configuration of Processing Services

  4. Configuration of Delivery Services

Here is the sample StreamX Mesh file:

defaultRegistry: sample-container-registry/streamx-images
defaultImageTag: 1.0.0

sources:
  source-name:
    outgoing:
      - "topic-for-incoming-channel-1"
      - "topic-for-incoming-channel-2"
processing:
  processing-service-name:
    image: processing-service-name
    incoming:
      incoming-channel-1-name:
        topic: inboxes/topic-for-incoming-channel-1
      incoming-channel-2-name:
        topic: inboxes/topic-for-incoming-channel-2
    outgoing:
      outgoing-channel-1-name:
        topic: outboxes/topic-for-outgoing-channel-1
      outgoing-channel-2-name:
        topic: outboxes/topic-for-outgoing-channel-2
    environment:
      PROCESSING_SERVICE_PROPERTY_1_NAME: "value-1"
      PROCESSING_SERVICE_PROPERTY_2_NAME: "value-2"
delivery:
  delivery-service-name:
    image: delivery-service-name
    incoming:
      delivery-channel-1-name:
        topic: outboxes/topic-for-outgoing-channel-1
      delivery-channel-2-name:
        topic: outboxes/topic-for-outgoing-channel-2
    environment:
      DELIVERY_SERVICE_PROPERTY_1_NAME: "value-1"
      DELIVERY_SERVICE_PROPERTY_2_NAME: "value-2"

Container registry configuration

The StreamX Mesh file declares a set of services. Each of them has a mandatory image field, which corresponds to a container image that delivers that service. It is possible to define a default container registry and a default image tag to be shared by all or some of the services.

Example:

defaultRegistry: sample-container-registry/streamx-images
defaultImageTag: 1.0.0

You can provide a publicly available container registry and a custom one (including your own).

These two fields are optional. If not specified, each image declaration in the services section must contain explicit container registry and image tag information.

Refer to canonical image declaration for example.

Sources

A sources is an optional section that defines the list of the external systems that are allowed to ingest StreamX Mesh. Each source has a list of channels where ingestion is allowed. Channels passed to outgoing section is a topic name from inboxes namespace

Processing Services

StreamX Processing Service is a category of a microservice that facilitates the processing of data pipelines. The Processing Service processes data originally published to StreamX by a Rest Ingestion Service. The Rest Ingestion Service is an internal service in StreamX, and therefore is not part of the StreamX Mesh configuration file.

You can find exhaustive information about Processing Services in StreamX Processing Service reference.

Processing Service name

StreamX Mesh file can include configurations for one or more microservices. Each service is assigned a name, which can be any name you prefer. It is recommended to use names that are clear and descriptive, because StreamX will create a Container for each service, using the service name for the container name.

In the example Mesh file above, processing-service-name is the name of the Processing Service. The sample Mesh contains only one Processing Service.

Container Image for the Processing Service

The image setting provides information about the location of the Service’s Container image.

Short form of specifying the image name is:

image: processing-service-name

The image will be searched in defaultRegistry, with defaultImageTag, if those settings are specified at the top of the current StreamX Mesh file.

If no defaults are given, or if some images are to be pulled from different repositories, the canonical form should be used for the image setting, such as:

image: sample-container-registry/streamx-images/processing-service-name:1.0.0

Channels

Every Processing Service must specify a list of incoming and outgoing channels. Both types of channels - incoming and outgoing - must be declared. That’s because the Processing Services, connected by these channels, create a data pipeline (the flow of data events within the Mesh). The pipeline starts at inboxes topics populated by the REST Ingestion Service and ends at outboxes topics consumed by the Delivery Services. Note: if a Processing Service does not use both incoming and outgoing channels, it might cause invalid default configuration setups and other issues.

The channels and topics are configured by following the syntax presented in the sample StreamX Mesh YAML file. Below you can find the reference of the configuration fields:

processing:
  processing-service-name:
    image: processing-service-name
    incoming: (1)
      incoming-channel-1-name: (2)
        topic: inboxes/topic-for-incoming-channel-1 (3)
      incoming-channel-2-name: (4)
        topic: inboxes/topic-for-incoming-channel-2
    outgoing: (5)
      outgoing-channel-1-name: (6)
        topic: outboxes/topic-for-outgoing-channel-1 (7)
      outgoing-channel-2-name: (8)
        topic: outboxes/topic-for-outgoing-channel-2
1 incoming is a fixed token, which allows StreamX to distinguish incoming channels from outgoing channels, while parsing the Mesh file.
2 incoming-channel-1-name is the actual name of the incoming channel.
3 inboxes/topic-for-incoming-channel-1 is the incoming channel topic, where the Service subscribes to incoming messages. By default, StreamX uses Apache Pulsar for a messaging system under the hood.
4 The example Processing Service declares two incoming channels, the second one is incoming-channel-2-name. There’s no explicit limit on the number of channels. It depends on the actual needs of a service.
5 outgoing is a fixed token, which allows StreamX to distinguish outgoing channels from incoming channels, while parsing the Mesh file.
6 outgoing-channel-1-name is the actual name of the outgoing channel.
7 outboxes/topic-for-outgoing-channel-1 is the outgoing channel topic, where the Service publishes its results. The results might be further processed by another Processing Service, or consumed by a Delivery Service, by configuring them to listen on the same channel and topic.
8 The example Processing Service declares two outgoing channels, the second one is outgoing-channel-2-name. As in the case of incoming channels - there’s no explicit limit on the number of channels. It depends on the actual needs of a service.

More detailed information about Channels and Topics can be found in the Processing Service reference.

Environment variables

Every Service or Component can be extended with a map of environment variables, which is applied to the Container of a Service while running it.

This is done by providing environment, which is a set of property name and value pairs. The environment field should be set on the same level with image / incoming / outgoing blocks in the Mesh YAML file.

The syntax for specifying environment properties is the following:

    environment:
      PROPERTY_1_NAME: "value-1"
      PROPERTY_2_NAME: "value-2"

Global environment variables

It is supported to define environment variables at the Mesh level:

defaultRegistry: sample-container-registry/streamx-images
defaultImageTag: 1.0.0
environment:
  PROPERTY_1_NAME: "global-value-1"
  PROPERTY_2_NAME: "global-value-2"
sources:
  source-name:
    outgoing:
      - "topic-for-incoming-channel-1"
      - "topic-for-incoming-channel-2"

These environment variables are global, meaning they are applied to every container.

Merging order is defined according to the following priorities (higher takes precedence):

  1. Global environment variables

  2. Service-level environment variables

  3. Component-level environment variables (for use with Composite Delivery)

Environment from configs and secrets

You can use environmentFrom to load environment variables from property files stored in the secrets or configs directories. These directories must be located in the current working directory. The secrets and configs directories are also shared with files specified by volumesFrom.

The declaration might look like this:

environment:
  ENVIRONMENT_KEY: "environment-value"
environmentFrom:
  secrets:
  - "env.properties"
  configs:
  - "env.properties"

environmentFrom is supported at the Mesh, Services, and Delivery Components levels.

Merging priorities are identical to explicitly defined variables; however, explicitly defined variables always override properties loaded from files.

Volumes from configs and secrets

It is possible to configure additional files or directories to be mounted to a container’s filesystem by specifying volumesFrom.

An example usage might look like this:

components:
  nginx:
    image: docker.io/library/nginx:1.26.0
    ports:
      - 80:80
    repositoryVolume: /usr/share/nginx/html
    volumesFrom:
      configs:
        - "nginx.conf:/etc/nginx/conf.d/default.conf" (1)
      secrets:
        - "private.key:/etc/ssl/default.key" (2)
1 A custom NGINX configuration file from the configs directory overwrites the default /etc/nginx/conf.d/default.conf file.
2 A private key is mounted from the secrets directory to /etc/ssl/default.key on the container’s filesystem.

Both the configs and secrets directories must be located in the current working directory. The directories are shared with environmentFrom source files.

Explicit and implicit mapping in volumesFrom

Mapping notation consists of two parts separated by a : character:

  1. The part before the : is the path relative to either the configs or secrets folder.

  2. The part after the : is the path where the file or directory is mounted in the container’s filesystem.

If the : character is not present, implicit mapping is applied by default.

You can define both explicit and implicit mappings in the volume source declaration.

volumesFrom:
  configs:
    - "file1.txt:/destination-file1.txt" (1)
    - "file2.txt" (2)
1 Explicit: the file file1.txt from $CWD/configs/file1.txt is mounted as /destination-file1.txt in the container’s filesystem.
2 Implicit: the file file2.txt from $CWD/configs/file2.txt is mounted as /file2.txt in the container’s filesystem.

The same rules apply to the secrets source.

Mounting files or directories

You can also mount individual files or entire directories, as shown in this example:

volumesFrom:
  configs:
    - "local-folder:/destination/folder" (1)
1 The entire folder local-folder from $CWD/configs/local-folder is mounted as /destination/folder in the container’s filesystem.

Delivery services

StreamX Delivery Service is a category of a microservice that facilitates the delivery of processed data. You can find exhaustive information about Delivery Services in StreamX Delivery Service reference.

Delivery Service name

StreamX Mesh file can include configurations for one or more microservices. Each service is assigned a name, which can be any name you prefer. It is recommended to use names that are clear and descriptive, because StreamX will create a container for each service, using the service name for the container name.

In the example Mesh file above, delivery-service-name is the Delivery Service’s name. The sample Mesh contains only one Delivery Service.

Container Image for the Delivery Service

This setting shares the concepts of Container Image for the Processing Service.

Channels

Every Delivery Service specifies a list of incoming channels. Delivery Services are used to deliver processed data to the expected destinations. The StreamX Mesh file allows specifying the destination channel / topic of the data.

The channels and topics are configured by following the syntax presented in the sample StreamX Mesh YAML file. Below you can find the reference of the configuration fields:

delivery:
  delivery-service-name:
    image: delivery-service-name
    incoming: (1)
      delivery-channel-1-name: (2)
        topic: outboxes/topic-for-outgoing-channel-1 (3)
      delivery-channel-2-name: (4)
        topic: outboxes/topic-for-outgoing-channel-2
1 incoming is a fixed token, which allows StreamX to distinguish incoming channels from outgoing channels, while parsing the Mesh file.
2 delivery-channel-1-name is the actual name of the incoming channel.
3 outboxes/topic-for-outgoing-channel-1 is the outgoing channel topic for the data producer (for example a Processing Service) - and the incoming channel topic for a Delivery Service, where the Service subscribes to incoming messages. By default, StreamX uses Apache Pulsar for a messaging system under the hood.
4 The example Delivery Service declares two incoming channels, the second one is delivery-channel-2-name. There’s no explicit limit on the number of channels. It depends on the actual needs of a service.

Note: In the above examples, outgoing channels of the Processing Service are connected to incoming channels of the Delivery Service by topic names. This approach allows creating the data flow is described in Processing Service channels reference.

More detailed information about Channels and Topics can be found in the Delivery Service reference.

Port

A Delivery Service can also be configured with a port:

    port: [numeric_port]

The port setting should be declared on the same level as image and incoming fields in the StreamX Mesh YAML file. In case when the implementation of your Delivery Service is an HTTP web server - the port allows specifying the port number to communicate with the web server.

Environment variables

This feature works identically for Delivery Services, which is described in the Environment variables section for Processing Services.

Composite Delivery

The Composite Delivery is a concept referring to the possibility of configuring nested Delivery components in the StreamX Mesh YAML file. This allows you to deliver incoming data to any destination, including a Web Server, making the data available in the form of web pages with images. Below, you can see a sample Mesh YAML file for such configuration:

defaultRegistry: sample-container-registry/streamx-images
defaultImageTag: 1.0.0
processing:
  processing-service-name:
    image: processing-service-name
    incoming:
      incoming-channel-name:
        topic: inboxes/topic-for-incoming-channel
    outgoing:
      outgoing-channel-name:
        topic: outboxes/topic-for-outgoing-channel
delivery:
  composite-delivery-service-name:
    image: composite-delivery-service-name
    incoming:
      composite-delivery-channel-name:
        topic: outboxes/topic-for-outgoing-channel
    port: 8088
    environment:
      REPOSITORY_RESOURCE_ROOT_DIRECTORY: /srv/www (1)
    repositoryVolume: /srv/www
    components:
      webserver:
        image: docker.io/library/nginx:1.26.0
        ports:
        - 8089:80 (3)
        repositoryVolume: /usr/share/nginx/html (2)
        volumesFrom:
          configs:
          - "test/overridden-nginx.conf:/etc/nginx/conf.d/default.conf"

This Mesh file contains one Processing Service (processing-service-name) and one Delivery Service (composite-delivery-service-name). The Delivery service contains a nested component, which is a NGINX web server.

To allow sharing files between the Delivery Service and NGINX component, additional settings are applied to the Delivery Service:

1 The environment.REPOSITORY_RESOURCE_ROOT_DIRECTORY and repositoryVolume settings point to a shared directory inside the Container for the Composite Delivery Service. Note: REPOSITORY_RESOURCE_ROOT_DIRECTORY is a required configuration of an example Delivery Service.
2 Using the webserver.repositoryVolume setting, the shared directory is configured to be a Container Volume associated with a directory in the filesystem of the NGINX web server.
3 In this example, the NGINX web server is also configured to expose its 80 port to the outside, mapping it to port 8089 from the Container.