This version is still in development and is not considered stable yet. For the latest stable version, please use StreamX Guides 1.0.0!

StreamX Command Line Interface (CLI)

The streamx command lets you run a StreamX Mesh locally and ingest it with payloads.

Prerequisites

  • You have installed Java 17 or higher.
    The StreamX CLI is built on Java, so you must have it installed.

  • You have installed Docker.
    The StreamX CLI uses Docker for containerization. Verify you have it installed and running on your system.

Docker on Windows requires Windows Subsystem for Linux (WSL).
Verify you have virtualization enabled in the BIOS (required by the WSL service).

  • You have a proper package manager:

    • Homebrew (for Linux/macOS)

    • Scoop (for Windows)

Installing the CLI

The StreamX CLI is available in two developer-oriented package managers:

  • Homebrew - for Linux and macOS

  • Scoop - for Windows

These package managers provide the easiest way to install and update StreamX CLI.

Homebrew

Homebrew is a package manager for macOS (and Linux). You can use Homebrew to install (and update) StreamX CLI.

To install the latest version of StreamX CLI by using Homebrew, run the following command:

brew install streamx-dev/tap/streamx

This command can also be used to upgrade StreamX CLI.

Once installed, streamx is in your $PATH. Running streamx --version prints the installed version:

streamx --version
# prints current version

You can also upgrade StreamX CLI with:

brew update
brew upgrade streamx
To list the available versions of StreamX CLI, use brew tap streamx-dev/tap && brew search streamx-dev/tap/streamx\*. You can then install the version of StreamX CLI you want with brew install streamx-dev/tap/streamx@x.y.z.

Scoop

Scoop is a package manager for Windows. You can use Scoop to install (and update) StreamX CLI.

To install the latest version of StreamX CLI by using Scoop, run the following commands:

scoop bucket add streamx-dev https://github.com/streamx-dev/scoop-streamx-dev.git
scoop install streamx

Once installed, streamx is in your $PATH. Running streamx --version prints the installed version:

streamx --version
# prints current version

You can upgrade StreamX CLI with:

scoop update streamx
The list of the available versions of StreamX CLI is published at releases page. You can then install the version of StreamX CLI you want with scoop install streamx@x.y.z.

Using the CLI

Use --help to display help information:

streamx --help

Usage: streamx [-hV] [[--accept-license]] [COMMAND]
      --accept-license   Automatically accept actual StreamX license
  -h, --help             Show this help message and exit.
  -V, --version          Print version information and exit.
Commands:
  run
  publish
  unpublish

streamx run

For development purposes, it is possible to run StreamX Mesh locally.

To do this specify the location of the StreamX Mesh definition. By default, StreamX CLI looks for mesh.yaml and mesh.yml in the current directory. If both are present, mesh.yaml is used. You can specify a custom path to the mesh definition by using the -f option.

streamx run -f ./streamx-mesh.yaml

This outputs information that STREAMX IS READY! with ports exposed by StreamX Mesh’s containers.

Example output of a mesh started with streamx run.
--------------------------------------------------------------------------------
STREAMX IS READY!
--------------------------------------------------------------------------------
Stack:
grafana                                                    http://localhost:3000
prometheus                                                 http://localhost:9090
otel-collector                                             http://localhost:4317
jaeger                                                    http://localhost:16686
pulsar                                                     http://localhost:6650

DX Mesh:
web-delivery-service                                       http://localhost:8081
relay                                                     http://localhost:62329
rest-ingestion                                             http://localhost:8080
--------------------------------------------------------------------------------
Network ID:
dbef5cd015a6637f4bd392de551ba7e2387e63d0eabca71a3d0bd7ee9d12b0d7
Mesh configuration file: ./mesh.yaml
--------------------------------------------------------------------------------
Sources:
websight token: eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJpc3M....
root token: eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJpc3M....

To shut down a running mesh, simply kill the process with ctrl + c. It also handles stopping containers and cleaning up existing data.

Shutting down StreamX takes a few seconds. Wait a while to avoid a startup error on restart.

Observability

By default, the StreamX CLI provides additional services that make it easier to monitor the operation of the mesh. The main services that facilitate development are:

  • Grafana - enables data querying, visualising and alerting on

  • Jaeger - provides workflows monitoring and troubleshooting

  • Prometheus - collects and stores its metrics in the form of time series data

  • Otel-collector - receives, processes and exports telemetry data

Each observability service runs in a separate container and consumes resources. If your machine has limited resources or if observability is not required, you can disable this feature. Refer to Run configuration for more details.

Container logs

StreamX CLI is based on Docker containers, so container-specific logs are available through the Docker CLI.

docker logs yourStreamXMeshContainer

You can also access logs from any GUI-based application that manages the Docker environment.

Run configuration

Configuration property Type Default

streamx.tenant

Distinguishes different tenants in StreamX.

It is also used in the underlying messaging system to set up dedicated channels and topics.

Environment variable: STREAMX_TENANT

String

streamx

streamx.container.startup-timeout-seconds

Maximum time to wait for each container to start expressed in seconds. If any container needs more time to start - increase this value.

Environment variable: STREAMX_CONTAINER_STARTUP_TIMEOUT_SECONDS

positive natural number

60

streamx.runner.observability.enabled

Enables additional observability containers

Environment variable: STREAMX_RUNNER_OBSERVABILITY_ENABLED

boolean

true

streamx.runner.generated-keys.location

The location of the folder to store private and public keys.

Environment variable: STREAMX_RUNNER_GENERATED-KEYS_LOCATION

String

${user.home}/.streamx/config/generated

Ingestion

StreamX CLI allows you to populate the StreamX Mesh with data. The CLI supports both publishing and unpublishing actions.

Ingestion requires providing the ingested channel and the key. This data is sufficient for unpublishing, while publishing requires more data - the content.

You can learn more about ingestion in ingestion reference.

streamx publish

To publish content with StreamX CLI, run the following template command:

streamx publish channel key pathToContentFile

For example, if you want to publish content loaded from the ./content.json file to the pages channel with the index.html key, you can use:

streamx publish pages index.html ./content.json

Content-defining options

Publication content is defined in JSON format. It can be defined not only by providing payload file for the third argument, but also by merging content fragments resolved from content-defining options.

There are three content-defining options:

  • -j defines a raw JSON fragment
    The provided JSON fragment will be rewritten to merged JSON without any changes.
    This option should be used for raw JSON fragment, along with boolean, number, null or undefined.

  • -s defines a JSON string fragment.
    Unlike -j it handles JSON string escaping. Use this option to create string fragment.

  • -b defines fragments containing binary data.
    In JSON Avro format (used when communicating with the Rest Ingestion Service), binary data is encoded with string JSON type.
    Use this option to create a JSON-encoded binary fragment.

Table 1. Summary of content-defining options.
Option Name Use cases

-j

JSON fragment

Full JSON data, boolean, number, null or undefined

-s

JSON string fragment

string

-b

JSON-encoded binary fragment

Binary data (for example images, movies, PDF files and others)

The simplest use of the content-defining option is to create entire JSON content with -j:

streamx publish pages index.html -j "{'content': {'bytes': 'Some string'}}"
The content-defining option, like any command-line option, requires the use of ' or " to preserve the literal value of the option.

The above command might seem unclear. To improve readability, you can specify a JSONPath expression, which should be replaced with the given value.

streamx publish pages index.html -j content.bytes='"Some string"'

When defining a string you still must use ' or ".

When defining a string value you can use -s instead of -j to skip double quotes.

streamx publish pages index.html -s content.bytes='Some string'
Ingestion payload source

By default, Content-defining options publish data that is literally specified in the option values.

However, it is possible to publish data loaded from a file. To do this, add the prefix file:// followed by the path to the content file.

streamx publish pages index.html -s content.bytes=file://relative/path/to/file.json

The CLI accepts both relative and absolute paths.

The absolute path always starts with /, so Content-defining options will effectively contain file:///.
Multiple Content-defining options

The publish command accepts multiple Content-defining options. This results in a single JSON that is computed from multiple separate options.

The target content is resolved in four steps:

  1. Resolve JSON fragments separately for each Content-defining option

  2. Create a temporary JSON with the first fragment

  3. For each Content-defining option merge temporary JSON with corresponding fragment resulting in a new temporary JSON

  4. After processing all JSON fragments, the temporary JSON becomes the target content

streamx publish pages index.html -j content.num=123 -j {"content":{"text":"text"}}
// published content is {"content":{"text":"text", "num":123}}

Sequential processing of JSON fragments allows you to override JSON nodes derived from previously processed Content-defining options.

streamx publish pages index.html -j content.num=123 -j '{"content":{"num":null}}'
// published content is {"content":{}}
streamx publish channel key pathToContentFile is effectively equal to streamx publish channel key -j file://pathToContentFile

streamx unpublish

To trigger the unpublishing of content with StreamX CLI, run the following template command:

streamx unpublish channel key

For example, if you want to unpublish content fed to the pages channel with the index.html key, you can use:

streamx unpublish pages index.html

Source systems simulation

StreamX CLI provides the ability to simulate any source system defined in the mesh.yaml file. Each source system is assigned a unique token, which is displayed after running the StreamX Mesh. To impersonate a specific source system, apply the token by using the streamx.ingestion.auth-token property.

root is the default token used by StreamX CLI when no authentication token is provided. In this case, no authorization checks are enforced, and the source system is allowed to perform actions on any channel.

Tokens are generated automatically with private and public keys. These keys are generated when the run command is executed. If the keys already exist, no new keys are generated, allowing you to use your own keys if desired. The path where the keys are stored can be controlled by setting the streamx.runner.generated-keys.location property.

Ingestion configuration

Configuration property Type Default

streamx.ingestion.url
The REST ingestion service URL

CLI Option: --ingestion-url
Environment variable: STREAMX_INGESTION_URL

String

http://localhost:8080

streamx.ingestion.auth-token
Auth token used for authentication on server side.

Environment variable: STREAMX_INGESTION_AUTH_TOKEN

String

streamx.ingestion.insecure
Disables SSL/TLS certificates validation

Environment variable: STREAMX_INGESTION_INSECURE

boolean

false

Configuration

StreamX CLI uses Quarkus Config for configuration resolution, so it reuses the same configuration sources.

There are also two configuration sources specific to StreamX CLI: StreamX Options and StreamX Home.

The StreamX Option config source allows you to set selected properties with CLI options. For example, passing the --ingestion-url option sets streamx.ingestion.url for CLI command invocation.

The StreamX Home config source allows you to configure a local development environment. This config source loads properties from the ${user.home}/.streamx/config/application.properties file.

All available configuration sources, starting from source with the highest priority source, are:

  1. StreamX Option config source.
    Only selected properties are exposed to be CLI options.

  2. Environment variables.
    All available properties can be set by defining an environment variable. However, environment variable property names must follow the conversion rules.

  3. .env file in the current working directory.
    This config source is specific to each environment, so all properties can be set by using .env file. Property names follow the same conversion rules that apply to environment variables.

  4. ./config/application.properties file relative to the current working directory.
    This config source is intended to be shared. To prevent sharing sensitive data such as auth tokens, not all properties can be set this way.

  5. StreamX Home config source.
    This config source is intended to be global for users. Only selected properties can be set this way. for example automatic license acceptation or disabling self-signed certificates verification.

Troubleshooting

Each invocation of the streamx command persists file detailed logs. This file is stored in the ${user.home}/.streamx/logs directory. Each log file contains invocation timestamp.

Container logs of the running StreamX Mesh containers might also be helpful.

Tracing information is available through Jaeger, which is part of the StreamX observability.

License

Use of the StreamX CLI requires acceptance of the CLI license. By default, the CLI prompts the user to accept the license on the first time the streamx command is run.

streamx --version

Distributed under StreamX End-User License Agreement 1.0
https://www.streamx.dev/licenses/eula-v1-0.html

Do you accept the license agreement? [Y/n]

Respond with y to accept the license.

If you want to use streamx in your CI/CD process set the streamx.accept-license property to true. This will accept the accept license and prevent reading from standard input.

streamx --accept-license --version

Distributed under StreamX End-User License Agreement 1.0
https://www.streamx.dev/licenses/eula-v1-0.html


Do you accept the license agreement? [Y/n]
Y -> "streamx.accept-license" property was set to "true".
Configuration property Type Default

streamx.accept-license
Automatic license acceptance enabled.

CLI Option: --accept-license
Environment variable: STREAMX_ACCEPT_LICENSE

String

false

Related content