StreamX updates: OpenSearch added, more features to come

Take it straight from the Project Manager: we've been quite busy at StreamX lately.

Significant improvements have been made in a few key areas: search functionality, data storage, and system monitoring capabilities.

We've expanded our documentation, adding detailed guides on configuring StreamX Mesh using YAML files and deploying StreamX on local Kubernetes clusters.

We're also working on something we think you'll love, called Mesh Manager. This new web app will give you a visual way to manage your StreamX mesh, making it easier than ever to configure and monitor the system.

Stay tuned – there's more features to come!

The key update is the transition from Lucene Search Engine to OpenSearch for StreamX search capabilities. 

While Lucene has been a robust search library, we anticipated challenges with its maintenance and scaling. That’s why we introduce support for OpenSearch that improves:

• Scalability: as it’s designed to handle large volumes of data and high query loads with ease,

• Maintainability: as it simplifies the operational aspects of search, and provides

• Service Independence: as operates independently from the StreamX Mesh, providing a more modular and flexible architecture.

Want to see it in action? Check it out with our tutorial: Set up search with StreamX.

We introduced support for RocksDB as a new store backend in StreamX. 

This allows switching from in-memory storage to persistent storage using RocksDB, either for specific stores or globally across the application. RocksDB optimizes performance for fast storage and large-scale server workloads by offloading data to disk, reducing memory consumption, and preventing OutOfMemoryErrors. 

We have introduced a set of reusable Terraform modules designed to set up long-lived environments efficiently.

In addition, we’ve developed a distribution testing framework using the Gatling Operator, which allows us to thoroughly assess the distribution of StreamX.

To enhance our observability and monitoring capabilities, we’ve adopted several industry standards:

• OpenTelemetry for tracing

• Micrometer for gathering metrics

• Loki as our logs database

• Grafana Dashboards and Jaeger for UI-based monitoring

The improvements enable us to monitor environment behavior and track its health in real-time, providing the team with extensive information. This setup facilitates quick reactions to incidents, ensuring more effective and responsive management of the environment.

• StreamX Mesh YAML File: that provides a comprehensive guide on configuring a StreamX Mesh using a YAML file. It details how to set up services, Docker images, environment configurations, and serves as the entry point for starting StreamX Mesh. Read more

• Run StreamX on Local Kubernetes Cluster: this guide walk you through setting up StreamX on a local Kubernetes cluster using Docker and Terraform. Read more

To support managing the mesh, the StreamX Development Team is working on the Mesh Manager - a web application to visualize and manage the StreamX mesh. After uploading the mesh from the StreamX Mesh YAML file, it can be adjusted and downloaded for further use.

The current preview version enables adding and removing connections, creating and removing topics and source services, and switching between overview (read-only) mode and manager mode.