This version is still in development and is not considered stable yet. For the latest stable version, please use StreamX Guides 1.0.0!

StreamX Key Rewrite feature

The StreamX Key Rewrite feature enables the transformation of the resource keys from the source system into the desired publication key format within StreamX Mesh. The feature is based on match-rewrite regular expression pattern pairs and uses the google/re2j package for regular expression processing.

Key Rewrite configuration

To configure the Key Rewrite feature, add the following environment variables in the StreamX Rest Ingestion YAML file:

ingestion:
  rest-ingestion:
    environment:
      STREAMX_INBOXES_KEY-REGEX-REWRITE_PATTERNS_CONFIG1_MATCH-PATTERN: "MATCH_PATTERN"
      STREAMX_INBOXES_KEY-REGEX-REWRITE_PATTERNS_CONFIG1_REWRITE-PATTERN: "REWRITE_PATTERN"

Where:

Key Rewrite pattern identifier

More than one pattern for transforming resource keys can be defined. To achieve this, add several entries under the 'environment' property, and assign each match-rewrite pattern pair a unique identifier. It is supported by chaining CONFIGX, where X pointing to testing matcher invocation order. See the following example:

ingestion:
  rest-ingestion:
    environment:
      STREAMX_INBOXES_KEY-REGEX-REWRITE_PATTERNS_CONFIG1_MATCH-PATTERN: "MATCH_PATTERN_1"
      STREAMX_INBOXES_KEY-REGEX-REWRITE_PATTERNS_CONFIG1_REWRITE-PATTERN: "REWRITE_PATTERN_1"
      STREAMX_INBOXES_KEY-REGEX-REWRITE_PATTERNS_CONFIG2_MATCH-PATTERN: "MATCH_PATTERN_2"
      STREAMX_INBOXES_KEY-REGEX-REWRITE_PATTERNS_CONFIG2_REWRITE-PATTERN: "REWRITE_PATTERN_2"

The order of processing match patterns is determined lexicographically by their identifiers. For the example given above:

CONFIG1 is processed before CONFIG2.

A resource key can only be rewritten by one matcher. Once a match is found, further matchers are not processed.

Match-rewrite pair definitions

Match pattern

A match pattern is a regular expression that identifies publication keys for transformation. It can contain groups that correspond to specific parts of the key. A group is denoted by ( and ), it can be used for matching parts of strings in match pattern and capturing the matched parts.

Rewrite pattern

The rewrite pattern formats the matched keys. It uses captured groups from the match pattern. Defined groups are referenced in the rewrite pattern with $1, $2, and subsequent, based on their order in match pattern.

Match-rewrite patterns use example

To transform the resource key from:

"example.com/${path}"

to:

"example.subdomain.com/${path}"

the following example can be used.

Match pattern example

The match pattern should capture the specific parts of the original resource key that are needed to transform and retain the necessary parts. The match pattern can look like this:

STREAMX_INBOXES_KEY-REGEX-REWRITE_PATTERNS_CONFIG1_MATCH-PATTERN: "^example.com/(.*)$"

Explanation of the match pattern:

  • ^www.example.com/: Asserts that the match must start at the beginning of the string.

  • (.*): Captures everything after the domain, which corresponds to the path portion of the URL. The group can be referenced with $1 in the rewrite pattern.

Rewrite pattern example

The rewrite pattern reforms the matched keys according to the desired output:

STREAMX_INBOXES_KEY-REGEX-REWRITE_PATTERNS_CONFIG1_REWRITE-PATTERN: "example.subdomain.com/$1"

Explanation of the rewrite pattern:

  • example.subdomain.com/: This part specifies a static string for the new domain, changing it from example.com/.

  • $1: references the first captured group, which corresponds to the path part of the URL, ensuring that this part is retained in the transformed key.

For guidance on constructing regular expressions and how to use groups, refer to the RE2J documentation.

Related content