Analytics Modules Overview

Analytics Modules Overview

Analytics Modules Overview

This section presents the analytics modules available in the Crosser Module library. Analytics modules are used to transform and harmonize message formats from different sources, as well as operate on the actual message content. The modules are presented in groups with related functionality.


Data and Property Mappers

The Data Mapper and Property Mapper modules are the Swiss army knifes for modifying message structure.

Use them to:

  • Rename properties
  • Move properties to new hierarchy levels
  • Remove properties
  • Add new properties with static values
  • Copy existing property values to new properties

You can basically do the same thing with both these modules. The difference is that the Data Mapper uses payload templates for input and output formats and then provides a visual interface for defining the mappings. The Property Mapper has a more configuration-oriented interface where each individual mapping is defined explicitly in the module settings.

Working with Arrays

Arrays are common both on inputs and outputs, but may not be the optimal internal format when applying streaming processing. For example, to apply processing on individual array elements in a streaming pipeline you need to split up the array into individual messages. Another example is when you want to write data into columns in a database, then you need an object with key/value pairs that can be mapped against the database columns.

There are several modules in the Crosser Module library that help you convert back and forth between these different formats. In this environment an array is typically an array of objects, i.e. each element contains multiple values and even hierarchical message structures.

Names in bold indicate the most common array modules.

Array SplitBreaks up an array into individual messagesArrayMessages
Array JoinCombine a stream of messages into an array, by time or message count (opposite of Array Split)MessagesArray
Array to ObjectConvert an array into an object with key/value pairs. Each item in the array must have a property holding the key value and another property with the value for that keyArrayMessage
Object to ArrayConvert an object with key/value pairs into an array. Each element in the array will have a property containing the name of the key and another with the value belonging to that key (opposite of Array To Object)MessageArray
Array RegexFilter out some elements from an array by matching the value on a specified property with a regular expression.ArrayArray
Array Property PickSelects some of the properties available in each array element. A list of properties to pick must be provided.ArrayArray
Array Property OmitRemove some of the properties available in each array element. A list of properties to remove must be provided.ArrayArray
Array Sort By ListSort array elements by comparing the values on a specified property against an ordered list of values provided. Typically used together with Array Property Get to create lists of values where the position in the array is used to identify a source.ArrayArray
Array Property GetSelect the values from a selected property. The output is an array with only values. Useful when feeding e.g. ML models where the position in the array is used to identify a source, rather than key/value mappings.ArrayArray of Values
JoinThis is a variant of the Array Join module that ensures that data for a specified number of sources are always available in each output. If no data has been received for a specific source within the given time period an output will be generated based on the strategy selected. This module is more complex to set up but is useful in cases where you must ensure that each output has a value for each source, e.g. when feeding an ML model.MessagesArray

Working with Strings

Text TemplateGenerates text messages based on a template provided in the settings. Template syntax can be used to insert data from the message together with static text.MessageString
String ReplaceReplace a substring with another substring.StringString
String SubstringSelect a substring based on start and stop positionsStringString
CSV Line ParserBreak up a line of text (string) into substrings based on a delimiter, e.g. “,”. The output is an object with one property per substring. These properties are named based on the position in the string, e.g. col1, col2...StringObject with substrings
CSV Text ParserSame operation as the CSV Line Parser but operating on multi-line strings. The output is an array with one element per line. Each element has the same format as with the CSV Line Parser.StringArray of objects with substrings

Format Conversions

Messages sent between modules use a custom .NET data type (FlowMessage) that supports dynamic structures with hierarchies of objects and arrays which can contain basic .NET data types. Sometimes, e.g. in the debug window, messages may look like Javascript objects (JSON) but that is only for presentation purposes. JSON is never used internally. When communicating with external systems you may get data in JSON or XML format, or the output is expected to be in one of these formats. To work with these types of data you must convert between the internal format and the external formats. To do that you use these conversion modules.

JSONConverts to and from JSON strings. If the input is a string it will be interpreted as JSON and the output is a FlowMessage.

If the input is a FlowMessage it will be converted into a JSON string.

This module operates on a selected property of the incoming message and the result can be assigned to a new property.

XMLSame as the JSON module but working with XML text.
Time StampConvert DateTime values into another format. Predefined formats are ISO8601 and Unix timestamp in seconds or milliseconds. Custom output formats can also be defined.

If no input is specified the module will add the current system time on the output.

Base64 EncodeConvert a binary array or a string into a base64 encoded string. For string inputs the output can be set to use base64url format, to create URL safe strings.
Base64 DecodeDecode a base64 encoded string into a byte array or string

Condition Logic

Message filters

In addition to the below modules each module has a configurable message filter, which can be found on the “Common” tab. By default modules will process every message they receive. By adding a message filter the module will only process messages matching some criteria. Messages not matching the filter criteria can either be dropped, or passed on untouched.

Message filters can be set to match conditions on numeric property values (=, <, >), string property values (contains, does not contain, equal/not equal, isNull, isNotNull, regex) and boolean values (isTrue, isFalse). Multiple conditions can be used.

Filtering modules

In addition to the message filters above the following modules can be used to control which messages that are passed on to the next module(s), based on different criteria.

Range FilterOnly let through messages where a selected value is either within or outside of a specified range. The same effect can be achieved by using message filters.
Report By ExceptionOnly let through messages when the value of a specified property changes. Can be used to remove duplicates.
SplitSplit a stream of messages into multiple paths by defining conditions on message data. Multiple conditions can be specified for each path. Each path will get a separate output on the module.
ThrottleLimit the rate of messages, e.g. only let through a maximum of one message per second. Can also be used to spread out messages in time by enforcing a minimum delay between messages, which can be used to smooth out bursty traffic.

Intelligent Logic

Operating on message Data

The following modules will create new data by processing the values in messages. They all work on numerical values and the result can either be assigned to a new property or the original value can be overwritten.

AggregateCalculate average, min and max on a selected property from a group of messages. The group of messages to use can either be selected by specifying a time period, or a number of messages. Calculations can be performed independently for messages coming from different sources by specifying a property that indicates the source. At the end of each interval a message is delivered for each source with the calculated statistics.
Math ExpressionExecute generic mathematical expressions on message data. The expression is specified using template syntax where message data can be inserted into the mathematical expression, e.g. ‘Abs({data.cur_temp} - {data.prev_temp})’.
Range ClassifierAdd a text classification based on ranges of values on a selected property, e.g. ‘Low’, ‘Med’, ‘High’
ScaleScale a value on a selected property ‘(value*scale)-offset’, e.g. to convert temperature values from Farenheit to Celsius. Note: This operation can also be performed with the Math Expression module.
SmoothApply an exponential smoothing filter on a property value, e.g. to remove noise.
StatisticsSame as Aggregate but operating on a rolling window of messages, i.e. an output message is generated for each input message. This module will also calculate the standard deviation of the values.
ToggleToggle the value of a boolean property, or toggle the value of an internal variable and add to the output message

Code Modules

With the code modules you can apply custom processing of messages using either C# or Python code. The code can either be entered directly in the settings UI, or code files can be uploaded to the Resource library and then referenced from within the modules.

CsharpExecute C# code. The code is compiled at runtime. Standard .NET libraries can be used but not 3rd party libraries.
IronPythonRun Python code using the IronPython interpreter which runs in .NET. Standard libraries can be used but not 3rd party libraries. Supports Python 2.7 code.
Python BridgeRun python code in a standard Python 3.9 environment outside of .NET. On Windows the Python environment must be installed separately and the local node configuration must be updated accordingly (see installation docs). Standard and 3rd party libraries can be used, including ML frameworks.

Counter Modules

Message CounterCounts the values seen in a selected message property over a time period. At the end of the period the number of occurrences of each value is listed together with the relative count of each value. Useful for KPI calculations, such as Yield.

If no source is specified the total number of messages received is reported.

Time CounterCounts the time spent in different states by looking at the values received on a specified property and the time between value changes. At the end of the period the total time spent in each state (value) is reported together with the relative times. Useful for KPI calculations such as Availability.
TimeoutMeasures the time since the last update on a selected property. If no update is seen within a specified timeout period a message is sent out. Use this module to monitor expected traffic patterns. For example, if you know that data should arrive every second this module can be used to trigger an alert if no update is seen within 2 seconds.

By keeping track of individual property values the module can operate on multi-source streams and trigger an alert as soon as one a value is missing from one of the sources.

Storage Modules

Sometimes you need to store message data temporarily and then these modules will come in handy.

Memory BufferThis module will keep messages until acknowledged by an external signal. It is designed to work with any output module to keep messages until successfully delivered. Output modules produce a ‘crosser.success’ value indicating if the external delivery was successful or not. This value can be fed back to the Memory Buffer module. The size of the buffer as well as retry strategies can be defined.
StateThe State module can store any message data, including complex structures, and add the stored value to all messages it receives. This can be used to store data that is updated infrequently but is needed on every message from another source. For example to add configuration data to each new sensor message.
    • Related Articles

    • Crosser Security Overview

      Crosser Security Overview November 2023 About this document This document describes security aspects related to the Crosser Streaming Analytics solution. The Crosser Streaming Analytics solution The Crosser Streaming Analytics solution has two main ...
    • Solution Overview

      Solution Overview The Crosser Solution is entirely focused on streaming analytics, i.e. analyzing data in motion. It is optimized for collecting data close to the source and then analyze the data in real time. The results can be delivered to ...
    • Messages, Modules and Flows

      The Crosser Edge node is a real-time engine designed to process Messages. A message can be pushed from an external source, such as a MQTT or HTTP client, or it can be pulled from the Edge node using connectors for different protocols/services. ...
    • Resources management and new Modules page

      Release Note Resources management and new Modules page Release date: 2024-05-07 Resources management The management of Resources has got a major overhaul in this release: Versions - A Resource now has versions, i.e. you don’t need to create a ...
    • Creating Your First Flows

      Create Your First Flow In this guide we will start by creating a basic example Flow. It doesn't connect to any external system, the goal is just to get you familiarized with the concept of building and testing Flows. We will use an internal data ...