This section presents the analytics modules available in the Crosser Module library. Analytics modules are used to transform and harmonize message formats from different sources, as well as operate on the actual message content. The modules are presented in groups with related functionality.
The Data Mapper and Property Mapper modules are the Swiss army knifes for modifying message structure.
Use them to:
You can basically do the same thing with both these modules. The difference is that the Data Mapper uses payload templates for input and output formats and then provides a visual interface for defining the mappings. The Property Mapper has a more configuration-oriented interface where each individual mapping is defined explicitly in the module settings.
Arrays are common both on inputs and outputs, but may not be the optimal internal format when applying streaming processing. For example, to apply processing on individual array elements in a streaming pipeline you need to split up the array into individual messages. Another example is when you want to write data into columns in a database, then you need an object with key/value pairs that can be mapped against the database columns.
There are several modules in the Crosser Module library that help you convert back and forth between these different formats. In this environment an array is typically an array of objects, i.e. each element contains multiple values and even hierarchical message structures.
Names in bold indicate the most common array modules.
| Name | Description | Input | Output |
| Array Split | Breaks up an array into individual messages | Array | Messages |
| Array Join | Combine a stream of messages into an array, by time or message count (opposite of Array Split) | Messages | Array |
| Array to Object | Convert an array into an object with key/value pairs. Each item in the array must have a property holding the key value and another property with the value for that key | Array | Message |
| Object to Array | Convert an object with key/value pairs into an array. Each element in the array will have a property containing the name of the key and another with the value belonging to that key (opposite of Array To Object) | Message | Array |
| Array Regex | Filter out some elements from an array by matching the value on a specified property with a regular expression. | Array | Array |
| Array Property Pick | Selects some of the properties available in each array element. A list of properties to pick must be provided. | Array | Array |
| Array Property Omit | Remove some of the properties available in each array element. A list of properties to remove must be provided. | Array | Array |
| Array Sort By List | Sort array elements by comparing the values on a specified property against an ordered list of values provided. Typically used together with Array Property Get to create lists of values where the position in the array is used to identify a source. | Array | Array |
| Array Property Get | Select the values from a selected property. The output is an array with only values. Useful when feeding e.g. ML models where the position in the array is used to identify a source, rather than key/value mappings. | Array | Array of Values |
| Join | This is a variant of the Array Join module that ensures that data for a specified number of sources are always available in each output. If no data has been received for a specific source within the given time period an output will be generated based on the strategy selected. This module is more complex to set up but is useful in cases where you must ensure that each output has a value for each source, e.g. when feeding an ML model. | Messages | Array |
| Name | Description | Input | Output |
| Text Template | Generates text messages based on a template provided in the settings. Template syntax can be used to insert data from the message together with static text. | Message | String |
| String Replace | Replace a substring with another substring. | String | String |
| String Substring | Select a substring based on start and stop positions | String | String |
| CSV Line Parser | Break up a line of text (string) into substrings based on a delimiter, e.g. “,”. The output is an object with one property per substring. These properties are named based on the position in the string, e.g. col1, col2... | String | Object with substrings |
| CSV Text Parser | Same operation as the CSV Line Parser but operating on multi-line strings. The output is an array with one element per line. Each element has the same format as with the CSV Line Parser. | String | Array of objects with substrings |
Messages sent between modules use a custom .NET data type (FlowMessage) that supports dynamic structures with hierarchies of objects and arrays which can contain basic .NET data types. Sometimes, e.g. in the debug window, messages may look like Javascript objects (JSON) but that is only for presentation purposes. JSON is never used internally. When communicating with external systems you may get data in JSON or XML format, or the output is expected to be in one of these formats. To work with these types of data you must convert between the internal format and the external formats. To do that you use these conversion modules.
| Name | Description |
| JSON | Converts to and from JSON strings. If the input is a string it will be interpreted as JSON and the output is a FlowMessage. If the input is a FlowMessage it will be converted into a JSON string. This module operates on a selected property of the incoming message and the result can be assigned to a new property. |
| XML | Same as the JSON module but working with XML text. |
| Time Stamp | Convert DateTime values into another format. Predefined formats are ISO8601 and Unix timestamp in seconds or milliseconds. Custom output formats can also be defined. If no input is specified the module will add the current system time on the output. |
| Base64 Encode | Convert a binary array or a string into a base64 encoded string. For string inputs the output can be set to use base64url format, to create URL safe strings. |
| Base64 Decode | Decode a base64 encoded string into a byte array or string |
In addition to the below modules each module has a configurable message filter, which can be found on the “Common” tab. By default modules will process every message they receive. By adding a message filter the module will only process messages matching some criteria. Messages not matching the filter criteria can either be dropped, or passed on untouched.
Message filters can be set to match conditions on numeric property values (=, <, >), string property values (contains, does not contain, equal/not equal, isNull, isNotNull, regex) and boolean values (isTrue, isFalse). Multiple conditions can be used.
In addition to the message filters above the following modules can be used to control which messages that are passed on to the next module(s), based on different criteria.
| Name | Description |
| Range Filter | Only let through messages where a selected value is either within or outside of a specified range. The same effect can be achieved by using message filters. |
| Report By Exception | Only let through messages when the value of a specified property changes. Can be used to remove duplicates. |
| Split | Split a stream of messages into multiple paths by defining conditions on message data. Multiple conditions can be specified for each path. Each path will get a separate output on the module. |
| Throttle | Limit the rate of messages, e.g. only let through a maximum of one message per second. Can also be used to spread out messages in time by enforcing a minimum delay between messages, which can be used to smooth out bursty traffic. |
The following modules will create new data by processing the values in messages. They all work on numerical values and the result can either be assigned to a new property or the original value can be overwritten.
| Name | Description |
| Aggregate | Calculate average, min and max on a selected property from a group of messages. The group of messages to use can either be selected by specifying a time period, or a number of messages. Calculations can be performed independently for messages coming from different sources by specifying a property that indicates the source. At the end of each interval a message is delivered for each source with the calculated statistics. |
| Math Expression | Execute generic mathematical expressions on message data. The expression is specified using template syntax where message data can be inserted into the mathematical expression, e.g. ‘Abs({data.cur_temp} - {data.prev_temp})’. |
| Range Classifier | Add a text classification based on ranges of values on a selected property, e.g. ‘Low’, ‘Med’, ‘High’ |
| Scale | Scale a value on a selected property ‘(value*scale)-offset’, e.g. to convert temperature values from Farenheit to Celsius. Note: This operation can also be performed with the Math Expression module. |
| Smooth | Apply an exponential smoothing filter on a property value, e.g. to remove noise. |
| Statistics | Same as Aggregate but operating on a rolling window of messages, i.e. an output message is generated for each input message. This module will also calculate the standard deviation of the values. |
| Toggle | Toggle the value of a boolean property, or toggle the value of an internal variable and add to the output message |
With the code modules you can apply custom processing of messages using either C# or Python code. The code can either be entered directly in the settings UI, or code files can be uploaded to the Resource library and then referenced from within the modules.
| Name | Description |
| Csharp | Execute C# code. The code is compiled at runtime. Standard .NET libraries can be used but not 3rd party libraries. |
| IronPython | Run Python code using the IronPython interpreter which runs in .NET. Standard libraries can be used but not 3rd party libraries. Supports Python 2.7 code. |
| Python Bridge | Run python code in a standard Python 3.9 environment outside of .NET. On Windows the Python environment must be installed separately and the local node configuration must be updated accordingly (see installation docs). Standard and 3rd party libraries can be used, including ML frameworks. |
| Name | Description |
| Message Counter | Counts the values seen in a selected message property over a time period. At the end of the period the number of occurrences of each value is listed together with the relative count of each value. Useful for KPI calculations, such as Yield. If no source is specified the total number of messages received is reported. |
| Time Counter | Counts the time spent in different states by looking at the values received on a specified property and the time between value changes. At the end of the period the total time spent in each state (value) is reported together with the relative times. Useful for KPI calculations such as Availability. |
| Timeout | Measures the time since the last update on a selected property. If no update is seen within a specified timeout period a message is sent out. Use this module to monitor expected traffic patterns. For example, if you know that data should arrive every second this module can be used to trigger an alert if no update is seen within 2 seconds. By keeping track of individual property values the module can operate on multi-source streams and trigger an alert as soon as one a value is missing from one of the sources. |
Sometimes you need to store message data temporarily and then these modules will come in handy.
| Name | Description |
| Memory Buffer | This module will keep messages until acknowledged by an external signal. It is designed to work with any output module to keep messages until successfully delivered. Output modules produce a ‘crosser.success’ value indicating if the external delivery was successful or not. This value can be fed back to the Memory Buffer module. The size of the buffer as well as retry strategies can be defined. |
| State | The State module can store any message data, including complex structures, and add the stored value to all messages it receives. This can be used to store data that is updated infrequently but is needed on every message from another source. For example to add configuration data to each new sensor message. |