Monitoring the Crosser Node

Monitoring the Crosser Node

Introduction

Once you have your first flows deployed, you might think about how to integrate the Crosser Node and Flows into your existing monitoring solution.
In this article we describe what options you have and how to utilize provided interfaces for integration into your monitoring solution.

Local log files

The logs of the Node are stored locally and roll-over once per day assuming they do not reach the size limit. A timestamp is added to the log name to make it easy to identify.
You can find the logs in the installation folder of your Node ./data/logs.
We decided to create one log per process running on the device which results in one log for the Node itself ./data/logs/host and separated logs for each flow ./data/logs/runtime/<Flow ID>. Assuming you had executed Remote Sessions at some point on that Node, you will also see a folder ./data/logs/runtime/remotesession/<Flow ID>.
Since you most likely do not want to integrate this one into your monitoring solution, we leave this folder out of scope.
Once you have located the logs you can use third-party tools like 'datadog' or 'filebeat' to catch and upload the logs into your monitoring solution.

Node's API

General information

The Node offers a local API interface (version 2.5.x and above) which you can use to extract information about the Node, Flows and Modules.
A documentation about this API is integrated into the Node. The API is exposed to port 9191 (default) for both Windows and Docker installations.
Assuming your port is exposed and not blocked by a firewall, you should be able to access your Node's local API and documentation via port 9191.
i.e. http://<Your Node IP>:9191/swagger/index.html

You should be able to access the Web UI with any browser:

Once you know that the API is accessible, check the endpoints and think about what information you would like to extract.
You can use the exposed Web UI to run API commands immediately to see how it works.

Now it is up to you how to utilize the API.


Consume the API from external systems

Assuming your monitoring solution has access to the local Node's API, you can think about implementing them directly (if your monitoring system does support that).
This approach is often seen in environments where customer have monitoring solutions on-premise or agents of their monitoring solution that are capable of consuming APIs.



Create a monitoring flow

Another way to consume the local API is to create a Flow that uses the API itself. You can simply use the HTTP Request module to call the API and extract the relevant information with the low-code approach. Once you have done that you can react on certain events and send out notifications to external Brokers, APIs, HTTP endpoints or notify a user with the extracted and relevant information.
With this setup you would push the data out to external systems. This is useful if your monitoring solution does not have access to the Node's local API but the Monitoring System's endpoint is exposed so we can i.e. use HTTP Request module to push data to it. You can deploy this flow on the same Node as your other flows are running.


The downside of this implementation though is that if the entire Node fails for some reason, also your monitoring flow will die.
To get around that, you can think about setting up a dedicated Node which only runs monitoring flows in a dedicated environment.
This setup can also help bridging network gaps between the Nodes (production network) and a monitoring system (office network/cloud).


MQTT (Node Version 2.6)

With Node 2.6 Crosser has introduced the possibility to utilize the Node’s internal MQTT Broker to get events when a flow/module status/state changes. This allows you to create an event driven notification architecture with external monitoring tools.

Similar to the HTTP approach, you can either connect to the Node’s local broker from an external MQTT client, or create a flow which subscribes to the relevant topics and send out notification in case something goes wrong.
The Node will publish messages when things occur that someone might want to know. The root topic for all the messages will be $crosser/. So for example $crosser/flows/{flowDefinitionId}/started will be published when a flow starts.

The $crosser topic will be forbidden by clients to publish to, only subscriptions are allowed. If a client tries to publish with a topic that starts with $crosser/ the client will be disconnected.

Note that MQTT Notifications are disabled for remote sessions. Only deployed flows will trigger MQTT Notifications.

Flow Topics

Topic: $crosser/flows/{flowDefinitionId}/state [RETAINED]
When a flow changes state (started, stopped) this topic will be triggered with the information:

{

    "flowId": "2220e7a7-9e98-4af3-afd4-a1b309128540",

    "flowName": "MQTT Notifications",

    "version": 9,

    "running": true,

    "status": "Ok",

    "reason": "Flow started",

    "origin": "Cloud",

    "timestamp": "2023-07-04T10:40:47.601+00:00",

    "nodeId": "27929e60-a586-4f51-a512-0583391b0c86",

    "nodeName": "Machine123",

    "timestamp": "2023-09-28T13:22:51.743+00:00"

}


Topic: $crosser/flows/{flowDefinitionId}/status [RETAINED]
When a flow change status to Ok, Warning or Error:

{

  "flowId": "2220e7a7-9e98-4af3-afd4-a1b309128540",

  "flowDefinitionId": "6faf4080-c7ce-e642-37a0-43e965904899",

  "flowName": "Example MQTT Notifications",

  "version": 9,

  "status": "Warning",

  "reason": "Flow status changed to Warning",

  "reasons": [

    {

      "moduleId": "9c3727c6-7d64-4cb3-95eb-a597596068b2",

      "moduleName": "OPC UA Reader",

      "moduleType": "Crosser.EdgeNode.Modules.OPC.UA.Reader",

      "version": "2.3.0",

      "status": "Warning",

      "reason": "Module OPC UA Reader, 2.3.0 changed to status Warning: This is a test",

      "timestamp": "2023-09-28T13:22:51.743+00:00"

    }

  ],

  "nodeId": "27929e60-a586-4f51-a512-0583391b0c86",

  "nodeName": "Machine123",

  "timestamp": "2023-09-28T13:22:51.743+00:00"

}


Topic: $crosser/modules/{moduleId}/status [RETAINED]
When a module changes status (Ok, Warning, Error):

{

    "moduleId": "9f5f82a2-1dc0-433d-805e-182be417fdef",

    "moduleName": "Queue & Persistence",

    "version": "2.3.0",

    "status": "Warning",

    "flowId": "2220e7a7-9e98-4af3-afd4-a1b309128540",

    "flowName": "MQTT Notifications",

    "reason": "Module Queue & Persistence, 2.3.0 changed to status Warning: Queue is full  (100), message was dropped, fullMode=DropWrite, persistent=True",

    "queue": {

        "mode": "DropWrite",

        "size": 100,

        "retries": 3,

        "retryDelay": 100,

        "delay": 0,

        "persistence": true

    },

    "nodeId": "27929e60-a586-4f51-a512-0583391b0c86",

    "nodeName": "Machine123",

    "timestamp": "2023-09-28T13:22:51.743+00:00"

}


Topic: $crosser/modules/{moduleId}/messages/dropped

When the module queue is full and the strategy is not wait.

{

    "moduleId": "9f5f82a2-1dc0-433d-805e-182be417fdef",

    "moduleName": "Queue & Persistence",

    "version": "2.3.0",

    "status": "Warning",

    "flowId": "2220e7a7-9e98-4af3-afd4-a1b309128540",

    "flowName": "MQTT Notifications",

    "reason": "Queue is full",

    "queue": {

        "mode": "DropWrite",

        "size": 100,

        "persistence": {

            "enabled": true,

            "retries": 3,

            "retryDelay": 100

        }

    },

    "nodeId": "27929e60-a586-4f51-a512-0583391b0c86",

    "nodeName": "Machine123",

    "timestamp": "2023-09-28T13:22:51.743+00:00"

}


Topic: $crosser/modules/{moduleId}/messages/deadletters

When persistence is on and a message has failed maximum number of times it will be moved to the deadletter queue. You will then get a message similar to this:

{

    "moduleId": "9f5f82a2-1dc0-433d-805e-182be417fdef",

    "moduleName": "Queue & Persistence",

    "version": "2.3.0",

    "status": "Ok",

    "flowId": "2220e7a7-9e98-4af3-afd4-a1b309128540",

    "flowName": "MQTT Notifications",

    "reason": "Retries maxed out",

    "queue": {

        "mode": "DropWrite",

        "size": 100,

        "persistence": {

            "enabled": true,

            "retries": 3,

            "retryDelay": 100

        }

    },

    "link":     "http://localhost:9191/api/flows/2220e7a7-9e98-4af3-afd4-a1b309128540/deadletters/043e9604-837d-41b7-8a63-ad85321bfe32",

    "nodeId": "27929e60-a586-4f51-a512-0583391b0c86",

    "nodeName": "Machine123",

    "timestamp": "2023-09-28T13:22:51.743+00:00"

}

Subscribing

  • By subscribing to $crosser/# you would get all messages
  • Subscribing to $crosser/flows/# would get you all flow messages
  • Subscribing to $crosser/modules/# would get you all module messages
Of course you can also use other combinations and use + as a single level wildcard
  • Subscribing to $crosser/flows/+/stopped would get you only stopped flow messages
  • Subscribing to $crosser/modules/+/status would get you module status messages for all modules
Get the current status for all deployed flows. When a MQTT client creates a subscription for the topic $crosser/flows/# or $crosser/flows/+/status the broker will send out the current status for all deployed flows.

    • Related Articles

    • Crosser Node 2.5.0 - Major update

      Crosser Node 2.5.0 - Major update October 20, 2021 New architecture This new version is primarily an internal architecture change, where flows are now even more separated by running as individual processes. This first version is designed to be ...

    • Crosser Node 2.5.1

      Crosser Node 2.5.1 October 29, 2021 This is a minor update with some changes that didn’t make it into the main release. Improved API documentation The built-in swagger UI now provides more details about each endpoint. Release-notes in UI Since v2.5.0 ...

    • Crosser Node 2.6.0

      Release Note Release date: 2023-06-29 Note: A critical bug was found in 2.6.0 and is now deprecated, use 2.6.1 instead. New features Retry on all modules Each module now has settings for Retry (Common Settings). When the Max Number Of Retries setting ...

    • Crosser Node as Azure Container App

      Introduction Beside running the Crosser Node on a stand-alone Docker setup, or using a container orchestration system like Kubernetes/OpenShift, you can also run the Node as an Azure Container App in your Azure infrastructure. This gives you the ...

    • Crosser Security Overview

      Crosser Security Overview November 2023 About this document This document describes security aspects related to the Crosser Streaming Analytics solution. The Crosser Streaming Analytics solution The Crosser Streaming Analytics solution has two main ...