Data Integrator (DI)
At a Glance
Section titled “At a Glance”
- Category: Data & Infrastructure
- Primary environment: MaaS Consumer, MaaS Provider
- Documentation focus: Design reference
- Maturity: TRL 4–5 (first release, M18)
- Related architecture docs: Reference Architecture Guide, Environments
Purpose
Section titled “Purpose”The Data Integrator is a middleware solution designed to enable seamless communication between ERP/MES systems and Edge Data Storage within the MaaSAI ecosystem. It ensures that information managed by enterprise applications can be reliably transferred, transformed, and stored at the edge, where production data is collected and utilised.
Primary Actors
Section titled “Primary Actors”The D5.2 usage viewpoint defines roles, permissions, constraints, a role-task matrix, and a use case diagram for this solution.
- External system: Provide continuous production schedule and execution data to the toolkit. Receive and update information coming from MaaSAI while keeping synchronisation between systems; constraint: Access is limited to the local data domain, requires authentication, and must use supported protocols or data formats.
- External system manager: Provide the information needed to develop API mechanisms for external systems. Check synchronisation between the MaaSAI platform and external systems and investigate errors when they occur; constraint: Cannot modify Data Integrator flows.
- Integration administrator: Configure and update integration flows. Check connection status and data flow. Manage DI control tools; constraint: This role is restricted to authorised personnel, and configuration changes must respect security policies.
Use-case diagram extracted from MaaSAI deliverable D5.2.
Functional Components
Section titled “Functional Components”DI links ERP/MES connectors, parsing services, message brokering, and edge-database integration so operational data can be harmonised across MaaSAI solutions. The functional view shows the ingestion path from enterprise systems to edge storage and the connector services that mediate that exchange.
Functional components diagram extracted from MaaSAI deliverable D5.2.
Design Baseline
Section titled “Design Baseline”At the heart of the Data Integrator lies Node-RED, the enabling framework for building lightweight, scalable microservices. Node-RED flows are used to implement the ERP/MES Connector, the Data Parser and Transformer, and the Edge Data Microservice. Each of these encapsulates specific functional responsibilities. Thanks to Node-RED’s visual programming model and extensive library of connectors, it is possible to quickly integrate with heterogeneous ERP/MES systems, parse different data formats (e.g. JSON, XML and iDoc) and prepare queries for SQL-based storage, all without the need for custom monolithic development.
A key element of the architecture is the MQTT broker (Mosquitto), which introduces decoupling, buffering and security to the data exchange pipeline. Rather than creating direct, synchronous dependencies between the ERP/MES system and the edge data storage system, all intermediate communication flows through MQTT topics. This enables messages to be queued and reliably delivered, even in the event of network disruption, while also allowing multiple subscribers (e.g. monitoring services or analytics engines) to access the same data streams. Security is reinforced by enabling Transport Layer Security (TLS) for encrypted communication and applying authentication mechanisms at the broker level.
These components, Node-RED for integration and transformation, and MQTT for secure and decoupled communication, allow the Data Integrator to act as a robust, extensible bridge between business systems and edge infrastructure.
The Data Integrator is a middleware component designed to bridge ERP/MES systems with edge data storage in the MaaSAI ecosystem, ensuring reliable, secure, and decoupled data exchange. Its architecture, as illustrated in the diagram, is built around Node-RED, which implements microservices for parsing, transforming, and preparing data for storage. Incoming messages from ERP/MES systems (in formats such as XML, JSON, or iDoc) are processed by the Data Parser Microservice, then routed through the Mosquitto MQTT broker. The broker provides buffering, decoupling, and secure communication using TLS and authentication. This approach circumvents the necessity for direct synchronous dependencies, thereby facilitating the consumption of the same data streams by multiple subscribers. Finally, the Edge Database Connector, also implemented in Node-RED, formats and executes upsert queries to the Edge Data Storage (MySQL), while exposing data to the MaaSAI backend via HTTP. All components are containerised using Docker Compose, ensuring scalability and easy deployment across heterogeneous provider infrastructures.
Architecture diagram extracted from MaaSAI deliverable D5.2.
Technology Stack
Section titled “Technology Stack”The DI is implemented as a lightweight, containerised middleware built around the Node-RED runtime. All components are open source, allowing free commercial use without paid licences.
| Category | Technology |
|---|---|
| Backend | Node-RED (Node.js) as the central middleware, orchestrating bidirectional data flows, time-based ERP/MES polling, and HTTP requests towards the EDGE Data Storage. |
| Transformation | JSONata and JavaScript within Node-RED, interpreting heterogeneous formats (XML, JSON, CSV) and producing standardised MaaSAI JSON payloads. |
| Communication | REST (HTTP/HTTPS) as the primary synchronous protocol; an embedded MQTT broker (node-red-contrib-aedes) is planned as a future asynchronous decoupling layer. |
| Containerisation | Docker and Docker Compose. |
Integrations & External Dependencies
Section titled “Integrations & External Dependencies”The DI bridges external legacy ERP and MES systems with the MaaSAI EDGE Data Storage (DS). It ingests API responses and structured files (CSV, JSON, XML, iDoc) from enterprise systems, harmonises them in line with the ISA-95 standard, and writes the results to the DS through REST calls. The solution has no major external dependencies: the host only needs Docker installed and appropriate network access (firewall/VLAN) for HTTP/HTTPS traffic. Keycloak integration for authentication is planned for a later release.
APIs & Interfaces
Section titled “APIs & Interfaces”As a backend microservice, the DI exposes no graphical user interface; the Node-RED editor provides a visual representation of flows and payloads. Its RESTful endpoints control the integration pipeline:
POST /sync/erp-orders— triggers the polling flow that extracts new production orders from the ERP within a date range.POST /webhooks/erp/orders— ingestion endpoint for production orders pushed directly by the ERP.POST /webhooks/erp/progress— ingestion endpoint for order progress updates (order id, status, produced quantity, timestamp).
User Interface
Section titled “User Interface”DI is operated through its Node-RED integration-control environment:
Node-RED backend orchestration with the time-based polling mechanism — MaaSAI deliverable D8.1.
Sequential execution flow creating Material, Operation, and Request entities — MaaSAI deliverable D8.1.
Deployment & Configuration
Section titled “Deployment & Configuration”Deployment is entirely based on Docker and Docker Compose: the whole solution is packaged as a Docker image rather than installed on the host. The host needs Ethernet HTTP/HTTPS access and, ideally, strictly defined firewall/VLAN routing.
| Requirement | Minimum | Recommended |
|---|---|---|
| CPU | 2 vCPUs | 4 vCPUs |
| RAM | 2 GB | 4 GB |
| Storage | 10 GB | 20 GB SSD |
| OS | 64-bit Linux (Ubuntu/Debian) or Windows Server 2022 | 64-bit Linux (Ubuntu/Debian) or Windows Server 2022 |