Introduction to Data Management at the Edge
In the era of connected devices and intelligent applications, data has become the cornerstone of innovation. With the radical growth of sensor networks, IoT devices, and AI systems, managing and processing data efficiently is more important than ever, especially in edge environments where data is generated at high volumes and requires immediate action. Traditional cloud-based infrastructures often fall short in meeting the demands of edge applications. This has led to the evolution of leveraged data management and fusion strategies tailored for the edge.
The Role of Data Fusion in Edge Computing and Key Benefits
Data fusion in edge computing refers to the process of integrating information from multiple data sources to generate more consistent, accurate, and useful insights. At the edge, this enables systems to make decisions locally and in real-time. By aggregating sensor data, log files, and performance indicators from operational processes, data fusion enhances the autonomous function of AI models and digital twins, with minimum reliance on a centralized infrastructure.
The key benefits can be summarized below:
- Real-Time Insight Generation: Edge data management systems allow the ingestion, processing, and transformation of data the moment it is collected.
- Resilient and Scalable Architectures: By building the system in separate, well-defined parts that follow common communication standards, each edge device can work on its own or with others, making the system more reliable and easier to grow as new devices or features are added.
- Flexible Integration with Diverse Data Sources: Edge-based systems can combine different types of data, like detailed reports describing problems (such as FMEAs) and numerical data from process controls (like APC or SPC), even if they come in different formats or from different sources.
- Privacy-Preserving Design: Processing sensitive data locally minimizes exposure to potential threats and aligns with regulatory frameworks on data security and country-specific regulations.
- Continuous Learning via Federated Architectures: Edge nodes can collaboratively improve AI models using federated learning, enhancing performance over time without centralizing raw data
GNT’s Contribution to Edge-Centric Data Fusion
Within the EdgeAI project, GNT is developing specialized data fusion mechanisms designed to address the specific needs of diverse use cases, ensuring effective and intelligent data processing at the edge.
Industrial Object Recognition.
In a partnership with Cognition Factory that produces devices for industrial object recognition, GNT provides a fusion mechanism that combines inputs from multiple readings from on-board sensors and AI-based classifiers into composite, streamlined messages. Such messages from multiple recognition devices deployed in an industrial environment are then dispatched to another, more centralized fusion mechanism that persistently stores all recognition information for subsequent AI model training, monitoring, post-analysis and longitudinal study purposes. By merging their data locally before sending it for centralized training and analysis, the solution efficiently handles vast amounts of information, supporting scalable and accurate recognition capabilities.
Air Quality Forecasting. In urban air quality monitoring GNT works closely with ITML, EDI and IMST to produce a comprehensive system for air quality monitoring and forecasting in the city of Riga. Data fusion takes place on two levels. Roadside units first collect sensor readings from multiple distributed sensors via a Lora mesh network and integrate them locally, enabling rapid and precise assessment of air quality in their immediate area. These local insights are then aggregated at a higher level, where data from multiple city locations are combined to create a comprehensive and real-time overview of the city’s air quality. This layered approach balances quick local responsiveness with broad situational awareness.
Drone-based logistics operations.
GNT has partnered with Neurocontrols to support logistics operations involving UAVs. In this case, data fusion focuses on pre-flight condition monitoring and asset detection. Sensor data from drone components are merged to provide a clear picture of their health status, supporting timely alerts and maintenance decisions to ensure safe and reliable flights. At the same time, drones deliver identification information of logistics assets that they detect in their proximity, which are then fused to enable a real-time visualization of detected asset information, supporting logistics operation monitoring and comprehensive situational awareness.
Across all these scenarios, these fusion mechanisms allow for smart local processing alongside centralized aggregation. This enables real-time analysis, visualization, and informed decision-making, aiming towards the full potential of edge computing.
How GNT’s Edge Data Management Systems Works
In EdgeAI, GNT has developed a layered, flexible data management architecture based on the pub/sub messaging pattern that allows loose coupling with services that can process the data close to where it is generated.
On resource-constrained devices positioned very close to data sources, GNT employs a lightweight technology stack involving MQTT, coupled with InfluxDB for persistent storage of time series data. In cases where another level of data aggregation is required (e.g., to combine data from multiple devices on a factory floor or from multiple buildings in a city area) on less constrained edge devices, GNT’s system can provide a more robust data fusion layer, based on a technology stack that involves Apache Kafka, coupled with OpenSearch for persistent storage.
GNT complements this two-layered data management backbone structure with various add-on tools, functions and services on a case-by-case basis, including pub/sub clients, data modelling and validation, data ingestion and data transformation pipelines, synthetic data generation and dataset playback, proxy gateway services, REST API communications and data visualization. Even in cases where data stays primarily in edge environments, security risks still exist, and cyber-security is of paramount importance in all data management operations and communications. For this reason, GNT implements a zero-trust security-by-design approach, involving authentication and mutual TLS encryption based on X.509 certificates in all data transfers, coupled with fine-grained Role-Based Access Control (RBAC) policies delivered via a thorough management of Access Control Lists (ACLs).
The data management architecture is deployed using container orchestration platforms, allowing seamless updates and scalability. This setup supports collaborative learning, where edge devices improve AI models collectively while keeping sensitive data local and secure.
Conclusion
Comprehensive data management constitutes the foundation of efficient and intelligent edge computing. By combining flexible middleware, advanced data processing, dynamic databases, and seamless AI integration, the EdgeAI project showcases how edge systems can become capable of managing, interpreting, and learning from data right where it’s generated. GNT’s contribution in developing powerful data fusion mechanisms highlights the future of edge computing: systems that are adaptable, deliver real-time insights, and continuously grow smarter to meet tomorrow’s challenges.
Blog signed by: GNT team