Message Queuing For Industrial Automation
Author: Ambar Prajapati
The industry has evolved enormously since the formal introduction of AMQP in 2006. The ISO and IEC approved OASIS AMQP Advanced Message Queuing Protocol came into existence in May 2014. Many practical implementations of message broker-based solutions are now available. This article discusses the offerings of AMQP and provides details of the RabbitMq as its implementation.
Research Question
There was a growing need for a standard middleware protocol with the existence of several proprietary systems. The research question is — which protocol fills that void and gives the most compelling example of its industrial implementation. This review paper attempts to demystify the crucial role of middleware in seamless data flow for industrial automation. It discusses AMQP as the significant backbone for inter-process data flow among diverse, distributed, and heterogeneous systems. It discusses RabbitMq as a successful technology that implements AMQP.
Research Gaps
The earlier research does not capture the recent advances in AMQP, such as its reorganization as an OASIS member. The previous studies also do not provide insight into many middleware challenges and the industry response to them.
Introduction
Industry 4.0 represents major technology shifts towards the fourth industrial revolution. Many engineering feats have shaped this revolution, such as — the Internet of Things (IoT), smart sensors, robotics & automation, data & analytics, augmented and virtual reality, 3D printing, wearables, safety innovation, cloud, cybersecurity, edge computing, etc. The focus is to seamlessly integrate data sources to achieve interoperability, autonomy, and decentralized control. Thus, a middleware gets center stage enabling seamless data flow into the distributed architecture of heterogeneous devices that operate over a commonly accepted message communication protocol. (Jasperneite, 2020, p. 30) The standard message protocols allow the flow of complex data sets over diverse sub-systems. A formal architecture is often required to connect processes for a hierarchy of data flow. We need to layout a data transportation framework using well-defined data models. There are various messaging styles technically feasible among the systems: publish/subscribe, transactions, request/response, one-way messaging, and store-and-forward. The most common is the publish/subscribe style of communication. The underlying data flows via the TCP layer. Large volumes of data can not flow over TCP unless serialized. The popular data serialization methods include — XML, JSON, message pack, and protocol buffers.
Advanced Message Queuing Protocol (AMQP)
The Advanced Message Queuing Protocol — AMQP is an open standard for Message Oriented Middleware (MOM) communication. (AMQP) The OASIS AMQP offers an open, secure, global standard for passing data streams in real-time over the cloud and in point-to-point messaging environments. (ISO and IEC Approve OASIS AMQP Advanced Message Queuing Protocol.)
The AMQP fits the enterprise scenarios requiring uncompromised reliability, performance, throughput, and scalability. For example, consider the AMQP’s role in the strict messaging requirements for financial trading or banking systems, where the delay for a fraction of a micro-second or the loss of data is hardly imaginable. (Vinoski, 2006, p.88)
AMQP provides a functional and transport layer and is well suited for high throughput peer to peer networks where fault-handling is mandatory. AMQP supports large volumes of data and provides message back-up and reprocessing, which HTTP does not offer. The fault-tolerant nature of the AMQP is a step ahead of RESTful HTTP web-services that expect the receiver to be alive at the time of the request; otherwise, the message is lost. The AMQP also beats HTTP while processing large volumes of data in a relatively shorter time. (Fernandes, 2013, p.815)
The message in a MOM (message-oriented middleware) is an independent encapsulated data entity with a header and a body. The AMQP based messaging covers all scenarios in a messaging system, e.g.,
Guaranteed response — The response is provided, followed by a request
Delivery acknowledgment — Response not sent, but the delivery confirmation is provided
Reliable message delivery — At-least-once, exactly-once, or at-most-once; sometimes with no response and no delivery confirmation
Prioritization — Assigning and delivering messages as per priority
The senders/receivers need not be active during actual communication. The broker system provides a fallback mechanism to retain messages in the queue when the recipients are offline. The retained messages are delivered when the recipient service goes live again — thus enabling fault tolerance. The acknowledged messages are removed from the queues. A queue can receive messages from multiple publishers. And a publisher can send messages to various consumers. The messages are stored with the queues in memory or on the physical disk. The queue settings define the life-time of a message. The whereabouts of a message are known while it is traveling physically from one system to another. The message routing is done by the Message Exchange between the publishers and consumers. The queue name serves as the address of the sender/receiver service. The sender has the Routing Key while the receiver has the Binding Key of the message. The keys hook the queues with the Exchange.
Depending upon how the messages need to be received, the exchanges are of four types -
Direct Exchange: The sender knows the recipient’s full queue name and sends the message to that queue.
Topic Exchange: Similar to direct but allows the sender to send the message to any queue that matches the regular expression, e.g., company_name.*.queue matches company_name.managers.queue and company_name.executives.queue
Fanout Exchange: The sender intends to send the message to all the bound queues.
Header Exchange: The sender uses AMQP message structure to send the message with additional metadata for -
§ Attaching priority, urgency, and expiry information to the messages
§ Clustering — combining several message brokers for further scaling or the Load balancing in the system
§ Use plug-ins for additional tasks, e.g., the management, authentication and authorization, back-up solutions, etc.
RabbitMq
RabbitMq is an open-source AMQP/MQTT message broker built over Erlang OTP and can support massive messaging for concurrent operations. It is the same Erlang that powers WhatsApp to transport billions of messages every day. (Rostanski, 2014, p.880) The RabbbitMq acts as a secure message transfer bus, generally over TCP/IP, for distributed and asynchronous systems. It uses a push model for the parallel distribution of messages.
Some of the salient features of RabbitMq are -
— Provides fine control over consistency or a set of guarantees on a per-message basis
— Allows configuring complex routing rules
— Allows mix-match messaging scenarios such as — publish/subscribe, point-to-point, request/response, synchronous/asynchronous, etc.
— Supports protocols including MQTT, AMQP, WebSockets, and STOMP
— Supports secure TLS/HTTPs/Certificate-based authentication further extensible with the use of plug-ins
— Provides API and CLI tools for monitoring, audit and application troubleshooting
It has well-supported client libraries, dev-tools, and documentation. The available plug-ins open up even more communication possibilities.
The RabbbitMq’s performance depends on the Exchange on the publisher’s side and the consumer side’s queues. A careful setup is required to achieve high performance and availability from RabbitMq. The general use case is three to seven RabbitMq nodes for excellent functioning — however, more resources (e.g., 30 nodes) are required when processing a million messages per second. There are recommended best practices to achieve optimum performance. (Part 1: RabbitMQ Best Practices)
RabbitMq for IoT: IoT systems can publish and consume messages using the MQTT plug-in in RabbitMq. It can also serve as a gateway for AMQP services to the IoT devices.
Future work.
Apart from Industrial automation, the concept of smart connected cities can be materialized by connecting internet-ready devices using data annotations/ domain language in real-time. For example, implementing information flow using APIs/AMQP/MQTT connects public services such as train/airport, mobile phones, sensors, and data exchange.
The multi-topic messaging is required for low powered IoT devices. There are many IoT protocols for IoT systems, including MQTT, MQTT-SN, AMQP, DDS, CoAP (HTTP based), and SOAP, but MQTT and CoAP are the most common protocols. They can also utilize the Google Cloud Publish/Subscribe system. Any Internet connection, such as Wi-Fi, Ethernet, General packet radio services (GPRS), and 3G or 4G data connections, can be used to communicate between IoT nodes.
References:
J. L. Fernandes, I. C. Lopes, J. J. P. C. Rodrigues, and S. Ullah. “Performance Evaluation of RESTful Web Services and AMQP Protocol.” 2013. doi:10.1109/ICUFN.2013.6614932.
Jasperneite, Juergen, Thilo Sauter, and Martin Wollschlaeger. “Why we Need Automation Models: Handling Complexity in Industry 4.0 and the Internet of Things.” IEEE Industrial Electronics Magazine 14, no. 1 (2020): 29. http://ezproxy.memphis.edu/login?url=http://search.ebscohost.com/login.aspx?direct=true&db=edb&AN=142397421&site=eds-live&scope=site.
M. Rostanski, K. Grochla, and A. Seman. “Evaluation of Highly Available and Fault-Tolerant Middleware Clustered Architectures using RabbitMQ.”2014. doi:10.15439/2014F48.
P. Sommer, F. Schellroth, M. Fischer, and J. Schlechtendahl. “Message-Oriented Middleware for Industrial Production Systems.”2018. doi:10.1109/COASE.2018.8560493.
S. Vinoski. Advanced Message Queuing Protocol. Vol. 10 2006. doi:10.1109/MIC.2006.116.
“AMQP.” Accessed October 18, 2020. https://www.amqp.org/.
“ISO and IEC Approve OASIS AMQP Advanced Message Queuing Protocol.” OASIS. Accessed October 18, 2020. https://www.oasis-open.org/news/pr/iso-and-iec-approve-oasis-amqp-advanced-message-queuing-protocol.
“OASIS Advanced Message Queuing Protocol (AMQP) TC.” OASIS. Accessed October 18, 2020. https://www.oasis-open.org/committees/tc_home.php?wg_abbrev=amqp.
“Part 1: RabbitMQ Best Practices.” Accessed October 18, 2020. https://www.cloudamqp.com/blog/2017-12-29-part1-rabbitmq-best-practice.html.