What is MQTT?

MQTT provides a networking system for IoT devices. Find out how IoT devices can communicate.

MQTT stands for Message Queuing Telemetry Transport. It is a communication standard for IoT devices. This protocol is an open standard, which means anyone can access it and implement it in a networked application.

The system of MQTT is not a software package but it can be used by the developers of new software as a standard to send messages to IoT devices and receive responses back. This opens up the possibility of writing new control instruction systems and also creating IoT device monitoring tools.

The MQTT system was first created in 1999, so it has been around for more than two decades. It was written as a standard to communicate with oil pipeline equipment. As an industrial communication system, MQTT was written within the framework of the Supervisory Control and Data Acquisition (SCADA) framework. The SCADA architecture is designed to communicate with shopfloor equipment in factories and sensors and controllers in process control.

Computerized things

Industrial systems were the forerunners of IoT. These are machines and devices that have computers in them without the computer being “the thing.” Onboard chips are now embedded in bigger things that consumers use, such as cars. They are part of the components of larger things. So, there are larger things that are made up of a group of little computerized things.

Those little hidden computers don’t have keyboards or monitors. They don’t look like computers and no one ever thinks of checking on their statuses. In industrial machinery, the computerized parts aren’t accessible by the operator; they are remotely managed by technicians in the office. Similarly, the driver of a car gets some feedback from some of the chips embedded in the car. Many of those embedded chips can only be interacted with through temporarily connected diagnostic tools, which provide the front-end that the normal operations of the chips do not need.

“Smart” devices remove the need to attach an interface for the brief periods of maintenance by connecting those embedded systems to a network, allowing remote control. Since MQTT was first conceived, the Internet has become a much cheaper and widely available communication medium.

The benefit the internet brings to industrial applications means that remote design offices and management departments can connect directly to shopfloor devices from a remote location. Embedded systems in consumer products can now be updated and managed in bulk by the producer. Smart devices all over the world can be automatically monitored and updated from one central office.

The combination of embedded systems with the Internet brings down production costs and enables producers to offer stronger and longer performance guarantees – improving reliability and sales. The assumption that millions of devices can be communicated with requires a communication standard. MQTT was already on hand to provide that service.

What does MQTT define?

Message Queuing Telemetry Transport is a bit of a misnomer. The protocol doesn’t lay out a method of transporting communications and it doesn’t implement queueing. The system assumes that all connection management is carried out by TCP/IP. MQTT is an application that is carried over the Internet on connections that have already been established by other systems. Think of it as a cross between HTTP and a mail server.

The participants in an MQTT session are all clients and their contact is mediated by a Message Broker. So, IoT devices are never contacted directly. The device posts messages to the Message Broker, where they are forwarded to devices that have subscribed to that message stream. Messages can also be stored at the Message Broker.

The standard also defines three message types. These are Connect, Disconnect, and Publish. The Connect message type is sent by the client to the broker, and the Disconnect message type is an “end of session” confirmation sent by the broker to a connected client. A Publish message type starts with a client and sends data to the Message Broker. This message structure is also sent by the broker to clients when it is forwarding information.

The Publish message is the main means of transmitting the information. There is greater detail in the MQTT standard about how this message is structured.

MQTT Publish message

The main structure of the MQTT standard is the Publish message. The fields in a Publish message are:

PacketID	A unique identifier for each message
TopicName	A hierarchical string
QoS	Quality of Service (QoS) rating
RetainFlag	True or False
Payload	The value being notified
DupFlag	True or False

The standard doesn’t rigidly define layouts for the content of each field in the Publish message – they are left up to the decision of the developers of applications. However, there are some conventions that are always used.

The hierarchy in the TopicName is usually delimited by slashes (“/”) and the Payload field usually contains a label and a value, separated by a colon, such as “temperature:31” with the inclusion of a unit of measure down to the developers.

The PacketID does not need to be universally unique. The use of a sequence can be useful to indicate a series of values and enable the receiving application to order incoming values. However, there is no requirement within the standard for the uniqueness of this value.

The PacketID and QoS fields are numeric with a list of possible values for the QoS field. The QoS field denotes a guarantee of delivery. This can have a value of 0, 1, or 2.

The RetainFlag and DupFlag fields are typed as Boolean variables and don’t need quotes around their values of true or false.

MQTT topics

The TopicName field in the Publish message can be seen as a “thread.” It is the anchor of the entire messaging system because it is the topic that ties together senders and recipients. Applications that want to receive information from a remote device “subscribe” to a topic. The mechanism of forwarding MQTT Publish messages works in much the same way as multicasting is implemented on routers.

Rather than being a category of data, the topic indicates the source of a message, such as a sensor. An example of a topic would be “Building1/Lobby/Sensor5.” It is typical for topics to include a hierarchy so that the application that receives data through MQTT can easily group statistics and roll up data.

Publish message Payload

The data being conveyed in the Publish message is held in the Payload field, so the combination of the Topic and the Payload contains all of the information that an interested application wants to receive. Rather than having a separate field for the label of the data, both pieces of information are contained within a single field, separated by a colon.

So, while the Topic field tells the receiver where the data has come from, the Payload gives the type of data and the current reading.

Publish message QoS

The Quality of Service (QoS) field in the Publish message dictates how many delivery attempts should be made. It can be set to 0, 1, or 2.

• QoS 0 means “at most once.” This offers no guarantee of delivery. This isn’t a connectionless transmission because the underlying services of TCP ensure that the message is actually received both by the Message Broker and the subscribers. However, there is no receipt confirmation organized within the MQTT system for this category of QoS.
• QoS 1 denotes “at least once.” That signifies that the receiver sends back a receipt confirmation. The Publish message will be resent if the client doesn’t receive back a PUBACK message. A resent message has its DupFlag field set to True.
• QoS 2 signifies “exactly once.” It has the highest overhead of all QoS options and gives the strongest delivery guarantee. It includes a four-way handshake. First, the client sends the Publish message, to which the Message Broker responds with a PUBREC message. If the sender doesn’t receive a PUBREC message, it resends the Publish message with DupFlag set to True. The sender will keep resending the Publish message until it gets a PUBREC message back. Once it gets a PUBREC message, the sender sends a PUBREL message, to which the Message Broker sends a PUBCOMP message.

The Publishing process is the same both when a message is sent from a client to a Message Broker and when a message goes from the Message Broker to clients. The QoS value in the same message does not have to have the same value when it is forwarded on by the Message Broker to other clients. The client sets the QoS level for all transmissions.

MQTT subscriptions

The MQTT Message Broker forwards incoming messages to one or many clients. The broker maintains a subscription list for each topic. The client gets onto a subscription list by sending a SUBSCRIBE message to the Message Broker. The subscription message format includes the following fields:

PacketID	A unique identifier for the message
TopicName	The identifier of the topic to be subscribed to
QoS	The QoS level expected by the client for the paired topic

The Subscribe message can contain multiple TopicName/QoS pairs. So, it is possible for the client to subscribe to many lists with one Subscribe list. The QoS specified can be of a different level for each topic. Whenever the Message Broker forwards on a message, it uses the QoS specified by this Subscribe message.

An MQTT transaction

The PacketID used in each message only needs to be unique per client, so messages traveling to and from the Message Broker can have the same PacketID if they are for different clients. Once a transaction has been completed, with all acknowledgments over, the PacketID can be reused. The PacketID on a forwarded message does not have to be the same as the PacketID on the original Publish message sent to the Message Broker.

All clients have to be permanently connected to the Message Broker in TCP/IP sessions in order to participate in the scheme. If the Message Broker receives a message on a topic for which there are no subscribers, it drops it.

There is no ad-hoc query method in the MQTT protocol. So, a client can’t send a demand for information. The standard is a one-way system. So, a system that requires data from a series of IoT devices for constant monitoring needs to set up a process to trigger the device to send status reports by some other interface that is external to the MQTT protocol.

As the topics are not standardized any implementation needs to include a process to define them and communicate them to the IoT devices. Also, the frequency with which an IoT device sends out messages cannot be defined within the MQTT protocol.

MQTT connections

All clients connect to the Message Broker with a CONNECT message. The Message Broker responds to this with a CONNACK message.

There are two types of sessions possible in MQTT: a Persistent session and a Clean session.

With a Clean session, the message broker does not retain messages if a subscriber is not online at the time that it needs to send a message. Even if the RetainFlag is set on an incoming message, the Message Broker will not check when a client comes online and send the packet if that client was not connected at the time a message came in for a topic to which it has subscribed.

In a Persistent session, the Message Broker will retain messages intended for that client if the client goes offline. It will send all stored messages that occurred while the client was offline once that client reconnects.

A client can change between persistent sessions and clean sessions. So, if a client connects with a Persistent session and then goes offline, the Message Broker stores all messages that it would otherwise have sent. If the client then connects with a Clean session, the Message Broker sends all of the stored messages and then continues on sending live messages to that client as they arrive. When the client goes offline again, the Message Broker won’t store new messages that arrive for that client’s subscriptions.

The Connect message includes the following fields:

ClientID	A unique identifier for the client
CleanSession	True or False
Username	Optional account name
Password	Optional password
LastWillTopic	A hierarchical identifier
LastWillQoS	0, 1, or 2
LastWillMessage	Text to send on unexpected exit
LastWillRetain	True or False
KeepAlive	Maximum duration of silence in seconds

Of these fields, only the ClientID and CleanSession fields are mandatory. The ClientID must be the same for that client in all of its dealings with the message broker. This is more important for devices that only intend to receive messages, such as monitors. For devices that originate messages, such as IoT devices, their identity is set by the topic of each message. The topic is sent on by the Message Broker, the ClientID is not.

The ClientID for the devices that connect in order to receive messages must be the same ClientID used by the device when subscribing to topics. Think of the ClientID as being the email address in this system.

Client Authentication

The MQTT standard doesn’t set out a process for device authentication on connections. However, it does include two fields in the Connect message that can be used by the writers of MQTT applications.

There is also no provision in the MQTT system for encryption. All messages are sent in plain text. However, as they sit on top of TCP, it is possible to implement TLS at the lower layer.

Keep Alive

The connection should ideally be maintained long-term. However, the MQTT protocol includes a service that checks whether a client is truly still available for communication. It is possible that a device’s network functions can maintain a connection while its MQTT client application loses the ability to interface.

The Keep Alive process in the MQTT specifications provides a checking message that is sent from the Message Broker to each client in order to check that they are still available. The KeepAlive field in a Connection message specifies the timeout for sending a Keep Alive message.

Last Will and Testament

The Last Will and Testament section of the MQTT specifications describe procedures that should take place if a device becomes unavailable unexpectedly. This situation is known as an ungraceful disconnection.

Four optional fields in the Connect message relate to the Last Will and Testament system. When a device connects to the Message Broker, it can use these fields to specify a message that will be posted in case the device becomes uncontactable. Naturally, this system, which is also known as LWT, is closely connected to the Keep Alive function.

The LWT fields in the Connect message include a topic. This should be a special pigeonhole that is reserved for connection or device failure messages. The hierarchical structure of the topic should identify the device and indicate a disconnection, for example, “Building1/Lobby/Sensor5/LWT.” The QoS for this message is set in a field of the Connection message plus the text of the message and a flag indicating whether the message should be stored on the Message Broker.

The Message Broker stores the specification of the LWT. If the Keep Alive timeout occurs with no message from a device, the Message Broker sends that device a PINGREQ message. If no reply comes back, the Message Broker sends out a message to the subscriber of the topic in the stored LWT specifications.

The LWT process requires monitoring applications to subscribe to the LWT topic for each of the devices that it monitors. That requirement also makes it necessary to build an error notification structure into the topic hierarchy at the planning phase.

History of MQTT

MQTT was developed by IBM. Its name was formed in order to tie in with the IBM MQSeries product line, so it was known as the MQ Telemetry Protocol. IBM kept the protocol as a proprietary system, refining it through redefinitions up to version 3.1. This version, in 2013, was handed over to the Organization for the Advancement of Structured Information Standards (OASIS) for management.

The charter that gave control of MQTT to OASIS, limited the amount of variation that could be introduced into the protocol, allowing only minor changes. However, since 2013, the OASIS Message Queuing Telemetry Transport Technical Committee has continued to improve the standard with small alterations. A major change to the system occurred in March 2019 with the release of MQTT version 5.

The two defining versions, and most frequently encountered are 3.1 and version 5. There is also an adaption of the system, called MQTT for Sensory Networks (MQTT-SN), which caters to small, battery-operated devices that uses wireless networks, such as the Zigbee communications standard instead of TCP/IP.

MQTT-SN

Lightweight IoT devices that communicate over wireless networks, particularly small consumer gadgets, are more likely to utilize the MQTT-SN standard. This uses less processing power and allows the possibility for devices to communicate with each other. It has shorter messages and runs over connectionless UDP or wireless protocols instead of TCP.

As this system is connectionless, The Keep Alive and Last Will and Testament functions are a little different. It is assumed that each device will turn off its communications system between message emissions in order to save power.

The Message Broker function is replaced by a Gateway. There is also a Forwarder unit. MQTT-SN systems can plug into MQTT implementations by collectively communicating with an MQTT Message Broker through a Gateway.

Each device has an autodiscovery service so that it can find its nearest gateway. The Topic Name field in the MQTT structure is replaced by a shortname, which has no hierarchy and has only two characters. The shortname can be replaced by a Topic ID, which has to be organized as a list of unique IDs for each expected topic either by the developer or by the user as part of a setup routine. The creation of topic IDs can also be implemented within the gateway – when the client sends a topic to create a subscription list for, the gateway allocates an ID and returns this to the client for future reference.

Gateways can be interlinked, creating intermediate gateways, called forwarders that serve sections of the network. Several devices can connect to the same gateway and, under wireless conditions, each device might have several gateways to choose from.

Although the MQTT-SN specification is more suitable for wireless-connected devices, this standard is not very well known and so MQTT is used instead by the manufacturers of IoT devices.