SMQ (Simple Message Queues)

SMQ is an easy to use Message Queues architecture solution that follows the publish subscribe broadcast design pattern. The message queues provide pub/sub features similar to that of historical architectures, however in addition, the newly designed protocol structure includes features that are specifically implemented to simplify today's IoT design.

How it Works

Clients initiate the IoT communication with the SMQ Broker by use of a standard web URL, which is either HTTP for non secure or HTTPS for secure communication. The device to broker communication is established in a similar fashion to how WebSockets are initiated.

Once a connected session occurs, the SMQ Client may then publish and subscribe to topic names. A topic name can be any string value, but is typically structured in a hierarchical fashion, which is by common comparison equivalent to that of a file system hierarchy found in a UNIX like environment.

Topic names are then translated into numbers by the server, thus enabling fast processing of messages by using lookup tables and values stored on the server. The client subscribes to a topic and the server responds with a topic ID that identifies the topic channel. Each client is also assigned a unique ID by the broker know as the ephemeral topic ID. The ephemeral topic ID is used in a unique way such as "publisher's address" by subscriber, thus enabling each subscriber the ability to send messages directly to a device by using the pehemeral topic ID. The ephemeral topic ID also enables an application to simulate a remote procedure (RPC) call.

The SMQ broker provides easy segmentation of products/customers by enabling an unlimited number of broker instances running in the server. Each Message Queue entry (URL) in the server is associated with one broker instance, thus the number of instances that can be installed in the server at any given time are only limited to any inherent memory limitations.

SMQ Architecture

Figure 1: Four clients connected to a broker, where one device utilizes a secure TCP connection, one device utilizes a non secure TCP connection, and two browser clients utilize WebSockets in either secure or non-secure mode.

Connecting to the Broker

Illustrated in the figure below, the protocol initially starts as HTTP or HTTPS and is then morphed into a persistent socket connection. The SMQ connection is channeled on top of WebSockets when a browser running the JavaScript client stack connects to the broker.

The WebSocket protocol uses the word "upgrade" when an HTTP(S) connection is morphed into a persistent socket connection. The two C client stacks are also initiating the connection using HTTP and HTTPS.

The difference between the browser and device connections is that a browser HTTP(S) connection is morphed into a WebSocket connection and the device HTTP(S) connection is simply morphed into a standard TCP socket. The WebSocket protocol adds a lot of processing and management specifically designed for browsers, and this management is not required for devices.

Our construction keeps the device code to a minimum. The SMQ Client morphs an HTTP connection into a persistent TCP connection, and SharkMQ morphs an HTTPS connection into a persistent and secure (encrypted) TCP connection.

SMQ Upgrade

Figure 2: SMQ protocol upgrading request to a persistent connection.

Publish and Subscribe

The SMQ protocol includes the publish/subscribe pattern found in other pub/sub protocols such as MQTT and AMQP. The publish/subscribe pattern (pub/sub) is an alternative to the traditional client-server model, where a client communicates directly with an endpoint. Pub/Sub decouples a client, who is sending a particular message (called publisher) from another client (or more clients), who is receiving the message (called subscriber). Traditional pub/sub is a messaging pattern where senders of messages do not program the messages to be sent directly to specific receivers, but instead characterize published messages into topics without knowledge of which subscribers, if any, there may be. Similarly, subscribers express interest in one or more topics and only receive messages that are of interest, without knowledge of which publishers, if any, there are.

SMQ is designed for IoT communication and extends the traditional pub/sub design pattern to include some additional services such as one-to-one communication and sender's address. One-to-one communication eliminates the need for creating Dynamic Topic Names, such as topic names based on the client's MAC address, when using SMQ for IoT communication.

SMQ pub/sub

Figure 3: Messages are always physically routed via the broker

Messages published by a publisher are sent to the broker, and it is the broker that is responsible for finding the subscriber(s) for the message and sending the message to those subscriber(s). However, we can ignore the broker when we look at the SMQ protocol from a conceptual/abstract perspective. Just remember that messages are always routed via the broker.

Topic Names

Traditional pub/sub protocols use topic names. A client may subscribe to any number of topics and publishers may publish messages to those topics. In SMQ, a topic is a 32 bit number created by the broker. You subscribe to topic names as you would in a traditional pub/sub protocol, however, the topic name (string) is converted, by the broker, into a random 32 bit number. The clients use the number and not the topic name when publishing and subscribing to messages.

A topic name (string) is converted to a number and this number exists for the lifetime of the broker -- in other words, the number will not change and should be considered static. The topic name converted to a number is a unique ID and is referred to as the Topic ID or tid for short.

The high level APIs in the JavaScript, Java, and Lua stack allow you to simply refer to everything by topic name (string); however, the low level SMQ C code stack requires that you use the Topic ID.

The broker also creates a unique number for each client connected. This number is referred to as the Ephemeral Topic ID or etid for short. The number is valid as long as the client is connected to the broker. The number will change if the client disconnects and reconnects with the broker.

Sender's Address

Sender's address is conceptually similar to caller ID on your phone. Once a person calls you, you acquire the person's phone number and can call the person back at any time.

Sender's Address is a concept that comes from SDL, where SENDER provides the identity of the process instance from which the last consumed signal was sent.

When a client publishes a message, the client's Ephemeral Topic ID (etid) is included in the published message. The etid embedded in a published message is referred to as the Sender's Address or stid for short. A subscriber receiving a message will learn the etid of the sender of the message.

pub/sub RPC

Figure 4: Emulating Remote Procedure calls by publishing to sender's address

When a subscriber receives a message published to a named topic or to the client's own etid, the client receiving the message learns the etid of the sender. The client may then immediately send a message to this Topic ID, thus simulating a Remote Procedure Call (RPC).

One-to-one Communication

A publisher may publish to the Topic ID of a named topic such as "alarm/sensor1". A publisher may also publish to the etid of any other client as long as the publisher knows the etid of the destination client. We refer to this as one-to-one communication.

A message published to a named topic may have any number of subscribers, but a message published to an etid has only one destination, the owner of the etid. All clients are automatically subscribed to messages sent to their own etid.

pub/sub one to one

Figure 5: One-to-one communication. Publish to client's etid.

A client cannot publish a message, using one-to-one communication, to another client unless the client knows the other client's etid. A client can learn the etid of another client when the client receives a message published to a named topic. There is one exception to this rule; an SMQ client running in the server can learn the topic ID of clients as soon as they connect. An SMQ client, implemented in Lua and running on the server, can for this reason learn the etid of all connected clients without having to receive a message sent to a named topic. An SMQ client running on the server has elevated rights and additional APIs to its disposal.

A client sending messages to an etid should first request the broker to supervise the destination and request the broker to send a "change" notification event to the client should the destination go offline. See supervising subscribers for details.

Secondary Level Topic Names (Sub Topics)

In object oriented design, it's common to have an object or interface with a number of methods. SMQ provides a similar feature for topics. This optional feature is referred to as a sub-topic and enables fine grained control of messages in IoT design. The sub-topic also enables multiple message types to be sent to an ephemeral tid. The sub-topic is simply a 32 bit number and this number is completely transparent to the broker. The number is included in all published messages and can optionally be used by applications using the SMQ protocol.

SMQ sub-topic

Many embedded systems use messages internally. It is common to use #defines for naming message numbers as shown below.

#define MESSAGE_A 100
#define MESSAGE_B 101
#define MESSAGE_C 102

The C code can then simply refer to the defined names instead of the numbers, thus making it easier for the programmer to remember the message types.

A C program using the SMQ protocol can use the same design philosophy for the secondary sub-topic since the sub-topic is simply a 32 bit number. However, a maintenance problem occurs since all devices and revisions must make sure they use the same #defines. This complexity escalates when communicating with non C code such as JavaScript code, which does not provide a preprocessor.

The SMQ protocol therefore includes management of naming sub-topic numbers by name. A client sends a message to the broker, requesting the broker to give the client a unique number for a specific string. All clients requesting the same sub-topic string will get the same number. This construction makes it easy to use sub-topic numbers when communicating between say a C program and a JavaScript program since both implementations can refer to the sub-topic number by name. When the number is resolved and when the number is included in a published message, the number is completely transparent to the server. Only the publisher(s) and subscriber(s) use this number.

Supervising Subscribers

Say you are using a pub/sub protocol for designing a temperature alarm system for a nuclear reactor. The temperature sensor publishes data when the temperature is too high. What if the temperature subscriber is offline and there are no subscribers to the temperature topic? What happens then? In a standard pub/sub protocol, the temperature would be discarded and the publisher would have no way of knowing this.

The SMQ protocol includes a feature that enables a publisher to discover how many subscribers are subscribed to a particular topic. The publisher sends an observe request to the broker and the publisher will then get a "change" notification event when the number of subscribers change.

A publisher can get change notification events from named topics and from ephemeral topic IDs. The number of connected subscribers for an ephemeral ID can only be one, which means the client is connected. Receiving a change notification for an ephemeral ID means the client has disconnected/terminated.

The SMQ protocol also includes a feature where an ondrop server side callback function can be installed in the broker. The "ondrop" callback function is called for each message published to a topic with no subscribers. Server side code can then perform various types of actions such as storing the data in a database, send an email to an email address, and so on.

Implementations

The protocol is provided in source code and can easily be implemented into any computer language when needed.

Supported Libraries

NameLanguageSecurityTarget
SMQ.jsJavasScriptSecureBrowser
SMQ ClientANSI-CNon SecureDevice
SharkMQ™ ClientANSI-CSecure (TLS)Device
JavaMQ ClientJavaSecure (TLS)Java & Android
smq/broker.lua (*)LuaSecureLua Broker (server)

(*) The broker, implemented in Lua, is designed for the cosocket API in the Barracuda Application Server. You can use the broker in any Barracuda Application Server derivative product. In addition, the broker comes pre-integrated with the Mako Server, a Barracuda Application Server derivative product.

The SMQ broker cannot be used on some RTOS ports due to limitations in TCP/IP stack. See limitations for cosockets for details.

Examples

NameLanguageLinksInformation
Chat ClientJavasScriptdemo
documentation
A basic chat client that shows how to use publish and subscribe.
Improved ChatJavasScriptdemo
documentation
Enhanced chat clients that shows how the basic chat can be improved by using one-to-one communication.
LED ControlJavasScript
Java
C
demo
documentation
Web based (HTML/Javascript) and Java (Swing and Android) control management user interface for controlling Light Emitting Diodes (LEDS) in one or multiple devices (C code).
Weather AppJavasScript
C
Lua
demo
documentation
The SMQ Weather Application illustrates an example of the SMQ Architecture capability as used in web-based control of an IoT Device Thermostat. The example provides cloud-based dynamic control of the client via on-demand content generation and settings manipulation from either a local display or a remote browser-based HMI.
Light ControllerJavasScript
C
Lua
documentationThe Christmas Light Controller is a fun project that lets you provide public access to your outdoor lights during the holiday season.

Download SMQ Source Code and Libraries for the above examples.