SMQ is an easy to use Message Queues architecture solution that follows the publish subscribe broadcast design pattern. The message queues provide pub/sub features similar to MQTT; however, SMQ also includes features that are specifically implemented to further simplify today's IoT design.
This introduction includes SMQ fundamentals and concise hands-on examples.
Clients initiate the IoT communication with the SMQ Broker by use of a standard web URL, which is either HTTP for non secure or HTTPS for secure communication. The device to broker communication is established in a similar fashion to how WebSockets are initiated.
Once a connected session occurs, the SMQ Client may then publish and subscribe to topic names. A topic name can be any string value but is typically structured in a hierarchical fashion, which is by common comparison equivalent to that of a file system hierarchy found in a UNIX like environment.
The broker translates the SMQ publish/subscribe topic names into numbers, thus enabling fast processing of messages by using lookup tables and values stored on the server. The client subscribes to a topic and the server responds with a Topic ID (etid) that identifies the topic channel. Each client is also assigned a unique ID by the broker known as the Ephemeral Topic ID (etid). The Ephemeral Topic ID is used in a unique way such as "publisher's address" by subscriber, thus enabling each subscriber the ability to send messages directly to a device by using the ephemeral Topic ID. The Ephemeral Topic ID also enables an application to simulate a remote procedure (RPC) call.
The SMQ Broker provides easy segmentation of products/customers by enabling an unlimited number of broker instances running in the server. Each Message Queue entry (URL) in the server is associated with one broker instance, thus the number of instances that can be installed in the server at any given time is only limited to any inherent memory limitations.
Figure 1: Four clients connected to a broker, where one device utilizes a secure TCP connection (SharkMQ), one device utilizes a non secure TCP connection (SMQ Stack), and two browser clients utilize WebSockets in either secure or non-secure mode (SMQ JavaScript).
The optional application shown in figure 1 can for example be a server side SMQ client, a server side web application using HTTP, an SMQ bridge connecting SMQ with any other protocol supported by the Barracuda App Server.
Illustrated in the figure below, the protocol initially starts as HTTP or HTTPS and then morphs into a persistent socket connection. The SMQ connection is channeled on top of WebSockets when a browser running the JavaScript client stack connects to the broker.
The WebSocket protocol uses the word "upgrade" when an HTTP(S) connection morphs into a persistent socket connection. The two C client stacks are also initiating the connection using HTTP and HTTPS.
The difference between the browser and device connections is that a browser HTTP(S) connection morphs into a WebSocket connection and the device HTTP(S) connection simply morphs into a standard TCP socket. The WebSocket protocol adds a lot of processing and management specifically designed for browsers, and this management is not required for devices.
Our construction keeps the device code to a minimum. The SMQ Client morphs an HTTP connection into a persistent TCP connection, and SharkMQ morphs an HTTPS connection into a persistent and secure (encrypted) TCP connection.
Figure 2: SMQ protocol upgrading HTTP(S) request to a persistent connection.
The SMQ protocol includes the publish/subscribe pattern found in other pub/sub protocols such as MQTT and AMQP. The publish/subscribe pattern (pub/sub) is an alternative to the traditional client-server model, where a client communicates directly with an endpoint. Pub/Sub decouples a client who sends a particular message (called publisher) from another client (or more clients) who receives the message (called subscriber). Traditional pub/sub is a messaging pattern where publishers of messages do not program the messages to be sent directly to specific receivers, but instead characterize published messages into topics without knowledge of which subscribers, if any, there may be. Similarly, subscribers express interest in one or more topics and only receive messages that are of interest, without knowledge of which publishers, if any, there are.
SMQ is designed for IoT communication and extends the traditional pub/sub design pattern to include some additional services such as one-to-one communication and publisher's address. One-to-one communication eliminates the need for creating Dynamic Topic Names, such as topic names based on the client's MAC address, when using SMQ for IoT communication.
Figure 3: Messages are always physically routed via the broker
Messages published by a publisher are sent to the broker, and it is the broker that is responsible for finding the subscriber(s) for the message and sending the message to those subscriber(s). However, we can ignore the broker when we look at the SMQ protocol from a conceptual/abstract perspective. Just remember that messages are always routed via the broker.
Traditional pub/sub protocols use topic names. A client may subscribe to any number of topics and publishers may publish messages to those topics. In SMQ, a topic is a 32 bit number created by the broker. You subscribe to topic names as you would in a traditional pub/sub protocol, however, the topic name (string) is converted, by the broker, into a random 32 bit number unless it has been programmatically pre-set. The clients use the number and not the topic name when publishing and subscribing to messages.
A topic name (string) converts to a number and this number exists for the lifetime of the broker -- in other words, the number will not change and should be considered static. The topic name converted to a number is a unique ID and is referred to as the Topic ID or tid for short.
The high level APIs in the JavaScript, Java, and Lua stack let you simply refer to everything by topic name (string); however, the low level SMQ C code stack requires that you use the Topic ID.
The broker also creates a unique number for each client connected. This number is referred to as the Ephemeral Topic ID or etid for short. The number is valid as long as the client is connected to the broker. The number will change if the client disconnects and reconnects with the broker.
Publisher's address is conceptually similar to caller ID on your phone. Once a person calls you, you acquire the person's phone number and can call the person back at any time.
Publisher's Address is a concept that comes from SDL, where SENDER (publisher) provides the identity of the process instance from which the last consumed signal was sent.
When a client publishes a message, the client's Ephemeral Topic ID (etid) is included in the published message. The etid embedded in a published message is referred to as the publisher's address or ptid for short. A subscriber receiving a message will learn the etid of the publisher of the message.
Figure 4: Emulating Remote Procedure calls by publishing to publisher's address
When a subscriber receives a message published to a named topic or to the client's own etid, the client receiving the message learns the etid of the publisher. The client may then immediately send a message to this Topic ID, thus simulating a Remote Procedure Call (RPC).
A publisher may publish to the Topic ID of a named topic such as "alarm/sensor1". A publisher may also publish to the Ephemeral Topic ID (etid) of any other client as long as the publisher knows the etid of the destination client. We refer to this as one-to-one communication since the message is sent from one client to another client via the broker.
A message published to a named topic may have any number of subscribers, but a message published to an etid has only one destination, the owner of the etid. All clients are automatically subscribed to messages sent to their own etid.
Figure 5: One-to-one communication. Publish to client's Ephemeral Topic ID (etid).
The randomly created etid number will change should a client disconnect and then re-connect. Using a random number prevents a client from sending a one-to-one message to another client unless the client knows the etid. A client cannot publish a one-to-one message to another client unless the client knows the other client's etid. A client can learn the etid of another client when the client receives a message published to a named topic. There is one exception to this rule; an SMQ client running in the server can learn the Topic ID of clients as soon as they connect. An SMQ client, implemented in Lua and running on the server, can for this reason learn the etid of all connected clients without having to receive a message sent to a named topic. An SMQ client running on the server has elevated rights and additional APIs to its disposal.
The server side SMQ client, if installed, has the hard coded etid one (1) -- in other words, the etid is not randomly created for the server client. This construction makes it possible for all clients to communicate directly with the server side SMQ client without having to learn the server's etid.
A client sending messages to an etid should first request the broker to supervise the destination and request the broker to send a "change" notification event to the client should the destination go offline. See supervising subscribers for details.
In object oriented design, it is common to have an object or interface with a number of methods. SMQ provides a similar feature for topics. This optional feature is referred to as a sub-topic and enables fine grained control of messages in IoT design. The sub-topic also enables multiple message types to be sent to an Ephemeral Topic ID. The sub-topic is simply a 32 bit number that is completely transparent to the broker. The number is included in all published messages and can optionally be used by applications using the SMQ protocol.
Figure 6: Object (class) and Methods v.s. Topic and Sub-Topics.
In Figure 6, an object (such as a C++ class instance) may have a set of functions that can be called. A Topic can be thought of as an object with a set of functions, which in SMQ terminology are sub-topics. Secondary topic names are useful when sending one-to-one messages i.e. when sending a message to an Ephemeral Topic ID (to an SMQ client, which can be thought of as an object). The sub-topics can be used to address specific functions in one particular client.
Say you are using a pub/sub protocol for designing a temperature alarm system for a nuclear reactor. The temperature sensor publishes data when the temperature is too high. What if the temperature subscriber is offline and there are no subscribers to the temperature topic? What happens then? In a standard pub/sub protocol, the temperature would be discarded and the publisher would have no way of knowing this.
The SMQ protocol includes a feature that enables a publisher to discover how many subscribers are subscribed to a particular topic. The publisher sends an observe request to the broker and the publisher will then get a "change" notification event when the number of subscribers change.
A publisher can get change notification events from named topics and from ephemeral Topic IDs. The number of connected subscribers for an ephemeral ID can only be one, which means the client is connected. Receiving a change notification for an ephemeral ID means the client has disconnected/terminated.
The SMQ protocol also includes a feature where an ondrop server side callback function can be installed in the broker. The "ondrop" callback function is called for each message published to a topic with no subscribers. Server side code can then perform various types of actions such as storing the data in a database, sending an email to an email address, and so on.
A broker must be programmatically instantiated at the server side. The broker is stored as a Lua module, and the module is loaded by using the "require" function:
local broker = require("smq.hub")
Example 1.a: Fetching the SMQ broker table.
Use of parenthesis () is optional in Lua for functions that take only one parameter and when this parameter is a literal string. We can therefore write the above line as:
local broker = require"smq.hub"
Example 1.b: Fetching the SMQ broker table.
The returned value, saved as the local variable 'broker' above, is a Lua table (object). This table includes one function, the create function, which is used in the example below.
local broker = require"smq.hub" -- Fetch "broker" module local smq = broker.create() -- Create an SMQ broker instance
Example 2: Fetching the SMQ broker table and creating an SMQ broker instance.
The "create" function creates an SMQ broker instance. We can simplify the above Lua module loading and SMQ instance creation as follows:
local smq = require"smq.hub".create() function smqConnect(request) smq:connect(request) -- Upgrade HTTP(S) request to SMQ connection end
Example 3: Fetching and creating an SMQ broker + adding public smqConnect() function.
The above construction is typically put in a .preload script, which runs when the application loads. The function smqConnect() calls the method smq:connect(), which morphs an HTTP(S) request into a persistent SMQ connection. Function smqConnect() is typically called from an LSP page when an SMQ client sends a connection request to the server.
<?lsp app.smqConnect(request) -- Upgrade HTTP(S) request to an SMQ connection ?>
Example 4: LSP page accepting SMQ connections and forwarding requests to the code in Example 3.
Recall how Figure 2 shows how an HTTP(S) request upgrades to an SMQ connection. The broker's URL entry is typically put in an LSP page. The smqConnect() function is available in an LSP page via the 'app' table. See the LSP page's Command Environment for details on how this works.
An SMQ broker is typically created at startup and maintained for the lifetime of the application. It is for this reason a broker should be created in a .preload script or a module. It is, however, possible to create a broker in an LSP page and prevent it from being garbage collected as follows:
<?lsp
local smq = page.smq -- fetch from 'page' table
if not smq then -- first time accessed
smq = require"smq.hub".create() -- Create one broker instance
page.smq = smq -- Store (reference) broker instance
end
smq:connect(request) -- Upgrade HTTP(S) request to an SMQ connection
?>
Example 5: Bundling Example 3 and 4 in one LSP page.
The (if) statement on line 3 will trigger the first time the code is accessed after a server start. The code on lines 4 and 5 creates a broker instance and then saves the resulting broker instance in the page table to isolate it from garbage collection.
Line 7 takes the LSP page connection object and passes it into the smq:connect() function. A SMQ connection request is either a standard HTTP or HTTPS request, which is upgraded to a persistent SMQ connection by the "connect" function. This connection is then, maintained until the SMQ client decides to disconnect.
Creating and storing a broker within an LSP page may be suitable for very basic operations or limited use of the broker as shown within the above example; however, it is not recommended (if) you are designing any extensive server side logic that will have interaction with the broker. In contrast, separation of the logic from the LSP page should be maintained by creating the broker within a separate module (or) within an application environment such as the .preload script.
The following includes hands-on examples design to be run on our online tutorial server https://tutorial.realtimelogic.com/, which has some limitations such as being restarted every hour. Persistent SMQ connections will therefore break when the online server restarts or is otherwise disrupted. All examples that include a run button can be executed.
A web page designed to interact using the SMQ JavaScript client must include JavaScript code for creating an SMQ client and for dynamically updating the HTML elements. You can work with the browser's DOM by using any modern toolkit such as React and Vue.js. We use JQuery in the following examples and example 6 serves as the scaffolding for the subsequent SMQ client examples.
<script src='/rtl/jquery.js'></script>
<script>
$(function() {
function print(txt) { $("#console").append(txt+"\n"); };
print("Hello World");
});
</script>
<pre id="console"></pre>
Example 6: Using JQuery for creating a basic console with a 'print' function.
Example 6 should be sent to our tutorial server when you click the above button. The code opens in a source code editor and you can modify and run the code. You should see the text "Hello World" being printed just below the source code editor when you click the run button on this page. Try adding another print line (duplicate line 5) and re-run the example.
We have used example 6 as a base in the next example. The new lines are 2, 6, and 7. Line 2 loads the SMQ JavaScript Client Library. Line 6 connects to our public SMQ test broker. Notice that an SMQ JavaScript client URL is the same as a WebSocket URL (wss://). Line 7 subscribes to topic "BasicChatMsg" with a callback routed to the print function.
<script src='/rtl/jquery.js'></script>
<script src='/rtl/smq.js'></script>
<script>
$(function() {
function print(txt) { $("#console").append(txt+"\n"); };
let smq = SMQ.Client("wss://simplemq.com/smq.lsp");
smq.subscribe("BasicChatMsg", {onmsg:print, datatype:"text"});
});
</script>
<pre id="console"></pre>
Example 7: Connecting to public test broker and subscribing to chat messages.
The Basic Chat Client shows how to use publish and subscribe for sending messages to all connected clients and how to receive the published messages.
See the tutorial A Basic Chat Client for details.
We have kept the JavaScript code (JS) to a bare minimum in example 7 and the naive implementation is susceptible to code injection attacks. Copy the following JS code, paste the JS into the Basic Chat Client, and press enter. You will then see unicorns popping up in example 7's console window.
The Basic Chat Client's source code includes a function called escapeHtml(), which mitigates these kind of attacks.
The following example dynamically installs an SMQ broker on our tutorial server using one LSP page. The example uses the logic explained in example 5 for persistently storing a reference to the required objects in the page table to prevent garbage collection. However, the LSP page is not used as the entry for SMQ connection requests. Instead, the example creates a directory object (line 7) and installs it in the virtual file system (line 11). Line 10 inserts a directory callback function, which routes HTTP requests to SMQ:connect().
local smq = page.example8
if smq then
print"Term. old broker" smq.dir:unlink() smq.broker:shutdown"bye bye"
end
smq={
broker=require"smq.hub".create(), -- Create one broker instance
dir=ba.create.dir("smq-example8")
}
page.example8=smq
smq.dir:setfunc(function(_ENV) smq.broker:connect(request) end)
smq.dir:insert() -- SMQ connection request entry URL: "/smq-example8/"
response:write("Installing new broker @ ",smq.dir:baseuri())
Example 8: Installing a broker at SMQ URL '/smq-example8/'.
Click example 8's run button to open the example window and click the run button in this window to install the example broker. However, do not close the example window. The following example shows a basic JS client connecting to the "example 8" broker.
<script src='/rtl/jquery.js'></script>
<script src='/rtl/smq.js'></script>
<script>
$(function() {
function print(txt) { $("#console").append(txt+"\n"); };
var smq = SMQ.Client(SMQ.wsURL("/smq-example8/"));
smq.onconnect=function(etid, rnd, ipaddr) {
print("onconnect, client's Ephemeral Topic ID="+etid);
};
smq.onclose=function(message,canreconnect) {
print("onclose, reason="+message+", canreconnect="+canreconnect);
};
});
</script>
<pre id="console"></pre>
Example 9: JS client with SMQ onconnect and onclose event handlers.
You can test the above sequence a few times to get an understanding of how SMQ connects and disconnects. The second argument provided in the onclose event (line 10) tells the client if it can automatically re-connect or not. The reason this is false when example 9 runs is because we shut down the broker by calling method SMQ:shutdown (line 3 in example 8). The SMQ JS client can be set to auto reconnect should the connection break, but it cannot auto connect if the broker is shut down.
The hands-on example Using SMQ for WebSocket Communication is similar to example 8 and 9 above but also shows how to send messages from a server side SMQ client to the SMQ JS client running in the browser.
Note: see the SMQ example list below if you are not planning on connecting an SMQ C/C++ client.
We provide several C/C++ examples and the SMQ publish/subscribe C++ example is the best starting point since the example shows several features that simplifies using the SMQ protocol with C/C++. For one, the message names use pre-registered Topic IDs (tids) making it super easy to use from C code. See the broker startup script line 10 and 21 for details. The messages can then simply be #defined in the C code; see subscripe.cpp line 18 for details. This construction makes it very easy to write a C switch statement when receiving the messages. See subscripe.cpp line 252 for details.
The example can be compiled for both Windows and Linux. The repository includes ready to use Visual Studio project files. The code can be downloaded and compiled for Linux as follows:
See the SMQ GitHub repository's readme file for additional details. The following video shows how to run the example bundle using three terminals on Linux. You would use the same startup sequence on Windows using three console windows.
See the SMQ-Examples on GitHub for details.
Several SMQ tutorials are provided on the Mako Server website. See SMQ IoT tutorials for details.
| Name | Language | Links | Information |
|---|---|---|---|
| Streaming camera images | JavaScript and Lua | Tutorial | This is a simple and fun introduction to SMQ, where you use an ESP32 cam board to stream real-time images to multiple browsers. |
| One to one communication | JavaScript and Lua | GitHub Source | SMQ is often used for asynchronous bidirectional communication between browsers and the server. The One-to-One communication example provides a foundational guide demonstrating direct communication between a browser and the server. |
| AJAX over SMQ | JavaScript and Lua | GitHub Source | SMQ can also be used as a foundation for calling server methods asynchronously from a browser. The RPC (aka AJAX) over SMQ example shows how easy this is. |
| Device Management | JavaScript, C, and C++ | Tutorial GitHub Source | Modern Approach to Embedding a Web Server in a Device. |
| Chat Client | JavaScript | Demo Tutorial | A basic chat client that shows how to use publish and subscribe. |
| Improved Chat | JavaScript | Demo Tutorial | Enhanced chat clients that shows how the basic chat can be improved by using one-to-one communication. |
| LED Control | JavaScript Java C | Demo Tutorial | Web based (HTML/Javascript) and Java (Swing and Android) control management user interface for controlling Light Emitting Diodes (LEDS) in one or multiple devices (C code). |
| Weather App | JavaScript C Lua | Demo Tutorial | The SMQ Weather Application illustrates an example of the SMQ Architecture capability as used in web-based control of an IoT Device Thermostat. The example provides cloud-based dynamic control of the client via on-demand content generation and settings manipulation from either a local display or a remote browser-based HMI. |
| Light Controller | JavaScript C Lua | Tutorial | The Christmas Light Controller is a fun project that lets you provide public access to your outdoor lights during the holiday season. |
The following protocol implementations are provided in source code and can easily be implemented into any other computer language when needed. See the SMQ Specification for details on how the protocol works.
Supported Libraries
| Name | Language | Security | Target |
|---|---|---|---|
| SMQ.js | JavaScript | Secure | Browser |
| SMQ Client | ANSI-C | Non Secure | Device |
| SharkMQ™ Client | ANSI-C | Secure (TLS) | Device |
| JavaMQ Client | Java | Secure (TLS) | Java & Android |
| smq/broker.lua (*) | Lua | Secure | Lua Broker (server) |
| smq/client.lua | Lua | Secure | Lua Client |
(*) The broker, implemented in Lua, is designed for the cosocket API in the Barracuda Application Server. You can use the broker in any Barracuda Application Server derivative product. In addition, the broker comes pre-integrated with the Mako Server, a Barracuda Application Server derivative product.
The SMQ broker cannot be used on some RTOS ports due to limitations in the TCP/IP stack. See limitations for cosockets for details.