JSON has become a popular inter-process communication (IPC) data interchange format for a variety of computer languages. UBJSON (Universal Binary JSON), the binary version of JSON, provides a more compact encoding format than standard JSON. Although the JSON and UBJSON encoding and decoding libraries are part of the Barracuda Application Server, they can also be used standalone. The libraries are implemented in C code and provide a C API and C++ API. The C++ API consists of inline C++ methods in the header files. The C libraries conform to the object oriented C APIs in the Barracuda Application Server.
The JSON and UBJSON C libraries are designed such that they can function without dynamic memory allocation, which is a requirement found in many embedded/microcontroller solutions. Consequently, the libraries are ideal for use "standalone" in small microcontrollers, where JSON or UBJSON can be used for encoding and decoding data.
JSON and UBJSON Use Cases
A typical JSON use case is to send encoded data over HTTP or other similar Remote Procedure Call (RPC) based protocols. The JSON and UBJSON libraries we provide extend well beyond the typical or traditional JSON use cases. For example, the encoders/parsers can be used for encoding/decoding data on a fully asynchronous bi-directional communication stream such as a real-time Machine-2-Machine (M2M) channel. Both parsers are stream and event based, thus capable of parsing data as it trickles in on a continuous data stream. The parsers do not need any start/end marker on an incoming multi JSON object stream and can be connected directly to a raw socket connection. Event based means that the parsers do not block and wait for new data. As such, the parsers can be used in an event driven bare metal embedded system as a means for providing encoding/decoding of real time data sent over the Internet.
Use cases include, but are not limited to:
- Encode/decode messages sent on a continuous data stream such as on a raw TCP/IP connection
- Encode/decode messages sent on a WebSocket connection
- Encode/decode messages sent via an MQTT topic
- Encode/decode messages sent/received via RPC protocols such as HTTP
- Encode/decode data for persistent storage on disk (mini database)
Using JSON or UBJSON:
UBJSON is friendlier to C/C++ design than JSON, is more compact, and supports sending binary data in objects without having to resort to B64 encoding, however, JSON is more ubiquitous and it is easier to find a library for other platforms that supports JSON. We support JSON and UBJSON via the C libraries and via the Lua scripting language in the Barracuda Application Server and derivative products such as the Mako Server. The UBJSON library is smaller and uses less resources than the JSON library.
Introduction
As we mentioned above, the JSON and UBJSON libraries are implemented in C code, with a C++ wrapper API defined in the header files. We will use a mixture of C and C++ in the examples that follow. In general, you can easily switch between C and C++ as the following example illustrates:
We will focus on using JSON and UBJSON for encoding and decoding C/C++ data structures. You can apply the information you learn here and use it for encoding and decoding data for various purposes such as encoding messages sent over a Machine to Machine connection. The following figure provides a high level view of how a C structure can be encoded and decoded by using method "set" and method "get".
In Figure 2, we use method "set" for encoding a C structure as a JSON stream and method "get" for decoding a JSON stream and converting it into a C structure instance. Method "set" is similar in functionality to function printf in that it provides formatted output. The first argument specifies the types and length of the variable number of arguments that follows. Method "get" is similar to function scanf in that it performs formatted input. We are also using a few optional helper macros in the above example. We will get more into how these macros are used later. The UBJSON API is similar, however, the format specifiers differ from the JSON API.
Introduction to Encoding Data
Encoding data is straightforward. You serialize the C structure by using method "set" or by using primitive methods that enable you to dynamically serialize data and/or C structures. The following C++ example copied from Figure 2 shows how to encode a C struct instance and it's 3 members:
The first argument provided to method "set" includes format flags that indicate the subsequent number of dynamic arguments and their type. The format flags for the "set" method closely matches the format flags for the "get" method. See the "get" method for how to use the format flags.
You can also encode data by using the primitive methods used internally by the "set" method. The following example shows how to use these methods to encode the data in the above example:
| C++ | C Code |
|---|---|
| encoder.beginObject(); | JEncoder_beginObject(&encoder); |
| encoder.setName("myInt");// Member name | JEncoder_setName(&encoder,"myInt"); |
| encoder.setInt(myInt); | JEncoder_setInt(&encoder,myInt); |
| encoder.setName("myString"); | JEncoder_setName(&encoder,"myString"); |
| encoder.setString(myString); | JEncoder_setString(&encoder,myString); |
| encoder.setName("myFloat"); | JEncoder_setName(&encoder,"myFloat"); |
| encoder.setDouble(myFloat); | JEncoder_setDouble(&encoder,myFloat); |
| encoder.endObject(); | JEncoder_endObject(&encoder); |
Figure 3 shows how to dynamically encode the C structure in Figure 2.
Introduction to Decoding Data
Decoding data is more complex than encoding data since the incoming JSON or UBJSON stream must be parsed and broken up into tokens. The decoder logic consists of a parser and a decoder that plugs into the parser. The JSON parser is designed for parsing a continuous JSON text stream, and the UBJSON parser is designed for decoding a continuous binary UBJSON stream.
Both parsers provide an event based callback API, where you can install your own callback function or use one of the ready to use decoders for JSON (JParserValFact or JDecoder) or UBJSON (UBJDecoder).
The JDecoder (JSON) and the UBJDecoder (Universal Binary JSON) are designed to make it easy to decode JSON/UBJSON and convert the incoming data directly to data represented by a C structure and/or C array.
JParserValFact is an optional JSON value factory builder that can be used instead of the JDecoder. JParserValFact provides an API that is much more advanced than the API provided by JDecoder and is more suitable for decoding complex types. Note: A value factory builder is not provided for the UBJParser.
The JSON and UBJSON Lua Value Factory builders are part of the Barracuda Application Server's LSP bindings and are designed to automatically build Lua tables and primitive values from the parsed data.
Using the JDecoder and UBJDecoder
JDecoder and UBJDecoder are designed for memory constrained systems where a value factory such as JParserValFact is not suitable. Note: JDecoder and UBJDecoder do not use dynamic memory allocations, but they do need a buffer as explained later. Method "get" builds a pointer value tree where the leaf nodes contain pointers to the primitive values in your C structure. The following figure shows how this works.
It is important to understand that the "get" method does not actually update any values in your C struct instance. What it does is to build a tree where the leaf nodes have pointers to the primitive values in your struct instance. For example, in the above figure, the green 'id' leaf node above contains a pointer to ld.green.id. When the parser calls the callback function in the JDecoder or UBJDecoder instance, the correct leaf node is found, and this is when the value in the struct instance is set.
There are some design limitations imposed by the JDecoder and UBJDecoder. JSON and UBJSON maps more naturally to dynamic data represented by scripting languages. The mapping to strict C structures and/or C arrays as managed by the "get" method limits the dynamic features of JSON and UBJSON. You must also know the representation layout of the parsed JSON and UBJSON data before parsing since the "get" method, which builds the pointer value tree, must be called before parsing. However, if you define a strict API, this should not be a problem. As an example, say you know the peer side will send you the exact command sequence: LED, MOTOR, TEMPERATURE. From this knowledge, you can call the "set" method and parse as the following pseudo code shows:
decoder.get(<<LED>>) /* Use as shown in Figure 5 */ until complete: parser.parse(stream data) decoder.get(<<MOTOR>>) until complete: parser.parse(stream data) decoder.get(<<TEMPERATURE>>) until complete: parser.parse(stream data)
Every time we call method "get", we re-arrange the internal pointer value tree in the decoder to represent the new struct instance thus making the decoder ready for the next object parsed by the parser.
The JDecoder and UBJDecoder "get" methods use a variable number of arguments. The first argument specifies the subsequent arguments.
- An object starts with '{' and ends with '}'
- An array starts with '[' and ends with ']'
Objects must include the member name; for example, the following C++ code example from Figure 2 includes the member name via the preprocessor expansion of the support macros.
The following C++ example shows how to call the "get" method without using the helper macros:
Notice that a string also requires the buffer size. The parser callback (in JDecoder and UBJDecoder) checks the size of the buffer when copying the decoded string to the buffer. The parser callback will return an overflow error to the parser if the decoded string is larger than the provided buffer. The maximum string length is "buffer size - 1".
The JSON Decoder Member Number and Member String macros:
The member name must also be included when with nested structures. The example in Figure 5 includes the member name "green" and "blue" when calling the "get" method.
You should note that if your format flag does not correspond to the arguments provided, the code will crash when calling method "get".
Arrays are easier to manage. The following C++ example shows how to decode an array of 3 elements:
As you can see from the above example, using format flags for extracting an array is easy, but not very efficient since we need to specify each element in the array. For large arrays or dynamic length arrays, you can either install your own parser callback directly in the parser or use the 'X' format flag, which enables you to install a callback for sections of the data being parsed. The following example illustrates how this works:
In the above example, the Decoder calls the chained parser callback for each element in the decoded array. The following example shows a more complex use, where an object includes two member arrays and where the member names in the object is called "array1" and "array2".
Using JParser and JDecoder
(Creating a JParser object and initializing it with a JDecoder callback object)
The following example shows how to create a JSON parser and decoder, in C++ and in C code.
JSON parser and decoder in C++ code:
JSON parser and decoder in C code:
Figure 6 shows creating a JSON parser and decoder in C++ and C code.
The JParser and JDecoder do not require dynamic memory allocation, but they do require temporary memory for processing the data.
The buffer used by the JDecoder is initialized and used when building a pointer value tree when the "get" function is called. The buffer must remain valid as long as the JDecoder is used. The buffer must be no less than N* 16 bytes on a 32 bit processor, where N is the number of format flag parameters used when calling method "get". See the reference manual for details.
The parser requires two buffers. The object member name buffer must be no less than the largest expected member name size, e.g. the member name "myFloat" in figure 2 requires 8 bytes (7+1). The second buffer is used for internal housekeeping such as temporarily storing parsed data including strings, if used. The parser will return an error code if it parses JSON data that does not fit into the provided buffers. In the above example, memory for the second buffer is provided by a dynamic memory allocator that can grow the buffer for large strings. An embedded system without an allocator can set a fixed size buffer, but the buffer must not be smaller than 256 bytes and no less than the largest expected string.
Creating a UBJparser and UBJDecoder is similar to creating the JSON version shown above. The UBJParser uses less memory than the JParser, especially for strings. The JParser must assemble and temporarily store a parsed string. The string is passed in whole to the parser callback. However, the UBJParser does not need to process string data and a large string may be sent as fragments to the parser callback as the data trickles in on the stream.
Using the Parser
As we mentioned above, the JParser and UBJParser are stream based and can parse data as it trickles in on a stream such as on a socket connection. In the following section, we will explain how this works, but before doing so, let's look at a more traditional way of using a JSON parser. In the following example, the file "oneobject.json" contains one JSON object. We have omitted error checking on reading the buffer.
Figure 7 shows how to parse one JSON object.
In the above example, a complete JSON string is loaded into a buffer and passed into the parser. Since we load one complete object, we expect the parser to be in state JParsStat_DoneEOS (Done parsing and end of stream) after parsing the data.
Now, let's assume that we instead load a buffer with multiple JSON objects as shown in the following example:
Figure 8 shows how to parse multiple JSON objects in one buffer.
In the above example, we continuously call the parser until the state is not JParsStat_Done (Done parsing, but not end of stream). When the parser state no longer is JParsStat_Done, we expect it to be JParsStat_DoneEOS (done and end of stream).
JParsStat_Done means that the parser parsed a complete JSON object, but that the buffer passed into the parser contains additional data. When the parser returns JParsStat_Done, the parser must be called again with the same buffer unmodified.
For the final example, we will show how to read data from a stream as it trickles in. The data could for example come from a raw socket connection. The following example emulates a data stream by reading one byte at a time from a JSON data file containing multiple JSON objects.
Figure 9 shows how to parse data as it trickles in on a stream.
We do not check the return value from the parser in figure 7 and 8, but instead check the parser status. In addition to using the status, the return value from the parser can be used as shown above. The parser returns zero if the provided chunk of data did not include a complete JSON object, but parsed successfully – in other words, the parser needs more data in order to complete parsing the JSON. The parser returns minus one on any error and one when a JSON object is completely parsed.
In the above example, we feed the parser only one byte at a time, thus the parser state can only be one of JParsStat_DoneEOS or JParsStat_NeedMoreData as long as no errors occur. The state cannot be JParsStat_Done, which means that the buffer contains more data to be parsed. In a real case scenario, you would not read one byte at a time from a stream, but say 1024 bytes. In such an application, you must also check for the JParsStat_Done state and make sure you call the parser again with the same buffer unmodified. Figure 11 includes an example that shows how this is done.
Using JParserValFact
The JParserValFact can be used as an alternative to JDecoder. We do not have an equivalent to JParserValFact for UBJSON, however, it is possible to also create one for UBJSON.
The JParserValFact builds a tree structure of JVal objects. A JVal instance represents one JSON value.
The difference between the JParserValFact and JDecoder is that the JParserValFact can dynamically build and represent any JSON type. However, the JDecoder can only decode JSON that exactly matches the pointer value tree created by calling method "get".
An instance of JParserValFact is initialized as follows (using C code):
Figure 10 shows how to Create a JParserValFact and a JParser.
The JParserValFact constructor takes two allocators as arguments, where one allocator is used for allocating JVal types and the other for allocating strings. We previously mentioned that the JParserValFact requires dynamic memory allocation, but it is possible to use it with static buffers. The reason for using two allocators is to make it easier to use static buffers. The JVal pointers must typically be aligned on word boundaries on most CPUs, but not strings, which can contain non aligned lengths. Releasing memory is not done before terminating the JParserValFact instance; thus, the allocator does not need to implement releasing or reallocating memory. An allocator for the JParserValFact can for this reason simply advance an internal pointer in a static buffer for each new allocation.
The following C example illustrates how one can use the JParser and JParserValFact for receiving and decoding asynchronous JSON messages received over a socket connection. Each JSON message includes a message number that uniquely identifies the message type received. All JSON messages include this message number.
Figure 11: Continuously parsing data on a socket stream (C code example)
When using the parser with HTTP, you typically read the complete HTTP input, create a parser, parse all data at once, and then terminate the parser. The parser can also be used when receiving HTTP chunk encoded data with unknown length. You would use the parser similar to above; i.e. parse data as it streams in on a HTTP chunk encoded stream.
Notice how we skip reading from the data stream if we parsed a complete object (state != JParsStat_Done). The reason for this construction is that a chunk read from the stream may contain parts of a new JSON object or contain multiple objects. We must call the parser again, without reading more data since the parser is at this point still using our input buffer.
In the above code, the JSON data is extracted from the JSON tree by the following code:
The function is the inverse of JEncoder_set(). The function is similar to scanf. A format flag specifies the data structure to extract from the JSON tree. The function is internally parsing the format string and using the primitive JVal_XX functions. For example, the formatting "{sd}" instructs the function to do the following in sequence on the JVal tree:
- getObject(): The curly bracket { instructs JVal_get to go one level down in the JVal tree.
- getString(): Set the pointer a1.msg to the pointer in the JVal string member.
- v=JVal_getNextElem(v): Advance to next element
- getInt():Copy data into the pointer to integer &a1.level
The sender of ALARM1 can use the JEncoder set method when sending the message as illustrated below:
M2M LED JSON Client
IoT protocols may sometimes add unnecessary memory and processing overhead. The M2M LED JSON example illustrate how one can design a secure IoT Edge Node merely by using a TCP/IP stack and optionally in combination with SSL where the payload data is exchanged between a server and the device using an encrypted connection.
The benefit with using JSON is that it is easy to scale up the message exchange types in any way you want, and you will not have the restrictions found in the primitive binary protocol used in the original demo. The source code for M2M LED JSON edition can be used as a template for designing IoT applications where JSON is used for message exchanges.
The code consists of the device side implemented in C code and the server side code implemented in the Lua scripting language.
The JSON protocol used for exchanging messages is defined as follows:
The message number enables us to decode the message objects used. The message number is used in a C "switch" statement as illustrated in the following example:
M2M LED JSON Message Types:
Message 'A':
A for Accept client connection request. The server sends this command to the client if the connection request was accepted.
messageObject: none. The accept message does not require a message body.
Message 'R':
R for Register device. The device client sends LED information to the server, such as number of LEDs and LED colors, after receiving the 'A' message.
messageObject:
The size of the dynamic-length-array depends on the device and how many LEDs the device has. This array contains object(s) of type:
The 'id' is used in message 'S' when switching LEDs on or off. The name and color is used by the server logic when building the web pages for browsers connecting to the server.
Message 'S':
S for Switching LED on or off. Both client and server can send this command. A device client sends this command if a button for the LED was clicked on the device. The server sends this command if a connected browser either switches the LED on or off.
messageObject:
Message 'P':
P for Ping or Pong. Device sends ping and server sends pong. The purpose with ping and pong is to keep the persistent connection alive when communicating via network devices such as NAT routers that will close idle connections after a certain time.
messageObject: none
Message 'C'
C for gracefully Closing the connection. Server sends this message if it wants the client to close the connection.
Using JDecoder or JParserValFact in the device client
When designing the device side we have to decide if we should use JDecoder or JParserValFact for decoding the parsed JSON stream. JParserValFact is easier to use since a value tree is dynamically built for any received JSON data. The C code can then simply use the value tree as soon as a complete JSON object is parsed. However, the JParserValFact uses more memory than JDecoder; thus JDecoder is favorable in a design where memory is limited. In this example, a JDecoder is selected since we want the code to use as little memory as possible.
Using JDecoder with multiple message types
The JDecoder requires that we build a pointer value tree before parsing which creates a problem since we can receive multiple message types from the server. However, this problem can easily be overcome by creating a parser callback function that chains the parser callback function in JDecoder.
The chained parser callback function is designed for our protocol, where it decodes the message number and instructs the JDecoder to build a pointer value tree for the message object.
The chained parser callback decodes the messageNumber and delegates decoding of messageObject to a JDecoder
The following code snippet shows the source code for the chained callback. The nesting level (nLevel) will be 0 for start array '[' and end array ']', The nesting level will be one for messageNumber and any value between 1 and N for messageObject. The callback can be in two states, decoding messageNumber or delegating the decoding of messageObject to the JDecoder.
Figure 13 shows how to decode multiple object types based on message number
The message types we can receive from the server are 'A', 'S', 'P', and 'C', where 'S' and 'C' contain a messageObject. Notice how we build a JDecoder value tree for the two commands 'S' and 'C', thus preparing the JDecoder for decoding the messageObject. We delegate the decoding of the messageObject when the callback is in the state "messageObject" – i.e. when the messageNumber is non zero.
Notice that you can eliminate the chained parser callback if you instead use the JParserValFact since a JParserValFact instance can dynamically build any type of JSON value tree.
Parsing Stream Data
The following code snippet shows how we parse a chunk of data received from the socket connection. The code would have been almost identical if a JParserValFact had been used instead of a JDecoder.
Figure 14 shows parsing and managing multiple message types
The M2MLED_manage method is called for each chunk of data received on the socket. Since a TCP/IP connection is stream based, we may not receive a complete JSON object or we may receive more data than one JSON object. The M2MLED_manage method is designed to manage these conditions. The do while loop checks if we received more than one object by checking the parser state. As we explained in Using the Parser, the parser must be called again with the same buffer if the state is JParsStat_Done.