Designing Socket Protocols in Lua

The auxiliary Lua socket library is an optional library that provides TCP Lua bindings for the Barracuda Server's socket API. The Barracuda Server provides a socket API that is portable across all supported platforms, including deep embedded systems. UDP is also supported on some platforms. The Lua socket library provides high level access to the system's or device's TCP/IP stack. The Lua socket library also includes client and server WebSocket functionality.

Did you know that your MQTT Client, OPC UA Client & Server, Modbus TCP Client, and SMQ IoT Broker are all designed in Lua and are built on top of our socket API?

The Lua socket library supports three socket modes: standard blocking sockets, asynchronous sockets, and cosockets. It is important that you familiarize yourself with these three socket modes since selecting the wrong mode may temporarily block and prevent the Barracuda Server from serving client requests.

Socket modes:

Examples

The source code for the examples in this document can be downloaded from GitHub.

Blocking Sockets

The following diagram from section Thread Mapping and Coroutines shows the native threads that may call into and interact with Lua code in a running system. We have highlighted the threads that can be used when working with blocking sockets.

Blocking sockets must run in the context of one of the Barracuda Server's worker threads (HTTP Server Thread Pool) or in a thread in the Lua Thread Library. Lua is not preemptive and native threads executing Lua code are designed to run cooperatively. The native threads only yield at known locations, such as when blocking on socket read. Each native thread is mapped to one Lua coroutine, thus making sure each call into the Lua VM has its own Lua stack. See the section Advanced Lua Bindings for information on how the native threads are designed to work cooperatively.

Lua Server Pages run in the context of one of the threads in the HTTP Thread Pool. A typical use case for blocking sockets is for a Lua Server Page to act as a network client to another service. The following diagram shows how a browser user sends a request to the Barracuda Server, where a Lua Server Page is triggered and executed as part of the request. The Lua Server Page opens a connection to another server, sends a request, and waits for a response.

Blocking Lua Sockets

The blocking socket connection in the Lua Server Page waits for the response from the other server, and, by doing so, it blocks the thread executing the Lua Server Page. The Lua Server Page resumes execution when it receives a response from the other server. The Lua Server Page then assembles the response message and sends the response message to the browser user. The Lua Server Page completes execution and the thread is then inserted back into the thread pool, where it waits for another request.

The following LSP example shows how to design a page that opens a blocking socket connection, sends a request to a time server, and waits for the response.

<?lsp
local function printTime()
   local s,err = ba.socket.udpcon("pool.ntp.org",123)
   if s then
      s:write(string.char(27)..string.char(0):rep(47))
      local d,err=s:read(5000)
      if d and #d == 48 then
         local secs = ba.socket.n2h(4,d,41)
         secs = secs - 2208988800 -- Convert from 1900 to 1970 format
         print("NTP Date and time:",os.date("%c",secs))
      else
         print"Invalid response"
      end
      s:close()
   else
      print("Cannot connect:",err)
   end
end
printTime()
?>

Example 1: Blocking socket example, implementing a request/response.

The following blocking socket example is similar to the one above, but uses the NTP UDP protocol.

<?lsp
local s,err = ba.socket.udpcon("pool.ntp.org",123)
if s then
   s:write(string.char(27)..string.char(0):rep(47))
   local d,err=s:read(5000)
   if d and #d == 48 then
      local secs = ba.socket.n2h(4,d,41)
      secs = secs - 2208988800 -- Convert from 1900 to 1970 format
      print("NTP Date and time:",os.date("%c",secs))
   end
   s:close()
end
?>

Blocking WebSocket Client Connection

Opening a WebSocket client connection is similar to opening a standard socket connection, with the exception that a WebSocket connection initially starts as an HTTP client connection. Example 2 below is similar to example 1, in that a browser user sends a request to a Lua Server Page, which in turn connects to a WebSocket server.

Blocking Lua Sockets

The following example provides an HTML form, in which the browser user can enter a text message. Once the user clicks the submit button, the message is sent to the Lua Server Page. The Lua Server Page opens a WebSocket connection to the WebSocket echo server echo.websocket.org and sends the message to this server. The LSP renders the HTML response when the echo server returns the echo message.

Example 2: Blocking WebSocket example (RPC message style)

See WebSockets for more information on using WebSocket connections.

Using blocking sockets outside of Lua Server Pages

As we mentioned above, blocking sockets are typically used in LSP pages. However, you can also run blocking sockets by using the Lua Thread Library. The following Lua script implements a very basic web server by using the socket library in blocking mode. The Lua implemented web server responds to secure HTTP requests (HTTPS requests). Each request is delegated to a native thread by using the thread library. The Lua script in example 3 below is designed to be loaded by a .preload script at system start.

-- Create a SharkSSL certificate by using the certificate stored in
-- the internal ZIP file.
local iovm = ba.openio("vm")
local certf=".certificate/MakoServer.%s"
local cert,err=ba.create.sharkcert(
     iovm, string.format(certf,"pem"),
     string.format(certf,"key"), "sharkssl")
if not cert then error"Certificate not found" end

-- Create a SharkSSL server object by using the above certificate
local sharkobj=ba.create.sharkssl(nil,{server=true})
sharkobj:addcert(cert)

-- Create a secure (SSL) server listening socket object.
local s=ba.socket.bind(9443, {shark=sharkobj})
-- Run the socket "accept" function.
s:event(acceptCoroutine,"r")

Example 3.a: Opening a secure listening socket.

The accept thread activated by code line 16 above:

-- Accept thread waiting asynchronously for client connections
local function acceptCoroutine(sock)
   while true do
      -- Method accept yields and resumes when a client connects.
      local s = sock:accept()
      if not s then break end -- Server terminating
      ba.thread.run(function() webServer(s) end)
   end
end

Example 3.b: Waits for client connections on port 9443.

The WebServer function, which runs in the context of a native thread in the thread pool, is activated by line 7 above:

-- Function running in the context of the Lua thread pool.
-- s is the blocking socket.
local function webServer(s)
    -- Read request
   local data,err = s:read(2000)
   -- Ignore request -- i.e. discard data.
   -- Send the same response for any URL requested
   local msg = "Hello World!"
   s:write(string.format(
               "%s\r\n%s\r\n%s%d\r\n\r\n%s", 
               "HTTP/1.0 200 OK",
               "Content-Type: text/plain",
               "Content-Length: ",
               #msg, msg))
   s:close()
end

Example 3.c: A basic HTTP 1.0 web server, running in the context of a native thread.

Note: when you test the Lua script in example 3, use a browser and navigate to https://server-address:9443

Asynchronous Sockets

Asynchronous sockets are blocking sockets attached to a Lua coroutine that wait asynchronously for incoming data. Asynchronous sockets make it possible to call the socket read method without requiring a native thread. Asynchronous sockets must also be used for server (listening) sockets and when calling method socket:accept.

Although asynchronous sockets appear to be blocking in the Lua script when calling the socket read method, the code is, under the hood, using coroutine.yield to temporarily freeze the calling coroutine if no data is available on the socket. The code is, on the C side, run in the context of the Barracuda Server's Socket Event Dispatcher, which is responsible for dispatching socket events to the system, including the web-server. It is therefore important to not block the thread for prolonged times as this will have a negative impact on the web-server, especially if the web-server is serving a high number of concurrent connections. For example, calling function ba.sleep() in a socket coroutine is a bad idea.

Asynchronous Lua Sockets

Example 4.a: A socket coroutine yields and waits for data.

The above Lua function is activated as a Lua coroutine. The function loops and waits for incoming data. The Lua binding for the socket:read knows that this call should be event driven and suspends the Lua coroutine when the Lua script calls this method. The suspended thread is resumed by the Lua socket bindings when the socket dispatcher sends a socket receive data pending event.

The above socket coroutine function is activated by calling method socket:event. This method can be called on both client and server socket objects. The following example shows how to activate the coroutine after a client connection is established.

local sock,err=ba.socket.connect(address,port)
if sock then
   sock:event(sockCoroutine)
else
   trace("ba.socket.connect failed:",err)
end

Example 4.b: Activating an asynchronous socket coroutine after a client connection is established.

Asynchronous sockets in LSP pages

Asynchronous server connections:

LSP pages can be used as the entry point for morphing incoming WebSocket requests into asynchronous WebSocket connections and for morphing regular HTTP requests into asynchronous socket connections. In this case, function ba.socket.req2sock morphs the incoming HTTP request either into a WebSocket connection or into a standard socket connection. Secure (HTTPS) WebSocket requests are morphed into secure WebSocket connections and regular HTTPS requests are morphed into secure socket connections.

The following LSP page illustrates how to morph an HTTP request into a WebSocket connection:

Example 5: Morphing an HTTP(S) WebSocket request into an asynchronous WebSocket connection.

See WebSockets for more information on using WebSocket connections.

Asynchronous client connections:

Asynchronous client connections are typically not used in LSP pages since the socket coroutine runs asynchronously and will run independently from the LSP request/response. The following example is a modified version of example 1, in which the blocking socket read method is converted to a non blocking read in an asynchronous socket coroutine.

Example 6: A modified version of example 1, where the socket is asynchronous.

Note that calling ba.socket.connect on line 2 above blocks the LSP page until the connection is established. Calling connect from an LSP page is okay, but you should avoid calling ba.socket.connect as shown above in the context of the Socket Event Dispatcher or in the context of any of the other event threads. The connect call can also be made asynchronous. See cosocket connect for more information.

Calling the socket write method from an asynchronous coroutine

We previously mentioned that asynchronous sockets are blocking sockets attached to a Lua coroutine. What this means is that the socket read method is asynchronous and non blocking, but writing data to the socket may block.

Blocking asynchronous Lua Sockets

Example 7: Asynchronous socket coroutine blocking in socket write.

Example 7 shows an asynchronous socket coroutine that acts as an echo server. Whatever data that is received is sent back to the peer side. A TCP socket implements internal buffering and a sliding window. When the buffer in the TCP/IP stack is full, the caller is blocked. What this means is that the C side thread executing the coroutine temporarily freezes until the TCP/IP stack can accept more data.

Cosockets run in the context of the Barracuda Server's Socket Event Dispatcher, and a blocking socket call will temporarily freeze the Event Dispatcher. The server will not be able to respond to new client requests as long as the Event Dispatcher is blocket. You should, for this reason, avoid calling the socket write method or any other call that may block for prolonged time from within an asynchronous socket coroutine. A simple fix to the above example is to delegate sending data to the peer side to a worker thread as shown in the example below:

local function echoCoroutine(sock)
   while true do
      local data = sock:read()
      if not data then break end
      ba.thread.run(function() sock:write(data) end)
   end
end
local sock=getMyServerSocket() -- See example 3.b for how to create a server socket
sock:event(echoCoroutine)

Example 8: Asynchronous socket coroutine with deferred socket write.

Example 8 is a modified version of example 7, in which calling the socket write method is delegated to a worker thread on line 5. Note: we also used function ba.thread.run in example 3.b.

Cosockets - Coroutine Sockets

The cosocket mode turns both reading from and writing to the socket into asynchronous events. The Lua socket bindings are specifically designed to remove the complexity found in the overly complex BSD non blocking socket paradigm. From the Lua script designer's perspective, both reading from the socket and writing to the socket appear to be blocking, but under the hood and inside the Lua bindings, complex logic makes sure these calls are mapped onto non blocking socket events. A cosocket is a special Lua coroutine with an attached socket which can be in the state connected or non-connected.

A cosocket is a self-referencing object as long as it has an active socket connection. This means that the cosocket object will not be garbage collected as long as the socket connection is maintained, even if there are no other references to the object. However, if you call the socket:disable method, the cosocket is no longer self-referencing, and it can be garbage collected even if it still has a valid socket connection.

local function echoCoroutine(sock)
   assert(sock == ba.socket.getsock()) -- true for cosockets
   assert(true == sock:owner()) -- true for cosockets
   while true do
      local data = sock:read()
      if not data then break end
      sock:write(data)
   end
end
local sock=getMyServerSocket() -- See example 3.b for how to create a server socket
sock:event(echoCoroutine, "s")

Example 9: A modified version of example 8, where both reading and writing is asynchronous.

The difference between example 8 and 9 is how we start the asynchronous coroutine. The "s" flag used on line 9 informs the socket Lua binding that you want the socket coroutine to be fully asynchronous. Line 5 writes to the socket just as we did in example 7, but unlike example 7, the socket write call in example 9 only appears to be blocking from the Lua script's perspective.

Method socket:event enables either asynchronous socket mode or cosocket mode. The socket event method is typically used for server connections and not for client connections. The reason for this is that the default action for function ba.socket.connect is to block until the client to server connection is established.

Note that only the thread owner can call socket:read when in cosocket mode. As shown in example 9 above, a cosocket typically stays in a forever loop waiting for new data. You can call socket:read in any location in the code, including deep down in functions being called from the main cosocket function. A generic library function can get the cosocket by calling ba.socket.getsock.

Cosocket connect

Function ba.socket.connect can also operate in cosocket mode if run in the context of a socket coroutine as shown in the following example:

local function sockCoroutine(sock,address,port)
   -- sock is not connected
   local socket,err=ba.socket.connect(address,port)
   if socket then -- If connect succeded
      while true do
         assert(socket == sock) -- we have one cosocket
         local data
         socket:write"hello"
         data,err = socket:read()
         if not data then break end
         trace(data)
      end
      if err then
         trace("Socket err:",err)
      end
   else
      trace("ba.socket.connect failed:",err)
   end
end
ba.socket.event(sockCoroutine, "MyEchoServer", 8080)

Example 10: Cosocket client connection example.

All functions are asynchronous in the above example. We have highlighted line 18, which activates the socket coroutine, and line 2, where ba.socket.connect executes in cosocket asynchronous mode.

The above example opens a standard non secure client socket connection in asynchronous mode. Function ba.socket.connect can also open a secure connection in asynchronous mode. The following example shows how to initiate a secure client connection:

local s = ba.socket.connect("google.com",443,{shark=ba.sharkclient()});

Example 11: Establishing a secure client connection.

When run in the context of a socket coroutine, the ba.socket.connect function completes the socket connection and the SSL handshake in asynchronous mode. The following diagram illustrates the connection sequence for establishing a secure connection:

Lua Asynchronous SSL Handshake

Figure 1: TCP (blue) and SSL handshake (red) sequence.

The blue colored lines show the TCP connect sequence, and the red lines show the SSL handshake sequence. The complete sequence is asynchronous, and all socket event management is handled under the hood in the Lua socket bindings.

Some protocols defer upgrading a socket connection to a secure connection. We support this via method socket:upgrade. The above sequence can be split in two, in which the blue lines are performed asynchronously by ba.socket.connect and the red lines are performed asynchronously by calling method socket:upgrade.

Method socket:event v.s. function ba.socket.event

Method socket:event either enables asynchronous socket mode or cosocket mode. Method socket:event binds an existing and valid socket connection to a socket coroutine.

Function ba.socket.event starts a socket coroutine in cosocket mode. The socket coroutine is bound to a non connected socket. The first socket function call in this coroutine must be one of: calling ba.socket.connect or opening a server listening socket by calling function ba.socket.bind. The coroutine is bound to one socket. Thus, you will get an error if you try to open subsequent sockets by calling one of ba.socket.connect or ba.socket.bind again.

Cosockets flow control

Lua cosocket flow control

Figure 2: cosocket flow control.

Flow control in the standard BSD non blocking socket paradigm is complex, and it is easy to make programming mistakes. Fortunately, the Lua socket bindings completely remove this complexity by making non blocking asynchronous socket calls appear as blocking socket calls in the Lua scripts. All cosockets have their own Lua stack and run in the context of the native thread powering the Barracuda Server's Socket Dispatcher (SoDisp). In other words, all cosockets, regardless of how many you create, run in the context of one native thread.

The Lua socket bindings include advanced flow control management that enables any cosocket to send data by calling socket:write on any another cosocket. The Lua bindings automatically suspend the calling coroutine when the TCP/IP stack reports that the non blocking socket cannot accept more data. The suspended coroutine(s) are automatically resumed when the coroutine socket can accept more data. Figure 2 shows how coroutine A is automatically suspended when calling the socket write method on cosocket B. cosocket A is resumed when cosocket B can accept more data or if cosocket B closes the connection. Multiple coroutines can write to the same socket and the message is guaranteed to be sent in a FIFO manner.

Non cosockets, such as Lua Server Pages, can also call the cosocket's write method. However, non cosockets cannot be suspended, and the message will instead be queued up internally in the cosocket. Note, the message sent is only queued up in the coroutine if the TCP/IP queue is full. The size of the queue can be read and changed by calling socket:queuelen. Method socket:write returns nil,queuelen if the queue is full, where queuelen is a number. Note: You may set the queue length for non cosockets to zero and implement your own message queue by using socket:disable and socket:enable. For example, the MQTT client implements its own queue (see the MQTT Lua source code; function sendMsg and sndCosock for details).

Flow control examples:

The example below creates two cosockets, in which cosocket B is connected and cosocket A is in the non connected state. Cosocket A does not need to be connected since we will only use the coroutine for sending out data on cosocket B.

The example below is full duplex, in which cosocket B waits for incoming data on socket B and cosocket A writes data to socket B.

Note: we have omitted some error checking in the examples below in order to shorten the examples.

local function cosocketA(sockA, sockB) -- SockA not connected
   local fp = io:open"Large-Firmware.bin"
   while true do
      local data = fp:read()
      if not data then break end -- EOF
      sockB:write(data)
      -- sockA:read(300) -- Enable for send throttling
   end
   fp:close()
end

local function cosocketB(sockB, address, port)
   ba.socket.connect(address, port) -- Returns sockB or nil
   ba.socket.event(cosocketA, sockB)
   while true do
      local data = sockB:read()
      print(data)
      if not data then break end
   end
end

ba.socket.event(cosocketB, "MySpecialService", 1234)

Example 12a: Cosocket asynchonous socket flow control example.

Function cosocketB is started as a cosocket on line 22 by calling ba.socket.event. The socket is unconnected until line 13 connects to MySpecialService, which we assume succeeds. The returned value from ba.socket.connect is the same socket object as the first argument to the cosocketB function when ba.socket.connect succeeds. Line 14 starts function cosocketA as a cosocket; notice that cosocket A stays in the unconnected state.

Cosocket A opens a large file and sends the large file to the peer side by writing data on socket B. The method call on line 6 suspends the cosocket when the TCP/IP stack is full. Cosocket B waits for incoming data from the peer side and prints this data. Cosocket B is suspended on line 16 until data is available from the peer side.

As a special case, a non-connected cosocket is allowed to call method read() on its own socket if a time value is provided. The function will then sleep for the specified time and return nil,timeout when the timeout elapses. You could enable line 7 in the above example and throttle the data being sent. A non-connected cosocket can call two socket methods: read(with a time value) and disable().

The above two coroutines are powered by one native thread, and the coroutines run cooperatively, yielding on line 6 and 16.

The following example shows the flow control detailed in Figure 2, when a non cosocket (an LSP page) calls method write on a cosocket. The LSP page keeps writing until the queue is full and then exits.

<?lsp
local sock = ba.socket.connect("localhost",8888)
-- Receive cosocket required, but not used
sock:event(function(sock) trace(sock:read()) end, "s")
while true do
   local ok,status=sock:write"hello"
   trace(ok,status)
   if not ok then
      if "number" == type(status) then
         -- status is now number of bytes in queue
         print("Socket queue full", status,sock:queuelen())
      else
         print("Socket error",status)
         assert("string" == type(status))
      end
      break -- exit
   end
end
?>

Example 12b: LSP page creating a cosocket and calling write() on the cosocket.

The above LSP page (example 12b) can be tested together with the following script. The code below can be copied to the .preload script.

local function rec(sock)
   sock:read()
   trace"received first message; now stop"
   sock:disable()
end

local function accept(s)
   while true do
      local s = s:accept(s)
      if not s then break end
      s:event(rec,"s")
   end
end
 
local s = ba.socket.bind(8888)
s:event(accept)

Example 12c: A server that receives one message and then stops (use with 12b).

Lua Socket Proxy Example

The WebSocket proxy is a fully functional proxy, aka tunnel, included in the Lua socket example package. The example further illustrates how easy it is to design fully asynchronous sockets using the API provided by the Lua socket bindings.

Figure 3 below illustrates the basic principles of the proxy. For each client request, a client and server cosocket is created. The server socket acts as the proxy server for the client (shown to the left in the image below). The client socket acts as the client for the destination server (shown to the right in the image below).

Cosocket Lua Proxy

Figure 3: A proxy implemented using cosockets.

The client and server coroutines call function webProxy as soon as the connection with the peer is established. The webProxy function is responsible for receiving data on one end (the source) and writing data on the other end (the sink). The client and server coroutines provide the same socket arguments to the webProxy function, but the client coroutine reverses the arguments v.s. the server coroutine. The proxy core code are the two code lines receiving and writing data (line 3 and 4).

The complete proxy source code, excluding error checking and printouts, is shown below.

local function webProxy(source, sink)
   while true do
      local data = source:read()
      if not data or not sink:write(data) then break end
   end
end

local function webProxyClient(client,server)
   -- ba.socket.connect returns socket or nil if it fails
   if ba.socket.connect(serverName,80) ~= client then return end
   server:enable(client)
   webProxy(client, server)
   client:close()
end

local function webProxyServer(server)
   ba.socket.event(webProxyClient, server)
   local client = server:disable()
   webProxy(server, client)
   server:close()
end

local function accept(s)
   while true do
      local s = s:accept(s)
      s:event(webProxyServer,"s")
   end
end

local s = ba.socket.bind(8080)
s:event(accept)

Example 13: A cosocket proxy implemented in Lua.

The above proxy source code manages all connections in one native thread and can, for this reason, scale up to a large number of connections. How many connections that can be accepted depends on how many socket connections the Barracuda Server's Socket Event Dispatcher can handle. The Barracuda Socket Event Dispatcher provides a platform independent layer for dispatching socket events. The Socket Event Dispatcher maps to the BSD function "select" on many systems, and on Linux the dispatcher typically maps to epoll. The epoll dispatcher is designed to handle up to 262,144 connections.

WebSockets

The Lua socket bindings include support for client and server WebSocket connections. We introduced WebSockets in the Blocking WebSocket Client Connection example.

WebSocket Examples:

Client and server WebSocket connections provide the same three socket modes as standard socket connections. However, the connection sequence differs from regular socket connections since a WebSocket connection is initiated over HTTP(S).

Client WebSockets

A client WebSocket connection is started by using the HTTP Client Library and this library cannot run in cosocket mode; thus a WebSocket client connection is usually started as shown in the example below.

local function openWebSocketCon()
   local h = require"httpc".create()
   h:request{url="ws://My-WebSocket-Server"}
   if h:status() == 101 then -- 101 switching protocol
      local s = ba.socket.http2sock(h)
      s:event(myWebSocketCoroutine,"s")
   end
end
ba.thread.run(openWebSocketCon)

Example 14: WebSocket client example.

Server WebSockets

A WebSocket server connection is similar to a standard server listening socket. However, instead of using socket:accept, a Websocket server connection is derived from an incoming HTTP request. WebSocket server connections are, for this reason, typically initiated in an LSP page. In the asynchronous server connection, we showed how to initiate a WebSocket server connection and how to morph the request into a asynchronous socket. The following code snippet is identical to the asynchronous server connection example, with the exception that the coroutine is started in cosocket mode. We do this by adding the "s" flag on line 5:

The Lua Server Page's environment is ephemeral and only exists for the duration of the request. It is, for this reason, we keep the Lua script managing the persistent WebSocket connection outside of the LSP page. The function "myCosocket" is declared in the application's environment, but it could also have been a global function.

A detailed explanation on the various Lua environments can be found in section The Command (Request/Response) Environment.

WebSocket authentication

WebSocket authentication uses standard HTTP authentication.

Client:

A client WebSocket connection is initiated by using the HTTP Client Library. You provide user credentials as part of the initial HTTP request. See http:request for more information on setting user credentials. Note: you only provide user credentials if the server requires authentication.

Server:

A Lua Server Page is typically used as the entry point for WebSocket requests and for converting the request into a WebSocket server connection. Use the Barracuda Server's authenticators if you require authentication from client WebSocket requests. Note that many browsers are not particularly good at using HTTP authentication; however, they do provide cookies when sending a WebSocket request. A browser that has previously been authenticated using form based authentication will automatically authenticate when opening a WebSocket connection, assuming the LSP page serving as the WebSocket entry is protected by a form based authenticator.

WebSocket Frames

The WebSocket protocol is a standard TCP connection with a frame based protocol. Websocket differs from TCP in that it enables a stream of messages instead of a stream of bytes. The WebSocket implementation in the Lua socket bindings enable frame sizes up to 65535 (0xFFFF) bytes. Peers sending larger frames are rejected.

The frame based protocol provides support for sending binary frames and UTF-8 encoded strings. Lua strings can store any type of data; thus both frame types are stored as Lua strings. The format used when writing data to the socket is by default binary.

socket:write(data) -- Send binary frame
socket:write(data,false) --  Send binary frame
socket:write(data,true) -- Send UTF-8 frame

Lua can manage UTF-8 encoding, but Lua does not contain any UTF-8 validation. You must make sure you use a text editor that supports UTF-8 and save the Lua script as UTF-8 when using UTF-8 frames.

When receiving data from the WebSocket connection, a second argument is returned by socket:read, and this return value is set to false for binary frames and true for UTF-8 frames. The following example shows how this works:

local data,utf8 = socket:read()
if data then
   if utf8 then
      trace('Received UTF-8 frame')
   else
      trace('Received binary frame')
   end
end

Example 15: Detecting binary/UTF-8 frames when receiving WebSocket data.

The data received by the Lua socket bindings is served as the return value from socket:read as it trickles in on the TCP stream. What this means is that the Lua socket bindings do not assemble a complete frame. You must manually assemble the frame if your implementation requires that you only work on complete frames. Method socket:read returns two additional return values, bytesRead and frameLen, when a WebSocket frame fragment is returned. Variable bytesRead is the number of bytes read so far, and variable frameLen is the length of the frame. The complete frame is read when bytesRead == frameLen. The following function illustrates how to assemble complete frames.

local function wsread(sock)
   local data,utf8,bytesRead,frameLen = sock:read()
   if frameLen then -- nil if not a fragment
      local t={} -- Assemble in table t
      repeat
         table.insert(t,data)
         data,utf8,bytesRead,frameLen = sock:read()
      until not data or frameLen == bytesRead
      if data then
         table.insert(t,data)
         data=table.concat(t) -- Assemble
      end
      -- else failed, utf8 is now errcode
   end
   return data,utf8 -- Returns data,utf8 or nil,ecode
end

Example 15: Assembling a WebSocket frame.