Advanced Lua Bindings

Our online Lua Binding Generator is a good starting point when it comes to building your own Lua bindings, thus enabling LSP pages and Lua scripts to call your own C functions.

The following documentation is for writing manually crafted Lua bindings. Manually crafted bindings are preferred for more complex interactions such as interactions with objects on the C side. The Lua book provides a great introduction in how to design Lua bindings.

The tutorials in this paper explain:

• How to design a Lua interface to a C object, such as a handle in C code or a C++ class.
• How to release the global mutex for code that is time consuming, thus enabling concurrent requests.
• How to call Lua code asynchronously from C code.

Object oriented access and releasing the global mutex

The following tutorial covers how to write a Lua interface to a C object. You can use the same method for exporting a C++ class to Lua. The tutorial also shows how to release the global mutex for C functions that may take some time to complete, thus enabling the web server to accept other HTTP requests while the C function is running.

The tutorial requires that you have an understanding of Lua user data and Lua metatables as described in the Lua book. An older version of the book is available online where you can read the two sections Object-Oriented Access and metatables.

The Lua interface to the C object will be as follows:

Dynamically create a Lua object that represents the C object.

local l2c = Lua2C.open()

The C object is created by calling the open function in the global Lua2C table. The Lua2C table and the open function are all part of this exercise. In other words, we will extend the Lua engine with this functionality.

The dynamically created object will have two functions that can be called.

l2c:fast()
l2c:slow()

The "fast" and "slow" methods increment a counter when called and return the number of times the function has been called. The "fast" function returns immediately and the "slow" function takes 5 seconds to execute. The slow function is therefore designed such that it releases the global mutex while processing, enabling other threads in the server to execute.

Lua can return multiple values so both the "fast" and "slow" functions return its own counter and the counter for the other function.

Creating the Lua2C table and binding the 'open' function to the Lua2C table.

We must start by creating a C function that registers our Lua2C table.

int luaopen_Lua2C(lua_State* L)
{
   luaL_requiref(L, "Lua2C", installLua2C, TRUE);
   return 1;
}

The luaopen_Lua2C is the only global function in the C file, and it is the function you must call from your startup code – i.e. from the code creating the main Lua state. The name of the function can be any name, but we have opted to use a naming convention that is compatible with the Lua function require. Our C file can be compiled as a standalone module, on operating systems that support DLLs/shared libraries, and loaded on demand by calling require("Lua2C").

Notice how we set the last argument in function luaL_requiref to TRUE. This construction registers the Lua2C table as a global variable and makes it possible to reference this table directly in the global namespace, without having to call require("Lua2C") from the Lua code.

In the typical embedded system, function luaopen_Lua2C will be called just after you created the main Lua state, and our C file with the Lua2C bindings will be compiled and linked with the system. In other words, it will not be loaded dynamically as a dynamic library.

L = balua_create(&blp);        /* create the Lua state */
lua_pop(L,luaopen_Lua2C(L)); /* pop and balance stack when installing Lua2C */

Function luaopen_Lua2C returns a value designed for dynamic loading via require and the return value is not needed when we link the code directly with the server code. What we do after creating the main Lua state "L" is to call luaopen_Lua2C and pop the values pushed by luaopen_Lua2C off the stack. This keeps the Lua stack balanced.

In function luaopen_Lua2C, luaL_requiref takes function installLua2C as an argument. The installLua2C function installs the "open" function in a new Lua table and returns this table. Lua will then register this table as "Lua2C".

static int installLua2C(lua_State *L)
{
   static const luaL_Reg mdfTable[] = {
      {"open", Lua2C_open},
      {NULL, NULL}
   };
   luaL_newlib(L, mdfTable);
   return 1;
} /* End */

Lua code can now call Lua2C.open() and a call to this function from Lua ends up in the Lua binding Lua2C_open.

static int Lua2C_open(lua_State *L)
{
   Lua2C* l2c = (Lua2C*)lua_newuserdata(L, sizeof(Lua2C));
   l2c->slowCnt = 0;
   l2c->fastCnt = 0;
 
   /* Get metatable for Lua2C */
   if(luaL_newmetatable(L, LUA2C))
   {  /* Failed getting meta i.e. first time this func is called. */
      static const luaL_Reg Lua2CLib[] = {
         {"fast",              Lua2C_fast},     
         {"slow",              Lua2C_slow},
         {"__gc",              Lua2C_gc},
         {NULL, NULL}
      };
      /* Create metatable */
      lua_pushvalue(L, -1); /* Push table (t) created by luaL_newmetatable */
      lua_setfield(L, -2, "__index"); /* t.__index == t */
       /* Set t.fast=Lua2C_fast, t.slow=Lua2C_slow, and t.__gc=Lua2C_gc */
      luaL_setfuncs(L, Lua2CLib,0);
   }
   lua_setmetatable(L, -2); /* Set meta for Lua2C userdata */
   return 1; /* Lua2C userdata */
} /* End */

The key to understanding the above code is to read the Lua documentation for luaL_newmetatable and read up on how metatable.__index = metatable works. This function creates a new table the first time it is called and retrieves the table on subsequent calls. The table is set as a metatable for the Lua userdata, i.e. for the C object Lua2C. The metatable binds the 3 Lua functions, "slow", "fast", and "__gc" to the dynamically created Lua2C object, thus making it possible for Lua code to use the userdata and call function "fast" and "slow". The "__gc" function is run by Lua when the garbage collector reclaims the memory for the userdata object. The "__gc" function is the destructor.

Lua code can now call function "fast" and "slow". The following will print out "1 0" and "2 0".

local l2c = Lua2C.open()
print(l2c.fast(l2c))
print(l2c:fast()) -- Syntactic sugar for the above code 'l2c.fast(l2c)'

The above Lua example code triggers the C function Lua2C_fast twice.

static int Lua2C_fast(lua_State *L)
{
   Lua2C* l2c = (Lua2C*)luaL_checkudata(L,1,LUA2C);
   l2c->fastCnt++;
   lua_pushinteger(L, l2c->fastCnt);
   lua_pushinteger(L, l2c->slowCnt);
   return 2;
}

Function Lua2C_fast verifies that the first stack argument is a Lua2C object, typecasts the Lua userdata object to a Lua2C object, increments the counter, and pushes the two return values onto the Lua stack.

Releasing the Global Mutex

Function Lua2C_slow is similar to function Lua2C_fast and performs the same checking, increments the counter, and pushes the two values onto the stack. The "slow" function also sleeps for 5 seconds, thus simulating a lengthy C call.

static int Lua2C_slow(lua_State *L)
{
   Lua2C* l2c = (Lua2C*)luaL_checkudata(L,1,LUA2C);
   ThreadMutex* m = balua_getmutex(L);
   balua_releasemutex(m);
   Thread_sleep(5000); /* 5 secs */
   balua_setmutex(m);   
   l2c->slowCnt++;
   lua_pushinteger(L, l2c->slowCnt);
   lua_pushinteger(L, l2c->fastCnt);
   return 2;
}

The above code fetches the server's main mutex and releases the mutex just before sleeping for 5 seconds. The mutex is reclaimed just after it sleeps. You must make sure you have the mutex locked when working with the Lua state L. The mutex will always be locked when the Lua binding starts, and the mutex must be locked when it returns.

When an LSP page calls the "slow" function, it will block for 5 seconds. This enables other threads in the server's pool to execute. An introduction to the thread mechanism in the server can be found in the introduction under section Thread Mapping and Coroutines. Many of the Barracuda Application Server's Lua bindings release the mutex when calling code that can block. Examples of such functions are the functions in the SQL library and functions in the response object such as response:write.

Each native thread in the server's thread pool is mapped to one Lua coroutine stack; thus the server can handle many concurrent Lua stacks at the same time. We will explain more on how to map a coroutine stack onto a native thread in the next section, where we explain how to call Lua from C code.

Download Example Code

Download the source code for the above example.

Automatic Mutex Wrapper Generation

It is tedious to create code that releases the mutex during lengthy function calls if a large number of functions must be modified. You may also have existing Lua bindings, or Lua bindings from the Internet that you want to download and quickly modify. Real Time Logic has prepared two scripts that you may use to automate this process.

mutex.lua: Mutex wrapper script generator

This script creates C code that wraps around the original functions. Each wrapper function releases the mutex, calls the original function, and locks the mutex when the original function returns.

Let's say you have the following function and you want to wrap this function into another function that releases the mutex when it is called.

int myfunction(int arg1, const char* arg2);

Create a file such as functions.txt and put all the function declarations you want to wrap into this file. In our case, we put the above function declaration into this file. The next step is to create the wrapper code. Run the mutex script as follows:

lua mutex.lua functions.txt my-generated-wrapper my-header.h

The Lua script mutex.lua parses the function declarations in functions.txt and produces my-generated-wrapper.h and my-generated-wrapper.c. The last argument passed into mutex.lua (my-header.h) is to a header file that provides the original function declarations. This file is included in the output (my-generated-wrapper.h).

my-generated-wrapper.h:

#include "my-header.h"
int  myfunction_L(lua_State* L, int arg1, const char* arg2);
#define myfunction(arg1, arg2) myfunction_L(L, arg1, arg2)

my-generated-wrapper.c:

#undef myfunction
int  myfunction_L(lua_State* L, int arg1, const char* arg2)
{
   int  ret;
   ThreadMutex* __tm = getDispMutex(L);
   ThreadMutex_release(__tm);
   ret = myfunction(arg1, arg2);
   ThreadMutex_set(__tm);
   return ret;
}

fext.lua: Function declaration extraction script

The file that lists all functions (functions.txt) can be manually created or you can use fext.lua to parse all your header files and automatically produce functions.txt.

$lua fext.lua
Usage: lua fext.lua [-p pattern] [-c cdata-blob-file] [-e exclude-file] input-files....

The optional pattern (-p) means that you should only include files that matches this pattern. See Lua patterns for details.

You do not want to create wrappers for functions not being used by your Lua bindings. The optional -c option is for loading a "blob file" that is a concatenation of all C files that are part of your Lua binding library. When a function is found in the header file(s), the "blob file" is searched. The function is only included in files.txt if it is found in the "blob file". You can, for example, create this file as follows: cat *.c > c-data-blob.txt

You may further want to limit the set of functions included. The -e option is for a file that lists all functions that should be excluded.

Download mutex.lua and fext.lua

For additional examples on how to use the above two Lua scripts, see the build scripts (build.sh) for the MongoDB and PostgreSQL Modules.

Calling Lua Code Asynchronously From C Code

The Barracuda Application Server enables multiple threads to call and execute Lua code. In the previous section, we explained how threads in the thread pool can simultaneously execute in the Lua VM instance by using coroutine thread stacks. Only one thread can execute at any time in the VM, but Lua scripts can call Lua bindings (C functions), which may then release the global mutex protecting the Lua VM. This construction enables multiple threads to co-operatively run the Lua VM instance.

Easy solution

The code required for calling asynchronously from C code into Lua is more complicated than calling from Lua to C code. We will go into the design below, but before doing so, you should consider an alternative implementation that is easier to implement. The Barracuda App Server provides an API where you can create native threads in Lua. The Lua code executing in the native thread may be designed to repeatedly call a Lua to C function that blocks until data is available. The C side of the Lua binding releases the global mutex (as explained above), uses an OS primitive to wait for data, reclaims the mutex, and returns the data to Lua, thus simulating an asynchronous call.

local function myThreadFunc()
   while true do -- endless loop
      local data1,data2,data3 = myBlockingCFunction()
      -- Send data as an event to the system by calling some function
   end
end
 
local thread=ba.thread.create()
thread:run(myThreadFunc)

Recommended solution

Let's revisit the diagram from the Thread Mapping introduction.

Suppose you have an asynchronous source that generates two integers, and you want to call a Lua function and pass those integers as arguments when the event occurs. If your C startup code created an LThreadMgr instance, you could use the Lua Thread Library to achieve this.

The following must be in your startup code. See Xedge's C code for example.

LThreadMgr ltMgr;
LThreadMgr_constructor (....)

The following code shows how to use the LThreadMgr API to call a Lua function:

typedef struct
{
   ThreadJob super; /* Class inheritance using C code */
   int myvar1;
   int myvar2;
} MyJob;
 
extern LThreadMgr ltMgr; /* from your startup code */
 
/* Your source calls this function when you have an event.
 */
void myEvent(int var1, int var2)
{
   /* Func getSoDispMutex() does not exist, but you need the mutex.
      The function fetches the server's mutex.
   */
   ThreadMutex* m = getSoDispMutex();
   MyJob* job = ThreadJob_lcreate(sizeof(MyJob), myCallback);
   /* Your event data */
   job->myvar1=var1;
   job->myvar2=var2;
   ThreadMutex_set(m);
   LThreadMgr_run(&ltMgr, job); /* Send to thread queue */
   ThreadMutex_release(m);
}
 
/* This function runs in the context of one of the threads in LThreadMgr
 */
static void myCallback(ThreadJob* tj, int msgh, LThreadMgr* mgr)
{
   /* Mutex is already locket */
   MyJob* job = (MyJob*)tj;
   lua_pushinteger(tj->Lt, job->myvar1);
   lua_pushinteger(tj->Lt, job->myvar2);
   /* Push the Lua function you want to call on the stack, then: */
   lua_pcall(L, 2, 0, msgh);
}

References: LThreadMgr, ThreadJob, ThreadMutex, SoDispMutex, see the Lua documentation for lua_pushinteger and lua_pcall.

The C file AsynchLua.c in the Xedge's source code directory shows how to implement a complete example. Here is a copy of the AsynchLua.c file:

/*
The following example is part of the Advanced Lua Bindings tutorial
and demonstrates how to call Lua code asynchronously from C code. This
example receives UDP messages on ports 8080 and 8090, forwarding these
events to Lua if the relevant Lua functions have been created (example 1) and
installed (example 2).
 
Introduction:
https://realtimelogic.com/ba/doc/en/C/reference/html/md_en_C_md_LuaBindings.html#AsynchC2Lua
 
Example 1:
The example listening on port 8080, referred to as 'ex1' below,
expects a global Lua function to be defined. If the function is found,
the example will call this function. To create this function, add the
following Lua code to the .preload script of a Lua application using
Xedge:
 
function _G.udpmsg(msg)
    trace("Global func received:", msg)
end
 
You can also insert this code into an LSP page to experiment with the
incoming data. Repeatedly refreshing the LSP page will replace the
previous function with a new function.
 
Example 2:
The example listening on port 8090, referred to as 'ex2' below, expects
a Lua callback function to be installed. To install the callback
function, use the following code:
 
UDPTST.install(function(msg)
    trace("CB received:", msg)
end)
 
You can insert this code into an LSP page as well. Calling the
UDPTST.install() function repeatedly will replace the previous
callback with the new callback.
 
The following Python script can be used for sending UDP broadcast
messages on port 8080 and generate UDP events for example 1. To test
example 2, change the port number in the Python script to 8090.
 
Here is the Python script:
 
#!/usr/bin/env python3
import socket
import time
client = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDP)
client.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
client.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)
client.settimeout(0.2)
cnt=1
while True:
    message = "UDP Message %d" % cnt
    cnt=cnt+1
    client.sendto(message.encode('utf-8'), ("255.255.255.255", 8080))
    print(message, flush=True)
    time.sleep(1)
*/
 
#include "xedge.h"
#include <stdlib.h>
 
#define PORT_EX1 8080 /* UDP listen port number for Example 1 */
#define PORT_EX2 8090
#define MAXLINE 1024
 
/* The LThreadMgr configured in xedge.c */
extern LThreadMgr ltMgr;
/* The Socket Dispatcher (SoDisp) mutex protecting everything. */
ThreadMutex* soDispMutex;
/* Lua REGISTRY index reference to callback installed when calling
 * UDPTST.install(). See function installUdpCallbackEx2() below.
 */
static int luaFuncRefEx2;
 
/* This callback is called by one of the threads managed by LThreadMgr
 * when a job is taken off the queue and executed. The callback
 * attempts to find the global Lua function 'udpmsg', and if the
 * function is found, it will be executed.
 */
static void runLuaEx1(ThreadJob* job, int msgh, LThreadMgr* mgr)
{
   lua_State* L = job->Lt;
   lua_pushglobaltable(L);
   lua_getfield(L, -1, "udpmsg");
   if (lua_isfunction(L, -1))
   {
      lua_pushstring(L, (char*)(job + 1));
      lua_pcall(L, 1, 0, msgh);
   }
   else
   {
      HttpTrace_printf(0, "Err: global lua function 'udpmsg' missing\n");
   }
}
 
/* This callback is similar to runLuaEx1() above, but instead of
 * looking up a global Lua function, it uses the Lua function
 * referenced by the luaFuncRefEx2 variable. This variable is set when
 * Lua code calls UDPTST.install().
 */
static void runLuaEx2(ThreadJob* job, int msgh, LThreadMgr* mgr)
{
   if(luaFuncRefEx2)
   {
      lua_State* L = job->Lt;
      lua_rawgeti(L, LUA_REGISTRYINDEX, luaFuncRefEx2);
      baAssert(lua_isfunction(L, -1));
      lua_pushstring(L, (char*)(job + 1));
      lua_pcall(L, 1, 0, msgh);
   }
   else
   {
      HttpTrace_printf(0,"Err: Lua callback function 'udpmsg' not installed\n");
   }
}
 
 
/* This is a socket dispatcher (SoDisp) callback function. The
 * function is called by the SoDisp instance when we receive an UDP
 * message. The function is responsible for reading socket data,
 * creating a job by calling ThreadJob_lcreate, and dispatching the
 * job by calling LThreadMgr_run. This function is used by both
 * example 1 and example 2 code.
 */
static void asyncDispRecEv(SoDispCon* con, int port, ThreadJob_LRun callback)
{
   char buffer[MAXLINE];
   struct sockaddr_in sin;
   socklen_t len = sizeof(sin);
   memset(&sin, 0, sizeof(sin));
   sin.sin_family = AF_INET;
   sin.sin_port = htons(port);
   sin.sin_addr.s_addr = INADDR_ANY;
   int n = recvfrom(con->httpSocket.hndl, (char *)buffer, MAXLINE-1, 
                    0, ( struct sockaddr *) &sin,
                    &len);
   if(n > 0)
   {
      ThreadJob* job;
      buffer[n] = 0;
      job=ThreadJob_lcreate(sizeof(ThreadJob)+n+1, callback);
      if(job)
      {
         strcpy((char*)(job + 1), buffer); /* copy received message to job */
         /* We do not need to lock the mutex since it was already
          * locked by SoDisp. This code validates that the SoDisp
          * thread is the owner.
          */
         baAssert(ThreadMutex_isOwner(soDispMutex));
         LThreadMgr_run(&ltMgr, job);
      }
   }
}
 
/* SoDisp callback for example 1
 */
static void asyncDispRecEvEx1(SoDispCon* con)
{
   asyncDispRecEv(con, PORT_EX1, runLuaEx1);
}
 
/* SoDisp callback for example 2
 */
static void asyncDispRecEvEx2(SoDispCon* con)
{
   asyncDispRecEv(con, PORT_EX2, runLuaEx2);
}
 
/* This function opens a UDP socket on a specified port number and
 * performs the necessary steps for installing the socket in the
 * socket dispatcher SoDisp. Although this function is required for
 * this example to function properly, you do not need to comprehend
 * the details of its implementation, since the main focus of this
 * example is to demonstrate how to use the LThreadMgr.
 */
static void openServerSock(int port,SoDispCon_DispRecEv callback)
{
   SoDispCon* con;
   struct sockaddr_in servaddr;
   int sockfd;
   if ( (sockfd = socket(AF_INET, SOCK_DGRAM, 0)) < 0 ) {
      perror("socket creation failed");
      exit(1);
   }
   memset(&servaddr, 0, sizeof(servaddr));
   servaddr.sin_family = AF_INET;
   servaddr.sin_addr.s_addr = INADDR_ANY;
   servaddr.sin_port = htons(port);
   if(bind(sockfd,(const struct sockaddr *)&servaddr,sizeof(servaddr)) < 0)
   {
      perror("bind failed");
      exit(2);
   }
   /* Quick and dirty; memory never released. */
   con = (SoDispCon*)baMalloc(sizeof(SoDispCon));
   SoDispCon_constructor(con, ltMgr.server->dispatcher, callback);
   con->httpSocket.hndl = sockfd;
   SoDisp_addConnection(ltMgr.server->dispatcher, con);
   SoDisp_activateRec(ltMgr.server->dispatcher, con);
}
 
/* Open and install the two UDP socket objects used in this example.
 */
static void initSocketServer()
{
   openServerSock(PORT_EX1, asyncDispRecEvEx1);
   openServerSock(PORT_EX2, asyncDispRecEvEx2);
}
 
/* This Lua binding is registered by the xedgeOpenAUX() function below
 * and is invoked when Lua code calls UDPTST.install(). The binding
 * saves a reference to the specified function (argument 1) in the
 * luaFuncRefEx2 variable, which is used by the runLuaEx2() function
 * to push the function onto the Lua stack.
 */
static int installUdpCallbackEx2(lua_State* L)
{
   if(lua_isfunction(L, 1))
   {
      if(luaFuncRefEx2)
      {  /* Release old callback */
         luaL_unref(L, LUA_REGISTRYINDEX, luaFuncRefEx2);
      }
      lua_settop(L, 1); /* Make sure we only have one arg. */
      /* Save reference to function in registry */
      luaFuncRefEx2=luaL_ref(L, LUA_REGISTRYINDEX);
   }
   else
      luaL_typeerror(L, 1, "function");
   return 0;
}
 
 
/* This function installs the Lua binding, enabling Lua code to call
 * UDPTST.install(). The code employs standard Lua syntax, which is
 * included in any literature that explains Lua bindings. The function
 * is called by the Xedge startup code.
 *
 * NOTE: See also led.c: xedgeOpenAUX(), as this code enables additional
 * Xedge features you may consider using.
*/
int xedgeOpenAUX(XedgeOpenAUX* aux)
{
   soDispMutex = HttpServer_getMutex(ltMgr.server);
   initSocketServer();
   static const luaL_Reg reg[] = {
      {"install", installUdpCallbackEx2},
      {NULL, NULL}
   };
   luaL_newlib(aux->L, reg);
   lua_setglobal(aux->L, "UDPTST");
   return 0;
}

Using request/response in Lua bindings

You may be porting an existing web application to the Barracuda App Server or you may simply need the server's request or response object in your Lua binding. The following example shows how to write a Lua binding that may use the request and response objects.

static int LuaCmdEnv2C_printParams(lua_State *L)
{
   HttpCommand* cmd=baluaENV_checkcmd(L, 1);
   HttpRequest* r = &cmd->request;
   HttpParameterIterator iter;
   HttpParameterIterator_constructor(&iter, r);
   for( ; HttpParameterIterator_hasMoreElements(&iter);
        HttpParameterIterator_nextElement(&iter) )
   {
      printf("Key/Val: %s = %s\n",
             HttpParameterIterator_getName(&iter),
             HttpParameterIterator_getValue(&iter));
   }
   return 0; /* No return value */
}

The above Lua binding uses function baluaENV_checkcmd for extracting the HttpCommand argument. In Lua the request/response objects are the same and is a Lua representation of the HttpCommand object.

Lua code can now call the function as follows:

LuaCmdEnv2C.printParams(request)
LuaCmdEnv2C.printParams(response) -- request = response

Notice the baluaENV prefix in function baluaENV_checkcmd. The ENV prefix means that the function requires a special environment defined for the Barracuda App Server. The environment is provided as a C Closure. We must set this environment when we register the Lua binding.

In the first example above, we used the macro luaL_newlib to register the Lua bindings in function installLua2C. This macro is defined as:

#define luaL_newlib(L,l) (luaL_newlibtable(L,l), luaL_setfuncs(L,l,0))

Notice that the macro passes in the value zero as the number of up-values to function luaL_setfuncs.

The macro luaL_newlib cannot be used for this example since we need to push the Barracuda App Server environment onto the stack and let function luaL_setfuncs set this environment for the Lua bindings. Instead, of calling macro luaL_newlib, we do the following:

luaL_newlibtable(L,funcs); /* The table LuaCmdEnv2C */
balua_pushbatab(L); /* Push the BA ENV required by func baluaENV_checkcmd */
luaL_setfuncs(L,funcs,1); /* nup=1 (arg 3): the BA ENV */

Download Example Code

Download the source code for the above example.