Our online Lua Binding Generator is a good starting point when it comes to building your own Lua bindings, thus enabling LSP pages and Lua scripts to call your own C functions.
The following documentation is for writing manually crafted Lua bindings. Manually crafted bindings are preferred for more complex interactions such as interactions with objects on the C side. The Lua book provides a great introduction in how to design Lua bindings.
The tutorials in this paper explain:
- How to design a Lua interface to a C object, such as a handle in C code or a C++ class.
- How to release the global mutex for code that is time consuming, thus enabling concurrent requests.
- How to call Lua code asynchronously from C code.
Object oriented access and releasing the global mutex
The following tutorial covers how to write a Lua interface to a C object. You can use the same method for exporting a C++ class to Lua. The tutorial also shows how to release the global mutex for C functions that may take some time to complete, thus enabling the web server to accept other HTTP requests while the C function is running.
The tutorial requires that you have an understanding of Lua user data and Lua metatables as described in the Lua book. An older version of the book is available online where you can read the two sections Object-Oriented Access and metatables.
The Lua interface to the C object will be as follows:
Dynamically create a Lua object that represents the C object.
The C object is created by calling the open function in the global Lua2C table. The Lua2C table and the open function are all part of this exercise. In other words, we will extend the Lua engine with this functionality.
The dynamically created object will have two functions that can be called.
The "fast" and "slow" methods increment a counter when called and return the number of times the function has been called. The "fast" function returns immediately and the "slow" function takes 5 seconds to execute. The slow function is therefore designed such that it releases the global mutex while processing, enabling other threads in the server to execute.
Lua can return multiple values so both the "fast" and "slow" functions return its own counter and the counter for the other function.
Creating the Lua2C table and binding the 'open' function to the Lua2C table.
We must start by creating a C function that registers our Lua2C table.
The luaopen_Lua2C is the only global function in the C file, and it is the function you must call from your startup code – i.e. from the code creating the main Lua state. The name of the function can be any name, but we have opted to use a naming convention that is compatible with the Lua function require. Our C file can be compiled as a standalone module, on operating systems that support DLLs/shared libraries, and loaded on demand by calling require("Lua2C").
Notice how we set the last argument in function luaL_requiref to TRUE. This construction registers the Lua2C table as a global variable and makes it possible to reference this table directly in the global namespace, without having to call require("Lua2C") from the Lua code.
In the typical embedded system, function luaopen_Lua2C will be called just after you created the main Lua state, and our C file with the Lua2C bindings will be compiled and linked with the system. In other words, it will not be loaded dynamically as a dynamic library.
Function luaopen_Lua2C returns a value designed for dynamic loading via require and the return value is not needed when we link the code directly with the server code. What we do after creating the main Lua state "L" is to call luaopen_Lua2C and pop the values pushed by luaopen_Lua2C off the stack. This keeps the Lua stack balanced.
In function luaopen_Lua2C, luaL_requiref takes function installLua2C as an argument. The installLua2C function installs the "open" function in a new Lua table and returns this table. Lua will then register this table as "Lua2C".
Lua code can now call Lua2C.open() and a call to this function from Lua ends up in the Lua binding Lua2C_open.
The key to understanding the above code is to read the Lua documentation for luaL_newmetatable and read up on how metatable.__index = metatable works. This function creates a new table the first time it is called and retrieves the table on subsequent calls. The table is set as a metatable for the Lua userdata, i.e. for the C object Lua2C. The metatable binds the 3 Lua functions, "slow", "fast", and "__gc" to the dynamically created Lua2C object, thus making it possible for Lua code to use the userdata and call function "fast" and "slow". The "__gc" function is run by Lua when the garbage collector reclaims the memory for the userdata object. The "__gc" function is the destructor.
Lua code can now call function "fast" and "slow". The following will print out "1 0" and "2 0".
The above Lua example code triggers the C function Lua2C_fast twice.
Function Lua2C_fast verifies that the first stack argument is a Lua2C object, typecasts the Lua userdata object to a Lua2C object, increments the counter, and pushes the two return values onto the Lua stack.
Releasing the Global Mutex
Function Lua2C_slow is similar to function Lua2C_fast and performs the same checking, increments the counter, and pushes the two values onto the stack. The "slow" function also sleeps for 5 seconds, thus simulating a lengthy C call.
The above code fetches the server's main mutex and releases the mutex just before sleeping for 5 seconds. The mutex is reclaimed just after it sleeps. You must make sure you have the mutex locked when working with the Lua state L. The mutex will always be locked when the Lua binding starts, and the mutex must be locked when it returns.
When an LSP page calls the "slow" function, it will block for 5 seconds. This enables other threads in the server's pool to execute. An introduction to the thread mechanism in the server can be found in the introduction under section Thread Mapping and Coroutines. Many of the Barracuda Application Server's Lua bindings release the mutex when calling code that can block. Examples of such functions are the functions in the SQL library and functions in the response object such as response:write.
Each native thread in the server's thread pool is mapped to one Lua coroutine stack; thus the server can handle many concurrent Lua stacks at the same time. We will explain more on how to map a coroutine stack onto a native thread in the next section, where we explain how to call Lua from C code.
Download Example Code
Download the source code for the above example.
Automatic Mutex Wrapper Generation
It is tedious to create code that releases the mutex during lengthy function calls if a large number of functions must be modified. You may also have existing Lua bindings, or Lua bindings from the Internet that you want to download and quickly modify. Real Time Logic has prepared two scripts that you may use to automate this process.
mutex.lua: Mutex wrapper script generator
This script creates C code that wraps around the original functions. Each wrapper function releases the mutex, calls the original function, and locks the mutex when the original function returns.
Let's say you have the following function and you want to wrap this function into another function that releases the mutex when it is called.
Create a file such as functions.txt and put all the function declarations you want to wrap into this file. In our case, we put the above function declaration into this file. The next step is to create the wrapper code. Run the mutex script as follows:
lua mutex.lua functions.txt my-generated-wrapper my-header.h
The Lua script mutex.lua parses the function declarations in functions.txt and produces my-generated-wrapper.h and my-generated-wrapper.c. The last argument passed into mutex.lua (my-header.h) is to a header file that provides the original function declarations. This file is included in the output (my-generated-wrapper.h).
my-generated-wrapper.h:
my-generated-wrapper.c:
fext.lua: Function declaration extraction script
The file that lists all functions (functions.txt) can be manually created or you can use fext.lua to parse all your header files and automatically produce functions.txt.
$lua fext.lua Usage: lua fext.lua [-p pattern] [-c cdata-blob-file] [-e exclude-file] input-files....
The optional pattern (-p) means that you should only include files that matches this pattern. See Lua patterns for details.
You do not want to create wrappers for functions not being used by your Lua bindings. The optional -c option is for loading a "blob file" that is a concatenation of all C files that are part of your Lua binding library. When a function is found in the header file(s), the "blob file" is searched. The function is only included in files.txt if it is found in the "blob file". You can, for example, create this file as follows: cat *.c > c-data-blob.txt
You may further want to limit the set of functions included. The -e option is for a file that lists all functions that should be excluded.
Download mutex.lua and fext.lua
For additional examples on how to use the above two Lua scripts, see the build scripts (build.sh) for the MongoDB and PostgreSQL Modules.
Calling Lua Code Asynchronously From C Code
The Barracuda Application Server enables multiple threads to call and execute Lua code. In the previous section, we explained how threads in the thread pool can simultaneously execute in the Lua VM instance by using coroutine thread stacks. Only one thread can execute at any time in the VM, but Lua scripts can call Lua bindings (C functions), which may then release the global mutex protecting the Lua VM. This construction enables multiple threads to co-operatively run the Lua VM instance.
Easy solution
The code required for calling asynchronously from C code into Lua is more complicated than calling from Lua to C code. We will go into the design below, but before doing so, you should consider an alternative implementation that is easier to implement. The Barracuda App Server provides an API where you can create native threads in Lua. The Lua code executing in the native thread may be designed to repeatedly call a Lua to C function that blocks until data is available. The C side of the Lua binding releases the global mutex (as explained above), uses an OS primitive to wait for data, reclaims the mutex, and returns the data to Lua, thus simulating an asynchronous call.
Full solution
Let's revisit the diagram from the Thread Mapping introduction.
First, we will demonstrate how to run Lua code asynchronously using the Lua Thread Library. Afterward, we will show how to run Lua code asynchronously using "User Defined Events", which is more complicated. See the above diagram for these two options.
Send Events Via The Lua Thread Library
Suppose you have an asynchronous source that generates two integers, and you want to call a Lua function and pass those integers as arguments when the event occurs. If your C startup code created an LThreadMgr instance, you could use the Lua Thread Library to achieve this.
The following must be in your startup code. See Xedge's C code for example.
The following code shows how to use the LThreadMgr API to call a Lua function:
References: LThreadMgr, ThreadJob, ThreadMutex, SoDispMutex, see the Lua documentation for lua_pushinteger and lua_pcall.
The C file AsynchLua.c in the Xedge's source code directory shows how to implement a complete example. Here is a copy of the AsynchLua.c file:
User Defined Events
The following code snippet is from the Web File Server C code wfs.c, and the function shows how to call an unload handler (Lua function) from C code just before the server terminates. The optional unload handler can, for example, be used to cleanup temporary resources just before the server exits.
We push the global variable onunload onto the stack and check if the variable pushed onto the stack is a function. We then call the function by using lua_pcall, which runs the Lua function in protected mode and catches any exceptions thrown by the Lua script.
The above code uses the main Lua state created by function balua_create and not a Lua coroutine state. This is okay since we are not running multiple threads just before exiting the Web File Server executable. However, this is not a solution you should use when calling Lua asynchronously from C code in a running system. You should not use the main stack since it could corrupt the system state. Instead, create a Lua coroutine thread for each native thread calling into Lua. You should have a one-to-one mapping between native threads and Lua coroutine threads.
A good place to create the coroutine thread(s) is just after you create the main state. The following code snippet is from the Web File Server code (wfs.c) and shows how to create the main Lua state 'L'.
Just after the above code, add code similar to the following:
The above code creates a new Lua thread Lt and it is the Lt state you will use in your native thread when calling Lua code from your C code. We must keep the main stack balanced at zero so we must pop the new thread off the main stack L. We cannot use lua_pop since this would make the garbage collector terminate the thread and make the Lt pointer invalid, thus causing your program to crash. What we do instead is to call luaL_ref, which pops it off the main stack, but keeps a reference so it is not garbage collected.
The following code snippet from wfs.c shows how the mutex variable of type ThreadMutex is initialized:
The native thread calling asynchronously into Lua code must keep a reference to the thread state Lt and to the main mutex (SoDisp mutex). The following code shows how you can design a function that uses the Lua thread state Lt and the main mutex when calling any global function.
Any error message is left on top of the Lua stack and it is safe to use this const pointer when the mutex is released. Any error is automatically erased the second time callLuaFuncFromThread is called by setting the stack top to 0.
The above example does not show how to pass arguments to the Lua function or check for any returned argument(s), except for 'error'. See the function lua_pcall for more information on passing in arguments to the Lua function and checking for return values.
Using request/response in Lua bindings
You may be porting an existing web application to the Barracuda App Server or you may simply need the server's request or response object in your Lua binding. The following example shows how to write a Lua binding that may use the request and response objects.
The above Lua binding uses function baluaENV_checkcmd for extracting the HttpCommand argument. In Lua the request/response objects are the same and is a Lua representation of the HttpCommand object.
Lua code can now call the function as follows:
Notice the baluaENV prefix in function baluaENV_checkcmd. The ENV prefix means that the function requires a special environment defined for the Barracuda App Server. The environment is provided as a C Closure. We must set this environment when we register the Lua binding.
In the first example above, we used the macro luaL_newlib to register the Lua bindings in function installLua2C. This macro is defined as:
Notice that the macro passes in the value zero as the number of up-values to function luaL_setfuncs.
The macro luaL_newlib cannot be used for this example since we need to push the Barracuda App Server environment onto the stack and let function luaL_setfuncs set this environment for the Lua bindings. Instead, of calling macro luaL_newlib, we do the following:
Download Example Code
Download the source code for the above example.