Data-Driven C++ Code using Lua

By nlguillemot

In this article, I would like to share a C++/Lua programming technique that I enjoyed using in a past project. This technique allows you to define functions in a data-driven way using Lua scripts. These functions are converted into a domain-specific “bytecode” at initialization time, which can then be evaluated in C++ without using a Lua runtime. By evaluating functions as bytecode in C++, we can simplify and optimize Lua use by minimizing garbage collection and inter-language function calls.

This article is split into two main parts. In the first part, I describe the motivating use case that inspired me to use this technique. In the second part, I describe the details of how the technique was implemented for the aforementioned use case.

Several code samples are raised from a project on Github, which can be seen in full here: https://github.com/nlguillemot/robdd

Motivating Use Case

The project where I used this technique was an implementation of the Binary Decision Diagram data structure. The input to the program is a Boolean function, and the output of the program is the number of possible solutions that make the Boolean function true. More specifically, I implemented the Multicore Binary Decision Diagram as described by Yuxiong He.

For this project, I wanted to specify the inputted Boolean functions through scripts, mainly because I thought it would be cute and fun. As a final result, I was able to specify Boolean functions as follows:

title = 'test'
display = true a = input.a
b = input.b
c = input.c r1 = a*b + a*c + b*c
r2 = b*c output.r1 = r1
output.r2 = r2

This script is broken down into four parts: The options, the inputs, the Boolean function itself, and its outputs.

The script begins by setting some options. The “title” is the name of the function, and “display = true” will display the binary function as a dot graph. Following these options, some inputs to the Boolean function are declared. Inputs are declared through the “index” table, and those inputs are assigned to variables with shorter names for convenience. After declaring the inputs, they are used in some Boolean operations. Here, the “+” operator means logical OR, and the “*” operator means logical AND. Finally, the results of these Boolean operations are marked as outputs, by putting them in the “output” table.

The result of running the script is the following diagram, which shows all ways in which the outputs of the Boolean functions can either evaluate to false (0) or true (1). By tracing a path from the top of the diagram to the bottom, while following dotted lines when a variable is false, and following solid lines when a variable is true, you can see the result of the Boolean function for a specific assignment of values to its inputs.

Simple Boolean function

By specifying the Boolean function as a Lua script, I was able to programmatically express various more complicated Boolean functions. For example, an n-bit ripple carry adder and the n-queens puzzle were programmatically specified with generic functions in terms of “n”. I was also able to make a general Lua function to generate Boolean functions for 3-colorings or 4-colorings of graphs, and I was able to conveniently import and reuse this function from other scripts to define Boolean functions for graph-colorings of the Petersen graph, the states of the United States, and the prefectures of Japan. Since I used Lua as a configuration language, I was able to easily create all these complicated Boolean functions, which wouldn’t be so conveniently possible if I was using JSON or XML as the input of my program.

If you’re curious about more details of my Binary Decision Diagram project, please consult this report pdf.

Implementation

At a high-level, the implementation is split into two parts:

  • Running the Lua script to generate bytecode.
  • Executing the bytecode.

These two steps are described separately in this section.

Bytecode Generation

The generated bytecode has two main purposes: declaring new variables, and describing Boolean operations with them. Each instruction of this bytecode has an “opcode” that defines what type of instruction it is, and stores references to its inputs and outputs. The bytecode is built by appending an “instruction” to a std::vector every time an operation happens in the Lua script.

The following code sample shows the structure of the instructions used in the bytecode, and the std::vector that stores the instruction bytecode:

struct bdd_instr
{ enum { opcode_newinput, opcode_and, opcode_or, opcode_xor, opcode_not }; int opcode; union { struct { int operand_newinput_ast_id; int operand_newinput_var_id; const std::string* operand_newinput_name; }; struct { int operand_and_dst_id; int operand_and_src1_id; int operand_and_src2_id; }; struct { int operand_or_dst_id; int operand_or_src1_id; int operand_or_src2_id; }; struct { int operand_xor_dst_id; int operand_xor_src1_id; int operand_xor_src2_id; }; struct { int operand_not_dst_id; int operand_not_src_id; }; };
}; std::vector<bdd_instr> g_bdd_instructions;

The inputs and outputs to each instruction are specified by Abstract Syntax Tree IDs (“AST IDs”). Initially, the IDs “0” and “1” are specially assigned to “false” and “true”, respectively. From there, monotonically increasing AST IDs are assigned to newly created variables, and to results of Boolean operations. Since a new AST ID is created for every intermediate result, the bytecode effectively represents a program where the result of every operation is immutable, even if the Lua code that created it might have syntactically reused a variable name.

The monotonically increasing AST ID numbers (and the specially reserved values for “false” and “true”) are represented by the following code:

enum { ast_id_false, ast_id_true, ast_id_user
}; int g_next_ast_id = ast_id_user;

Beside the AST IDs, a different type of monotonically increasing ID is used to uniquely identify input variables to the Boolean function specified by the script. These variable ID numbers are used to keep track of the inputs of the function (such as “input.a” and “input.b”), and they are used to record associations between AST IDs and input variables.

The monotonically increasing variable ID counter (starting at 0) is simply declared as follows:

int g_num_variables = 0;

The code that connects the Lua script to these C++ data structures is implemented mostly through Lua metatables. In particular, the metatable for the “input” table, and the metatable for each “ast” node of the computation. The implementation of these metatables will now be described.

The Input Table

The “input” table has two defined metamethods. The first one is “__newindex”, which is simply defined to forbid assignment to inputs. Obviously, inputs should only be read, not written. It is defined as follows:

int l_input_newindex(lua_State* L)
{ luaL_error(L, "Cannot write to inputs table"); return 0;
}

The second metamethod, “__index”, is much more interesting:

int l_input_index(lua_State* L)
{ int* ast_id = (int*)lua_newuserdata(L, sizeof(int)); *ast_id = g_next_ast_id; g_next_ast_id += 1; auto varid2name = g_varid2name.emplace(g_num_variables, luaL_checkstring(L, 2)).first; g_num_variables += 1; bdd_instr new_instr; new_instr.opcode = bdd_instr::opcode_newinput; new_instr.operand_newinput_ast_id = *ast_id; new_instr.operand_newinput_var_id = varid2name->first; new_instr.operand_newinput_name = &varid2name->second; g_bdd_instructions.push_back(new_instr); luaL_newmetatable(L, "ast"); lua_setmetatable(L, -2); lua_pushvalue(L, -2); lua_pushvalue(L, -2); lua_rawset(L, -5); return 1;
}

When the “__index” metamethod is called, a new input variable is created. Each variable created with this method is given a monotonically increasing variable ID, and an AST node (with its own AST ID) is created to represent this variable in the computation. From there, a instruction that creates a new input is appended to the bytecode. This “new input” instruction stores the AST ID, the variable ID, and the name of the variable (for display purposes.)

The Lua API calls at the end of the function above do a few different things. First, it associates the “ast” metatable to a Lua object that contains the AST ID. After that, it uses “rawset” to insert the newly created Lua “ast” object to the “input” table. Finally, it returns the newly created “ast” object.

Note that the step with the “rawset” is crucial. The “__index” metamethod is called only if the key used to access the “input” table does not exist in the table itself. Therefore, inserting the object into the “input” table guarantees that each variable will only be created once. In other words, an expression like “input.a + input.a” will not create two duplicate variables or duplicate AST nodes.

AST Nodes

After a Lua object with the “ast” metatable is returned by accessing the “input” table, this object will be used in various operations. These operations generally take “ast” objects as input, and return a new “ast” object as output. For example, a logical OR (indicated by “+”) appends a corresponding OR instruction to the bytecode, and the result of the operation returns a new Lua “ast” object to the script. This newly returned “ast” object stores its own unique AST ID, and it can be used as input to further operations.

Note that these Boolean operations are deferred: Rather than immediately computing the result of these operations, the inputs and outputs of the operations are recorded, which allows the C++ code to replay the sequence of operations later.

As an example of a Boolean operator, consider the following code that implements logical OR:

int l_or(lua_State* L)
{ int ast1_id = arg_to_ast(L, 1); int ast2_id = arg_to_ast(L, 2); int* ast_id = (int*)lua_newuserdata(L, sizeof(int)); *ast_id = g_next_ast_id; g_next_ast_id += 1; bdd_instr or_instr; or_instr.opcode = bdd_instr::opcode_or; or_instr.operand_or_dst_id = *ast_id; or_instr.operand_or_src1_id = ast1_id; or_instr.operand_or_src2_id = ast2_id; g_bdd_instructions.push_back(or_instr); luaL_newmetatable(L, "ast"); lua_setmetatable(L, -2); return 1;
}

The code above gets the AST IDs of its inputs, and creates a new AST ID for its output. A logical OR instruction is then added to the bytecode. The output’s AST ID is stored in the Lua object with the “ast” metatable, and this newly created “ast” object is returned from the function.

Note that the “arg_to_ast” function exists to allow Lua’s built-in “true” and “false” values to be used in Boolean operations with “ast” objects. It simply returns the specially pre-allocated IDs if the input is a “true” or “false” value, and otherwise just returns the ast node’s ID, as follows:

int arg_to_ast(lua_State* L, int argidx)
{ if (lua_isboolean(L, argidx)) return lua_toboolean(L, argidx) ? ast_id_true : ast_id_false; else return *(int*)luaL_checkudata(L, argidx, "ast");
}

Putting it together

At the program’s initialization time, the metatables for the “ast” nodes and the “input” table are created, as well as the “input” and “output” tables themselves. After everything is set up, the Lua program itself is executed. This is done as follows:

lua_State* L = luaL_newstate();
luaL_openlibs(L); luaL_newmetatable(L, "ast");
{ lua_pushcfunction(L, l_and); lua_setfield(L, -2, "__mul"); lua_pushcfunction(L, l_or); lua_setfield(L, -2, "__add"); lua_pushcfunction(L, l_xor); lua_setfield(L, -2, "__pow"); lua_pushcfunction(L, l_not); lua_setfield(L, -2, "__unm");
}
lua_pop(L, 1); luaL_newmetatable(L, "input_mt");
{ lua_pushcfunction(L, l_input_newindex); lua_setfield(L, -2, "__newindex"); lua_pushcfunction(L, l_input_index); lua_setfield(L, -2, "__index");
}
lua_pop(L, 1); lua_newtable(L);
luaL_newmetatable(L, "input_mt");
lua_setmetatable(L, -2);
lua_setglobal(L, "input"); lua_newtable(L);
lua_setglobal(L, "output"); if (luaL_dofile(L, infile))
{ printf("%s\n", lua_tostring(L, -1)); return 1;
}

After the program finishes running, the C++ code iterates over the contents of the “outputs” table in order to keep track of the AST IDs that correspond to the output variables. I think this code is not interesting enough to explain in detail, but feel free to look at it yourself.

Bytecode Execution

After the bytecode list has been fully recorded, it is passed as an input to a “decode” function. At its core, this “decode” function is just a simple loop with a “switch” statement inside, which executes each Boolean operation one-by-one based on their opcode ID. Each operation is implemented by simply making the equivalent call using the API of the Binary Decision Diagram data structure. Namely, the “new input” instruction is implemented by calling a “make_node” function, and the other instructions are implemented using an “apply” function. The details of these functions are strictly related to the implementation of the binary decision diagram, which is outside the scope of this article.

As a more visual example of what the bytecode looks like, consider the following debug output produced by the simple Boolean function that was shown near the start of this article. In this debug output, each number represents an AST node ID. Therefore, a statement like “7 = 5 OR 6” means that the AST nodes with IDs 5 and 6 are OR-ed together to produce a new AST node with ID 7. Also, a statement like “2 = new 0 (a)” means that a variable called “a” was created with variable ID 0, and stored in AST node ID 2.

2 = new 0 (a)
3 = new 1 (b)
4 = new 2 (c)
5 = 2 AND 3
6 = 2 AND 4
7 = 5 OR 6
8 = 3 AND 4
9 = 7 OR 8
10 = 3 AND 4

When the bytecode has been fully executed, the “decode” function simply outputs the binary decision diagram nodes that correspond to the outputs of the Boolean function, which are used to display the output of the program.

Conclusions and Possibilities

In this article, I discussed the implementation details of a programming technique that allows functions to be specified in Lua and converted into a bytecode that can be conveniently evaluated from C++. Although this article focused on a particular implementation for the sake of explanation, I hope this is also helpful to solve similar problems.

If this technique is implemented for a different purpose, the details of the code will likely change a lot. For example, the bytecode’s representation, the “ast” nodes, and the bytecode interpretation would likely be different. With such tweaks, it may be possible to adapt this technique to various applications.

Potential applications include: SQL-like queries, particle effect specifications, render pass specifications, AI behaviors, and RPG battle system damage calculations.

As a further step, the “bytecode” could be used as a source of optimizations. For example, dead code elimination could be applied by traversing the list of instructions, or independent data structure lookups could be batched. The kinds of optimizations that can be applied depends highly on the application of the technique.

I hope you found this useful or interesting. Thanks for reading!