Welcome everyone, to my talk about Lua.
My name is Andreas.
You might have seen me on the nickname of ComicSansMS
which is a really silly nickname, so it's usually not taken,
except on Twitter, because Twitter's full of crazy people,
so I had to use an even sillier nickname there.
I'm one of the organizers of the Munich C++ User Group,
so yeah, if you ever happen to be in our around Munich
and want to get in touch with the local community,
give us a call, we'd be happy to set up a meeting with you.
And my day job is incredibly working as a software architect
for BMW on the autonomous driving software platform.
So today I'm going to talk about Lua.
So can I maybe get a quick show of hands,
how many people have used Lua in like a serious project,
like not a toy project but like,
something production, yeah, couple of hands.
For those of you who have not used it,
hopefully I can convince you that it's a language
that is worth investigating even for serious projects.
So, what is it that is distinctive about Lua
that sets it apart from other languages.
I'd say there is two features that are kind
of unique to it, the first one is that it's actually
not designed to be a standalone language.
It's an embeddable language, how they call it.
So usually you have like some application,
like the most famous use case for Lua is being games,
that is written in some other
programming language, often C++.
And then you have Lua in there to customize
like some small parts.
But, it's also being used
for programming microcontrollers, embedded microcontrollers,
or also like serious, serious desktop applications,
like Adobe Lightroom.
The other thing that is kind of unique about Lua
is that it's small.
It's really small.
So you're not supposed to be able
to read this thing on the left,
but that is actually the full EBNF syntax
of the language, which fits exactly on one A4
sheet of paper, the compiled binary is really small.
These 180 KB is actually the full Lua package
including the standard library
if you strip out the libraries
and the parser if you only need to run compiled BiCode,
you can make this significantly smaller.
The complete reference manual is only 82 pages
is really the complete documentation including
the full CAPI, that is required for integrating it
with a enclosing program,
it only has eight basic data types.
As you can imagine, from these numbers,
there's not a whole lot of features in there.
Sort of if you take like the Python
approach of batteries included,
Lua is sort of the opposite of that conceptually.
But, the cool thing about Lua is that,
for them is actually not a limitation.
Is actually a virtue.
And not only because this allows them to run on devices
where other languages wouldn't fit, it's also
that from a language design perspective,
they really embrace this property of being small,
and turn that into a very elegant language.
So, I always like to say, the whole language
fits into your head at the same time.
Which is something that I cannot say of most
other programming languages.
I certainly cannot say that of C++,
unless maybe you're one of those guys,
but like for me, if I use C++, I have to constantly
swap in and out of the different features of my head,
and with Lua I don't need to do that.
Which is really nice.
So, hopefully I will be able to convince you,
in the progress of this talk, that although Lua
is small it is quite a powerful and interesting language.
However, I have to say, the problem with this talk,
is like, um, I cannot assume
that there are any Lua experts in the audience,
so on the one hand it's going to be an introductory talk
to Lua, but on the other hand I also want to show
interesting examples, right.
And the problem with that is that these interesting
examples, they're sometimes a little bit contrived,
so and a little bit too clever, so, what you see here
might not be representative
of what you would write in every day Lua.
Like, typically it might be a lot simpler.
In particular what I want to say is,
if at any point during the talk you have the impression
that oh no, this is like, really unnecessarily complex,
and I don't get it and I don't really
want to use this stuff, please blame that
on the talk, and not on the language.
That out of the way, this is Hello World.
Notice that although my syntax highlighter
likes to make the print blue, it's actually
not a keyword, it's just a function
from the standard library, which in Lua's case means
it's just an ordinary function it does not
get any special treatment whatsoever.
So this is how we declare a function.
It's a dynamic language, so we don't have to give any types,
we just say that we have three arguments
and give them names, the interesting thing about Lua,
this is the first thing where we actually see it,
it being small and lightweight,
is that functions are actually not like this, this
special kind of thing that they are in C,
but all functions are actually lambdas,
all functions are anonymous, and this declaration
that we see up there, is just syntactic sugar
for what we see down there, which is the construction
of an anonymous function, which is then being assigned
to a variable of type f.
So, f is just a variable like any other,
and you just assign to it a value of type function.
And what that really means is like functions
are true first-class values.
Function values are not any different from strings
or numbers or anything else.
So that way we can reuse all of the properties
that we have for the other values also for functions.
That keeps it small, but that also enables
some interesting things.
So, for example, let's say I'm not happy
with the print function here.
I want to replace it by my own.
I can just do that.
It's just a value.
I just assigned to it a different function to it.
Let's say I want to count the number of print calls.
That could be a use case for this, right?
I just replace the vanilla print function
by the new function, and there's actually valid syntax,
this is a very adept function, which just increases
the counter, and then calls
the original print function within.
Now, you notice that there's actually not
any variable declarations in this code.
I just assigned values directly to variables
that were not previously declared.
This is something that most scripting language
actually will allow you to do, so if you just assign
to a new variable, you don't need to declare it,
it just automatically creates a new global variable
and puts the value into it.
Which is not a behavior that is always useful,
but we will get to that later how we can.
Maybe work around that if we don't like it.
What is really not nice about this code
is that we now polluted our global name space
with a variable named count.
So, if we only ever count the print function,
don't do much else in the program, that might be okay,
but really it doesn't look that nice.
So what we would like to do instead is keep
this count function at a smaller scope,
so, for example, have a function enable counting,
which like performs this hooking of the print function,
and have the count function and the old print function
as local variables in there.
So here notice if I prefix the variable with the local
keyword that declares that variable as a local variable,
it works pretty much exactly the same
as local variables in C++.
Now the interesting thing is if I now write the same code
that I wrote before, what will happen is that
the local variables from the end closing scope
will be implicitly captured in that new local function.
So like in C++ if you wanted to do this,
you would have to write in the area brackets
of lambda, the captures explicitly.
So like the names of count and old print,
you would have to specify them, write them out.
In Lua you don't need to do this.
Just as lexical scoping, so it knows
that it can pull in the stuff from the outer scope.
The interesting question is now.
I just made the count variable a local variable,
so how can I access my count now.
Like if I want to find out of often print was called.
I can just return a function that gives me back
the value of the counter.
And this is interesting now, right.
Because the count is an integral value, so it has
value semantics, and it also has value semantics
in Lua, just as it has in C++.
So, now have captured the same value
in two different lambdas, in two different closures.
And it still works.
So this is actually where the full
in full lexical scoping comes from.
That allows you to do something like this.
This is actually really difficult to pull off
in the implementation, and Lua couldn't do this
from the beginning, but since version 5.0, I think,
they support it, and it's a really powerful feature.
Notice that of course since we capture stuff
that means that our function objects are actual objects,
so whenever you read function, interpret that
as construction of a closure.
So there's actually memory potentially being allocated there
and this needs to be garbage collected,
potentially, at some point.
Let's talk about tables.
Tables are the most important data structure in Lua.
In fact it's the only complex data structure in Lua.
And, I actually don't know of any other language
that can get by with just one data structure.
Which is quite fascinating.
So what is a table, like, this is the simplest table,
it's just constructing an empty table with nothing in it.
But a table, imagine it as being an associative array.
So, you can use it just as you would use a C array,
but you can also use it as a dictionary.
Using a, using it to map keys to values.
And the interesting thing about Lua tables
is that they actually allow you to map any type of value,
to any other value, so for example, you can use functions
as keys, you can use other tables as keys, if you want.
This is again, very much, the Lua philosophy.
Right, like you only provide very few features,
but you make those as flexible as possible.
Notice that tables, unlike the numbers that you saw before,
have reference semantics.
So, for example, what we do here, is we basically built
assembly link list, we construct like two note tables,
each of them holding a value,
and then we link them together,
through the next field of the first note.
You don't usually need to build link lists like this,
because like if you just use plain tables,
they usually powerful enough for what you need to do,
but it's useful to know that you can do it.
As with functions before, whenever you,
read the curly braces, think of it as table construction,
so this means that you will potentially allocate memory.
Since tables are only complex data type,
if we want to build like a struct, or like a record type,
you also have to do that with tables,
which means like we simply create entries
in our dictionary, in our table,
for the different fields of the struct.
And since the syntax that we already know for accessing
the tables is a bit cumbersome for this use case,
Lua provides syntactic sugar with the,
for accessing the fields with a dot.
So the last line is just syntactic sugar
for the line buff there, completely equivalent otherwise.
So we want to add member functions to our record now,
we can just assign a function to one of the fields,
and then call it as member function.
So notice Lua's tables have reference semantics,
so this function actually mutates,
the value inside the table, and the call down there,
this is, think of it as a member function calling
in C whether to give this pointer explicitly.
Seems we don't want to pass it explicitly
because it's kind of silly, we also get syntax sugar
for that, with the colon syntax.
So, the thing about tables is that,
we don't really have types right,
so all of our complex numbers have the same type table.
So if we want to build multiple values of this table,
we usually have to supply a constructive function
like we see here.
So let's say now we have these complex numbers
and we want to add to some.
For this we would actually need operator overloading, right?
And Lua provides this through a feature called Metatables.
So everything in Lua is a table right?
So Metatable, the idea here is basically,
I provide a table which I then attach to a value,
and this table tells me how to treat this type
that is attached to, in the language.
So, there's a couple of special fields,
and they allow you to specify how it treats
the other authentic operators, the comparison operators,
how it treats the element access
with the outright brackets and so on.
So, I just changed my construction function
to add this Metatable, call that,
assigns the Metatable to my newly constructed
complex number, and then I can
just add them with the plus operator.
So, let's say now I have more complicated data type,
where I actually have invariance between the different,
on the different fields.
Might not want to allow the user to access them directly,
but rather encapsulate them.
So, instead of exposing your month and date
of the date directly, I actually want to provide
getter and setter functions.
And the way that we do this is in the construction,
we actually build a local table that contains
the actual data, then what goes in the table that we return,
is just to getter and setter functions,
which again access the actual data through the closure.
So some of the similar trick that we saw
with the counter before, and that allows us to give,
to implement encapsulation without any explicit
language support for it.
Which is kind of cool.
Of course, since a table is the end
just an associative array,
we also get a certain reflection properties for free.
Like, a table is a complex number if it has these fields,
real and imaginary.
And of course I can also inspect all of the fields
by just iterating over the table
which in Lua I do with this syntax, so the pairs function
just gives me back an iterator function,
which with each invocation returns the next field
of the table, and then I have this generic form,
that processes iterators.
So this allows me to perform reflection on individual
table that I have already in my hand,
but what about global variables?
I might need to know, which objects
are aligned around in my environment.
So Lua has a solution for that.
Global variables are actually not global variables,
they are entries in a table called _G.
So, as I said, everything was a table,
and global variables are just syntactic sugar
for accessing elements in this global table _G.
And this has a really interesting implication.
Cause like, as we saw before, we don't need
to be clear variables.
So what people really don't like about this,
if I make a silly typo like this, where I assign
to fobar instead of foobar, the language
will just silently create a new global variable.
Which is not what I wanted here.
But, G is just a table, right?
So this creates a new entry fobar into this table.
And I can just use Metatables to forbid this.
So, I can use, say, you're not allowed to create
any new entries into this table.
And Lua actually provides on their homepage module
that does this in a little more complicated way,
and the idea there, is that you can sort of restrict
where you are allowed to create global variables.
So like only allow to declare them in like certain
parts of your script, or only be allowed to declare
them through declare function,
which then bypasses the Metatable.
And this is I think a pretty cool feature
and it basically comes for free
from the properties of the tables,
and the fact that everything in Lua is a table.
So, this is all pretty cool, but if you have ever tried
to integrate scripting language with C++,
you know that this is usually not a lot of fun.
So, how that does work with Lua.
So as we said before, Lua is an embedded language,
that means the main functions still belongs to C++.
And in there, we use Lua more or less as a library.
So, we create a Lua State object, and then
we can just run some Lua code.
So who thinks that doing something like this
with a scripting language should
be any more complicated than this?
Okay, that's what I thought.
So, just running Lua, chunks of Lua code
in isolation is not very interesting.
We also want to communicate with our C++ program.
So in order to export functions from C into Lua,
so that Lua code can call them, we actually
have to wrap them in functions of this signature.
Like any function with this signature can be directly
injected into the Lua VM.
And as you might notice there, this actually
does not take any function parameters
and it also does not provide any return values.
It just takes the Lua State object.
And that is by design.
So the Lua API is all designed around
this concept of a stack.
And the idea of the stack is that whenever
you want to get a value out of the VM
or into the VM, you have to put it on the stack,
and then you can grab it from the stack.
And this is, this might seem a limited design at first,
because like if you imagine if you want to set a value
inside a table, you have to put the table on the stack,
you have to put the key on the stack,
and you have to put the value on the stack.
And then you have to make the call that does the actual
assignment, which is a little bit verbose.
But there are two fundamental advantages here.
The first advantage is that it keeps the API very small,
because we have only eight data types overall,
and you only need two functions per data type,
one to put it on the stack and one to get it from the stack.
The second implication is a little bit more subtle
but I think it's even more important,
is that it solves the ownership problem.
Cause the problem with garbage collector languages is,
if I now grab like a table, from inside the Lua VM,
and expose it to the C++ code, I actually need
to make sure that the VM's aware of the fact
that I am pointing to this table,
so that the garbage collector
doesn't pull it out from under me.
And with Lua, I cannot access the table
if it's not on the stack.
But as soon as it's on the stack,
the stack has ownership on it.
This might not seem like a big deal,
but if you're using this in everyday code,
this is actually really powerful.
So, pushing values on the stack,
I'm just gonna use number and string
as the example types here,
some of the types are a little
more complicated, but not much.
So let's say I just have two push functions,
so two overloads for push functions,
one that pushes a number, so Lua numbers
are usually double, but if you compile
for a platform where you don't have double available
you can actually change that
very easily to like pure interval types.
I would push a string.
And then we can very easily write a function like this
in C++ 17, which uses the fold expression,
to just push an arbitrary number of arguments to the stack.
Is everyone familiar with this syntax?
I've seen it quite a lot this week already.
So if you don't have C++ 17, available,
you can also use a library to do the same,
so here's an example that uses boost hana.
And if that also doesn't work for you
there was actually a talk, I think yesterday
by Joe Fan-Cue, where he have like some examples
on how you can emulate this feature with all the C++
versions, I just put the code up for reference,
you don't have to read it,
you're certainly not expected to understand it.
Because it's horrible.
But it's possible.
So what about the other way around.
How can I get values from the stack,
so out of the VM into C++.
So here the thing is that since Lua is a dynamic language,
I don't know what the type is of, of the value on the stack.
So I need to do the switch case here.
But, the thing is that, this is actually the only,
the only point where I'm willing to stick
to this factor where I don't know the type.
Because like, as soon as this function returns,
I'm actually again in the C++ type system,
so I want this to be like, as rigorous as possible again.
So, what can we use to as a return value here?
So we could of course use a base class,
like, solve the problem with virtual inheritance,
but that's just so 90's, right.
So, in this example, you have to define what like
your unified interface of all the different types is,
in my case it's just a type function.
So each value can tell you what it's type is.
And then you could just use a C++ 17 variant, for example.
So, like in a full implementation you would have,
there's only eight types in Lua.
And you know them all beforehand.
So variant is actually really good fit,
and you can also implement it with
very little size overhead.
Like you might lose some bytes when saving numbers
or bullions in there, but for tables, functions
and the more complex types, it's pretty much uniform
for memory consumption.
And then of course, this is how you get stuff
out of the variant with visitation.
If you don't like variants
and you would like to use a different kind of type erasure,
Louis Ti-Yong was giving a very nice talk
yesterday about runtime polymorphism,
where he explained all the different options that you have
if you don't want to use a variant.
Okay, so let's call a function now from C++.
Let's call a function.
So, we load the function on the stack,
get it by its name, put it on the Lua stack,
and you push all of our arguments onto the stack.
And we call the function, and then we inspect
the stack to get the return values
that were returned by the Lua function.
And I noticed that in Lua, a function can have
an arbitrary number of return values,
so I actually have to check, after the Lua call,
API call returns, I have to inspect the stack
to see like how many values were put on there.
This is also the reason why here I have to return
a vectoral value, because I'm not sure how many
arguments am I going to get.
And then I can just call a Lua function,
like here I'm calling a print function,
from the Lua standard library with a couple of arguments,
and we just print them to the console.
So of course this is the most general
signature that you can give
for such a core function.
You could constrain it, like if you know the number
of return values and the number of arguments beforehand,
or maybe even know their types.
So, you probably want to go to like the most
restrictive signature that applies for your case
to make sure you get the maximum support
from the C++ type system.
So to wrap up.
This is actually the description that the Lua people
give like how they understand the language,
their elevator pitch.
It's a powerful efficient lightweight
embeddable scripting language.
And I hope through this talk I could give you an idea
of what this attributes in the context of Lua.
I compiled a small list of literature,
the reference manual for Lua
is actually really good, there's a book by the authors
of Lua called Programming in Lua, which is,
it's not a big book, it's around 300 pages,
if you're interested in the language
I would really recommend purchasing the book,
not only because it's a great book,
but also because it's a great way to support
the creators of Lua financially.
Because it's created by a university,
and like they don't get a lot of funding otherwise.
And that, I think we have some time left for questions.
- [Moderator] If you have a question,
please go up to the microphone.
Thanks for the talk.
You mentioned that you have an application
that uses several libraries, for example, dynamic libraries,
and each of these library want to do some
of its own stuff, and as separate Lua interpreters.
So, how do they get, do they conflict with this
or okay for them to have several
interpreters in several application?
You mean, I think, several Lua interpreters,
in one application?
Yes, I mention several modules are developed
independently, and all of them rely on their own Lua,
something, so I want those to be independent.
Yeah, so what you can always do, like we saw
in the simple example that you have to create
this Lua State at one point, and if you want
the different modules to be truly independent,
you can just have each module use its own Lua State.
And then they can even run concurrently
in multiple threads, there's like zero overlap
between the different VMs, but then of course
you have to take care that if you do want communication
from one Lua VM to the next, than you need to model that,
but if you're fine with them being totally separate
then that's actually a good way.
The problem there is that there's some memory overlap
to those Lua States, so if you're in a very constrained
environment, you might not have that memory.
And what you can do there,
is you can also restrict the environment.
So this _G table, that we saw, that stores
all the global variables, you can also restrict that.
So saying that I'm now executing a chunk of Lua code
under this environment.
And sometimes that's enough of a separation,
but it's a little bit weaker
than having truly separate States.
And the second question is,
it seems like to use Lua functions conveniently
from C++ coding, it's some kind of boilerplate,
but we demonstrate this, so I wonder,
is there some standard solutions.
Standard solutions, some libraries
which contains such boilerplate.
A library which takes care of the bindings for you.
Well that provides such functions,
functions like Kol, like that is shown.
So, there are libraries that help with the integration,
a pretty famous one is I think called Luabind,
you can use those, but the interesting thing,
is that the API is so simple, and you want to roll
your own, this is not an unreasonable thing to do.
Like you can probably get a decent
library up and running within a weekend.
So, there are libraries available, but you don't
have to use them, like it's easy enough
to do it by hand, if you want.
[Audience Member 1] So how do you debug Lua?
Oh yeah, that's a good point.
So the interesting thing is actually part
of the standard library of Lua is actually
a debug module, so you can actually write
your debugger for Lua inside Lua.
Which is pretty cool.
[Audience Member 1] You can separate points
and things like that.
And you can inspect like local variables
from closure and stuff like that.
[Audience Member 1] And then you write your own.
Like if you want it to go to Standard Out,
and have some sort of interactive mode, you --
The thing is that this is just an API.
This is not a ready-to-use debugger.
So you would still need to write a user interface
in order to be able to actually debug.
But since Lua's an embedded language,
so you don't know what the environment is going to be,
they don't take care of that for you.
So if you only use vanilla Lua, you will have
to write it for yourself, but it gives you
all the tools that you need to be able to this.
[Audience Member 2] Just wanted to make a quick comment
if you don't mind, on the question of something
to help generate binding?
Sol2, that's S-O-L two, is a really well-written modern
C++ binding generator for Lua.
Thanks for the info.
And I think we're through.
Thank you very much.