- Welcome to Modern C++ Testing with Catch2.
We'll get onto exactly what Catch is,
and what the difference between
Catch and Catch2 is, shortly, and so I'm Phil Nash.
I am a developer advocate at JetBrains.
So, if you want to know anything about JetBrains products,
come and catch me afterwards or down on the JetBrains booth.
Always happy to talk to people.
We're not gonna be talking about JetBrains products
in this talk today, we'll talk about Catch,
which is also something that I'm heavily involved with,
the original author and still the lead maintainer.
But before we get onto that, just want to
talk a little bit about this photo.
So, you maybe wondering what this is a photo of.
I was searching, just googling, for photos to do with catch,
just to see what comes up,
and this was one of the search results.
It came from Flickr and the photographer had a little story
about it, which was that it reminded him of
a superhero catching a speeding bullet,
you can sort of see that there.
Something kinda catching a bullet-like shape.
But, what it actually is, is a handrail on a bridge
somewhere just outside Toronto, I believe.
It's taken a very narrow depth of field
and that's where the blur's coming from.
So it's got nothing to do with catch at all,
but I really liked the photo, so I decided to keep it in.
So that's what that is.
So, almost ready to start.
Just wanna point out that all my slides are brought to you
with the help of Classic Programmer Paintings.
If you haven't seen this Tumblr,
you really need to check it out.
It's just lots of classic oil paintings with the titles
changed to something to do with software development.
Some of them are really, really amusing,
so, you'll see plenty of those.
Also, a little bit of help from Twitter,
I have one or two sprinkled through,
just to lighten the mood. (audience laughing)
But I think we should get started
with something a bit more serious.
And so, for our first oil painting, and I just want to talk,
just for a moment, about the cost of bugs.
Bugs are not the only reason that we do unit testing,
definitely a big part of it.
We want to get a low defect rate in our code.
I mean, who here doesn't want to lower
the defect rate in their code?
No, I didn't think so.
We all want to lower the defect rate,
so, we know that testing is a big part of that,
but it's nice to put some numbers to it.
There was a study in 2002 by NIST and they had...
There's this quite a long article about it,
there'll be references at the end if you want to
look that up, which I encourage you to do.
I just got a few choice quotes from that study though,
which I think are quite interesting.
First of all, that software bugs are costing the US economy,
so that's just the US economy, in 2002 remember,
an estimated $59.5 billion each year.
That's quite staggering if you think about it.
Just wasted on bugs.
And that improvements in testing could reduce this cost
by about a third, or $22.5 billion.
So, testing alone won't solve the problem,
but a big chunk of it, they've estimated, could be improved.
This is back in 2002.
And the final bit, the general consensus seems to be,
that the current state of the art with respect to testing
is poor and can be sufficiently improved.
So, do bear in mind this is in 2002,
just sort of before the start of the Agile revolution.
Could be argued that the state of the art,
with regard to testing, has improved since then, but I think
we still have a little way to go, particularly in C++
where it's not been quite as big
a part of the ecosystem as other languages.
And in fact, we'll come onto how that
fits into our topic today in a moment,
but I think that's a little bit too heavyweight,
so quick I go back to Twitter.
(audience member laughing)
Why did I write Catch in the first place?
So, back in 2010, I started work on a project
where the approach to software quality
was basically, you know, hope and pray
and was running into a lot of problems because of this,
so I decided to have a look around
at what the state-of-the-art, with regards to testing,
was the time in C++, and there is a Wikipedia page
with all the test frameworks, broken down by language,
and the list of test frameworks for C++ is quite big.
I don't expect you to read it all, there's actually
76 entries in there, and that's not exhaustive.
There'll be plenty that aren't on there.
So, why did I decide to write yet another testing framework?
Well, I can't say I tried all of these, but I did try
all the main ones, the popular ones the time, the big names
that you'd recognize, won't name names,
but I didn't particularly like any of them.
Not that I didn't respect the work that had gone into them,
but it was just too hard to get them, to set them up,
get them running, and just to write code in them
when you do get them up and running,
too much ceremony, too much friction.
Getting people to write tests in the first place
is hard enough, but if you make that even harder
they're just not gonna do it, they're gonna give up,
be disheartened and think
that it's just too much extra work.
I wanted to cut through all of that, give people something
that was simple, easy to get, easy-to-use,
and actually fun to test.
So, whether I achieved that, I'll leave that up to you.
I'm gonna show you some examples, some demos,
go through some features.
I'll let you decide, I'm not going to give you
any leading titles at all, but what I'm gonna do
is drop to a demo,
while I switch that over, okay.
So whoa, too far.
So, here I've got CLion, JetBrains's IDE,
did say we weren't gonna be talking about that, but I lied.
What we're gonna do is just create a new project,
let's call it CatchTest4, because
I've run through this demo a couple of times now.
So, it's brand-new project, completely fresh and,
like any good IDE, it puts a load of junk in there
that you don't really want.
Get rid of that, so now we have no code.
So it's a good place to start, really.
What we're going to do, in true TDD faction,
is write some code that won't compile
because we don't even have this header file yet.
So, Catch is a single header library.
Just a single file.
We don't have it.
So, of course that's not gonna compile.
So, put the file over here just to say save me getting it
from Github, but that's what you'll have to do.
We drag that in
and now that does compile, but it doesn't link
because we don't have a main.
So what we do is, just before that include, we define
CATCH_CONFIG_MAIN, and that
should now compile and link.
Just make that full screen, there we go.
And that's a complete, working application,
believe it or not.
We've run that.
You'll see, not particularly interesting,
we haven't written any tests,
but it does tell us that, at least, that's quite nice.
And if we go to a terminal...
Let me just grab the path,
and what did I call it, CatchTest4.
There we go, so we can see that we've also got a
complete command line processor in there
we'll look at a bit more later.
So it's quite rich, quite a lot you can do
before we've even written more than two lines of code.
Now, we can go ahead and start writing code in this file.
The trouble is, because we've written that
CATCH_CONFIG_MAIN, it's gonna compile the whole library
into that file, so it's gonna be a bit slow.
So, what I'm gonna do is create a new file,
Let's include Catch again.
Now we're ready to start writing some tests,
so, write the test case.
We don't even need to give it a name, and if you've used
other testing frameworks, you might be surprised
we haven't had to give it a class as well,
because test cases in Catch are really just functions.
They're generated for you behind the scenes.
And now we can write an assertion, which is REQUIRE.
In Catch there's another form we'll look at later.
So, some pretty straightforward maths.
I'm sure this is gonna work the first time.
Let's run that and see what happens.
And it failed.
Well now, we can see what the failure looks like.
So you can see it has named our test
just Anonymous test case.
We'll come back and name it in a moment.
Given us all the file and line numbers.
Come down here, we can see the assertion we wrote,
but here we can see it's been able to expand the values
that we wrote, even though we haven't had to separate it out
as you might have seen in other test frameworks.
Instead of saying assert equals, assert not equals,
we just use a simple expression and Catch
uses expression templates to decompose that expression
and then reconstruct it for you, so that it gives you
rich output, without you having to
contort the way you write the code.
Let's go in and give it a name as well.
Only you'll see again, unlike many other test frameworks,
we can just write a plain string here, rather than having to
contort our test names into legal identifiers.
And now, why is this failing?
I'm pretty sure the maths is right.
What I forgot to do was to it in base 13, of course.
So, we don't have a base 13 function.
What I've done is, I prepared one earlier,
so I don't have to type it all out, here we go.
Pretty simple. (audience laughing)
If we ran that now, we can see that
it all works, and we're all green.
So it's got nice color coding there, as well.
Though, if you are using CLion or ReSharper C++,
it has a built-in Catch test runner you can also use.
So, find Catch in the list there.
If we run that again now,
we'll see it runs in this hierarchical view.
Maybe if we make that fail it will be more interesting.
There we go, see the output again.
Okay, so that's our first demo, for the moment.
Let's go back to our slides.
So that is just Catch, plain Catch,
as it's been for about ten years, eight years, sorry.
But, Catch was originally written for C++98/03,
'cause it came out in 2010, it predated C++11.
In fact, throughout the years, a large proportion
of the user base was still stuck in prehistoric C++.
But, what I really wanted to do was
turn the dials up to 11, C++11 of course.
And I didn't want to break compatibility for everyone else,
so I put it off, put it off, put it off, until last year
when I said, "Enough is enough," and decided
to take a branch of the major version,
so we're now on Catch2, and breaking compatibility,
so it's C++11 only, but we still have Catch1.x on a branch
so, if you are stuck with an older compiler,
you can still use it, and we are maintaining that
with, at least, important bug fixes.
It's been two or three over the last nine months or so,
so it's not very high traffic.
But if you are stuck on that, then I'm sorry,
but Catch has moved on.
So, what's new in Catch2, apart from the language change,
because, well, one thing that was really nice about
moving to C++11 was the Catch code base is big enough
for that to be interesting and meaningful, but small enough
that we could go in and just do complete update to C++11,
find all the nooks and crannies,
and all the things that have been
weighing us down over the years, update it C++11.
It's great for us as maintainers,
but it doesn't really do a lot for the users.
So what else is new in Catch2?
Well, let's have a little dig below the surface,
'cause some of the changes do run quite deep,
but we're gonna look at just a number of new features.
Here's a list of most of the major things.
We're not gonna cover them all now.
I just want to dig into just a few of these,
and I want to start with a new name.
And first of all, why does it need a new name?
Well, a lot of people tell me that it's very hard
to find information about Catch by searching on Google,
in particular on Stack Overflow.
It's such overloaded term in C++, and testing,
and the intersection of the two, so even if you put
extra keywords in there, it doesn't seem to help much.
People were complaining about this for long enough
that I decided I really need to do something,
but I didn't want to break the momentum behind the name,
as well, it's already starting to become popular.
I didn't want to lose that.
So, the new name for Catch is Catch2.
And it might sound like a silly thing but,
over the last nine months since we've released Catch2,
that situation has improved.
It's now much easier to find
information online about Catch2.
So that's one of the reasons I'm doing these talks,
is to try and publicize that name change, so that people
write more about Catch2 and we get better search results,
but it's working, and we haven't
really had to break too much.
Another one I wanna mention briefly is micro-benchmarks.
I started putting support for this in
'cause I was doing a lot of benchmarking.
I was mostly using a framework called Nonius.
I don't know if you've used that at all.
I wanted something similar but
was built into Catch and ran more like Catch.
'Cause there's a lot of shared infrastructure involved.
So, I started putting support in.
It's not really complete.
It's definitely not production-ready.
You can try it out, but please don't rely on it.
But what's interesting is, since I did that,
the author of Nonius, R Martino Fernandez, or RMF,
you'll often see his name as, he approached me
and talked about merging our efforts, actually merging
a lot of the Nonius codebase maybe into Catch.
So that might actually help just to round it out.
Unfortunately, I haven't had time to follow through
on that yet, but that's something that's coming,
something for the future.
Okay, another big feature is matchers.
This actually came in just at the end of Catch1,
but it's actually been rounded out a lot more in Catch2,
so I think it's worth putting in here.
I think I've, okay...
We'll come back to matchers 'cause I accidentally
put this in the wrong place.
I decided this just before we started,
so I forgot about this.
Just recently we've added support for -fno-exceptions,
so if you do run an environment where exceptions
are not possible, you can now use Catch for the first time.
It's been a long requested feature.
They said it couldn't be done, but we did it.
I believe, but if a test does fail, which would normally
involve throwing an exception to terminate,
it would terminate the whole application,
so it's not great, but you can use it.
But, what I wanted to talk about was matchers.
So there it is, I just put the slide in the wrong place.
If you've used matchers in other languages
this should all be familiar, and if you haven't,
I'll just walk you through it a little bit.
So, we have an alternative set of macros.
We've got REQUIRE_THAT and CHECK_THAT, and then
the first argument is the thing you want to actually,
the value you want to test,
and the second art argument is the matcher.
So, in this case we're looking at string matchers.
So we want to see whether this string contains a substring,
or starts with a substring, ends with the substring,
so pretty useful already.
It gets even more powerful when you compose them together
using the logical composition operators And and Or,
also Not, you can use brackets for precedence,
so it's a very rich compositional approach
to building some quite complex matchers
just out of very simple components, that's really nice.
It's not just strings, we've got
some built-in matchers for vectors, for example.
So, these ones test whether a vector contains
a specific element, and then we've got variants that will
test for if a vector contains another vector.
We've got some similar matchers for maps,
and things like that as well.
But, matchers are a completely open customization point,
so you can write your own matchers for anything you like.
- [Audience Member] You had CHECK_THAT,
and then there was another.
What was the other one?
There were two kinds CHECK_THAT and--
- Yeah, I skipped of the difference
between the CHECK and REQUIRE forms.
They basically work exactly the same,
but with a REQUIRE, if a test fails,
it will abort the test, if the check fails it will carry on
but it will just report the failure,
so that's the difference.
- Thank you. - May I ask another question?
Why do you guys have vector contains and just contains?
Couldn't it be just overloaded?
- That's here, so the question was,
"Why do we have vector contains and contains?"
So they mean different things,
and I wanted to do this using overloading.
I can't remember exactly what the problem was now,
but it wasn't possible.
I think there was some ambiguity, somewhere.
So I had to disambiguate one of them.
That's well enough, but I'm never quite happy with it,
but I was more interested in getting functionality in.
If you've got any ideas for improving it,
then do let me know.
So, custom matchers:
So, all you have to do
is write a class that has a couple of overloads.
Just walk you through it very briefly.
So you just need to derive from this base class,
where it's templatized on the thing that you're matching,
in this case SpecialException, then we have a constructor
that takes the thing that you want to test against,
and then stores it as a member variable.
Then the first overload is match, that just takes the thing
being tested against to compare with
the thing you're holding, however you want to do that,
that's where you do your matcher logic, of course,
and then describe, just to be able to print out
what's happened in the event of a failure.
So that will depend on the value you're holding, as well.
And that's it, with all those, you've got your
custom matcher, so it's really simple
to knock up your own matchers as well.
The reason I used an exception here as an example
is we've got another couple of new assertion macros
specific to using these types of matchers with exceptions.
So, this can now test whether a function throws
a particular type, and if it does,
wherever it then matches whatever your conditions are.
So you can get some quite powerful exception matching going.
Was never quite happy with the names of those,
but again the functionality's the important part.
So that's matchers.
The next feature I want to talk about
is the command line processor.
So, back in the early days of Catch, I realized
that the command line processing was getting so
full-featured that it was almost like a library in itself,
and so I spun it out as a separate library called Clara.
But, similar to Catch at the time,
constrained to C++03 or 98, with a couple of
slightly dead-end design decisions as well,
so I took the opportunity with Catch2
to completely rewrite Clara as well, not only for C++11,
but also have a much nicer design, I think,
which allows it to be composable as well,
and that's the really important point,
that's why I want to highlight this,
because another one of the long-requested features in Catch
was the ability to hook into
the command line argument processing itself.
So, if you're writing a test, you might sometimes also want
to take additional arguments in the command line
and incorporate those in your test,
and there was no easy way to do that.
But now that Clara is composable,
that actually makes things a lot easier.
And I'll show you what I mean,
want to do another quick demo.
Do that first...
So this is where we got to earlier.
Let's bring that down.
Now you remember, back in our main file, we said that
by writing CATCH_CONFIG_MAIN,
Catch will write its own main for us
and also compile the Catch library in there.
There's another form, which is CATCH_CONFIG_RUNNER.
If we do that, it's gonna still compile the implementation
in there, but we now have to supply our own main.
Just a shortcut that a bit,
I'm going to use another snippet, so here we go.
So we've got a typical main, then we declare
a Catch session object, which does some cleanup at the end.
We can pass the command line arguments
into applyCommandLine and then run our tests.
That now gives us some points that we can hook into.
So first one to do is, let's imagine we want
to be able to set this integer from the command line.
So the first thing we need to do is to grab hold
of the current command line interface object,
and now that gives us our own copy of it,
and now we can just compose directly on top of that
using the pipe operator and a new Opt.
Oh, before we do that I need
to bring in the Clara namespace.
So, this Opt, I want to bind it to our number.
And then we give it a string, and that's just a hint
for the command line documentation.
So, let's put a number.
And then, we say what arguments, what options
we want to bind that to.
We're gonna use -m, because -n is already taken,
and --number, and we can also give it
another string just as a description.
Here we go.
And that's it.
We now have a new command line parser
incorporating all of Catch's original one
and our new argument, but this is now a separate parser,
'cause it's all value semantics,
so we now need to get that back to Catch,
like that, and we're done.
And just to prove that that's all working,
we'll print out our new value at the end.
Should be it, if I did that right.
Now it's building the whole of Catch again,
so it takes a bit longer.
There we go, let's go to the terminal and run that there.
So, bringing up the command line documentation again,
as we did before, we can see, at the end there,
we've got our new documentation line,
all nicely formatted with line wrapping and everything.
We didn't have to do any work for that, it's automatic.
It's ported out of the values we put in.
And if we run with -m 42, of course,
you can see up here,
it's telling us our number we provided.
So, they're working, just with a few lines of code and,
because Clara is a separate library,
you can actually go and pick this up so outside of Catch.
You can do this yourself.
If you've got an application that has different components
that need to be configured on the command line,
but they don't need to know about each other,
they can provide their own parsers up to a top level
that composes them together with just
a couple of lines of code, and you've got
a rich command line interface to the whole system.
This is quite powerful.
So, let's go back to the slides.
And, in fact, for generators, I'm gonna go back
to the code in a second but,
just to explain briefly what these are:
A generator is just a way of providing
potentially large sets of test data to a single test case,
so you don't have to write
the same test over and over again,
and, in some cases, even provide combinations of test data.
So it can be quite powerful, and originally had
an initial form of generators
in one of the very early versions of Catch,
but it had a problem that, in fact, we didn't get to
look at exceptions, but one of the powerful features
of Catch is the ability to break a test up
into multiple sections that could be nested and,
for each section, it will actually run through
the whole test case multiple times.
So, some bookkeeping behind the scenes to see
how many times it was run, and that
wasn't interacting very well with generators,
which needed to do sort of the same thing.
So, I've had to rewrite all of that from scratch
to support generators and they are now in,
so I'm gonna show you that, just look at some examples.
We go across to here, okay.
In fact, I shall put that in presentation mode.
So, this is actually from some of the test cases
in the self test suite for generators.
So, this first one, all of the generators
use this GENERATE macro, that's what does
all of the bookkeeping behind the scenes.
And then in here, we can compose one or more generators.
So, this one's just a simple range
between one and eleven, inclusive.
And what will happen is, each time it comes in
to the test case, our x, which will be an integer
in this case, will get a new value between one and eleven.
So we'll come into it ten times.
But in this case, we've got two generators.
We've got a y, as well, different range.
So, this test case is gonna get executed 100 times
for that cross product of all those values.
So, just in a few lines of code, we've covered
100 different possibilities and that's just a small one.
Just looking at some other possibilities here.
We have another generator for strings.
We've had to include this as bit here because,
obviously, these are char stars.
If we want them to be generated as strings,
we pass it through the as generator.
So this will come through and generate a load of strings,
and here, instead of a range,
we're providing specific values.
We can combine these, so in this section,
now we see sections as well, we have a range of integers
again, and the value on its own, so we can actually compose
quite complex generators too.
Often, we want to get a whole range of values
and then concentrate on some edge cases,
and we can do that in just a single line.
And down here we've got some more, some doubles.
So again, we get the cross product of all of these running,
but we only have to write one assertion each time.
And then the other one I wanted to show you was...
Well, there's a few ways to do this one.
Here, we're generating some pairs of values,
in this case a string and a size_t,
so we can see them here, which is nice.
It's a little bit verbose here, but by using
C++17 structured bindings, we can actually
capture them out quite nicely, just in one go.
But, what can you do about the noisy bit?
Well, there's a table generator which just does that for us,
just wraps the noisy bit.
So we're just gonna generate a table of strings and size_ts,
everything else is exactly exactly the same,
and if you don't have access to C++ 17,
we can do it by binding onto structs instead.
So, we've got our same two values, two types here,
just as members of this data struct,
and we're just gonna generate values of that.
Otherwise, works exactly the same way.
And we can even generate random numbers.
So here, we're gonna generate random numbers
between -10,000 and +10,000,
and I think, by default,
it would just take 100 of those, you can control that,
and then we can do something with those random numbers.
With Catch, when you run it on the command line,
you can control a random seed,
which you can have seeded from the current time,
and if you do that, it will print out what the seed is.
So in the case of a failure, you can then rerun it
with that seed again, manually,
so actually using random numbers in test
can be useful if you do it that way, and, you can also...
Here's another example with a table but,
mix it with BDD macros,
GIVEN, WHEN, THEN.
So that, don't think it actually shows you here, does it?
No, it'll actually incorporate the generated values
in the test data that it prints out, which,
if you use these macros,
will read like a BDD scenario description.
So, it would be human readable,
have all the information you need,
and it's just taken a few lines of code to write it.
So, if we have time at the end, I'll come back
and show you what that looks like when you run it.
So, let's go back to the slides and see where we got to.
So that's generators.
Now, if you've been to any of my Catch talks before,
you'll know that I usually round out on a slide like this:
Future directions, what's coming up?
And I'm gradually chipping away on them.
It seems like they're taking a long time,
but we can see, we're already starting to do generators.
There's still some more work to do,
so I've left it up there, but that's a really big one.
The other big one that I've been waiting for C++11 to do,
so now I've got no excuse, is threading support,
which was really hard before C++11, almost trivial now.
And there is some work underway
at the moment, actually, to do that.
And the third one's really interesting,
property-based testing, and this builds on generators,
but, takes it up to the next level.
Can I ask you, who here already knows
what property-based testing is?
So, nobody basically, that's interesting.
So, let me tell you what property-based testing is.
So, property-based testing is similar to what I showed you
with generators, so working with ranges of data
that are generated for us, usually randomly,
and you just take a sample of 100 or 1000 values.
What it does differently is, in the case of a failure,
it will then try to work out what
the simplest failure mode is and give you that.
It's called shrinking in property-based testing parlance.
So you can run it with lots of different random values.
They'll be different every time,
so if there are any edge cases you've forgotten about,
you'll eventually find them.
You don't have to come up
with examples for every possible case.
What it does find, it will try to simplify it
as much possible, so that when you go in to debug,
you're not working with ridiculous numbers.
So, that shrinking part is the bit that
still needs to be done, but I think that's gonna be
a really important addition to Catch.
So, it's a different way of thinking about writing tests.
It doesn't completely replace unit testing,
but, in many cases, it's often the preferred first approach.
And then we have type-parameterised tests.
So, whereas generators are data-parameterised tests,
it's often nice to be able to write one test
that will cover many different template instantiations.
And you can do that right now with Catch
by using helper functions, doing a lot of that work
manually, but it would be nice if there was better support
in Catch for doing that, as well.
So, that's also coming, now that we've got generators.
And the final one I've got in here, because it's a big one
that frameworks like Google Test have that Catch doesn't,
a lot of people say they're using
Google Test for that reason,
is death tests or out-of-proc calls in general.
So, death test is just where you want to test
whether a run of your application
actually terminates the program.
It's honestly really hard to do
if it's all on the same process,
but, if you spawned another process and see that terminate,
then you can actually track that condition.
Of course, that requires you to be able to spawn
an external process, which...
So, doing a portable way and rich enough way
to do this is not quite so trivial.
So, that's just why that's been put off,
but it will come, at some point.
So, that's all I had on the slides,
but I've still got enough time to go back,
and I wanted to actually run some of those generator tests,
because seeing the output is quite interesting.
Was it here?
No, here we go.
Let's see if I can run just this.
Need it to come out of presentation mode.
Do it the long way.
Alright, do it from here.
Not this way.
Okay, maybe I'm not gonna be able to show you that.
I need to rebuild it.
For some reason, I'm not able to run that.
Let me see if I can run a different one.
- [Audience Member] It seems like you have to
create another macro.
- [Audience Member] It's surrounded by a macro which...
- Surrounded by a macro?
- [Audience Member] Which would put all the angles
in the upper tiers.
- Ah yes, you're absolutely right, this one, you're right.
Okay, in that case, let me show you two tests.
Thanks for that, that's a good observation.
I've got some tests here that will
demonstrate the BDD macros.
So, here we can see
that the test name
is actually printed out as a BDD scenario description,
so Given, When, and Then which makes...
There's some better ones up here.
No, that's all a...
And then, that can also run.
Let's run the, where is it,
Yeah, so here we can see, even though there was
only one test case, it's actually run 18 assertions
and we can see all of the values that it ran with,
that whole cross product of different values.
So, it's quite powerful.
And I think, with that,
we'll just come back to the final slide,
and point out that all the references and links
that I mentioned earlier are on my website,
If you can't remember that, I've also got
extralevelofindirection.com that redirects there,
so you don't have to remember them all.
And, with that, I'll say thank you and any questions?
- [Audience Member] May I ask a question?
- [Audience Member] I might not be very familiar with other
kind of solutions in that domain, I'm just
a little bit surprised to see that everything
is kind of macro-based, and we just had
the first session today where Tanya said,
"Don't use macros, they are bad."
I'm just wondering, is there any limitation
that does this a lot?
Take like something object-oriented approach,
why it matters.
- So, lemme try and summarize the question.
Why do you use macros?
It's a good question, and I hate using macros.
They're used all over the place in Catch,
and part of it's historical, 'cause when I first started
in the C++03 days, pretty much
the only solution we had to a lot of this.
I even wrote a blog post about this a few years ago
where the title for this comes from.
Modern C++ Testing was the title of the blog
addressing that exact question.
What would a modern test framework look like now,
if you were trying to avoid too much use of macros
by using modern language features instead?
And my conclusions were, although there are some things
you could do to avoid some of the macros,
it probably wouldn't look that different.
Most of the macro usage,
we don't currently have better alternatives for.
In particular, we don't yet have a way
to easily capture the file and line number.
That's in the pipeline.
We don't yet have a way to get
the stringized form of the expression.
We don't yet have a way to minimize sufficiently
the boilerplate of setting up
and registering a test case as a function.
That one you can do with a bit more, if you can tolerate
a bit more boilerplate, you can now do it with a lambda
that will effectively self register when it executes.
And I think there are some test frameworks that do do that.
There's some frameworks that lose some of the richness
in reporting so that they can avoid using macros.
So they don't have the file and line number for example,
or they get the string form.
So, you can do it, but at the moment it's still a trade-off.
And I do look forward to a future where we can do more.
So, as well is the file and line number,
reflection would get us a long way as well, when that comes.
So, we're not there yet.
When we get there, that'll definitely be
worth looking at again, but at the moment,
I'd rather use macros
and get everything I need out of it.
- [Audience Member] So one of the things that...
We're currently on Google Test.
One of the things we make some use of is Google Mocks.
Do you know, is there a mocking framework
that plays well with Catch currently?
- Yes. So, the question was,
"Is there a mocking framework that plays well with Catch?"
I usually recommend either HippoMocks or Trompeloeil.
- [Audience Member] What was the second one? I'm sorry.
- Trompeloeil, don't ask me to spell it.
It's a French word which means trick of the eye,
or something like that.
- [Audience Member] Oh, Trompeloeil, okay.
- And they're very similar in some respects.
They try be modern C++ frameworks that work well with Catch,
but they make different trade-offs.
So, Trompeloeil tries to stick entirely
within defined behavior, but at the expense of
a bit more boilerplate, and a bit more set up,
whereas HippoMocks is okay with a bit of undefined behavior,
because it just works for the moment,
but it's a lot simpler to use, with less boilerplate
as a result, so it depends on where you want
to make that trade off yourself,
which one you go for.
And I know there are one or two others,
I can't remember the names of them now,
that people have got working nicely with Catch.
And then you've got Google Mocks, which doesn't.
- [Audience Member] Thank you.
- Yep, you're welcome.
- [Audience Member] I've got a question.
So, I saw you evaluated for the values,
it can basically print out
what's a valid expression and like that.
Is that like you're doing something under the hood,
like parse it and run by itself,
'cause like I wanna know how powerful it is.
- So, I think the question is, when I showed
right at the start where I had the 6 × 9 = 42
and it broke down the values to print them out,
is that what you're talking about?
- [Audience Member] Yes.
- How is it doing that, and how powerful is it?
I think I mentioned at the time, it's using
expression templates to decompose it.
So what it's actually doing is, within the macro,
and this is another reason that I need to use a macro
to hide this all away, it introduces an object
called a decomposer, which has an operatoral match,
I think it's currently...
I've been changing it a few times, I think it might be
the Right Arrow, Right Arrow,
and that then binds to the left-hand side of the expression,
because it's got a higher precedence
than anything up until the logical operator,
so it captures that as a value, and it then
captures that in another temporary object,
which has overloads for all the logical operators,
which then captures the right-hand side of the expression.
So, most types of expression will
bind quite nicely to those parts.
It'll also do single expressions as well, unary expressions.
Where it can break down a bit, is if you try to have
things like 1 + 1 =, actually it does that now.
It didn't do before.
There may be some limitations around that,
but because the operator I use to introduce the bind
is very high precedence, it pretty much eats
the whole of the left-hand side
of the expression quite readily.
- [Audience Member] So that means, so can you
include a function call, I guess?
- Oh yeah, you can do function calls,
so you can do most types of operation there.
In fact, I'm struggling to think now what does break it.
It used to be that you couldn't do the arithmetic stuff
on the left hand side, but you can now,
so, yeah, it's pretty stable.
- [Audience Member] Left shift won't break it?
- [Audience Member] Shift will break it, I know.
- Yes, although, yeah maybe
it would actually, for that reason.
Yeah, don't do that, you can put in brackets, though.
- [Audience Member] Cool, cool, okay.
- Yeah, in the back.
- [Audience Member] Two questions actually,
and these are really Catch questions
as much as Catch2 questions.
Do you support something along the lines of compile fails
or compile succeeds as an assertion, or you can
put a code snippet in and it attempts to compile it or not.
- So, if I got the question right, you want to be able
to write a test that tests whether something compiles.
Not as such, and this is a very interesting
area of investigation, and I know some people
have been thinking about this for a long time.
I think Roland Bach has some interesting thoughts on this,
mostly to do with, can you construct some sort of
heavily templated type and test
whether it would compile or not?
I think that's usually what we're interested in.
And I think he's come up with ways of being able to do that,
so basically it reduces it down to some sort of
boolean expression you can test at runtime,
but it's not something I've built into Catch yet,
but that is something I'd like to look at more.
- [Audience Member] And another question is, does Catch
provide memory leak detection?
- So the other question was,
"Does Catch provide memory leak detection?"
And the answer is yes, at least on Windows,
'cause, I put that in.
I think somebody was working on a Linux version, as well.
I can't remember offhand whether that's in or not,
but it definitely is for Windows,
just a basic
WinCRTDB, DBG leak detector.
- [Audience Member] Alright, I'm curious,
I think it was someone else,
perhaps you wouldn't know, but I'm curious how that was
implemented, because some test frameworks do that using some
rather draconian steps
and employ steps that sort of interfere with code.
- To what extent does that interfere with code?
I can talk about the Windows one,
because I know about the implementation.
So, it's using the Windows debugger APIs,
which just take a snapshot of the beginning,
and then a snapshot of the end, and print out any symbols
that are linked in the meantime.
So, the main thing we have to do within Catch,
and this is why they have that Catch session object,
in the destructor of that, 'cause the thing
with test frameworks, you'll have loads of singletons.
You can't get away from them, so it goes through and it
cleans up all of the singletons, makes sure any memory
they're holding onto is released, so that that doesn't
interfere with the link results.
That's the main thing.
There shouldn't really be anything you need to worry about
within your own code, unless of course,
you're also using the same APIs.
That would probably not play well.
- [Audience Member] I'm not, so that's alright, thank you.
- Yeah, you're welcome.
Any other questions, yeah?
- [Audience Member] Do you have report outputs
for being consumed by CI systems?
- So, the question is,
"Do I have reporter outputs for different CI systems?"
That's a great question because there should've
been a slide on that, or a bit in the demo at least.
So Catch has a modular reporting system.
The one that I was showing you was
the default console reporter.
There's also an XML reporter, which is Catch's own format.
There's a JUnit reporter,
which should work with most CI systems.
The JUnit format itself is not great, which is why
I did a custom Catch one, but there's also,
I did a TeamCity reporter, before I joined JetBrains,
as it happens, and I think some other people
have contributed for other CI systems.
Some of those are built in.
Can't remember off the top my head which ones.
Probably the TAP reporter, Test Anything Protocol,
is in there, and I think an Autotools one, as well,
and you can write your own relatively easily,
as well, if you need to.
So yeah, that's a good question.
- [Audience Member] Do you have a
recommended approach for...
So, suppose you have exceptions.
You want to check if they fire or not,
and it was about a year ago I was working with this.
I think I found a workaround, but I'm not sure if that's
what you had in mind, or if you have a special function
for dealing with those sorts of tests?
- So you want to test whether an exception fired or not?
- [Audience Member] Yeah.
- It wouldn't be a great test framework called Catch
if it couldn't catch exceptions, so yes.
There's a number of macros,
assertion macros to do with testing for exceptions,
testing that something doesn't throw an exception,
testing that something does throw an exception,
testing that it throws an exception of a certain type,
and then there's the one that I showed, where we also
passed that on to a matcher as well, so you can do
further testing, and even if you don't write any of those,
if you throw an exception that's not caught
in your test case, it'll catch it
outside of there and still report it.
And any exception that's thrown, it will put it through
an exception translator which will effectively re-throw
the exception and then try to match it
in a number of catch blocks, so it will catch anything
that derives from std::exception, it'll call What on it.
But if you've got your own exception type, you can write
your own catcher or exception translator for that type,
as well, and you can then register with that,
so it's very rich exception reporting.
- [Audience Member] Okay.
- Hope that answers your question.
- [Audience Member] Yeah, I'll have to go back,
as I say it's about a year ago, but...
- Yeah, if it doesn't answer your question
then let me know and I'll ...
- [Audience Member] Okay thanks, appreciate it.
- I'll look into it more, thanks, in the back.
- [Audience Member] A couple years ago,
when we first started using Catch,
I think we had issues trying to get
it to work with precompiled headers.
Is that just by the nature of it being kind of header only,
with the main bit defined,
or is there any way to get that to work?
- So, the question was about
using Catch with precompiled headers, which is tricky
because it could have been a single header library
with the separate compilation of the main,
and that has been tricky, and I know that recently,
as in maybe six months ago I'm thinking,
there was some work done to make that
easier and certainly possible.
I wasn't involved in that work, I just saw it going on,
so I can't talk too much of the details, but I know
people have been using it, so I think there might be a page
on the documentation about using it with precompiled headers
now, so I'd take a look at that, and if not then
you can reach me and I'll look into it more.
But, you can definitely do it.
Okay, any more questions?
Okay, then I think we're done.
Thank you very much for coming.