I can't see any of this.
I can't see the mountaintop, or the yawning chasm in front of me,
and this isn't some magical drone camera hovering out in mid-air.
This is the Mixed Reality Lab at YouTube Space New York,
and if I want to see anything around me other than these green walls,
I'm going to have to wear this headset.
The challenge with virtual reality is that
while the person who's in it is amazed,
and waving their arms around and experiencing it,
the rest of us don't know what's going on.
And so mixed reality is a way of letting everyone else see
what it would be like if we were there too.
We have an actual camera that is filming you in the headset,
in front of a green screen,
and then the game just needs to know where the virtual camera should be.
We're using an HTC Vive for the VR, and if we use a third hand controller
and attach that controller to the camera,
now the game understands where the camera is
and so we can have both the actual camera and the virtual camera
moving through the space and recording what it would be like
if we were inside there with you.
We're using a PC that's specced-out with the highest-end Intel processor,
32 gigs of RAM, and the current high-level consumer video card,
which is an NVIDIA GTX 1080.
We're rendering at 4K,
and anything less than that is going to run into an issue!
What's needed to make the mixed reality work
is a small config file that sits right in the root folder of the game that you're playing
and that config file contains the offset between where that controller sits
and where the actual lens of the camera is.
And then you also need to match the rotation and the field of view of the camera
so other than that, it should work with any recent built of Unity.
Unity is a framework for building 3D stuff that a lot of programmers use.
And because the system ties into it,
the Mixed Reality Lab can work with a lot of different VR games.
Not all of them, but a lot.
And because the tracking on the HTC Vive is so accurate,
we can do things like go in for a close-up on the laser gun
that I reckon I'm holding right now.
Granted, there's a Vive controller sticking out of the middle of it,
but the positioning is pretty much perfect.
When you're playing in VR normally,
all the computer needs to output is the view that you see in the headset.
When we're doing mixed reality, we also output the view
as seen from this third controller, this virtual camera.
And so what that gives us is four views instead of just the one.
The four views are, yes, the headset view as usual,
we also see the background, as seen from the perspective of this virtual camera,
and then we have two layers of foreground,
one of them is the image itself, everything that should appear in front of the player,
and then we have just a luma layer to help you key out the black space.
The SteamVR plugin automatically selects the foreground and the background
based on the position of the headset.
And anything that is between the headset and the third controller is foreground
and anything that's beyond the headset becomes background.
Here in our studio, we're doing a live composite on the fly.
And this is a rough idea of what you'll get in your final edit.
The live composite is a bit rough around the edges,
but you can also take the original assets
and do a much more detailed composite in post-production later.
Now, virtual studios aren't a new thing.
The BBC have been using one for election night for years,
although it uses carefully pre-arranged graphics,
lots of expensive broadcast-quality equipment,
and the host has to rehearse and hit all their marks exactly for things to look right.
And let's face it, replace this headset with a Helmet of Justice
and you've got 1990s British kids show Knightmare.
All those were fancy, custom-built affairs, though:
this uses more off-the-shelf components.
If you want to do this at home,
you're going to need a large, wrap-around green screen,
but you could paint your walls!
You're going to need a very powerful computer,
you'll need a camera that can feed a signal back into your computer,
you're going to need a capture card and third controller,
but if you have all of those things,
you can build a mixed reality setup at home!
Thanks to everyone at YouTube Space New York, and the Mixed Reality Lab.
You can find out more about them at the links in the description.
(Are you going to keep the bump?)
Yeah, that's nice!
All right, that's good!
That's a wrap!