Introduction

Video mapping is a very promising technology for enhancing user immersion into new technologies. Multi projector systems can create an artificial 2D and 3D environments for new game experience and design. There are many video mapping solutions on the market and even more graphic engines to create those environments. The problem occurs when you need to connect those technologies to create a worthwhile experience for users. This post will introduce a possible solution for this problem using Unreal Engine 4 (graphical engine), GrandVJ XT (video mapping software) and Spout (a framework for streaming video from graphics engine to video mapping software).

Note
Unreal Engine 4 and GrandVJ XT was picked because of authors preference. 
Many more software can utilize Spout, to name only Unity3D and Resolume Arena.

Possibilities

If you ask yourself “What can I use it for?” I can point you to the resulting application that was a foundation for this article:

Of course this article is only a mere beginning of the game production process. Nevertheless, power that video mapping can give your game is worth a few minutes of reading.

Prerequisites

Skill

Considering the power of Spout it is very easy to use after proper configuration. User needs to be able to install software on his own. Unreal Engine 4 knowledge will be useful but is not required.

Software

Spout

Although presented solution uses commercial software, both ends of the system (graphical engine and video mapping software) can be substituted by other suitable software as long as they can support the most important element – Spout – a free open source software for managing streams inside of Windows platform. Spout offers both sender and receiver functionality in multiple instances. We will use a single sender instance on the UE4 side and a single receiver on GrandVJ XT side (it doesn’t allow more Spout sources after all).

Spout SDK needs to be installed and every platform needs to be provided with suitable plugin.

GrandVJ XT has SpoutCam source out of the box. Unreal Engine 4 needs a little more work with obtaining and installing a suitable plugin. We will get to this shortly.

Spout SDK can be downloaded from here.

Unreal Engine

This tutorial uses commercial software, but it can be created without any cost. Unreal Engine 4 can be downloaded and used for free until project doesn’t generate profits.

Solution was tested on version 4.12.5.

You can obtain UE4 from official website.

GrandVJ XT

GrandVJ XT is available to download for free and can be used on a trial license.

Solution was tested on version 2.1.4.

It can be obtained from here.

Operating System

You will need a Windows 7 or higher (this solution uses Windows 8.1 Pro 64-bit, but it was also tested on Windows 10).

Example project

You can find the Unreal Engine 4 project here. You will need to run the SpoutPuzzleTest.uproject file to prepare the game to be run.

Hardware

Video mapping operates mainly on projectors, however the Spout configuration can be done using monitors. It should work with projectors without any changes (not considering tweaking the final view)

Configuration

At this moment you are ought to have installed Unreal Engine 4, Spout SDK and Grand VJ XT, and verified that all of them work (Spout SDK has a DEMO folder where you can use SpoutSender and SpoutReceiver to verify that it works).

Unreal Engine 4 Spout plugin installation

Next step is to install Spout plugin for UE4. I have encountered 2 potential plugins gathering state of the art. UE4ToSpout and Spout-UE4. Although UE4ToSpout looks far more professional and it even has an half of an hour long video that even concludes building Unreal Engine 4 from sources (good tutorial in this part though) it failed to work for me. I got only a single frame and later a black screen and couldn’t move forward. In contrary Spout-UE4 has little information on how to install it and of low quality. Altogether skill acquired using UE4ToSpout was very useful when installing Spout-UE4 so it worked finally.

First we will need to create a basic project. Let us start with Puzzle Project which view we will try to lay on the floor using video mapping.

Open Unreal Engine 4 Editor, go to File -> New Project, select C++ tab and a Puzzle project. Let’s name it SpoutPuzzleTest.

Note
We will not use C++ in this tutorial, but it is easier to be prepared for it 
rather than add it later, if you would like to go on with the project.

new-project

Before doing anything with the project let’s take care of the plugin part. From official repo download the project (you can clone repo using GIT, but it would be perfectly suitable to just download ZIP archive).

Extract the archive, rename the main folder to SpoutUE4 and copy it whole into (assuming that you created your project where I did):

C:\Programming\UE4\Unreal Projects\SpoutPuzzleTest\Plugins.

If Plugins folder doesn’t exist just create it.

Now close Unreal Editor. Go to C:\Programming\UE4\Unreal Projects\SpoutPuzzleTest and open SpoutPuzzleTest.uproject (it will restart the project).

Unreal Engine will ask to rebuild Spout plugin using current Engine. Just click Yes.

spoutquestion

If it rebuilds properly your project will open as usual. Now check if Spout plugin is enabled. Go to Edit -> Plugins, scroll down to Spout plugin and make sure it states “Enabled“.

spout

Utilizing SpoutSender

SpoutSender Function

SpoutSender is a Blueprint Function from SpoutPlugin that provides us the possibility to insert textures into the stream. It contains several parameters that need to be provided.

spoutsenderfunction.PNG

SpoutName is a name of the stream that Spout will later recognize as our stream.

SendTextureFrom is an Enum that states what is the source of the texture for stream. We will use TextureRenderTarget2D.

TextureRenderTarget2D is the texture source, which will be added to the stream.

Target Gamma is ommitable.

SpoutSender function puts a single frame into a stream. We need to do it multiple times every second so the stream is fluent. We will use an Event Tick for sending textures about 60 frames per second.

Creating SpoutSender Blueprint

To correctly manage our stream we need to create a Blueprint that will offer a interface for controlling source and state of SpoutSender.

Let’s create a new Blueprint that inherits from Actor. Click Add New -> Blueprint Class and select Actor.

spoutsenderbp.PNG

Put it into Content folder and name it BP_SpoutSender.

bpspoutsenderready.PNG

Now open BP_SpoutSender and select EventGraph.

First we need to create variables that will be exposed to other Blueprints (we will manage this BP_SpoutSender instance from LevelBlueprint).

Variables

Variables to be created are:

SpoutName of type Name – it will hold the name of our stream. You can hard code “SpoutPuzzleTest” here.

State of type ECheck Box State – it will tell us that stream is ready to be generated (after the level loads, etc.).

TextureRendererTarget2D of type Texture Renderer Target 2D – it will provide the stream with fresh textures.

SendTextureFrom of type ESpout Send Texture From – it will configure the type of source SpoutSender expects.

The Variables set should look like this:

variables.PNG

Note
DefaulsSceneRoot Component is added automatically to every Actor.

EventGraph

EventGraph needs to be configured as on the image below.

bpspoutsendereventgraph.PNG

This is what happens here:

On every tick Actor checks if it is supposed to generate another frame (Switch on ECheckBoxState), next it invokes the SpoutSender function using variables that need to be configured by external source (i.e.  LevelBluperint).

On the contrary stream is being closed (Close Sender) when Actor is being destroyed or game ends.

Putting SpoutSenderActor into the Scene

Finally we need to create an instance of BP_SpoutSender. Grab BP_SpoutSender and drag it into the Scene. Next change it’s name to SpoutSenderActor.

Creating a SceneCapture2D for SpoutSender

The next thing we need to do is to create an input for SpoutSender. This input is called SceneCapture2D. SceneCapture2D works similar to a simple camera but it stores what it points to into a texture. This texture can be later utilized as a frame in a stream.

Let’s add SceneCapture2D to the scene. Search for this class in the class search box and drag it into the scene.

scenecapture2d.PNG

And put it as a child of CameraActor named Camera, so it points to the same point as the camera.

child.PNG

Our new SceneCapture2D Actor is named SceneCapture2D1 as default and we will leave it this way.

Level Blueprint

Adding references to Actors

Now select SceneCapture2D1 and open Blueprints -> Open Level Blueprint.

open level blueprint.png

This way we will access a Level Blueprint, a Blueprint that has access to all Actors on the Scene. Being there click RMB (Right Mouse Button) on the grid and select Create a Reference to SceneCapture2D1.

create reference.png

This way we can later use SceneCapture2D1 captured frames.

Repeat the same process with SpoutSenderActor.

Creating logic

Puzzle template came with some logic in the LevelBlueprint.

levelblueprintdefault.PNG

It only specifies that the default camera for player will be the camera provided on the Scene. We will leave it that way and continue with our initialization.

levelblueprintinitialization.PNG

What happens here? First we create TextureRenderTarget2D, which basically means that we initialize the stream on Spout. Next we set the TextureTarget on CaptureComponent2D of SceneCapture2D1 to this TextureRenderTarget2D so it knows where to transfer the generated texture. After that we set the TextureRendererTarget2D variable on SpoutSenderActor so it can use it to generate the frame. Finally we set the SendTextureFrom on SpoutSenderActor to TextureRenderTarget2D so it knows that he needs to use it and we change the stream State to Checked (which means that it should generate frames).

That’s it!

Our SpoutSender configuration is ready. We can test if it works using Spout SDK Demo Receiver. Go to Spout installation directory (it should be located in Program Files (x86)). Go to folder DEMO and run SpoutReceiver.exe. Now click Play in the Unreal Editor and watch magic happens:

spoutsenderreceiverworks.PNG

Puzzle template allows to change color of blocks when clicked on. Try it. It will change in Spout Receiver instantly.

Note
Unreal Engine 4 is optimized for performance wherever it can, so when game goes 
into background it loses FPS. To prevent this behavior go to 
Edit -> Editor Preferences, Miscellaneous and uncheck 
"Use Less CPU when in Background"
config
A similar problem occurs with sound - it is not playing when losing focus. 
To enable it open DefaultEngine.ini from Config folder of your project and add
[Audio]
UnfocusedVolumeMultiplier=1.0

If everything works we can go to GrandVJ XT configuration.

GrandVJ XT configuration

When running GrandVJ it will ask for the license key. Please just select Try GrandVJ XT for the purpose of this tutorial.

Next select New… on the What would you like to do ? dialog and select Blank template – Mixer mode in the next dialog.

Now go to Sources tab, scroll to the top and select Spout Cam. If the SpoutPuzzleTest project is playing you should see the preview (if not, this is the time to let it run!).

grandvjsourcespoutcam.png

Now drag the Spout Cam source onto the mixer Layers 8th channel. You should see the out stream in the Master Preview. Now click the Open Video Mapper button on the top.

8thchannel.PNG

GrandVJ XT video mapper setup

What you see in the Video Mapper depends on your system’s video configuration. You should see every video device that is plugged into your video card. If there are projectors you can start your mapping. If not, let us just test it with a monitor. In my configuration there are 2 monitors. To view our stream just pick the rectangle symbol on one of them.

videomapperconfig.png

To start video mapping press CTRL+F (for Fullscreen).

Note
Remember the CTRL+F combination. This is the only way to go back to desktop as 
GrandVJ XT takes control of your monitor.

That’s all!

Summary

The process of combining game engine with video mapping software can be cumbersome, but Spout is a very versatile and considerably easy to use software to combine various technologies on either side. Spout can be used in numerous ways, e.g. as a Winamp and Virtual DJ visualization source, to name a few. Unreal Engine 4 and Grand VJ XT can be substituted with any software that supports Spout (which should be a standard for video mapping purpose, if you ask me), e.g. Unity3D and Resolume Arena.

Next step

Please let me know if you would like me to write posts about going deeper into related topics such as:

Video Mapping configuration

The video mapping is a complex problem. You need to place projectors correctly, plan how does your resources (projectors) can be used to create best effect, finally to transform the view on each of them properly considering compensating trapeze effect and linear stretching of the view.

Game Engine inputs for video mapping

Using game engine as a source of video mapping gives as a massive opportunity for creating novel gaming experiences. The biggest problem is potentially player input. Player needs a controller to join the game, which without it is just an animation. Gesture tracking controllers are the most immersive, so the user feels as he would be in the game. One of the most advanced (and accessible) devices is Microsoft Kinect v 2.0. Utilization of Kinect and video mapping can result in experience similar to Virtual Reality Helmets like Oculus Rift, HTC Vive and more, but without the additional equipment on the head and in the hands of user.