What Is Matchmove?
Matchmoving, also referred to as camera tracking, is the “process of matching CG elements into live-action footage”. Without matchmoving, visual effects could not exist, as the effects would not move with the footage. On a basic level, matchmoving is solving a real camera’s location and movement using tracking markers and parallax, and applying the data to a CG camera. By using parallax, camera solving software can work out the position of the camera by evaluating the difference in movement from different tracking points. Matchmove allows digital assets to integrate seamlessly into live-action footage, without appearing to slide around.
There are two types of matchmoving within visual effects, camera tracking and object tracking. Camera tracking refers to solving a cameras movement in order to place a digital asset into live-action footage. Object tracking is the process of tracking an object’s motion so that an asset can be copied to it. The biggest difference between the two processes is that camera tracking produces a moving replica CG camera in 3D space, whereas object tracking produces a still camera with a moving object however, both can be used in conjunction with one another.
Camera tracking is required when a camera needs to be replicated in 3D space in order to add CG assets to a scene. Object tracking is required when a CG asset needs to be stuck to a moving object within the shot or an actor needs to interact with a CG element.
Matchmoving Pipeline
It is important to define a workflow within matchmoving, as completely camera solves correctly and accurately can save costly errors from occuring further down the pipeline.
Below shows the ideal matchmoving pipeline and what should happen at each stage. Following these steps should allow for an accurate camera track and solve and should minimalise the amount of slipping within a track.
- Source FootageSending and receiving of the captured plate footage.
- Evaluate FootageReview of the footage to highlight any potential problems and to understand and outline the intended tracking approach.
- Camera Data / Data WranglingInputting any recorded camera data and on-set information into the matchmove fotware. This can include camera focal length, location dimensions, photogrammetry/LiDAR scans, frame rate, and many more.
- 2D TrackingThe placement of tracking markers within the footage. Ideally markers want to last throughout the duration of the footage however, when this is not possible, it is important to ensure there are enough markers at every point within the footage to reduce solve errors.
- Camera SolvePerforming the camera solve, which allows the matchmove software to evaluate the position of the camera based on the parallax of the tracking markers within the scene.
- Orient SceneWith the camera solve completed, the CG camera can be imported into 3D software and the scene lined up to the camera so that the floor is in the correct position.
- Test SolutionTest geometry can then be placed within the scene to ensure there is no slipping and that the camera track and solve has been successful.
- Geometry LayoutOnce the solve has been deemed successful, the set geometry can be built and laid out within the user’s chosen 3D software, such as Maya, Blender or Houdini.
- DeliveryOnce the footage has been tracked, solved and the set has been rebuilt, the scene file can be saved and sent to the next department in the studio’s pipeline.
It is also important to mention the possible workflows for when things go wrong, including trying to reduce solve errors for a more accurate track. There are many ways to approach this, but the standard workflow would be to look at the deviation browser and begin by deactiviting any points that cause spikes in solve error. This should be repeated and resolved until the solve error has dropped. Another method that can be used to reduce the error within a track is to adjust the camera parameters incase there are any small deviations. Common practice is to adjust parameters such as the focal lentgh and lens distortion. Both of these things can provide a more accurate track.
3D Matchmoving Principles
Camera Data
Camera data is essential for creating accurate tracks and camera solves of footage, including information such as the camera focal length and viewing angles. Information such as film back size is directly related to camera lens focal length and the field of view, which depicts how much we can see within a frame. On top of this, information about any lens filters have been used is important to know what has been added on top of the camera’s settings. Lens filters can be used to correct the colour of a shot, or reduce light or glare, so it is important to know what has been added so the software can take all the additional information into account. The film back size refers to the dimensions of a film’s frame, with focal length referring to the optical magnification of the lens. For this reason, a camera can have the same film back size and different focal lengths and they will look different, and vice versa if the camera film back size is different, hence why it is important that the software has all the details.
It is important for the matchmoving software of choice to have this information as it will increase the accuracy of the solve. With the information, the software has a better idea of where each tracking point is in 3D space. If the software didn’t have this information, it would have to try and calculate the position based off different values, which can produce incorrect results. This stands true for incorrect data too. For example, if the focal lentgh was wrong, the software would struggle to locate where the points are in 3D space as it the software has been told the point is closer/further away from the camera that it actually is.
Set Data
Set data needs to be recorded alongside camera data. Whilst camera data refers to all the modifications made to how the shot appears from the lens of the camera, set data refers to all the information on the location of the shoot, such as the angle of the camera, the height of the camera, the camera’s distance from the subject. All of this information is useful; however, it is more commonly used during scene line-up to get the scale of the scene correct. The more set data that is collected, the better, as it gives layout artists a significantly greater idea of where everything actually is, so that it can be replicated in the 3D environment, making for a better overall shot.
Technical Requirements
Parallax
Parallax is “the amount of perspective in an image. Generally, it is used to mean the apparent change of perspective from one image to another, although this is technically known as parallax shift”. Parallax, essentially, is the idea that objects far away move at a slower pace to objects that are closer to the camera. Parallax is essential for matchmoving as it used to trangulate the position of each of the points in 3D space.
A scene with very little parallax would be extremely hard to track as the software would not be able to locate where all of the points are in space. When creating 2D tracking markers, it is common practice to make sure you have points in the foreground, midground and background and also to avoid false parallax wherever possible.
Nodal Pan
Nodal pan is the significant lack, or complete absence, of parallax. It refers to a live-action shot where the camera doesn’t move in any direction, but does pan on an angle. As stated before, parallax is what is used to drive a camera solve, so having none makes it incredibly difficult to solve well. Since perspective is used to drive the camera solve, having a fixed camera means that the perspective within the shot will never change, meaning perspective cannot be used to calculate the triangulation of the points in 3D space.
Though some software’s do offer the ability to change the settings to fixed camera, the lack of perspective will make the footage extremely hard to track. In the case of nodal pan shots, it is very important to get set measurements such as distance and object size so that methods such as surverying can be used.
Lens Distortion
It is known that all lens contains some level of distortion, with some suffering more than others. Lens distortion refers to how an image can be warped and appear bent. Lens distortion becomes an issue within matchmoving such as being problematic for camera solving and scene line-up. For this reason, lens distortion needs to be removed from plate footage before being tracked.
Tracking Points
Tracking points refer to the locations in footage that are being tracked. Correctly using tracking points can be used to create efficient workflows with matchmoving. If there are a selection of tracking points along one, or more, of the cartesian coordinates (X, Y, Z), they can be set to a plane, helping to solve the remaining points within the scene. This also significantly helps with scene orientation, after the camera track has been solved. Tracking points should be points of the footage that contain unique features such as areas with largely contrasting pixels or unique shapes.
Placement
Placement of tracking points is very important to ensure that the correct information is being fed into the solver, instead of confusing it and generating solve errors. An example of things to avoid would be reflections, shadows, moving objects and false parallax. False Parallax refers to when points are placed in areas, typically corners, outlining 2 objects that are different distances away. False parallax during auto-tracking, such as points being generated on the outline of 2 buildings that overlap, for example. Because of parallax’s importance, these need to be avoided. On top of this, it is important to consider the spread of points throughout the footage; it is important to ensure that there is always a good selection of tracking points visible in the scene. Making sure the footage always has constant tracking points will allow for better parallax calculation.
3D Matchmoving Workflow
The 3D matchmoving workflow, in 3DEqualiser, begins with the user changing the software “environment” to tracking mode and then importing the footage.
Once the footage has been imported, the settings can be adjusted. This is where the recorded camera data from the production shoot is used. The film back size can be added, along with the focal length, and any other information that has been saved from the shoot.
Tracking points can be placed on the footage by control clicking on the footage. The user can then access the tracking point options, which will allow them to change the mode of the tracking marker between pattern and corner/edge. With the settings set for the tracking point, the regions
can be adjusted. As shown below, the tracking point is split into the point, the reference pattern, and the search region. The tracking point should be placed roughly in the centre of the tracking marker, with the reference pattern outlining the patern that is to be tracked and the search region
telling the software the region it should search in to find the pattern that has been defined in the reference pattern.
Once the settings for the tracking point have and the regions have been set, the point can be tracked against the reference. This process can then be repeated until there are enough tracking points within the scene, with enough parallax, to solve the camera movement. Once the user is happy with all the tracking points, the user can solve by using the shortcut “Alt + C”.
With the camera solved, the user will be able to view the solve error and deviation chart, showing them which points are less accurate than others. This information can then be used to remove tracking points making the solve less accurate and create new ones where necessary.
When happy, the user can export the matchmoved camera and import it into Maya ready for scene orientation and build-up, so that assets can be imported into the scene.
2D Image Restoration Workflows
Introduction
Rig removal and image restoration is the process of removing unwanted elements from plate footage. This can include rigs, such as wires or camera dolly tracks, or unwanted features, such as brandings on buildings, people, and grafitti. The image below shows an example of how image restoration and rig removal can be used to remove unwanted features.
Marker Removal Node
The marker removal node workflow consists of utilising Nuke’s in-built marker removal node workflow. The setup for this workflow is very simple and consists of 3 nodes: tracker, roto and marker removal. In order for this method to work, the footage needs to be tracked so that the position of the marker removal can follow the marker, making it look seamless. Though this can be done manually through keyframing, tracking the footage significiantly improves the process.
To start the workflow, the user must connect the tracker node to the footage. The user can then create a new tracker node, add a new track and position it on the marker. After the marker is positioned, it can then be tracked.
With the marker tracked, a roto node can be created and used to place an ellipse on the footage. Selecting the transform options in each node will display the keyframed transformations and centre point. This data can be copied from the tracker node, which contains all the data of how the marker moves through the scene, onto the ellipse in the roto node; this is done by holding control whilst dragging the translate data from the tracker onto the transform data on the roto shape. This will create a link between the nodes, shown with a green arrow. This process can be followed again for copying the centre point data.
With all the data copied across, the marker removal node can be connected to the roto node, sampling pixels from the surrounding area in the scene and hiding the marker.
Live Patch
The live patch workflow follows a similar process to using the marker removal mode method, but substitutes the removal node for a rotopaint node.
Following the same process as before, with tracking the marker, the user can then make a rotopaint node. With the rotopaint node selected, the user can draw an ellipse over the marker.
Patch
The final method for 2D image restoration/rig removal is patch. The patch method is the most complex method of the three, but also provides more accurate results, making it the method that is typically used within industry. The reason this method is seen as more complex than the others above is because it consists of removing the noise from the footage, then applying a clone to cover the marker, before reapplying the grain back on top.
The method starts with using a tracker node and creating a tracking point to track the marker. This follows the same method as the other techniques.
With the footage tracked, the user can then add a denoise node. The settings required to remove the noise from the footage varies. Largely, the user will adjust the “amount”, “roll off” and “smoothness” values within the settings of the node to create the desired result.
Once the user is happy with how the footage has been denoised, they will need to apply a frame hold. This will set a selected frame to play for the duration of the project. The frame selected within the frame hold will also be the same frame that is used when drawing the rotoshape and will also be used as a reference frame.
With the frame selected and held, the user can use the clone brush from a rotopaint node to paint over the tracking marker within the scene. With the tracking marker covered, the user can then use a normal roto node, partnered with a copy node, selected to alpha, to create an alpha mask around the tracking marker.
Applying a premult node after the previous step will hide the plate footage and reveal the area within the roto shape. With this in the viewer, the user can then add a re-add grain, using the grain node. To help with this, the user can switch between the red, green, and blue channels to match the grain to the grain in the original footage.
When the user has finished adding the grain to the rotoshape, the now-hidden tracking marker can be merged back on top of the original footage. If there is anything that doesn’t look correct, a grade node can be added in before the merge node to fix any lighting problems.
Comparison
Of the three methods, for the best results, the patch method is by far the best. It gives the user the most control over how the removed area should look, allowing them the most freedom to make it look the most accurate. Methods such as the marker removal node provide quick results to easy to remove markers, such as ones that are never obstructed, but don’t provide good results when more complex solutions are required. An example of this is when a marker gets obstructed by a deforming object – the marker removal node would not be sufficient for this problem, so patch would be required to give the user the most control.
