Digital Compositing on Mobile

Node-based Compositing is the process of combining multiple seemingly simple nodes to render and achieve a desired result. This process has wide applications in the graphics software industry, especially in procedural image generation, motion graphics, animation, 3D modeling, and visual effects VFX.

The paradigm of a node-based tool involves linking basic media objects onto a procedural map or node graph and then intuitively laying out each of the steps in a sequential progression of inputs and outputs. Any parameters from an earlier step can be modified to change the final outcome, with the results instantly being visible to you, and each of the nodes, being procedural, can be easily reused, saving time and effort.

To visualize what it looks like, the user interface of a node compositing software is shown below. Videos or images are first read in and then processed by a sequence of operations before being merge into one final outcome. The sequence of operations, the node graph - Directed Acyclic Graph (DAG), is represented by a flowchart.

 Read media  Read media
 |           |     
 V           V     
 Premult     Color Correction
 |           |
 |           V
 |           Transform
 |           |
 |           V
 |           Gaussian Blur
 |           |
 V           V
     Merge
       |
       V
     Viewer

As the number of nodes increases, the node graph, together with a large monitor screen, provides an overview of what is happening while also enabling one to focus on a particular progression step (node) and make changes to that step to achieve the end results.

The Small Screen Problem on mobile
On a mobile device, a pure node-based app is not common. Resource constraint is one limiting issue. Putting that aside, since mobile has come a long way, with mobile chips enabling one to do many advanced image and video capabilities, the other limiting issue is the small screen size of a mobile device for displaying the node-based user interface.

As a start, the arrangement of the multiple node properties screen, the node graph, the dope sheet, and the curve editor together in one small screen presents a problem. Compositing a node graph with a flowchart-like user interface is also more difficult especially when trying to connect nodes while panning the small screen frequently. This is made worst when you need to connect to a distant node with multiple inputs; or if you need to link node properties, which may involve the timeline, for tracking, animating objects, or rotoscoping.

Main Idea

A Node Pipeline is proposed to represent the node graph on a mobile device instead of using a flowchart-like user interface.

Node Pipeline for Node Graph

A node graph is commonly represented as a flowchart with a Directed Acyclic Graph (DAG) in most digital compositing software. This is shown below:

 Read media  Read media
 |           |     
 V           V     
 Premult     Color Correction
 |           |
 |           V
 |           Transform
 |           |
 |           V
 |           Gaussian Blur
 |           |
 V           V
     Merge
       |
       V
     Viewer

On a small screen mobile device, the DAG could be decomposed into multiple series of nodes shown below.

For example, the node graph above can be represented as a simple list below:

 Read media
 |                
 V               
 Premult     


 Read media
 |              
 V             
 Color Correction
 |           
 V
 Transform
 |           
 V          
 Gaussian Blur      


 Merge
 |
 V
 Viewer

Note the break from the first series after the 'Premult' node and the second break after the 'Gaussian Blur' node. From here, we will refer to the above arrangement as a pipeline **(or **Node Pipeline with breaks).

This arrangement has the obvious advantage of being represented easily on a mobile device with a simple list as shown below.

1. Read media         >
2. Premult            >

3. Read media         >
4. Color Correction   >
5. Transform          >
6. Blur               >

7. Merge 2,6          >
8. Viewer             >

The lines joining the nodes could be further represented by listing the input nodes used, like using a spreadsheet formula (e.g., =SUM(A1,A2)). In the above, the 'Premult' node uses node 1 (Read media) as the input, while the 'Merge' node uses node 2 and 6. We can also represent this information on the pipeline.

1. Read media       (input none)  >
2. Premult          (input 1)     >

3. Read media       (input none)  >
4. Color Correction (input 3)     >
5. Transform        (input 4)     >
6. Gaussian Blur    (input 5)     >

7. Merge            (input 2,6)   >
8. Viewer           (input 7)     >

Simplifying it further

1. Read media       (none)  >
2. Premult          (1)     >

3. Read media       (none)  >
4. Color Correction (3)     >
5. Transform        (4)     >
6. Gaussian Blur    (5)     >

7. Merge            (2,6)   >
8. Viewer           (7)     >

Implications

With the above, let's think a little further about the implications if we use a node pipeline as the user interface for managing a node graph on a small-screen mobile device.

• A node pipeline will require a shift in thinking when you are using it to manage the node graph. Instead of three parallel series of nodes in the node graph above, a pipeline has three sequences (or series) of nodes. The lines joining nodes in a node graph are now represented by the target node referencing input nodes. For example, node 2 references node 1 (as the input). Node 7 references node 2 and 6.

• It should be noted that, mathematically, a node graph, which is a Directed Acyclic Graph, can be decomposed to a node pipeline(the series of nodes above) without any loss of information. This means a pipeline user interface can be as flexible as a flowchart-like node graph.

• A pipeline makes it easy to navigate the node graph on a small screen as you can quickly scroll up and down the list. Tapping on any item in the list (a node) can further bring up a screen for changing node properties. This user interface should be familiar to many mobile users as it is similar to the user interface for managing mobile phone 'Settings'.

• The contextual overload of a user frequently panning a flowchart-like node graph on a small screen is significantly reduced. This will improve the usability and ease of managing a node graph on a small screen.

• The referencing model for linking nodes on a pipeline is similar to using a spreadsheet. 'Merge' below is like a formula applying to cell 2 and 6.
  

7. Merge (2,6) >

The pipeline approach leads to other interesting productivity gains in compositing on a mobile device. Please check out our open source Nodef repo on Github or Nodef app on the app store for the following:

• Auto Chaining & Reverse Compositing
• Viewer Cycling
• Directed Acyclic Graph (DAG) Generation/Import

In the above, we will address how to join nodes by Auto Chaining and Reverse Compositing, ponder further on linking node properties, and also about setting the Viewer to a selected intermediate node on the pipeline.