DEV Community: Daniel Chou Rainho

[Unity] Loading GLTF Models at Runtime for Android

Daniel Chou Rainho — Wed, 19 Jul 2023 10:24:59 +0000

The goal of this project is to load .gltf files at runtime in Unity for Android, with specific focus on the Oculus Quest. This can be challenging due to differences in rendering pipelines and limited native GLTF support in Unity.

The Problem

While Unity provides support for common 3D formats like .fbx, .obj, and .dae, it lacks native support for GLTF files.

The Contenders: UnityGLTF, GLTFUtility, and UniGLTF

On reviewing the available options, three libraries emerged as the front runners: UnityGLTF, GLTFUtility, and UniGLTF

Selection

Taking into account all three options, I opted for GLTFUtility for this project. The native URP support is critical for the seamless loading of GLTF models on the Oculus Quest platform without any shader issues.

UnityGLTF, despite its strong feature set, was less suitable due to its lack of URP support, requiring additional workarounds or library extensions.

The key differences lie in their complexity and features. While UnityGLTF boasts more features, it seems like an extremely messy project, and having to compile a DLL for it to work makes it inaccessible to most of the people who actually need it.
GLTFUtility is built with simplicity as the main focus, uses no external libraries, and is easy to navigate for potential contributors. It's plug and play.

Thor Brigsted, creator of GLTFUtility

UniGLTF was a close competitor and offered much of what GLTFUtility provided, including URP support. However, I leaned towards GLTFUtility, as it is also designed with a focus on simplicity and performance which could prove beneficial when loading models at runtime on a device like the Oculus Quest 2.

Implementation

Getting GLTFUtility up and running was quite straightforward. After adding it to my Unity project, I set up a basic script to load a .gltf file into my scene.

The GLTFUtility package also provides an asynchronous loading option which is useful for loading complex 3D models without freezing the main thread.

using Siccity.GLTFUtility;
using System.Collections;
using System.IO;
using UnityEngine;

public class GLTFLoader : MonoBehaviour
{
    private string relativeFilePath = "YourFile.gltf";

    private void Start()
    {
        string absoluteFilePath = Path.Combine(Application.streamingAssetsPath, relativeFilePath);
        StartCoroutine(LoadGLTF(absoluteFilePath));
    }

    private IEnumerator LoadGLTF(string filePath)
    {
        GameObject loadedObject = null;

        Importer.ImportGLTFAsync(filePath, new ImportSettings(), (result, animations) =>
        {
            loadedObject = result;
        });

        while (loadedObject == null) yield return null;

        loadedObject.transform.SetParent(transform);
        loadedObject.transform.localPosition = Vector3.zero;
    }
}

Note: For this project, place your .gltf file in the StreamingAssets folder, which is located at YourUnityProject/Assets/StreamingAssets. Ensure the relativeFilePath variable in the script matches your .gltf file's name.

The source code for this project can be found on GitHub.

[Unity] GameObject to Shader Communication: MaterialPropertyBlocks

Daniel Chou Rainho — Sun, 18 Jun 2023 17:56:01 +0000

Intro

Each GameObject in my scene is given a unique ID, and I want this ID value to be used inside the Shader that is part of the Material of every GameObject in my scene.

Specifically, I want this identifier to be able to be stored in a Color channel, i.e. it has to be a float between 0 and 1 that is unique to every GameObject in the scene.

Architecture

In the following 3 sections I outline the 3 main scripts I used for this post.

They interact as follows:

My Identification script is attached to each GameObject in the scene that I want to attribute to an unique ID.
This Identification script calls a function from the script IDManager (which I have attached to another GameObject in my scence) to get a unique ID.
Then, the Identification script creates a MaterialPropertyBlock object to which it assigns the ID.
Finally, in my Shader which I assign to the material associated with a custom Render Objects Render Feature, I can read this ID value from the MaterialPropertyBlock to then return it in the red color channel in the fragment shader.

IDManager

using System.Collections;
using System.Collections.Generic;
using UnityEngine;

/**
* The first id starts at 0.0001.
* We don't use 0 because that's reserved for the background.
*/

public class IDManager : MonoBehaviour
{
    public static IDManager instance; // Singleton instance
    private int lastAssignedID = 0;  // Keep track of last assigned ID
    private int totalObjects = 10000; // Total number of objects you expect

    void Awake()
    {
        // Singleton setup
        if (instance == null)
        {
            instance = this;
            DontDestroyOnLoad(gameObject); // Ensure the manager persists between scenes
        }
        else
        {
            Destroy(gameObject);
        }
    }

    public float GetUniqueID()
    {
        lastAssignedID += 1;
        return (float)lastAssignedID / totalObjects; // Normalize the ID
    }
}

Identification

using System.Collections;
using System.Collections.Generic;
using UnityEngine;

public class Identification : MonoBehaviour
{
    private float _id;
    private Renderer _renderer;
    private MaterialPropertyBlock _propBlock;

    void Awake()
    {
        _id = IDManager.instance.GetUniqueID();
        _propBlock = new MaterialPropertyBlock();
        _renderer = this.gameObject.GetComponent<Renderer>();

        // TEMP
        Debug.Log("ID: " + _id);

        _renderer.GetPropertyBlock(_propBlock);
        _propBlock.SetFloat("_ID", _id);
        _renderer.SetPropertyBlock(_propBlock);
    }
}

URP Shader

Shader "Custom/UniqueColor2"
{
    Properties
    {
        _ID ("ID", Range(0, 1)) = 0.01
    }
    SubShader
    {
        Tags { "RenderType"="Opaque" }
        LOD 100

        HLSLINCLUDE

        #include "Packages/com.unity.render-pipelines.universal/ShaderLibrary/Core.hlsl"

        CBUFFER_START(UnityPerMaterial)
            float _ID;
        CBUFFER_END

        struct VertexInput
        {
            float4 position : POSITION;
        };

        struct VertexOutput
        {
            float4 position : SV_POSITION;
        };

        ENDHLSL

        Pass
        {
            HLSLPROGRAM
            #pragma vertex vert
            #pragma fragment frag

            VertexInput vert(VertexInput v)
            {
                VertexInput o;
                o.position = TransformObjectToHClip(v.position.xyz);
                return o;
            }

            float4 frag(VertexOutput i) : SV_Target
            {
                return float4(_ID, 0, 0, 1);
            }

            ENDHLSL
        }
    }
}

[Unity] Encoding Data into Pixels (Part 2): Precision Considerations

Daniel Chou Rainho — Thu, 08 Jun 2023 09:09:46 +0000

32 bits.
That's how much data can be stored in a float.

In my previous post I noted the following:

In a 32-bit system, each bit can represent two possible values: 0 or 1. Since there are 32 bits in total, we can calculate the number of possible values by raising 2 to the power of 32: 2^32 = 4,294,967,296.

However, in our case, we are only considering floats between 0 and 1.

So the question becomes: How many numbers can you represent in a float between 0 and 1?

Theory

The total number of representable numbers between 0 and 1 in a 32-bit floating point format is a bit tricky to answer directly because the format doesn't distribute precision evenly across all possible values.

A 32-bit floating-point number follows the IEEE 754 standard for floating-point arithmetic. The standard specifies that a float is composed of 1 sign bit, 8 exponent bits, and 23 fraction (or significand) bits.

The significand provides the precision of the number. With 23 bits for the fraction part, we have 2^23, or about 8.39 million, unique values. But this doesn't mean that we can represent exactly 8.39 million unique values between 0 and 1.

The reason for this is because floating-point numbers distribute more precision close to 0 and less precision as values increase. This property allows them to represent very large numbers and very small numbers at the cost of precision.

So while there are roughly 8.39 million unique fractional parts, they are not evenly distributed between 0 and 1. Instead, there are far more representable numbers close to 0 and gradually fewer as you get closer to 1.

A precise count of the total representable numbers between 0 and 1 is not straightforward due to this varying precision, but know that there are far more than just 8.39 million due to the ability of the exponent to shift the location of the decimal point.

Example

Let's look at a simplified example using binary fractions, which are analogous to the fraction part of floating-point numbers.

In binary, a fraction is represented the same way as in decimal: each digit (bit) after the binary point represents a negative power of 2. So, the first bit after the binary point is 1/2 (2^-1), the second bit is 1/4 (2^-2), the third bit is 1/8 (2^-3), and so on.

If we only have 3 bits to represent fractions (as opposed to 23 in a real float), we can only represent 8 (2^3) unique fractions:

000 in binary -> 0 in decimal
001 in binary -> 0.125 in decimal (1/8)
010 in binary -> 0.25 in decimal (1/4)
011 in binary -> 0.375 in decimal (1/4 + 1/8)
100 in binary -> 0.5 in decimal (1/2)
101 in binary -> 0.625 in decimal (1/2 + 1/8)
110 in binary -> 0.75 in decimal (1/2 + 1/4)
111 in binary -> 0.875 in decimal (1/2 + 1/4 + 1/8)

Now, if you look at the range between 0 and 1, you will notice that these fractions are not evenly distributed. There are more representable numbers closer to 0 and fewer as you approach 1.

[Unity] Encoding Data into Pixels (Part 1): CG Shaders

Daniel Chou Rainho — Sun, 04 Jun 2023 16:32:38 +0000

For a project, I want to implement a feature which allows the picking of gameobjects by color. I was first introduced to this technique here.

I want pixel perfect object selection, and this approach is relatively straightforward. All it involves is painting each object in a distinct color, this color being a unique ID associated to each object. Then, we can simply see what color, and thus what object the player is pointing at.

Ideation

We create a new Universal Renderer Data since we want to color each gameobject in a different color, but don't want this effect to be visible on the main camera. We want this effect to be exclusive to a secondary camera which we will use only for the color picking action.

Setup

Locate the Universal Render Pipeline Asset used in your project. (To identify this one you can open Project Settings and then navigate to the Graphics section.)
Add a new field to its Renderer List under the Rendering section.
Create a new Universal Renderer Data asset. (Create -> Rendering -> URP Universal Renderer)
See its details in the Inspector, and then click on Add Render Feature, selecting the option Render Objects (Experimental). This one in particular allows us to apply the same material to every gameobject in our scene.
Inside the section for the newly created Render Feature, under the Filters section, select the Everything option for the Layer Mask field. In this case, I will want to paint every single GameObject.
Inside the section for the newly created Render Feature, click on Overrides. A Material field should be visible.
Create a new Material.
Create a new Shader, we will use this one later to color each gameobject in a unique color.
Assign the Shader to the newly created Material.
Assign the newly created Material to the Material field in the Render Feature section.
Assign the Universal Renderer Data to the previously created Renderer List field, created in step 2.
Inspect your secondary camera.
Select the newly created Renderer from the Renderer field in the Rendering section.

Hello World Shader

If you have followed along the steps above, recreating the Renderer, I have added this small section that tests if you did the setup correctly.

To test the correctness of the setup, you can use the following Shader code.

Shader "Custom/UniqueColoring" {
    SubShader {
        Tags { "RenderType"="Opaque" }
        LOD 100

        Pass {
            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag

            #include "UnityCG.cginc"

            struct appdata {
                float4 vertex : POSITION;
            };

            struct v2f {
                float4 vertex : SV_POSITION;
            };

            v2f vert (appdata v) {
                v2f o;
                o.vertex = UnityObjectToClipPos(v.vertex);
                return o;
            }

            fixed4 frag (v2f i) : SV_Target {
                // always return red color
                return fixed4(1.0, 0.0, 0.0, 1.0);
            }
            ENDCG
        }
    }
}

If everything was setup correctly, any GameObject you see through the second camera should now be red.

Bits & Bits

As of right now, our Shader only returns the color red for each fragment, which is determined by the following line:

return fixed4(1.0, 0.0, 0.0, 1.0);

We want to store a number in at least one of the color channels, say the red one for example.

The fragment Shader can be defined to return either a fixed4, half4, or a float4 data structure, which represent respectively, 4 11-bit numbers, 4 16-bit numbers, and 4 32-bit numbers. [Source]

To maximize the amount of data we can encode in the color, will choose to go with our fragment Shader returning a float4 data structure.

32-bits

The following section is very important to make sure everything works as expected.

We need to pay attention that when the fragment Shader returns 4 32-bit values, that any other functions these are passed to, do not make us lose any precision by converting the values to a data structure of lower precision.

Specifically, for reading colors I will be using my approach from my last post.

The following outlines the entire pass from the Shader to the final reading:

The Shader returns 4 32-bit values.
The image of my second camera is rendered into my Render Texture.
My Compute Shader reads a pixel from the Render Texture, and stores it in a buffer.
A script I have reads the value from the buffer.

Something that I missed early on when developing this solution is the precision value of the color channels for the Render Texture, which by default, are set to a very low precision value.

Best Render Texture

A Render Texture object has many properties for its RenderTextureFormat enumeration.

Here's a screenshot from the official documentation including some of them:

In our case we select the last option visible in the screenshot above: "ARGBFloat", making each color channel 32-bits, such that we preserve the precision level from the Shader.

In my case, I am not creating the Render Texture programmatically. I select the R32G32B32A32_SFLOAT option under the Color Format field.

In any case, we need to make sure that there is no loss in precision, and that at each step we preserve the 4 32-bit values without making any change to them.

In line with another great article:
Additional Considerations:

We use point filtering, since we don't want anti-aliasing or filtering.
We don't need MipMaps.
The Skybox should just be a solid color, filled with 0s as can be seen below:

How much is 32-bits?

In a 32-bit system, each bit can represent two possible values: 0 or 1. Since there are 32 bits in total, we can calculate the number of possible values by raising 2 to the power of 32:

2^32 = 4,294,967,296

Therefore, there are 4,294,967,296 different values that can be represented in 32 bits.

Conclusion

Everything should work now.
Try changing the Shader's color to have something like 0.5 or 0.001 in its red color channel, to then see if this is indeed the value output in the console (assuming you are logging every reading).

[Unity] Fast Pixel Reading (Part 2): AsyncGPUReadback

Daniel Chou Rainho — Tue, 30 May 2023 19:55:54 +0000

From my previous post, I felt a bit disappointed with the results.

Although my approach for reading pixels was significantly faster, in the end, the performance hit on the running fps count was approximately the same.

Hence this second post, in which I build on the previous post to create a solution that leads to a negligible drop from the baseline fps count for a default empty scene, as compared to the optimized approach from my previous post, which leads to a baseline drop of 22.89%, all while using the optimized Compute Shader from my previous post.

AsyncGPUReadback

The following implementation is an adaptation of my code from the previous post, being an implementation of the ReadPixels method using a Compute Shader for optimized performance.

In particular, in my new approach, I use the AsyncGPUReadback method, which allows me to read data back from the GPU in a non-blocking way.

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.Rendering;
using System;

public class PixelReader2 : MonoBehaviour
{
    [SerializeField] private RenderTexture inputTexture;
    private Texture2D texture2D;

    [SerializeField] private ComputeShader ReadFirstPixel;
    private ComputeBuffer outputBuffer;
    private int kernelID;

    private void Start()
    {
        texture2D = new Texture2D(inputTexture.width, inputTexture.height, TextureFormat.RGBA32, false);

        // Create the output buffer
        outputBuffer = new ComputeBuffer(1, sizeof(float) * 4);

        // Get the kernel ID of the ReadFirstPixel function
        kernelID = ReadFirstPixel.FindKernel("ReadFirstPixel");

        // Set the input texture and output buffer to the shader
        ReadFirstPixel.SetTexture(kernelID, "inputTexture", inputTexture);
        ReadFirstPixel.SetBuffer(kernelID, "outputBuffer", outputBuffer);
    }

    private void Update() {
        StartCoroutine(ReadPixel());
    }

    private IEnumerator ReadPixel()
    {
        // Dispatch the compute shader
        ReadFirstPixel.Dispatch(kernelID, 1, 1, 1);

        // Request the data from the GPU to the CPU
        AsyncGPUReadbackRequest request = AsyncGPUReadback.Request(outputBuffer);

        // Wait for the request to complete
        while (!request.done)
        {
            yield return null;
        }

        if (request.hasError)
        {
            Debug.Log("GPU readback error detected.");
        }
        else
        {
            // Extract the color components from the output array
            float[] outputArray = request.GetData<float>().ToArray();
            Color color = new Color(outputArray[0], outputArray[1], outputArray[2], outputArray[3]);
        }
    }

    void OnDestroy()
    {
        // Release the output buffer
        outputBuffer.Release();
    }
}

Test Setup

For evaluating performance I use the following script, which logs the average fps count over the first 30 seconds of running the game.

using UnityEngine;

public class AverageFPSCounter : MonoBehaviour
{
    private float startTime;
    private int frameCount;

    private void Start()
    {
        startTime = Time.time;
        frameCount = 0;
    }

    private void Update()
    {
        frameCount++;

        if (Time.time - startTime >= 30f)
        {
            float averageFPS = frameCount / (Time.time - startTime);
            Debug.Log("Average FPS over 10 seconds: " + averageFPS);

            // Optionally, you can stop the script from updating further.
            enabled = false;
        }
    }
}

Performance Evaluation

FPS Baseline Drop from 57.53fps to
old implementation: 22.89% (44.36fps)
new implementation: -0.83% (58.01fps)

[Unity] Fast Pixel Reading (Part 1): Render Textures & Compute Shaders

Daniel Chou Rainho — Wed, 24 May 2023 19:44:55 +0000

Intro

For a project, I needed to call ReadPixels every single frame, i.e. in the Update function.

As it turns out, this is a very costly operation, because it involves communicating information from the GPU to the CPU.

The following post details how I improved the speed at which I get information from the GPU to the CPU from 0.257ms to 0.075ms, an improvement of 70%.

Foreword

For this post I replicate my findings from a Unity project I built from scratch so that it can be replicated easily if wanted.

The following are the steps I took, starting in Unity Hub.
I also include simple performance evaluations at every step.

With "baseline" I shall refer to the fps count level measured in my "3D (URD)" Unity scene, before having done anything to it after creating it.

Testing

For evaluating performance I use the following script, which logs the average fps count over the first 10 seconds of running the game.

using UnityEngine;

public class AverageFPSCounter : MonoBehaviour
{
    private float startTime;
    private int frameCount;

    private void Start()
    {
        startTime = Time.time;
        frameCount = 0;
    }

    private void Update()
    {
        frameCount++;

        if (Time.time - startTime >= 10f)
        {
            float averageFPS = frameCount / (Time.time - startTime);
            Debug.Log("Average FPS over 10 seconds: " + averageFPS);

            // Optionally, you can stop the script from updating further.
            enabled = false;
        }
    }
}

Setup

Create a new Unity project using the "3D (URP)" template. (I'm using Unity version 2021.3.16f1)
Create a new "Camera" game object.
Remove the "Audio Listener" component.

Running the game now with 2 active cameras in the scene naturally tanks performance by quite a bit.

Performance Evaluation: Baseline drop of 43%

RenderTexture

In my case I just want to read the color of pixels from a separate camera, which does not need to be rendered to the player, which is why I just render it into a RenderTexture, wich is a lot more efficient.

Create a RenderTexture.
Assign it to the "Output Texture" of the secondary camera.

Performance Evaluation: Baseline drop of 11%

ReadPixel

I created a simple script that reads the color of the first pixel from the RenderTexture, and does this every single frame.

using System.Collections;
using System.Collections.Generic;
using UnityEngine;

public class PixelReader : MonoBehaviour
{
    [SerializeField] private RenderTexture _renderTexture;
    private Texture2D texture2D;

    private void Start()
    {
        texture2D = new Texture2D(_renderTexture.width, _renderTexture.height, TextureFormat.RGBA32, false);
    }

    private void Update()
    {
        ReadFirstPixel();
    }

    private void ReadFirstPixel()
    {
        // Copy the RenderTexture's pixels to the Texture2D
        RenderTexture.active = _renderTexture;
        texture2D.ReadPixels(new Rect(0, 0, _renderTexture.width, _renderTexture.height), 0, 0);
        texture2D.Apply();

        // Get the first pixel's color
        Color firstPixelColor = texture2D.GetPixel(0, 0);
    }
}

Attach the script to the secondary camera.
Drag previously created RenderTexture into the visible "Render Texture" field.

Performance evaluation: Baseline drop of 23%

Line-by-Line Code Performance Evaluation

Executing the ReadFirstPixel function 100 times, the average runtime for it is 0.257ms.

I also made separate evaluations for all lines within the ReadFirstPixel function. The number in brackets next to the line number indicates the average runtime for the line across 100 runs.

Line 1 (0.011ms):

RenderTexture.active = _renderTexture;

Line 2 (0.213ms):

texture2D.ReadPixels(new Rect(0, 0, _renderTexture.width, _renderTexture.height), 0, 0);

Line 3 (0.064ms):

texture2D.Apply();

Line 4 (0.007ms):

Color firstPixelColor = texture2D.GetPixel(0, 0);

Conclusion: As expected, ReadPixels is indeed the function that takes the longest to execute. Let's try to optimize this!

Compute Shader

Instead of using ReadPixels, we will be creating a Compute Shader to transfer the color data from the GPU to the CPU in a faster way.

The following is my Compute Shader.

#pragma kernel ReadFirstPixel

// Input texture
Texture2D<float4> inputTexture;

// Output buffer
#pragma bind_buffer(name: outputBuffer, binding: 0)

RWStructuredBuffer<float4> outputBuffer;

[numthreads(1, 1, 1)]
void ReadFirstPixel(uint3 id : SV_DispatchThreadID)
{
    // Read the color of the first pixel
    float4 color = inputTexture.Load(int3(0, 0, 0));

    // Store the color in the output buffer
    outputBuffer[0] = color;
}

The following is my new version of the PixelReader script, which now uses the compute shader.

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using System;

public class PixelReader : MonoBehaviour
{
    [SerializeField] private RenderTexture inputTexture;
    private Texture2D texture2D;

    [SerializeField] private ComputeShader ReadFirstPixel;
    private ComputeBuffer outputBuffer;
    private int kernelID;

    private void Start()
    {
        texture2D = new Texture2D(inputTexture.width, inputTexture.height, TextureFormat.RGBA32, false);

        // Create the output buffer
        outputBuffer = new ComputeBuffer(1, sizeof(float) * 4);

        // Get the kernel ID of the ReadFirstPixel function
        kernelID = ReadFirstPixel.FindKernel("ReadFirstPixel");

        // Set the input texture and output buffer to the shader
        ReadFirstPixel.SetTexture(kernelID, "inputTexture", inputTexture);
        ReadFirstPixel.SetBuffer(kernelID, "outputBuffer", outputBuffer);
    }

    private void Update()
    {
        FastReadPixel();
    }

    private void FastReadPixel()
    {
        // Dispatch the compute shader
        ReadFirstPixel.Dispatch(kernelID, 1, 1, 1);

        // Read the result from the output buffer
        float[] outputArray = new float[4];
        outputBuffer.GetData(outputArray);

        // Extract the color components from the output array
        Color color = new Color(outputArray[0], outputArray[1], outputArray[2], outputArray[3]);
    }

    void OnDestroy()
    {
        // Release the output buffer
        outputBuffer.Release();
    }
}

Performance evaluation: Baseline drop of 25%

Line-by-Line Code Performance Evaluation

Executing the new FastReadPixel function 100 times, the average runtime for it is 0.075ms.

I also made separate evaluations for all lines within the FastReadPixel function. The number in brackets next to the line number indicates the average runtime for the line across 100 runs.

Line 1 (0.005ms):

ReadFirstPixel.Dispatch(kernelID, 1, 1, 1);

Line 2 (0.001ms):

float[] outputArray = new float[4];

Line 3 (0.073ms):

outputBuffer.GetData(outputArray);

Line 4 (0.001ms):

Color color = new Color(outputArray[0], outputArray[1], outputArray[2], outputArray[3]);