Every question about 3D generation reduces to which representation is being generated. A mesh, a signed distance field, a radiance field and a set of Gaussian splats are four different objects with four different failure modes, and only one of them is what a game engine actually consumes.
The short answer
Diffusion works best on a regular grid of continuous values. A mesh is not that — it is a graph with variable topology, and there is no natural fixed-size tensor for it. So generative approaches either work in a grid-shaped proxy representation and convert to a mesh afterwards, or generate 2D images and reconstruct geometry from them. Both leave work to be done before the result is usable.
Four representations, and what each costs
| Representation | Description |
|---|---|
| polygon mesh | Vertices, faces, UV coordinates, texture maps. What engines and DCC tools consume. Variable topology and irregular connectivity make it hard to generate directly with a fixed-shape network; almost everything produces it by conversion rather than natively. |
| voxel grid or signed distance field | A regular 3D grid of occupancy or distance-to-surface values. Diffusable in exactly the way an image is, and convertible to a mesh by marching cubes. Its cost is memory: resolution scales as n³, so detail is expensive in a way it is not for images. |
| radiance field (NeRF and relatives) | A network mapping a 3D position and view direction to colour and density, rendered by marching rays. Excellent at photorealistic novel views including transparency and reflection. It is not geometry: there is no surface to collide with, no topology, and extracting a clean mesh from one is its own lossy problem. |
| 3D Gaussian splats | Millions of anisotropic Gaussians with position, covariance, colour and opacity, rendered by rasterisation rather than ray marching, which makes it fast. Also not geometry: a point cloud with no surface, no UVs, no collision mesh, and file sizes that are large for what they represent. |
Why voxel and SDF resolution is the constraint:
a 2D image at 1024² = 1,048,576 positions
a 3D grid at 256³ = 16,777,216 positions (16× the image)
a 3D grid at 512³ = 134,217,728 positions (128× the image)
A 256³ grid over a one-metre object gives about 4 mm of resolution.
That is coarse for anything with fine surface detail, and going to
512³ costs eight times the memory for twice the linear detail.
This is why detail in 3D generation is usually carried by the texture
rather than by the geometry, and why the geometry that comes out is
often smoother than the render suggests.
Three ways to generate one
Score distillation from a 2D model
Optimise a 3D representation so that its renderings, from random viewpoints, score well under a frozen 2D image diffusion model. No 3D training data is needed at all, which is the entire appeal. The approach was introduced as score distillation sampling in DreamFusion by Poole and co-authors in 2022.
Two structural problems come with it. It is an optimisation per asset rather than a forward pass, so generating one object takes minutes to hours rather than seconds. And every viewpoint is scored independently against a 2D prior that was trained mostly on front-facing photographs, which produces the well-documented multi-face artefact: a generated animal with a face on the back of its head, because the prior thinks every view should look like a front view.
Multi-view generation then reconstruction
Generate several views of the same object that agree with each other, then reconstruct geometry from them with a reconstruction network or classical photogrammetry. Much faster than optimisation, because both halves are forward passes.
The difficulty concentrates in the word “agree”. Independently generated views of the same prompt are different objects, so the generator has to produce views jointly — typically by treating the set of views as one tensor with attention across them, which is structurally the same solution as temporal attention in video. Residual disagreement between views shows up as blurred or doubled surface detail in the reconstruction.
Native 3D diffusion
Train an autoencoder over 3D shapes — occupancy fields, signed distance fields, or a set of latent tokens representing a shape — and run diffusion in that latent space. This is the direct analogue of latent diffusion for images and produces the most coherent geometry, because geometry is what it is modelling rather than what it is inferring.
Its constraint is data, discussed next, and it is a hard one.
The data constraint
This is the single most important difference between 3D generation and image generation, and it is not an engineering problem.
Image models were trained on corpora of billions of image-text pairs, assembled from a web that had been accumulating photographs for decades. There is no equivalent corpus of 3D assets. Publicly available 3D datasets are counted in the millions of objects at most, many are low quality, and the text descriptions attached to them are sparse and inconsistent. Producing a 3D asset requires deliberate human work in a specialist tool, so the supply was never going to resemble the supply of photographs.
Every approach above is in some sense a response to that. Score distillation exists to avoid needing 3D data. Multi-view generation exists to borrow the image corpus. Native 3D diffusion is the most principled approach and the one most limited by what it can be trained on. Expect approaches that leverage 2D data to keep mattering for as long as that asymmetry holds.
From generated asset to engine-ready
A generated object is not an asset. This is the part that generally gets omitted, and it is where the time goes.
- Surface extraction. Marching cubes on the field, or a meshing pass over the splats. Produces a dense, irregular triangle soup — often hundreds of thousands of triangles with no useful edge flow.
- Retopology. Rebuild that as a low-polygon mesh with sensible quads and edge loops. Required for deformation, for LODs, and for anything a person will edit afterwards. Automatic retopology tools exist and are adequate for props, and are generally not adequate for anything that must animate.
- UV unwrapping. Assign a 2D coordinate to every vertex so textures can be applied. Automatic unwrappers produce usable but wasteful layouts, and a mesh with poor UVs cannot be hand-painted afterwards.
- Texture baking. Transfer colour from the generated representation — vertex colours, a radiance field, splat colours — onto the new UV layout.
- Material separation. Split the baked colour into physically based channels: base colour, roughness, metallic, normal. This is where generated assets most often fail, because the generated colour has lighting baked into it. Shadows and highlights painted into a base colour map will be lit again by the engine, and the object looks wrong under every light.
- Scale, orientation and pivot. Set real-world units, the up axis, and a pivot at a sensible point. Trivial and universally forgotten, and it is what makes an imported asset appear microscopic, sideways, and rotating about its own ear.
- Collision, LODs and rigging. A simplified collision shape, several detail levels, and — if it must move — a skeleton and skin weights. Rigging remains substantially manual for anything non-standard.
Which steps are actually hard
Of those seven, three are where a generated asset costs real human time, and they are the ones worth watching for improvement.
- Retopology, for anything that deforms. A character needs edge loops around joints and around the mouth and eyes. Nothing in a density field knows where those should be, because they are a property of how the mesh will be animated rather than of how it looks.
- Delighting the texture. Separating albedo from baked-in lighting is an inverse problem, and getting it wrong is immediately visible the moment the asset is lit. A generator that output correctly separated material channels would remove more downstream work than one that produced better geometry.
- Rigging. Automatic skeleton fitting works for humanoids matching a standard template and degrades quickly outside it.
The useful way to evaluate anything in this area is therefore not “does the render look good” but “how many of those seven steps does it leave me”. A watertight mesh with clean quads, sensible UVs and separated PBR channels is a categorically different deliverable from a good-looking splat, even when they look identical in a viewer.
Nothing on this page asserts what any current tool produces. The representation properties and the pipeline steps are stable; which steps a given generator automates is exactly what is changing, and it is the question to ask of any tool being evaluated.
Top comments (0)