TL;DR This is notes of my first trial of synthetic dataset generation for machine learnings. Just generating simple torus rendered images and trained machine learning model to detect them.
Training method
I referred this article about training a model for Tensorflow.js to make a model. I tested with the kangaroo dataset and tomato dataset on Kaggle. Both worked fine and the trained model gave me good predictions. I applied the procedure to my dataset generated by Blender.
Steps
Set up scene
Let's put a torus at the origin of the scene. I left the default point light and the camera as it was.
Next, go to the Shading
tab and create a material. By default, Principled BSDF
is used as the surface shader. I left as it is. As I talk in the next section, I changed the Base Color
randomly in the Principled BSDF
to set the color of the torus.
Scripting
On Blender, we can write python scripts to control objects' properties, set up rendering parameters etc. I automated rendering images, moving / rotating the torus and changing colors randomly. After setting those parameters, I calculated the bounding boxes (x min, y min, x max, y max) of torus in each rendered images to generate annotation dataset for training and rendered images as inputs. I generated 600 images for training and 200 images for testring. Let me share a part of my script. (The full script is on Github. Thanks for a Blender forum article. I use the code on the article to calculate the bonding box.)
base_dir = "d:/rendering_result"
geometry_file_path = os.path.join(base_dir, "geometry.txt")
donut = bpy.data.objects["Torus"]
with open(geometry_file_path, "w") as file:
for i in range(800):
# Setting rotation
donut.rotation_euler[0] = random() * math.pi
donut.rotation_euler[1] = random() * math.pi
donut.rotation_euler[2] = random() * math.pi
# Setting rotation
donut.location.x = random()
donut.location.y = random()
donut.location.z = random()
# Setting material color
color = (random(), random(), random(), 1)
donut.active_material.node_tree.nodes['Principled BSDF'].inputs["Base Color"].default_value = color
# Naming a file and render an image
f = "image" + str(i) + ".png"
path = os.path.join(base_dir, f)
bpy.context.scene.render.filepath = path
bpy.ops.render.render(write_still = True)
# Calculating the bounding box
b = camera_view_bounds_2d(bpy.context.scene, bpy.context.scene.camera, bpy.data.objects['Torus'])
# Generating annotation CSV row and writing it in a file
row = "%s,%i,%i,donut,%i,%i,%i,%i" % (f, 512, 512, b.x, b.y, b.x + b.width, b.y + b.height)
file.write(row + "\n")
The followings are the example images and the annotation data
filename,width,height,class,xmin,ymin,xmax,ymax
image0.png,512,512,donut,207,143,333,285
image1.png,512,512,donut,262,157,324,311
image2.png,512,512,donut,253,122,411,267
image3.png,512,512,donut,268,146,420,301
image4.png,512,512,donut,243,185,401,286
...
Training with the dataset
I do not explain the details of my training because the steps are written in the article I referred. However, I trained my models on my PC although the article is using Colab. My notes may help if you want to make your models in your local PC.
I used 1000
as the training step. First I tried 7500 as the same as the kangaroo model training in the article, but it was too much for this training.
Result
With the 200 testing dataset, I checked detection results. Almost all torus are detected with confidence over 70% confidence.
However, the model did not recognized the torus in the images taken by my webcam (I took pictures of my PC screen with the webcam). I guess the training torus in the dataset is too clean comparing to the real world. I should include real photo image as the background and some noise, motion blur and so on in the dataset.
Through this process, I learned synthetic dataset generation somewhat work but I still have many improvements I should introduce to use my model in the real world (like AR). Please share your experiences about synthetic dataset generation as comments!
Top comments (2)
So this data only contain torus?
Yeah, my data set only contains toruses