For most ladies, applying makeup is a daily routine. Some procedures are simple and can be done within a few minutes while others require a professional makeup artist. What if you had a personal stylist who is always available to do your makeup whenever you want? That's basically what this project is about; a physical robot (SO-101) that does your makeup, controllable by clicking a button on a web interface or just... talking to it. I call it Artsy.
So what does it actually do?
The robot does exactly what you tell it. Currently it can perform 3 basic actions i.e apply lipstick, draw eyebrows, and put on some blush on the cheeks. I settled on these simple actions as proof-of-concept before adding more functionality.
Disclaimer alert; this is still a work in progress and so it runs on the Cyberwave simulation environment (though the live robot’s movements can be seen in live mode).
You can control Artsy in two ways:
• Web interface: the web app provides you with an interface where you can select color, thickness, and what makeup to apply.
• Voice command: with this, you can issue instructions verbally e.g “put on some dark-blue lipstick” or “draw purple eyebrows”.
Either way, you're watching the same robot do the same motion. The web interface and the voice command are just two doors into the same room.
Since there is no 3D face model added to the simulation environment yet, the web app also allows you to upload a face image where you will be able to see as the makeup is being applied.
This started as a demo project for showing off a physical AI agent doing something tangible and fun rather than another chatbot in a text box. Makeup application turned out to be a great choice for that because it is visual, satisfying to watch, and also a nice showcase of a robot arm doing something precise and gentle rather than the usual industrial-arm-lifting-boxes cliché. It's equally a fun way to explore what physical AI can look like when you let a language model drive real hardware, safely, through natural conversation.
How to use it
1. Fire up the server — this is the one process that actually talks to the robot (or simulates it, if you don't have hardware handy).
2. Open the web app, upload a clear face photo, pick a makeup action and color from the sidebar.
3. Hit Apply Makeup and watch the arm move. You will also see the makeup being applied on the uploaded photo.
Want to control it by voice instead? Run the voice/text assistant in a terminal, say something like "draw both my eyebrows in dark brown," and watch the same thing happen.
For detailed instructions on how to run the program, check the readme.md file in this repo: https://github.com/Rabinnnn/artsy.git
The tech stack
• Cyberwave Python SDK - owns the actual connection to the SO-101 arm and allows you to control it.
• HTML/JS – for the web app interface that the user interacts with to give commands and also visualize the makeup application.
• MediaPipe FaceMesh – open source library for detecting facial features.
• LLM (DeepSeek) – planner that listens to what you say and figures out the makeup actions that need to be taken to satisfy your command.
• Mistral Voxtral – the speech-to-text engine that translates your voice commands to text.
• Websockets – the glue that links everything together, so the robot, the web app, and the voice assistant are all watching the same live stream of what the arm is doing.
What's next
Here are some of the things that I intend to work on next.
• Real vision – right now facial detection is done on an uploaded image. It would be great to have the camera attached to the robot do the actual recognition of facial features.
• More looks – currently the robot is only capable of applying lipstick, blush, and drawing eyebrows. More actions are required e.g eyeliner, contouring etc.
• Smarter color talk - right now you can specify the color e.g "coral" or "teal" and it works, but pairing that with actual style/vibe requests e.g "something that'd look good for a night out" would be a fun next step.
Top comments (0)