DEV Community: Alexandre Donciu-Julin

I asked ChatGPT to generate my Spotify playlists

Alexandre Donciu-Julin — Wed, 22 Nov 2023 09:03:24 +0000

A command-line tool to generate Spotify playlists based on ChatGPT prompts.

Project description

The goal of this simple project was to learn how to interact with the OpenAI API and get useful information from ChatGPT in a format I can easily manipulate (JSON dictionaries of artists and songs). On the other side, it uses the Spotipy module to interact with Spotify and create playlists. Linked together, these two modules can help you create creative playlists from songs suggested by ChatGPT according to a given prompt (Example: "Peaceful songs to listen to when it's raining"). It's a great way to discover new tunes!

This project is built upon a tutorial by Colt Steele: Mastering OpenAI Python APIs that I highly recommend, as it covers the basics of how ChatGPT works and how to interact with it. On top of the automatic playlist creation mode covered by the tutorial, I created an interactive mode, which lets you decide which songs should be added and gives you the possibility to blacklist artists and songs. I completed the project with additional methods to play and print playlists.

Prerequisites

To use this tool, you need both an OpenAI and a Spotify Developer account.

OpenAI

It's the company that created and offers access to different ChatGPT models. Go to the OpenAI website and create an account. It comes with some free credit to start and test the API. See documentation to generate your own API-Key.

Spotify

Create a Spotify account if you don't already have one, it's free. After that, log in to Spotify for Developers and create a project. In the project settings, you can generate the client-ID and -Secret needed to connect and send requests to Spotify. See the documentation for more info.

Once you have your credentials to access both APIs, you can add them to playlist_generator.py. I am using the keyring module to store and retrieve mine, but you can use an .env file, store them in environment variables or just use them directly in the code. In this last case, be careful not to share your code with anyone!

# Set openai key
openai.api_key = keyring.get_password('openai_key', 'main_key')
# Set spotify client id and secret
client_id=keyring.get_password('spotify', 'client_ID')
client_secret=keyring.get_password('spotify', 'client_secret')

Spotify Desktop

If you want to be able to play songs during the playlist creation (in the interactive mode for instance), you will need Spotify Desktop installed on your machine. If you want to skip this part, comment all calls to the method play_song_in_spotify in playlist_generator.py, as it will not be able to ask your Spotify player to play the given songs.

Installation

Clone project and install required libraries (in a virtual environment for instance)

git clone https://github.com/alexdjulin/spotify-playlist-generator.git
python -m venv venv
venv\Scripts\activate # on windows
source venv/bin/activate # on macOS/linux
pip install -r requirements.txt

How to use it

The most convenient way to start the playlist generator is to call main.py.

python main.py

It will ask for the 4 required input variables:

Playlist Prompt (-p): A short description sent to ChatGPT to generate the playlist. Example: *Best songs for a road trip with your best friend.
Playlist Length (-l): How many songs should be in the playlist. Default value is 10.
Playlist Name (-n): The playlist name on Spotify. Optional, leave blank to use the prompt as a name. Example: Road Trip Songs.
Interactive Mode (-i): Y to activate or N to deactivate the interactive mode. When active, each song will be played in Spotify and the user can choose to Add the song, blacklist it or blacklist the artist.

Playlist Prompt: Best songs for a road trip with your best friend
Playlist/Batch Length: 10
Playlist Name (leave blank to use prompt): Road Trip Songs
Interactive Mode? [y]es or [n]o: n
Creating playlist. Please wait...
>> end of playlist creation

Additionally, you can call playlist_generator.py and pass these arguments in the command line directly. Don't forget the quotes around the strings. The interactive mode flag is TRUE if you pass -i, FALSE if you don't pass it:

# interactive flag off
python playlist_generator.py -p "Best songs for a road trip with your best friend" -l 10 -n "Road Trip Songs"
# interactive flag on
python playlist_generator.py -p "Best songs for a road trip with your best friend" -l 10 -n "Road Trip Songs" -i

Automatic Playlist Creation

When the interactive mode is off, the playlist is created and filled automatically. It is great to create quick playlists, without worrying too much about the contents... Providing that you trust ChatGPT's suggestions, which can be sometimes surprising!

Output when the interactive mode is off:

----------------------------------------------------------------------------------------------------
------------------------------------ SPOTIFY PLAYLIST GENERATOR ------------------------------------
----------------------------------------------------------------------------------------------------
Prompt: Best songs for a road trip with your best friend
Length: 10
Name: Road Trip Songs
Interactive: False
----------------------------------------------------------------------------------------------------
1. Journey - Don't Stop Believin' | Escape (Bonus Track Version)
2. Smash Mouth - All Star | Astro Lounge
3. Bruce Springsteen - Born to Run | Born To Run
4. Guns N' Roses - Sweet Child O' Mine | Appetite For Destruction
5. John Denver - Take Me Home, Country Roads - Original Version | Poems, Prayers and Promises
6. ABBA - Dancing Queen | Arrival
7. Bon Jovi - Livin' On A Prayer | Slippery When Wet
8. Queen - Bohemian Rhapsody | Bohemian Rhapsody (The Original Soundtrack)
9. Lynyrd Skynyrd - Sweet Home Alabama | Second Helping (Expanded Edition)
10. Whitney Houston - I Wanna Dance with Somebody (Who Loves Me) | Whitney
----------------------------------------------------------------------------------------------------

Spotify should open automatically, the songs are added to the playlist and the first one from the list should play when done.

Interactive Playlist Creation

If the interactive flag is on, we get a first batch of suggestions from ChatGPT (using the input playlist length as batch length) and we play the songs one by one in Spotify. For each song, we have the choice to:

Add the current song to the Playlist
Blacklist this song (don't suggest it anymore)
Blacklist this artist (don't suggest any songs from this artist anymore)
Quit generating the playlist and keep it as it is

Gary Jules - Mad World   (1/5)
[1] Add to Playlist
[2] Not this song
[3] Not this artist
[q] Quit playlist generation
Your choice:

When the choice is done, we jump to the other song. When the batch is done, we get the following choice:

Ask ChatGPT for another batch of songs
Quit generating the playlist and keep it as it is

Do you want another batch of songs?
[1] Yes, give me more songs!
[2] No, I'm down with this playlist
Your choice:

When asking ChatGPT for a new batch, we provide it with the list of songs we already have, as well as the lists of blacklisted songs and artists, to take into account for the next suggestions.

Ouput when the interactive playlist is done:

----------------------------------------------------------------------------------------------------
------------------------------------ SPOTIFY PLAYLIST GENERATOR ------------------------------------
----------------------------------------------------------------------------------------------------
Prompt: Lonely songs to play when it's raining
Length: 10
Name: Rainy Mood
Interactive: True
Artists Blacklist: {'The Carpenters', 'Willie Nelson'}
Songs Blacklist: {'Adore', "I Can't Make You Love Me"}
----------------------------------------------------------------------------------------------------
1. System Of A Down - Lonely Day | Hypnotize
2. a-ha - Crying in the Rain | East of the Sun, West of the Moon
3. Brook Benton - Rainy Night in Georgia | Brook Benton Today
4. Green Day - Boulevard of Broken Dreams | Boulevard of Broken Dreams
5. The Doors - Riders on the Storm | L.A. Woman
6. R.E.M. - Everybody Hurts | Automatic For The People
----------------------------------------------------------------------------------------------------

See GitHub readme for additional notes and contributions.

Conclusion

This has been a fun little project and a good first experience playing around with the OpenAI API. The Spotipy library on the other side is very intuitive and you can quickly get around with it and interact with your Spotify account. Obviously, some prompts like "80's hits" will return you more accurate results than creative ones like "Sad songs to listen to when you just broke up and your fridge is empty". But I think switching the model to GPT4 and future models will probably improve on this.

My Runing Map using python and folium

Alexandre Donciu-Julin — Thu, 07 Sep 2023 20:27:13 +0000

A map of race events I took part in, with gpx traces and pop-up race information, entirely generated from a google spreadsheet.

Click on map to display the interactive one.

Link to the Github project

I'm a huge fan of maps, probably since I played around with my dad's roadmaps as a kid (he hated it, as he could never fold them back properly!). I cannot spend a day without fiddling
with Google Maps. Although my trail running hobby and my GPS-watch addiction plays a big part in that!

I have a running blog where I share reviews of events I took part in 🏃 But over the years, I kind of lost track of them. How many was there? Where did I run? How was the route? I need some kind of maps to visualise them. What if I could make it so simple, that I would only need to update a google spreadsheet with race information and the map on my blog would update accordingly? Let's see what we can do...

FOLIUM. That's our code-word for today. Folium is a python module that is going to provide exactly what we need: Manipulate data in Python, then visualize it in a Leaflet map.

Let's start with a simple example: Creating a map from one race event logged in this spreadsheet.

See tutorial.py on my git repo.

Download the spreadsheet

Let's first download the google spreadsheet as a csv file, it will be easier to work with the data. I shared the document so anyone with the link can access it. The following command will download the spreadsheet.



import os
sheet_url = 'https://docs.google.com/spreadsheets/d/1WghWJbxdCeKpbi3-H_6FmBAv_jILD_1_woRhuKGJ190/export?exportFormat=csv'
current_folder = os.path.dirname(os.path.abspath(__file__))
csv_filepath = os.path.join(current_folder, 'run_events.csv')
command = f'curl -L {sheet_url} -o {csv_filepath}'
os.system(command)

You should now have a run_events.csv file next to your script.

Load CSV data

Who doesn't love Pandas 🐼? Load the contents of the csv file into a python dictionnary.



import pandas as pd
data = pd.read_csv(csv_filepath).to_dict(orient='records')[0]
print(data)

# Output:
# {'Date': '28.09.2014', 'Race': '41. Berlin Marathon', 'Latitude': 52.51625499, 'Longitude': 13.37757535, 'Time': '4:17:35', 'Link': 'https://www.bmw-berlin-marathon.com/'}

Create the html map

Now we can start having fun! You will find Folium tutorials everywhere and the documentation itself is pretty straightforward. Let's generate a simple map and populate it with our data.



import folium

# create map object and center it on our event
import folium
run_map = folium.Map(location=[data['Latitude'], data['Longitude']], tiles=None, zoom_start=12)

# add Openstreetmap layer
folium.TileLayer('openstreetmap', name='OpenStreet Map').add_to(run_map)

# save and open map
run_map.save('run_map.html')
import webbrowser
webbrowser.open('run_map.html')

Run the code, the map should be generated, saved and opened in your default web browser. As you can see, it's empty and just centered on the race location (Berlin). Click images to see html maps.

Populate the map

Let's add a marker point for our race to the map. We want it to be part of a 'Marathons' feature group, display the race name as tooltip, assign a color to it and display a short legend in the corner.



# add feature group for Marathons
fg_marathons = folium.FeatureGroup(name='Marathons').add_to(run_map)

# create marker and add it to marathon feature group
folium_marker = folium.Marker(location=[data['Latitude'], data['Longitude']], tooltip=data['Race'], icon=folium.Icon(color='red'))
folium_marker.add_to(fg_marathons)

# add legend in top right corner
run_map.add_child(folium.LayerControl(position='topright', collapsed=False, autoZIndex=True))

Our marker now shows up at the given coordinates. It displays the name of the race if we hover over it and we can display or hide it from the corner legend.

Add pop-up windows

It would be nice to make that marker clickable and offer more information about the race. Let's create an html iframe that will pop-up when the user clicks on the marker. You will need some basic html knowledge for that, but nothing fancy at this point.



# create an iframe pop-up for the marker
popup_html = f"<b>Date:</b> {data['Date']}<br/>"
popup_html += f"<b>Race:</b> {data['Race']}<br/>"
popup_html += f"<b>Time:</b> {data['Time']}<br/>"
popup_html += '<b><a href="{}" target="_blank">Event Page</a></b>'.format(data['Link'])
popup_iframe = folium.IFrame(width=200, height=110, html=popup_html)

# modify the marker object to display the pop-up
folium_marker = folium.Marker(location=[data['Latitude'], data['Longitude']], tooltip=data['Race'], popup=folium.Popup(popup_iframe), icon=folium.Icon(color='red'))
folium_marker.add_to(fg_marathons)

There it is. We even created a link on the url, opening the event page in another tab. And since it's an html iframe, we can now basically display anything we want in it (pictures, videos, links, css stying, and so on).

Add GPX trace

Cherry on top, it would be amazing to display the route, like you usually see on the even't website.

This was actually easier than I thought. GPX files recorded by gps-watches or phones are XML-documents easy to read and parse. I used the gpxpy library for that, which is doing exactly what we need: read the gpx file and extract points/segments from it to display on our map. You can find GPX files everywhere, on Strava or Komoot for instance.

Here is how I opened and extracted segments from my own GPX file recorded during the race. I am using a step value when slicing all the points, to smooth out the curve (loading 1 every 10 coordinate points).



# parse gpx file
import gpxpy
gpx_file = 'berlin_marathon_2014.gpx'
gpx = gpxpy.parse(open(gpx_file))
track = gpx.tracks[0]
segment = track.segments[0]

# load coordinate points
points = []
for track in gpx.tracks:
    for segment in track.segments:
        step = 10
        for point in segment.points[::step]:
            points.append(tuple([point.latitude, point.longitude]))

# add segments to the map
folium_gpx = folium.PolyLine(points, color='red', weight=5, opacity=0.85).add_to(run_map)

# add the gpx trace to our marathon group
folium_gpx.add_to(fg_marathons)

Wonderful, the trace is showing up as expected on the map. Notice that, as we added it to the Marathons feature group, it inherits the red color and is affected by the legend checkbox too. You can now play around with the segments weight, opacity, color and with the step value to fine-tune it.

That's it, you have all you need to build a kick-ass map. Just add multiple lines to the spreadsheet and modify your dataframe to load all the data into lists.

Improve the map

If you browse through my run_map.py script, you will notice that it is a bit more advanced. Here are some improvements I added to my map and to the project itself, to make it more appealing and to fit my needs:

Add additional tile layers (ArcGIS)
Group my events by type (halfs, marathons, ultras) and assign to each one a different color, also affecting the pop-up title and the gpx trace
Customize the pop-up window with a picture of the race, a nice font, links, etc
Move all paths and map settings to a json file, so there are no hard-coded values in the code and it is easier to change a setting (gpx trace weight or opacity for instance)
Put all race events info into the google spreadsheet
Add an ftp upload method at the end to send the html, jpg and gpx files onto the online storage my blog is using

You can see the final result in the Events tab of my running blog.

Finally, you may notice that my script does not only update the map but also a table of events information displayed below it, as well as a little event-o-meter gadget in the sidebar. Both are generated when I run the script and updated according to the updated google spreadsheet information.

Conclusion

I therefore succeeded in my holy quest to put all my running events on a pretty nice-looking map. All I need to do now, after completing a new event, is to fill in the information in the google doc and provide a jpg thumbnail and the gpx trace of my run, then run the script to generate a new map and update the table and gadget. This last step could be automated of course, if we had our script running in the cloud and checking any update done on the spreadsheet.

Have fun playing with folium and don't forget to share your maps!
Take care and see you on the trail 🏔️🏃‍♂️

Draw and email your Secret Santas using Python and SendGrid

Alexandre Donciu-Julin — Tue, 14 Dec 2021 09:23:41 +0000

Project on GitHub here.

It's that time of the year, AGAIN! It's been a fast one, but I do feel like I learned a lot, on my holy quest to become a rock-star programmer. That say, I would like to finish the year on a fun simple project. So when my mom asked me to draw our Secret Santas this year, I had it.

If you are not familiar with the concept, it's a nice way to celebrate xmas amongst a large group of friends or family members, withoug having to spend a salary on gifs. Every member of the group is assigned one recipient, and as his/her Secret Santa (since these random assignments are kept secret), you need to buy a gift for this person only. This way everyone will get a nice present instead of dozens of cheap ones.

In the past years, I used a basic random name generator for that, but with the downside that I knew all the assignments and who my Secret Santa was. So this year, let's be clever and use a python script to do the draw and contact Santas per email.

Get list of Santas from a CSV file

I leaned the Pandas library this year, which is great for extracting information from a CSV file. It's a user-friendly way to input data, easier than writing in the script directly. Here is our CSV file, with our six friends and the information we need to contact them.

The Black List

Black list, quid est?
I like to spice things up and I thought it would be handy to add a black list feature, i.e. a list of names that one Santa should not get for any reason. In our case for instance:

Ross and Rachel should not get each other, cause they had a big fight and they are 'on a break'
Monica and Chandler should not get each other, as they are married and they wanna give each others more expensive presents
Joey is broke, he bought a cheap present but it's for a guy so he cannot be one of the girl's Secret Santa
Phoebe likes everyone so her black list is empty

Create the SecretSanta class

I learned programming using C++ and I have the CLASS word inprinted in caps inside my brain. So why not organise our script a bit by adding a SecretSanta class? Here are the class members and methods we need for the draw:

class SecretSanta:

    def __init__(self, name, email, black_list = None):
        """ initialize class variables """
        self.name = name
        self.email = email 
        self.recipient = None  # will be allocated later
        self.black_list = list()
        self.black_list.append(self.name)  # adding own name
        if black_list:
            self.black_list += black_list.split('|')

    def __repr__(self):
        """ override print method (optional) """
        # return string to print

    def contact_secret_santa(self):
        """ contact recipient """
        # use SendGrid example here

Load settings from a json file

Again, I find it more friendly to use external files to load variables instead of editing the code directly. So I usually add a settings.json file to all my projects. Here are the variables that we need to define there:

Path to our CSV file containing Secret Santas information
Path to the email template (.txt or .html)
Max attempts that should be done while trying to assign recipients to secret santas. Due to the black lists, an assignment may not be possible and therefore this variable will break the while loop and raise an error.
Email of your SendGrid account
Personal SendGrid API key (private, don't share it with ANYONE)

{
    "csv_file": "data/secret_santas_list.csv",
    "email_file": "data/email.html",
    "attempts_limit": 100,
    "sg_sender_email": "youremail@domain.com",
    "sg_api_key": "your_api_key"
}

The json library helps you import and store these parameters in a SETTINGS dictionnary, that you can use further in your script.

import json
with open("data/settings.json", 'r') as json_file:  
    SETTINGS = json.load(json_file)
print(SETTINGS['csv_file'])
# output: 'data/secret_santas_list.csv'

Main - Create Santas

We now have everything we need to write our main procedure.

To extract the CSV infos into a Pandas dataframe and create our SecretSanta instances is a child game.

from pandas import read_csv
df = read_csv(csv_file_path).fillna('')
secret_santas = []
for i in range(len(df)):
    new_name = str(df['Name'][i])
    new_email = str(df['Email'][i])
    new_black_list = str(df['Black List'][i])
    new_santa = SecretSanta(name=new_name, email=new_email, black_list=new_black_list)
    secret_santas.append(new_santa)

Main - Draw

Finally we can draw our Secret Santas!

Since we introduced the black-list option, we face the issue that a draw might not be possible due to too many constraints (for instance, some poor guy who would be on everyone's black list!). I'm sure we could come up with a clever algorithm that would analyse all black-lists first and determine if a draw is possible or not. But let's keep it simple and take the easy road of trying, until we succeed or decide it's not gonna happen!

We will therefore wrap everything inside a while loop, that will break if the draw is successful (everyone has been assigned as Secret Santa) or after a maximum numner of attempts (defined in our settings file).

Here are the main steps I am following, check the GitHub project for the (well-commented!) code:

Initiate the while loop with our two conditions (success or reaching max attempts)
Shuffle our list of names and delete any previous assignment
Go through our list of SecretSanta instances and look for a recipient, which is not on the black-list. If we find one, we assign it and we jump to the next SecretSanta. If not, the draw fails and we start a new one (the while loop restart, we are back at step 2)
If we assign a recipient to all Secret Santas, the draw is successful and the while loop will break. If it never happens, the loop will break once reaching the max attemps.
Process the result: A succesful draw will trigger contacting the Secret Santas, while a failed one will raise an error and ask the user to review the input parameters.

E-mailing Santas

This is our last step. We are going to use Twilio SendGrid for this, which provides a cloud-based service that assists businesses with email delivery. This is a very easy step thanks to their crystal-clear tutorials. Here are the steps you need to follow to use SendGrid services.

Create a SendGrid account
You need to register for an account first. The free offer allows you to send up to 100 emails a day, which is good enough for our little project.

Create a Single Sender Verification
Your recipients will receive emails from a specific address. Terefore this e-mail needs to be existing and verified as your own.

Create a personal API key
This is an important step. The API key is personal and should never be shared. It will serve as a badass password (seriously, just look at it!) that will need to be passed everytime e-mails are sent from your account. For simplicity, you can add it to your environment variables.

Install the SendGrid python library
Finally you can install the python library. The documentation gives you some simple examples on how to send your first emails. I did not need to dig much further, those are great. This project uses the "Without Mail Helper Class" example, that I implemented in my contact_secret_santa() class method.

Personalising the email content

This is an optional but nice little step. I found it more joyful and christmassy to write the email contents as from the hand of Santa himself, asking people to help him spread the joy. Sendgrid let you use a text or html format for the contents of the email. So why not draft an html file with some pictures, colors, a handwriting font, etc? You can also put tags like [NAME] and [RECIPIENT] that will be replaced with your class attributes to address the person directly. Put some CSS to make it look more personal and believable. I don't know much about web dev but feel free to check my html template.

Once all these steps are done and if your draw was successful, you can now call the contact_secret_santa() method of all your instances. They will be notified within a few minutes and receive an email based on your template. Hey, you got one too!

Conclusion

I hope this little project taught you something new, whether you are playing with CSV and json files, trying to send emails or simply improving your python skills. It was a fun last projet to wrap up this year, which has been for me rich in learning new stuff and hopefully a few more steps towards my goals.

I wish you all a wonderful Christmas. Have a great Secret Santa sharing time with the ones you love (and the ones from your black list too!). Take care and read you next year :)

__
Friends pictures © Warner Bros. Entertainment Inc
Cover picture from Rachel Middleton

Live Link Face to Unreal MetaHuman Retarget

Alexandre Donciu-Julin — Wed, 03 Nov 2021 07:41:04 +0000

How to retarget facial animations recorded with the Live Link Face iPhone app onto an Unreal MetaHuman character, using MotionBuilder and Python.

You can find the project files on my Github repo.

About MetaHumans

I recently started playing around with the mind-blowing MetaHuman (MH) feature from Unreal Engine. Through the Quixel Bridge editor, you can create and customize a highly realistic avatar in a few clicks, and send it to your Unreal project. Everything is fine-tuned to get the best of Unreal shaders and will leave you speechless 😍 I invite you to take a peek at it, plenty of resources out there.

About Live Link Face

As a mocap director, my interest quickly shifted over the live-retarget capabilities, i.e. how to transfer an actor's facial performance onto a metahuman, in real-time. Epic offers an app for that, which takes advantage of the iPhone's TrueDepth camera: Live Link Face (LLF). It also takes a few click to link your phone to your Unreal project and start driving your MetaHuman's face. Again, it's working 'Epic-ly' fast and well!

But there is the snag... The app was clearly designed to work in real-time with Unreal, where you can use the Sequencer to record facial data along body mocap, blueprints functions and so on. However, if used as a stand-alone app, it's another story.

LLF saves for each take 4 files:

A CSV file containing the animation
A video file of the performance
A json file containing some technical information
A jpg thumbnail

No maya scene, no unreal asset, not even a simple fbx file that we could use to import and clean the motion in a 3rd-party-software like MotionBuilder. Just a CSV file filled with random blendshape names and millions of values. Like that zucchini-peeler you got from aunt Edna for xmas, it's a nice gesture but what the heck am I gonna do with that!? 🤔

A quick look into the documentation confirmed the idea:

This data file is not currently used by Unreal Engine or by the Live Link Face app. However, the raw data in this file may be useful for developers who want to build additional tools around the facial capture.

So, I guess we are on our own now!

Let's dig in

It's not the most convenient format, but if you take a good look at this CSV file, it's all there: For each timecode frame (first column), you get a value of all the shapes defining the facial expression at that frame. These shapes come from Apple's ARKit Framework, which uses the iphone's TrueDepth camera to detect facial expressions and generate blendshape coefficients. All we have to do is extract those time/value pairs and store then in some handy container. This is where the Pandas library 🐼 and its dataframes are gonna do wonders!

Now what should I do with all this?

I need to recopy the animation values onto a target model that Unreal can import and read. Let's have a look at the MetaHuman project, that you can find on the UE Marketplace.

Facial and body motions are handled separately. A blueprint is applied to a facial version of the skeleton, which contains only the root, spine, neck, head and numerous facial joints, as Unreal relies on joint-based motions.

I can get this skeleton out of Unreal by creating an animation pose and exporting it as FBX. Now we can open it in motionbuilder and... HOLY MOLY GUACAMOLE, what's all this!? 😵 I'm not talking about the Hellraiser facial rig but about the root joint properties. It contains a huge amount of custom properties!

I run a quick line of code and count about 1100 entries:

254 properties starting with CTRL_expressions_, which are clearly blendshapes, like CTRL_expressions_jawOpen
757 properties starting with head_, which seem to be additional correctives (fixing for instance a collision between two blendshapes activated at the same time).
35 properties starting with cartilage_, eyeLeft_, eyeRight_ or teeth_, which are probably more correctives?
Last but not least: 53 properties, whose name are identical to the ARKit shapes in the CSV files from LLF, like EyeBlinkLeft or JawOpen.

We can therefore assume that these last 53 properties are correspond to the ARKit blendshapes from the app and should receive our key values during retargeting.

How is Unreal computing animations?

Before jumping into our favourite code editor, let's see how Unreal is reading and applying an animation onto the MetaHuman. I open the Face_AnimBP blueprint from the project, which is the one handling live-retargeting facial animations from the LLF app onto the rig.

BINGO, there it is! Inside a Live Link Face box, I find what seems to be a mapping block. It's doing all the connections. For each ARKit blendshape used by LLF, it is mapping the corresponding CTRL expressions and head correctives on the metahuman, by specifying an influence value between 0 (no influence) and 1 (full influence). You can see below how BrowOuterUpLeft triggers the 3 expressions browLateralL, browRaiseInL and browRaiseOuterL with different weights, while having no effect on any other expression.

To confirm, I create random keys on the different properties in MotionBuilder and export the takes as FBX files to Unreal. It's working like a charm!

Keys on the ARKit custom properties are read and retarget, but only if the animation goes through the mapping block first, so the animation blueprint is mandatory.
However, if I put keys on the CTRL shapes and head correctives, I can apply my animation directy onto the MH, without the need of an animation blueprint.

As you can see, we have two solutions taking shape here: The easy road where we simply transfer animation values from the CSV file to the ARKit blendshape properties, or the long road where we convert them into MH expression and corrective values. Trying both?

Retargeting on ARKit blendshapes

As said, this is the easy stroll in the park. Our CSV file and our root bone are sharing the same ARKit blendshape information. All we need to do is transfer the keys. Here are the main steps I am following:

Use Pandas to read the CSV file and extract all data. I create for this a BlendShape class, which receives the ARKit shape name, as well as animation keys, as pairs of timecode/values.
For all BlendShape objects created, I search on the skeleton root for the corresponding property name and create keys for all timecode/value pairs.
The scene framerate is very important here and should match the app's one, or keys won't be created at the correct frames.

Once done, adjust your timespan to frame the animation and have a look at the ARKit properties of the root. They should be animated.

Export your animation as FBX to Unreal and test it in the Face_AnimBP blueprint. Don't forget it needs to go through that mapping block so Unreal can convert ARKit blendshape values to MH property values, which will be then turned into joint information. Cool beans!

Retargeting on MetaHuman properties

Now let's get our hands dirty and explore that second solution! We want to be able to use a retarget file directly onto our MH character without going through all this mapping knick-knacks. But for that we need to find a way to recreate this mapping node in MotionBuilder. As a reminder, it is linking each ARKit shape to the corresponding MH ctrl and head properties with weighted values ranging from 0 to 1. So we first need to extract these values and weights from Unreal 🥵

After looking around for a while, I noticed that Unreal nodes can be exported as T3D ascii files, which contain all information on what the node is doing. I tried and I must admit that it wasn't love at first sight. Fortunately, it's only 13 lines long!

But let's not judge a book by its cover. Again, if you take a closer look, everything we need is here. Time to see if my regex ninja training has paid off. After playing around with three patterns, I manage to extract from the T3D file:

The ARKit blendshapes names
The MH properties names
The mapping weights for each property I add to my BlendShape class (characterising an ARKit shape), the target MH properties and the mapping weights. Done! I now have all the information I need for retargeting.

Back to MotionBuilder, I can run my magic loops again. This time we have to be a bit more careful though:

Read the T3D file and create BlendShape instances for all ARKit shapes.
Use Pandas to read the CSV file and extract the animation keys for each ARKit shape. I fill up these timecode/value pairs into my BlendShape class instances.
MotionBuilder is not very fast at finding root properties, maybe due to the high amount. To avoid searching for the same ones multiple times, I loop through the MH properties first and then through the timecode. For each frame, I check which ARKit blendshapes triggers my property and get the animation value weighted by my mapping information. For instance, if the ARKit value is 0.8 and the mapping value for this property is 0.5, then my animation key will be 0.8x0.5=0.4. If multiple ARKit shapes are triggering it, I take the max value corresponding to the highest influence.
Again, be careful to retarget at the same framerate you used when recording takes.

I see keys on my root's properties! Let's export it as FBX to Unreal and assign it to my Face component. WUNDERBAR! Unreal is reading the animation and transfering it to the facial joints without having to go through that mapping node anymore.

Wrapping up

To finalise the script and make it more user-friendly, I added the following features:

A batching option, in case you point to a folder containing multiple CSV files
The possibility to offset the starting timecode, to synchronise the animation file with other sources
A simple PySyde UI for the user to enter the required paths files
Some logs and exception handlings to make sure that our batch is performing properly

We are done, good job everyone! 😉

Conclusion

It only took patience and observation, as well as some ancient Regex voodoo to transfer the animation values recorded by the Live Link Face app as CSV files onto our brand new MetaHuman in Unreal. While not perfect, the result is good enough to move the animation to a cleaning stage.

This is therefore a quick way to record facial animations from your actors, even if they cannot perform in real-time. The ability to process them at a later stage using an animation software like MotionBuilder and a fairly simple python script like this one makes of Live Link Face a cheap and reliable facial motion-capture solution for MetaHumans characters.

I hope this will help you write your own solution. Feel free to message me if you need any help or buy me a croissant and cappuccino next time you're in town 🥐☕ Cheers!

A python regex to validate roman numerals

Alexandre Donciu-Julin — Fri, 09 Apr 2021 23:13:41 +0000

Note: This is my first post, I hope you'll like it :)

I'm not gonna lie to you... I LOVE REGEX!

As a kid, I grew up playing adventure games full of puzzles and riddles. Looking for the solution was a personal quest, a treasure hunt. Finding it was so exciting, but not as much as jumping on another riddle!

When I discovered regular expressions on my journey to become a (good) python programmer, I felt that same excitement. I was blown away by the countless possibilities these were offering. Deciphering one was like suddenly being able to read hieroglyph, writing one was like discovering I could speak a foreign language. Although I know they should be used with caution and in special cases only, I keep pushing myself to use them everywhere I can.

That's why, when Codewars challenged me to write a function to convert roman numerals from/to Arabic numbers, I could not resist writing a regex to help me solve that problem.

Enough chit-chat, let's get our hands dirty.

My first task was to validate if the user input was a valid roman numeral. To sum up, roman numerals consists of the following symbols:Source: Wikipedia

It seems that the thousands unit [M] does not extend past [MMM], which means that the biggest roman number would be [MMMCMXCIX], or 3999. I'm not sure if numbers could go higher than that and why the limit, anyway for the sake of this problem I limited myself with numbers between 1 and 3999.

Now the trick is that the symbols placement is very important. If you don't put them in the right order, the resulting number would be invalid and unreadable. As listed in the table up there, numerals should start with thousands [M] 1000, followed by hundreds [D/C] 500/100, then dozens [L/X] 50/10, and finally units [V/1] 5/1.

BUT, that's not it! Numerals can only repeat 3 times like [CCC] 300 before switching to a combo of two numerals like [CD] 400. So you can still have a [C] 100 before an [M] 1000, like in [CM] 900 for instance.

Bit confusing, isn't it?

Alright, let's recap our conditions:

Roman numbers are ranging from [I] 1 to [MMMCMXCIX] 3999
Numerals should follow a precise order: [M] 1000 / [D] 500 / [C] 100 / [L] 50 / [X] 10 / [V] 5 / [I] 1
A numeral cannot repeat more than 3 times, it then uses a pair
The following pairs are allowed: [CM] 900 / [CD] 400 / [XC] 90 / [XL] 40 / [IX] 9 / [IV] 4

Do you start to see our REGEX showing up? :)

Let's translate this into code.

For this we are going to use a tag I find really helpful when writing regex is the verbose one (re.VERBOSE or re.X) It allows you to spread your pattern on multiple lines and be more readable. Let's try it!



import re

def is_roman_number(num):

    pattern = re.compile(r"""   
                                ^M{0,3}
                                (CM|CD|D?C{0,3})?
                                (XC|XL|L?X{0,3})?
                                (IX|IV|V?I{0,3})?$
            """, re.VERBOSE)

    if re.match(pattern, num):
        return True

    return False

Wow, that looks amazing already! Let's take a closer look at these 4 lines:

^M{0,3} = Between 0 and 3 [M] at the beginning [^] of the string
(CM|CD|D?C{0,3})? = One pair [CM] or one pair [CD] or [D], followed by up to 3 [C]. Each element is optional [?], as well as the whole block [()?]
(XC|XL|L?X{0,3})? = One pair [XC] or one pair [XL] or [L], followed by up to 3 [X]. Each element is optional [?], as well as the whole block [()?]
(IX|IV|V?I{0,3})?$ = One pair [IX] or one pair [IV] or [V], followed by up to 3 [I]. Each element is optional [?], as well as the whole block [()?], which should be at the end of the string [$]

Let's test our code

I'm using a simple fstring calling my function and comparing the string against our pattern to validate the numeral or not:




num_valid = 'MMDCCLXXIII'
num_invalid = 'CCCMMVIIVV'

print(f"{num_valid} is {'not' if not is_roman_number(num_valid) else ''}a roman number")
print(f"{num_invalid} is {'not ' if not is_roman_number(num_invalid) else ''}a roman number")

# Output:
# MMDCCLXXIII is a roman number
# CCCMMVIIVV is not a roman number

That wasn't so bad after all! Now look at this and tell me it's not the most beautiful thing you've seen in your life:



^M{0,3}(CM|CD|D?C{0,3})?(XC|XL|L?X{0,3})?(IX|IV|V?I{0,3})?$'

That's all folks! Let me know if you are interested by the second part of the challenge: converting roman numerals from/to Arabic numbers and I will share my solution.

Stay safe out there and read you soon :)