At the start of February 2019, Uber made their code-free machine learning toolbox, Ludwig, open-source.
Ludwig runs on top of the popular and powerful TensorFlow library and offers CLI access to experiment and train machine learning models and predict using TensorFlow models
As an engineer, I'm absolutely not a data scientist. I know enough around TensorFlow to build the most basic of models using tutorials but really couldn't create anything from scratch. Ludwig offered that opportunity.
Let's dive in and run through a basic example. We're going to try to recreate the Keras tutorial at https://www.tensorflow.org/tutorials/keras/basic_regression with zero lines of code.
The dataset shows basic data to cars in the Auto MPG dataset. Our task is to predict the MPG from the features provided. I've grabbed this and converted it to a CSV file for use in this example.
Ludwig uses a model definition file to determine the parameters for building the model. The internals of Ludwig deals with your data. It creates train, test and validation datasets. It also modifies the data into the best format for training depending on the data type you've specified.
The Keras example needs us to manipulate the data in order to train and test the model. Ludwig does all of this for us. It allows us to train the model immediately by setting up a model definition file at
modeldef.yaml. Here we define the input features and their data type. There are a number of other parameters against each feature which can be set for more complex models. We also define the output feature and its parameters.
input_features: - name: Cylinders type: numerical - name: Displacement type: numerical - name: Horsepower type: numerical - name: Weight type: numerical - name: Acceleration type: numerical - name: ModelYear type: numerical - name: Origin type: category output_features: - name: MPG type: numerical
Our first experiment can now be run with the following command:
ludwig experiment --data_csv cars.csv --model_definition_file modeldef.yaml --output_directory results
This gives the following results:
===== MPG ===== loss: 52.9658573971519 mean_absolute_error: 6.3724554520619066 mean_squared_error: 52.9658573971519 r2: 9.58827477467211e-05
After 200 epochs complete, I have a mean absolute error (MAE) of 6.4 (yours may vary slightly depending on the random train/test split). This means that on average MPG prediction on a car is 6.4MPG from the actual value. Bearing in mind that values are generally between 10MPG and 47MPG, that 6.4MPG represents quite a large error.
If you were watching the log scrolling as Ludwig was running, you'd have seen the MAE against the validation set reducing with each epoch.
The Keras example was suggesting a final MAE of ~2 so we may need a bit of tweaking to get closer. There was fair indication that the MAE was still decreasing as the run ended. We can increase the amount of epochs with a simple addition to the addition to the model definition
training: epochs: 400
and continue from the previous training model with the command
ludwig experiment --data_csv cars.csv --model_definition_file modeldef.yaml --output_directory results -mrp ./results/experiment_run_0
Our MAE only comes down to 5.3MPG. Still not that close.
In a real life example, we'd start amending the hyperparameters, retraining, amending and retraining again while our target MAE still reduces.
We'll skip this step by replicating the hyperparameters from the Keras tutorial:
training: batch_size: 32 epochs: 400 early_stop: 50 learning_rate: 0.001 optimizer: type: rmsprop
In addition, we set early stop at 50 epochs - this means that our model will stop training if our validation curve doesn't improve for 50 epochs. The experiment is fired off in the same way as before. It produces these results:
Last improvement of loss on combined happened 50 epochs ago EARLY STOPPING due to lack of validation improvement, it has been 50 epochs since last validation accuracy improvement Best validation model epoch: 67 loss: 10.848812248133406 mean_absolute_error: 2.3642308198952975 mean_squared_error: 10.848812248133406 r2: 0.026479910446118703
We get a message that our model has stopped training at 132 epochs because it's hit the early stop limit.
MAE is down to 2.36MPG without writing a line of code and we've got our example to similar results to the Keras tutorial.
Now we'd like to validate that our test and validation loss curves are getting pretty close but not showing overfitting. Ludwig continues to deliver on its promise of a no-code solution. We can view our learning curves with the following command:
ludwig visualize -v learning_curves -ts results/experiment_run_0/training_statistics.json
The curves remain following a similar trajectory. Should the validation curve start heading upwards while the training curve remain on this trajectory, it would suggest that overfitting is occurring.
Ok, this is all well and good but tutorials notoriously pick and choose data so the output "just works". Let's try our model out with some real data.
With a bit of investigation, I've dug out the required stats of the DeLorean DMC-12 (https://en.wikipedia.org/wiki/DMC_DeLorean):
Cylinders: 6 Displacement: 2849cc (174 cubic inches) Horsepower: 130hp Weight: 1230 kg (2712 lb) Acceleration: 10.5s Year: 1981 Origin: US
and converted it to the same CSV format as the training data:
Now, to predict the fuel economy of this, we run the
predict command through Ludwig:
ludwig predict --data_csv delorean.csv -m results/experiment_run_0/model -op
We specify the
-op flag to tell Ludwig that we only want predictions. Inputting a CSV file with MPG column and not adding this flag will run the predictions but also provide us with statistics against actual values supplied in the file.
The result given by my model is 23.53405mpg. How good is this? Unfortunately our Wikipedia article doesn't show the published fuel economy but I did manage to find it in this fantastic article about the amazing car - 22.8mpg. A pretty decent real life test!
I appreciate that the data scientists out there are screaming that we didn't run through any analysis on the input features to create a meaningful feature set and that we didn't run specific analysis on the test data predictions. I also appreciate that MAE isn't necessarily the ultimate measure of accuracy as it may be skewed heavily by outliers which we could have validated through further analysis.
What we have shown is that using Ludwig, we can experiment and train a machine learning model and then predict using the model we've trained.
Machine learning is becoming more and more accessible. Ludwig seems to be big step forward in that regard.