Django + Celery + CPU/GPU + Tensorflow, Keras + Docker

We held up a situation where we build tensorflow model using a GPU machine and calling the model in the same machine working fine as expected.

We wanted to consume this model in our application which runs under Django framework with the help of celery tasks, while calling the model in the application flow inside celery worker it didn't work or got hang up.

We tried following possible solution but nothing worked.

1. Increased the CPU machine number of cores and RAM size.

2. Created a AWS instance with GPU enabled and tried to consume the model. We haven't added configuration to use the GPU since tensorflow automatically detects GPU and use it for model prediction.

3. Since the Django application and celery running in docker container, we thought docker couldn't able to use the GPU automatically, so we changed our base image to tensorflow CPU/GPU as base docker image and tried to consume the model but failed :(.

4. We created a new API to consume the model but failed to get the output again.

5. We noticed from the beginning the model call is working fine without application flow but via application it's not working. so we tried to call the model through a python script by using multiprocessing and found out that with multiprocessing(since celery does that) it's not working.

The solution:

After we found that with multiprocessing tensorflow model predict was not working, we came through the following post which given a clear picture what's happening inside.

why tensorflow model predict not running under multiprocessing

We added the flow in our requirements.txt

eventlet

And in celery command while building the docker added --pool as below,

celery -A Proj worker -l info -Q Q1,Q2 --pool=eventlet -c 5

By this change we were able to call the model and get the prediction in the application flow without GPU enabled machine!!!

To get the prediction alone from a tensorflow model we don't need a GPU machine.

We spent a quite few days by doing many changes apart from what I mentioned above. Thought to share this so someone's day will be saved..

Happy Coding!!!