Serving PyTorch Models in Production

Sangam SwadiK — Tue, 29 Nov 2022 14:39:06 +0000

Intro to serving models with PyTorch

Machine learning/Deep learning models are rarely deployed across the industry. The polls (Venturebeat, KdNuggets) indicate roughly 80-90 percent of the models developed never make it into production. This could be due to various reasons such as ROI, ineffective leadership, business needs or failure to incorporate MLOps.

As a Data Scientist/ML engineer, one of your responsibilities is to ensure a well designed and functional pipeline. And model deployment is an important part of the pipeline.

This is where the PyTorch ecosystem comes to rescue! PyTorch, TorchServe and many other projects built on PyTorch, can be used from model development until model deployment. There has also been an upward trend in PyTorch due to ease of usage.

Intro to Event

This talk is for a data scientist or ML engineer looking to serve their PyTorch models in production. It will cover post training steps that should be taken to optimize the model such as quantization and TorchScript. It will also walk the user in packaging and serving the model through Facebook's TorchServe.

Video

Resources

Repo: GitHub Repository

Section Timestamps of Video

00:00:00 About session
00:00:47 About Data Umbrella
00:04:18 Introduction
00:05:16 Session agenda
00:06:01 Machine learning at Walmart
00:12:11 Review of some deep learning concepts
00:15:24 BERT: Different architectures
00:16:07 Bi-LSTM vs BERT
00:21:59 Model inference
00:24:21 Load the model
00:25:21 Test prediction
00:28:01 Inference review(inference time vs accuracy tradeoff)
00:29:17 BERT large
00:30:03 Distilled-BERT
00:33:54 Optimizing model for production
00:34:03 Post training optimization: Quantization
00:35:50 Types of Quantization
00:37:35 Quantization results
00:38:23 Post training optimization: Distillation
00:39:44 Distillation results
00:40:35 Eager execution vs Script mode
00:42:02 TorchScript JIT: Tracing vs Scripting
00:43:11 TorchScript Timing
00:45:21 Optimizing the model(Hands On)
00:47:36 Quantization(Hands On)
00:52:00 TorchScript(Hands On)
00:56:33 Deploying the model
00:57:13 Options for deploying Pytorch model
00:57:42 Benefits of TorchServe
00:59:41 Packaging a model/MAR
01:00:00 Pytorch BaseHandler
01:03:00 Built in handlers
01:04:15 Serving
01:05:10 APIs
01:05:32 Deploying the Model(Hands On)
01:22:11 Lessons Learned
01:23:50 Q/A

About the Speakers

Bio

Nidhin Pattaniyil is a Machine Learning Engineer in Walmart Search.

Connect with the Speaker

Nidhin's LinkedIn: Nidhin Pattaniyil
Nidhin's GitHub: @npatta01

Key Links

Tutorial on Intro to Rust Programming

Sangam SwadiK — Fri, 25 Nov 2022 18:04:55 +0000

Summary posted by: Sangam SwadiK

Intro to Rust

The Data Science community has greatly benefitted from the python ecosystem(scipy, numpy, scikit-learn etc.) which has tools at every step of the Data Science workflow. Under the hood most of these python libraries use C/C++ which has bindings with python to improve performance. Although this has reduced developmental time, it is not without the reduction in performance as python is not known for its performance.

This is where Rust comes in! It provides performance similar to C/C++ but
better memory use and concurrency. It can be used with python just as C/C++ but with improved speed and safety.

There has been an upward trend in opensource tools written in Rust with interfaces to python eg: pydantic (moved to Rust in the recent release), polars which is very fast as indicated in the H2Oai benchmarks.

Into to event

An introduction to Rust Programming for complete beginners. This 2-hour video tutorial covers:

what is Rust
how to install Rust
how to use Rust online
basic structure of a Rust program
core concepts of Rust
a little CLI (command line interface) program
Q&A

Video

Resources

Event outline and notes
GitHub repo: rust-intro
Data science libraries in Rust: https://www.arewelearningyet.com/
Rust book: https://www.rust-lang.org/learn
micro (terminal editor): https://micro-editor.github.io/

Section Timestamps of Video

00:00:00 Beryl introduces Data Umbrella
00:05:35 Bruno begins presentation
00:13:00 Hello World
00:18:10 Cargo and Projects
00:24:50 Variable Definition
00:36:00 Type Inference
00:41:12 Mutability
00:46:56 Strong Typing
00:49:00 Variable Shadowing
00:53:30 Constants
00:58:21 Data Types
01:14:00 Static, Stack & Heap
01:23:58 Function
01:26:02 Docs
01:28:43 Ownership
01:34:36 Borrowing
01:37:37 Console Input
01:40:18 DEMO ERROR ignore this part :)
01:43:02 Read user input
01:45:22 Combinators
01:46:01 Result type
01:50:00 External Dependency
01:52:07 Random numbers
01:54:38 Type coercion
01:59:09 Complete program
02:01:47 Q & A

About the Speaker

Bio

Bruno is a Software Engineer who works developing the Red Hat Ansible Platform, he is member of the Python Software Foundation, creator of Dynaconf settings manager and a Rust enthisiast for 5+ years.

Connect with the Speaker, Bruno Rocha

YouTube: @rochacbruno
LinkedIn: Bruno Rocha
Twitter: @rochacbruno
GitHub: @rochacbruno

DEV Community: Data Umbrella

Serving PyTorch Models in Production

Intro to serving models with PyTorch

Intro to Event

Video

Resources

Section Timestamps of Video

About the Speakers

Bio

Connect with the Speaker

Key Links

Tutorial on Intro to Rust Programming

Intro to Rust

Into to event

Video

Resources

Section Timestamps of Video

About the Speaker

Bio

Connect with the Speaker, Bruno Rocha