DEV Community: Stephen Portanova

62 Best Uber Machine Learning Blog Posts

Stephen Portanova — Thu, 05 Dec 2019 23:29:49 +0000

Here are 62 engineering blog posts on how Uber uses machine learning to route drivers and pick up millions of passengers:

Meet Michelangelo: Uber’s Machine Learning Platform
Uber Engineering introduces Michelangelo, their machine learning-as-a-service system that enables teams to easily build, deploy, and operate ML solutions at scale.
Scaling Machine Learning at Uber with Michelangelo
Uber built Michelangelo, their machine learning platform, in 2015. Three years later, they reflect their journey to scaling ML at Uber and lessons learned along the way.
Engineering Uncertainty Estimation in Neural Networks for Time Series Prediction at Uber
Uber Engineering introduces a new Bayesian neural network architecture that more accurately forecasts time series predictions and uncertainty estimations.
Evolving Michelangelo Model Representation for Flexibility at Scale
To accommodate additional ML use cases, Uber evolved Michelangelo's application of the Apache Spark MLlib library for greater flexibility and extensibility.
Michelangelo PyML: Introducing Uber’s Platform for Rapid Python ML Model Development
Uber developed Michelangelo PyML to run identical copies of machine learning models locally in both real time experiments and large-scale offline prediction jobs.
Searchable Ground Truth: Querying Uncommon Scenarios in Self-Driving Car Development
When developing Uber's self driving car systems, engineers found a way to identify edge case scenarios amongst terabytes of sensor data representing real-world situations.
Science at Uber: Improving Transportation with Artificial Intelligence
Uber Chief Scientist Zoubin Ghahramani explains how artificial intelligence went from academia to real-world applications, and how Uber uses it to make transportation better.
Three Approaches to Scaling Machine Learning with Uber Seattle Engineering
At an April 2019 meetup on ML and AI at Uber Seattle, members of their engineering team discussed three different approaches to enhancing their ML ecosystem.
Science at Uber: Powering Machine Learning at Uber
Logan Jeya, Product Manager, explains how Uber's machine learning platform, Michelangelo, makes it easy to deploy models that enable data-driven decision making.
Introducing LCA: Loss Change Allocation for Neural Network Training
Uber AI Labs proposes Loss Change Allocation (LCA), a new method that provides a rich window into the neural network training process.
Using GraphQL to Improve Data Hydration in their Customer Care Platform and Beyond
Uber Engineering details how GraphQL integrated into their Customer Care platform, making for more targeted queries and reducing server load.
Science at Uber: Making a Real-world Impact with Data Science
Suzette Puente, Uber Data Science Manager, shares how she applies her graduate work in statistics to forecast traffic patterns and generate better routes.
Science at Uber: Applying Artificial Intelligence at Uber
Zoubin Ghahramani, Head of Uber AI, discusses how they use artificial intelligence techniques to make their platform more efficient for users.
Science at Uber: Powering Uber’s Ridesharing Technologies Through Mapping
Dawn Woodard, Director of Data Science, considers travel time prediction one of Uber's most interesting mapping problems.
Science at Uber: Bringing Research to the Roads
Uber Principal Engineer Waleed Kadous discusses how they assess technologies their teams can leverage to improve the reliability and performance of their platform.
Science at Uber: Building a Data Science Platform at Uber
Uber Director of Data Science Franziska Bell discusses how they created data science platforms at Uber, letting employees of all technical skills perform forecasts and analyze data.
Making Apache Spark Effortless for All of Uber
Uber engineers created uSCS, a Spark-as-a-Service solution that helps manage Apache Spark jobs throughout large organizations.
Gaining Insights in a Simulated Marketplace with Machine Learning at Uber
Uber's Marketplace simulation platform leverages ML to rapidly prototype and test new product features and hypotheses in a risk-free environment.
No Coding Required: Training Models with Ludwig, Uber’s Open Source Deep Learning Toolbox
Uber AI's Piero Molino discusses Ludwig's origin story, common use cases, and how others can get started with this powerful deep learning framework built on top of TensorFlow.
Using Causal Inference to Improve the Uber User Experience
Uber Labs leverages causal inference, a statistical method for better understanding the cause of experiment results, to improve their products and operations analysis.
Power On: Accelerating Uber’s Self-Driving Vehicle Development with Data
A key challenge faced by self-driving vehicles comes during interactions with pedestrians. In their development of self-driving vehicles, the Data Engineering and Data Science teams at Uber ATG (Advanced Technologies Group) contribute to the data processing and analysis that help make these interactions safe.
Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask
Uber builds upon the Lottery Ticket Hypothesis by proposing explanations behind these mechanisms and deriving a surprising by-product: the Supermask.
Improving Uber’s Mapping Accuracy with CatchME
CatchMapError (CatchMe) is a system that automatically catches errors in Uber's map data with anonymized GPS traces from the driver app.
Solving Big Data Challenges with Data Science at Uber
How engineers and data scientists at Uber came together to come up with a means of partially replicating Vertica clusters to better scale their data volume.
Accessible Machine Learning through Data Workflow Management
Uber engineers offer two common use cases showing how they orchestrate machine learning model training in their data workflow engine.
Using Machine Learning to Ensure the Capacity Safety of Individual Microservices
Uber leveraged machine learning to design their capacity safety forecasting tooling with a special emphasis on calculating a quality of reliability score.
Horovod Adds Support for PySpark and Apache MXNet and Additional Features for Faster Training
Horovod adds support for more frameworks in the latest release and introduces new features to improve versatility and productivity.
Modeling Censored Time-to-Event Data Using Pyro, an Open Source Probabilistic Programming Language
Censored time-to-event data is critical to the proper modeling and understanding of customer engagement on the Uber platform. In this article, they demonstrate an easier way to model this data using Pyro.
Introducing Ludwig, a Code-Free Deep Learning Toolbox
Uber AI developed Ludwig, a code-free deep learning toolbox, to make deep learning more accessible to non-experts and enable faster model iteration cycles.
Why Financial Planning is Exciting… At Least for a Data Scientist
In this article, Uber’s Marianne Borzic Ducournau discusses why financial planning at Uber presents unique and challenging opportunities for data scientists.
How Uber Leverages Applied Behavioral Science at Scale
Uber Labs utilizes insights and methodologies from behavioral science to build programs and products that are intuitive and enjoyable for users on their platform.
Manifold: A Model-Agnostic Visual Debugging Tool for Machine Learning at Uber
Uber built Manifold, a model-agnostic visualization tool for ML performance diagnosis and model debugging, to facilitate a more informed and actionable model iteration process.
POET: Endlessly Generating Increasingly Complex and Diverse Learning Environments and their Solutions through the Paired Open-Ended Trailblazer
Uber AI Labs introduces the Paired Open-Ended Trailblazer (POET), an algorithm that leverages open-endedness to push the bounds of machine learning.
Horovod Joins the LF Deep Learning Foundation as its Newest Project
Horovod, Uber's distributed training framework, joins the LF Deep Learning Foundation to help advance open source innovation in AI, ML, and deep learning.
Faster Neural Networks Straight from JPEG
Uber AI Labs introduces a method for making neural networks that process images faster and more accurately by leveraging JPEG representations.
NVIDIA: Accelerating Deep Learning with Uber’s Horovod
Horovod, Uber's open source distributed deep learning system, enables NVIDIA to scale model training from one to eight GPUs for their self-driving sensing and perception technologies.
Applying Customer Feedback: How NLP & Deep Learning Improve Uber’s Maps
To improve their maps, Uber Engineering analyzes customer support tickets with natural language processing and deep learning to identify and correct inaccurate map data.
Introducing Petastorm: Uber ATG’s Data Access Library for Deep Learning
Uber's Advanced Technologies Group introduces Petastorm, an open source data access library enabling training and evaluation of deep learning models directly from multi-terabyte datasets in Apache Parquet format.
Scaling Uber’s Customer Support Ticket Assistant (COTA) System with Deep Learning
Uber built the next generation of COTA by leveraging deep learning models, thereby scaling the system to provide more accurate customer support ticket predictions.
Forecasting at Uber: An Introduction
In this article, they provide a general overview of how their teams leverage forecasting to build better products and maintain the health of the Uber marketplace.
An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution
As powerful and widespread as convolutional neural networks are in deep learning, AI Labs’ latest research reveals both an underappreciated failing and a simple fix.
Transforming Financial Forecasting with Data Science and Machine Learning at Uber
Uber developed its own financial planning software, relying on data science and machine learning, to deliver on-demand forecasting and optimize strategic and operations decisions.
How Trip Inferences and Machine Learning Optimize Delivery Times on Uber Eats
Using GPS and sensor data from Android phones, Uber engineers develop a state model for trips taken by Uber Eats delivery-partners, helping to optimize trip timing for delivery-partners and eaters alike.
Advanced Technologies for Detecting and Preventing Fraud at Uber
To detect and prevent fraud, Uber brings to bear data science and machine learning, analyzing GPS traces and usage patterns to identify suspicious behavior.
Food Discovery with Uber Eats: Building a Query Understanding Engine
Uber engineers share how they process search terms for their Uber Eats service, using query understanding and expansion to find restaurants and menu items that best match what their eaters want.
From Beautiful Maps to Actionable Insights: Introducing kepler.gl, Uber’s Open Source Geospatial Toolbox
Created by Uber's Visualization team, kepler.gl is an open source data agnostic, high-performance web-based application for large-scale geospatial visualizations.
Engineering a Job-based Forecasting Workflow for Observability Anomaly Detection
Uber’s Observability Applications team overhauled their anomaly detection platform’s workflow to enable the intuitive and performant backfilling of forecasts, paving the way for more intelligent alerting.
Differentiable Plasticity: A New Method for Learning to Learn
Artificial intelligence researchers develop new method to let neural networks continue to learn, even after initial training.
COTA: Improving Uber Customer Care with NLP & Machine Learning
In this article, Uber Engineering introduces their Customer Obsession Ticket Assistant (COTA), a new tool that puts machine learning and natural language processing models in the service of customer care to help agents deliver improved support experiences.
Welcoming the Era of Deep Neuroevolution
By leveraging neuroevolution to train deep neural networks, Uber AI Labs is developing solutions to solve reinforcement learning problems.
Gleaning Insights from Uber’s Partner Activity Matrix with Genomic Biclustering and Machine Learning
Uber Engineering's partner activity matrix leverages biclustering and machine learning to better understand the diversity of user experiences on their driver app.
Engineering More Reliable Transportation with Machine Learning and AI at Uber
In this article, they highlight how Uber leverages machine learning and artificial intelligence to tackle engineering challenges at scale.
Uber AI Labs Open Sources Pyro, a Deep Probabilistic Programming Language
Pyro is an open source probabilistic programming language that unites modern deep learning with Bayesian modeling for a tool-first approach to AI.
Turbocharging Analytics at Uber with their Data Science Workbench
Uber Engineering's data science workbench (DSW) is an all-in-one toolbox that leverages aggregate data for interactive analytics and machine learning.
Meet Horovod: Uber’s Open Source Distributed Deep Learning Framework for TensorFlow
Uber Engineering introduces Horovod, an open source framework that makes it faster and easier to train deep learning models with TensorFlow.
Engineering Uber Predictions in Real Time with ELK
Uber Engineering architected a real-time trip features prediction system using an open source RESTful search engine built with Elasticsearch, Logstash, and Kibana (ELK).
Detecting Abuse at Scale: Locality Sensitive Hashing at Uber Engineering
In this article, they discuss how Uber Engineering uses Locality Sensitive Hashing on Apache Spark to reliably detect fraudulent trips at scale.
Building an Intelligent Experimentation Platform with Uber Engineering
Composed of a staged rollout and intelligent analytics tool, Uber Engineering's experimentation platform is capable of stably deploying new features at scale across their apps. In this article, they discuss the challenges and opportunities they faced when building this product.
Hudi: Uber Engineering’s Incremental Processing Framework on Apache Hadoop
Uber Engineering's data processing platform team recently built and open sourced Hudi, an incremental processing framework that supports their business critical data pipelines. In this article, they see how Hudi powers a rich data ecosystem where external sources can be ingested into Hadoop in near real-time.
Engineering Uber Systems to Combat Fraud
Fraud prevention is one of Uber's fastest growing areas of research and development. As their platform has grown, so has the international underworld that tries to undermine it. Here’s how Uber engineers systems to fight fraud in 2016 and beyond.
How Uber Engineering Increases Safe Driving with Telematics
The engineering behind how Uber's Driving Safety team is using telematics to raise awareness of driving patterns to their partners.
Engineer Q&A: Doing Data Science at Uber Engineering
This week, Emi Wang dishes out data knowledge on what she’s been up to at Uber since she joined in September 2012.

This post first appeared on the FullJoin blog at https://fulljoin.io/blog/uber-machine-learning

75 of The Best Software Engineering Blogs

Stephen Portanova — Wed, 20 Nov 2019 18:38:09 +0000

Here are a bunch of my favorite company engineering blogs, with “tags” on what they’re especially good at. In no particular order:

Spotify: profiles of team members, big data, scalability
Artsy: team culture, in-house tech
MemSQL: database design
NextRoll: diversity, erlang, adtech, big data tech
Uber: scalability, in-house tech, machine-learning
The New York Times: journalism, in-house tech, team culture
Airbnb: machine learning, scalability, in-house tech
Booking.com: machine learning, in-house tech
Gojek: in-house tech
Github: open source, in-house tech, Ruby on Rails,
Netflix: scalability, in-house tech
Skyscanner: in-house tech, team culture
Shopify: scalability, team culture, Ruby on Rails
Grofers: team culture, in-house tech
Twitter: in-house tech, scalability
Azavea: machine learning, geo spatial data
Coinbase: in-house tech, team culture
Findmypast: in-house tech, elixir
Tinder: in-house tech
Instagram: Python, in-house tech
Pinterest: scalability, in-house tech, big data
Feedzai: team culture
commercetools: team culture, in-house tech
Discord: scalability
SoundCloud: in-house tech
Dropbox: scalability, in-house tech
Ebay: in-house tech
Etsy: team culture, in-house tech
Benchling: bio tech, in-house tech
Instacart: in-house tech
Hootsuite: Scala
Adidas: in-house tech, team culture
Showmax: in-house tech
Giphy: in-house tech
Jane Street: ocaml, machine learning
Eventbrite: team culture
Yelp: in-house tech, diversity
Walmart Labs: in-house tech
Postman: APIs
Entelo: team culture
FINN: accessibility
Cockroach Labs: database design, team culture
Gusto: team-culture
Google: machine learning
Slack: in-house tech
Babbel: junior engineers, diversity, in-house tech
Indeed: machine learning
Just Eat: iOS
BuzzFeed: team culture, in-house tech
Haptik: team culture, machine learning, Python
Lyft: machine learning
OLX: team-culture, in-house tech
TransferWise: team culture, in-house tech
TrueCar: in-house tech
Zendesk: in-house-tech
Target: team culture
Wayfair: machine-learning
BlaBlaCar: in-house tech
Allegro: performance, in-house tech
Deliveroo: in-house tech
Vena: team-culture, in-house tech
Jet: F#
Stripe: in-house tech
LinkedIn: big data, in-house tech, kafka
Facebook: open source, machine learning
BigCommerce: team culture, in-house tech
Grab: machine learning, in-house tech
Asana: diversity, profiles of team members, team culture
Intercom: team culture, in-house tech
XING: team culture
Trivago: team culture, in-house tech
Riot Games: in-house tech
BBC: in-house tech
WePay: in-house tech, big data
Cloudera: open source, machine learning

The Many Faces of Scaling

Stephen Portanova — Thu, 07 Nov 2019 22:33:10 +0000

You can scale things other than just servers. I used my engineering blog search engine FullJoin.io to find blog posts on how companies scaled their processes, data centers, and yes - servers.

1. Airbnb - Used caching to buy time to fix their architecture issues
Key take away: If you hit a wall with the load that your databases can handle, lots of caching can give you breathing room until you can rearchitect your app for increased traffic.

2. The New York Times - Tips for scaling an engineering team
Interesting tidbit: They used pair programming as a way of transmitting the cultural values of their team to new hires.

3. Coinbase - Scaling the deployment process as they added more services and engineers
Interesting tidbit: They went from deploying 128 times per year to 580. Investing in the deployment process made it safer and less scary to do for new engineers.

4. Facebook - Created a second data center on the east coast to lower latency
Key takeway: When you're at Facebook scale, you can't just spin up a few extra AWS servers. It's more cost effective to create your own data center.

5. Dropbox - Migrating from S3 to their own exabyte sized data center (series)
Key takeway: Similar to Facebook, they had the scale to need their own datacenter, but they continued using AWS where it made sense.

6. Jet - Different ways of scaling microservices
Key takeway: There are different ways at varying levels of the stack to tune microservice performance.

7. Intercom - Interview with VP of engineering on scaling the engineering team
Interesting tidbit: Intercom hires for potential, since they're growing so fast and they need to hire people who can adapt to future challenges.

8. Cockroach Labs - How the hiring process scaled the company from 56 to 117
Interesting tidbit: To avoid being biased by someone's background, the engineers at Cockroach Labs don't look at resumes.

9. LinkedIn - Sending 2.75 gigabytes of data over Kafka
Interesting tidbit: Having core Kafka committers on the team has let LinkedIn push their Kafka infrastructure to the limit, while stilling keeping things maintainable.

10. Pinterest - Autoscaling servers during peak hours
Key takeaway: Sometimes something from your cloud platform doesn't work exactly how you need it to, so you need to supplement it with something built in-house.

11. Slack - Adding Kafka to their Redis job queue system to prevent downtime
Key takeaway: Sometimes when you hit scaling problems, rather than rearchitect your whole system, you can add a component that stabilizes the system.