What is machine learning?
Machine learning is the practice of training a piece of software, called a model, to make predictions or generate content from data. In traditional programming you write the rules yourself. With ML you show the software a large number of examples and it works out the rules on its own.
Delivery time estimation is a simple illustration. The traditional route is to write the rules yourself, something like 15 minutes of base time plus 4 minutes per kilometer, which falls apart the moment traffic, rain, or rush hour gets involved. The ML route is to feed a model months of past deliveries until it learns the relationship between distance, time of day, and how long the trip actually took. After that, you give it a new order and it predicts the delivery time.
A few terms will show up everywhere from here on. The inputs a model learns from are called features, such as distance, hour of day, and day of week. When the correct answer is known, it’s called the label, such as the number of minutes the delivery actually took. Training is the process of fitting the model to historical data, and a prediction is what the trained model produces for new data.
The Four Types of ML
ML systems are grouped by how they learn. Each type asks for different data, which is why it matters to know which one you’re dealing with.
Supervised Learning
Supervised learning is where most business ML lives. The model trains on data where the correct answer is already attached, the way a student prepares by working through old exams that include the answer key. After enough practice, the student can take a new exam alone. The system is “supervised” because a human supplied the known correct results.
It comes in two flavors. Regression predicts a numeric value. The delivery estimate from earlier is a regression problem, and so is predicting what a house will sell for or how much electricity a commercial building will use next month. Classification predicts which category something belongs to, such as spam or not spam, fraud or legitimate, churned or retained. Binary classification has two possible outputs, while multiclass classification has more, like a model that sorts support tickets into billing, shipping, returns, or other.
Unsupervised Learning
Unsupervised learning gets no answer key. You hand the model a dataset and it looks for meaningful patterns by itself. The most common technique is clustering, which groups similar rows together.
Clustering differs from classification because you don’t define the categories in advance. Run it over a year of customer behavior and you might find weekly shoppers, quarterly bulk buyers, and people who signed up once and never returned. The model produces the groups. Naming them is a human job afterward.
Reinforcement Learning
Reinforcement learning works through rewards and penalties. A model takes actions inside an environment, gets feedback on how each action turned out, and gradually builds a policy, which is a strategy for collecting the most reward over time. It’s how robots learn to walk around a room and how AlphaGo learned to play Go.
It needs an environment to act in and a clear reward signal, which is why you’ll read about it far more often than you’ll build it in your first couple of years. Knowing it exists and what makes it different is enough for now.
Generative AI
Generative AI is a class of models that create new content from an input. That content can be text, images, audio, video, or code, and many models handle combinations. People describe them by what goes in and what comes out: text-to-text for a chatbot, text-to-image for an image generator, text-to-code for a coding assistant, image-to-text for a model that describes a photo.
At a high level, these models learn the patterns in huge amounts of data so they can produce new material that resembles it. They’re usually trained first by learning to mimic unlabeled data, which is an unsupervised approach, and then refined with supervised or reinforcement learning for specific jobs like summarizing an article or editing a photo.
For a data engineer, the work around these models mostly involves collecting, cleaning, and refreshing the documents and text they draw from.
Which One Do You Need?
Most of the time, the data you already have answers this for you. Start with one question: do you have historical examples with the correct answer attached?
Run your own problems through it. A delivery time estimate has past deliveries with known durations and the answer is a number, so it’s regression. A churn flag has a known outcome for past customers and the answer is yes or no, so it’s classification. Grouping customers by behavior has no answer to learn from, so it’s clustering.
The last two branches work differently because the task decides them, not the data you have. If the job is producing text, images, or code, you’re in generative AI territory. If a system has to learn by acting and collecting rewards, that’s reinforcement learning. In your first couple of years as a data engineer, you’ll spend most of your time with the first three, and supervised learning will account for the bulk of it.
One last check before reaching for any of them: if you can’t say what the correct answer is, or when it becomes known, you aren’t ready to train a model yet.
Top comments (0)