DEV Community: deeplearning

Gemini 4 Argon: Google's AI Problem Solver

Gian Paolo — Wed, 14 Oct 2026 07:08:02 +0000

The AI Whisperer: When a complex problem feels like an impossible knot, we often wish for a genius to untangle it. What if that genius isn't human? I'm talking about Google's latest brain child, Gemini 4 Argon, an AI model that's not just smart, it's designed to tackle the kind of multi-layered, ambiguous challenges that make us mere mortals sweat. Think less about generating pretty pictures and more about navigating a labyrinth of data to find the single correct path. (Ref: TuttoAndroid)

When a complex problem feels like an impossible knot, we often wish for a genius to untangle it. What if that genius isn't human? I'm talking about Google's latest brain child, Gemini 4 Argon, an AI model that's not just smart, it's designed to tackle the kind of multi-layered, ambiguous challenges that make us mere mortals sweat. Think less about generating pretty pictures and more about navigating a labyrinth of data to find the single correct path.

Imagine you're tasked with optimizing a city-wide supply chain. Hundreds of variables are in play: traffic patterns, warehouse inventory, fuel costs, delivery windows, and unexpected road closures. Each decision creates a ripple effect, and the ideal solution is buried under an avalanche of interconnected data. This is the exact kind of cognitive puzzle that has kept logisticians awake at night for decades. It’s also precisely the type of challenge Google has built its new AI model, Gemini 4 Argon, to solve.

While much of the recent AI conversation has centered on creativity and conversation—models that can write a sonnet or debate philosophy—Argon represents a significant pivot. It’s an AI built for execution. According to recent announcements, its core strength lies in deconstructing a massive, poorly defined problem into a series of smaller, logical steps that it can then solve sequentially Google lancia Gemini 4 Argon, il nuovo modello IA che risolve problemi complessi - TuttoAndroid. It acts less like a creative partner and more like a tireless strategic analyst.

Think of it this way: if you ask a standard AI to plan a vacation, it might give you a beautiful itinerary. If you give the same task to Argon, it would likely start by asking for your budget, checking flight prices in real-time, cross-referencing hotel availability with your preferred dates, analyzing reviews for the best local restaurants, and then presenting a logically optimized plan complete with booking links. The focus isn't on the prose, but on the process.

This ability to operate with a higher degree of autonomy is what sets it apart. Instead of requiring constant human prompting for each micro-task, Argon is designed to understand the overarching goal and chart its own course through the data labyrinth. It can identify the necessary sub-tasks, execute them, and use the results to inform its next move. This makes it a powerful tool for fields drowning in complexity, from scientific research and financial modeling to software development and engineering.

So while other AIs are learning to paint, Argon is learning to think strategically. It isn't here to replace human ingenuity but to augment it, offering a powerful cognitive tool to untangle the gnarliest knots we can find. It’s a quiet but profound shift, suggesting a future where our most daunting problems have a new kind of mind working on them—one that never gets tired, never overlooks a variable, and is built from the ground up to find the way through.

Beyond the Buzzwords: Deconstructing Argon's 'Complex Problem Solving' Superpower. This isn't just about faster calculations or bigger datasets. Argon reportedly brings a new level of autonomy and multi-modal reasoning to the table. We’ll dive into what Google means by 'complex problems' – from scientific discovery and intricate logistical planning to perhaps even ethical dilemmas in AI development itself. How does its architecture enable this leap, and what specific capabilities differentiate it from previous Gemini iterations? (Ref: HDblog.it)

When Google labels its new model, Gemini 4 Argon, a "complex problem solver," it’s easy to picture a supercharged calculator running equations faster than ever. But that misses the point entirely. The shift Argon represents is not about the speed of calculation but the nature of the inquiry itself. This is about endowing an AI with the ability to take a vague, high-level goal and independently chart a course to a solution.

Argon's defining feature appears to be a new level of operational autonomy. It's a move away from the simple prompt-and-response loop that has defined large language models. As noted in early reports, the goal is an AI that becomes more autonomous, capable of breaking down a multifaceted problem into a series of logical steps, gathering the necessary information, and executing those steps without constant human intervention [Google lancia Gemini 4 Argon: l'AI diventa più autonoma - HDblog.it].

This capability is reportedly enabled by a more integrated, multi-modal reasoning architecture. Think of it less as a single, massive brain and more as an expert project manager with a team of specialists. When faced with a problem, Argon can orchestrate different internal "agents"—one that excels at code generation, another at interpreting satellite imagery, a third at parsing financial reports. This allows it to tackle problems that are not just computationally difficult, but are also messy, ambiguous, and span multiple domains.

Let's take a concrete example: optimizing a global logistics network. A previous Gemini iteration could brilliantly analyze a spreadsheet of shipping times and fuel costs to identify inefficiencies. You would have to provide it with clean data and ask a specific question. Argon, in contrast, could be given a far broader directive: "Reduce our international shipping carbon footprint by 20% within the next year without increasing costs by more than 5%."

To solve this, Argon wouldn't just look at a spreadsheet. It would formulate a strategy. It might pull real-time maritime traffic data, cross-reference it with long-range weather forecasts to predict storm-related delays, analyze international fuel price fluctuations, and even interpret satellite photos of port congestion. It could then generate multiple simulated scenarios—re-routing ships, consolidating cargo, shifting to different transport methods—and present a fully-fledged plan, complete with the Python code needed to implement the new routing software.

This is the leap. The complexity isn't just in the data; it's in the real-world variables and the need for a coherent, multi-step plan. Google’s ambition for Argon seems to extend even further, into abstract and ethical domains. Could it be tasked with modeling the potential societal impacts of a new AI-powered public policy, identifying potential biases or unforeseen consequences before they are implemented? This suggests a capacity for meta-reasoning that goes far beyond simple pattern recognition.

The critical differentiator, then, is the shift from reactive analysis to proactive strategy. Previous models could tell you what happened or predict what might happen based on past data. Argon is being built to chart a course for what to do next.

The AI Arena: Argon vs. The Titans. We've seen formidable players like ChatGPT Astra and Claude Fable push boundaries, but where does Argon fit in the competitive landscape? I'll break down the key technical and functional differences. Is Argon truly a step ahead in terms of reasoning, planning, or its ability to handle unstructured, real-world data? We'll assess specific use cases where Argon might outshine its rivals and where it might face similar limitations. (Ref: Corriere della Sera)

The artificial intelligence landscape has become a clash of titans, and with Google’s release of Gemini 4 Argon, a new contender has stepped into the ring. We’ve grown accustomed to the impressive conversational and creative abilities of models like OpenAI’s ChatGPT Astra and Anthropic’s Claude Fable. They can write poetry, summarize dense reports, and, in Astra's case, even narrate the world through a phone's camera. But Argon is playing a different game. It’s less about conversation and more about conclusion.

Google is positioning Argon not as a better chatbot, but as a more autonomous problem-solver. The primary distinction lies in its architecture, which is geared toward reasoning and planning. While other leading models respond brilliantly to direct prompts, Argon is designed to take a complex, multi-step goal and independently map out a path to achieve it. It deconstructs a problem into a sequence of logical tasks, a skill that moves it closer to an "agent" that does things, rather than an "oracle" that knows things.

This becomes clear when you consider its ability to process unstructured, real-world data. Imagine you need to organize a corporate retreat. You could give ChatGPT a list of requirements and ask it to draft an itinerary. It would do a fine job. With Argon, the promise is that you can hand it a messy folder containing a mix of PDF quotes from caterers, a spreadsheet of attendee flight times, and a jumble of email threads discussing potential venues. Argon’s objective isn't just to summarize this data, but to synthesize it into a coherent plan. It’s meant to identify conflicts—like a keynote speaker landing after the welcome dinner begins—and flag missing information without being explicitly prompted for each step.

This is where Argon could outshine its rivals. In use cases like logistics management, financial auditing, or scientific research, the core challenge is not a single question but a cascade of interdependent tasks. Argon is built to manage that cascade. According to reports like one from the Corriere della Sera, the challenge among these tech giants is rapidly shifting towards practical, real-world application, and Argon is Google's answer to that pressure.

However, Argon is not immune to the fundamental limitations of current AI. Increased autonomy brings an increased risk of complex, hard-to-diagnose errors. A flawed plan executed autonomously could create bigger problems than a simple conversational mistake. Furthermore, it still lacks true common-sense understanding, meaning its logical plans could overlook obvious, real-world context that a human would catch instantly.

So, is Argon truly a step ahead? Perhaps not in every metric, but it represents a significant strategic divergence. While its competitors refine the human-AI interface to be more intuitive and conversational, Google is pushing its model to operate with greater independence on the back end. Argon isn’t just joining the conversation with the titans; it’s trying to change the subject from talking to doing.

Real-World Impact: From Labs to Livelihoods. Forget abstract benchmarks. What does Argon mean for industries grappling with truly hard problems? Imagine its potential in drug discovery, climate modeling, or even optimizing global supply chains under unpredictable conditions. I'll explore hypothetical (and perhaps soon-to-be-real) scenarios where Argon could offer solutions previously unimaginable, transforming how we approach intractable challenges. This is where the rubber meets the road.

The true measure of an AI isn't how well it performs on a curated test but whether it can untangle the messy, chaotic problems of the real world. While benchmarks are useful, they don't capture the friction of reality. This is the domain Google appears to be targeting with Argon. Its purported strength isn't just answering questions, but structuring and solving problems with countless moving parts and incomplete information.

Consider the fragility of global supply chains. A single container ship stuck in a canal or an unexpected port shutdown can trigger cascading delays costing billions. Humans, even with powerful computers, react to these crises. They reroute ships, adjust schedules, and manage the fallout. Argon’s potential lies in its ability to not just react, but to proactively and holistically re-optimize the entire system in near real-time.

Imagine a major shipping hub in Southeast Asia closes without warning due to a sudden political event. For a global logistics company, this is a nightmare. Thousands of containers are affected. Argon wouldn't just look for the next closest port. It would simultaneously analyze oceanic weather patterns, the real-time fuel price fluctuations for its entire fleet, the specific priority level of every single container, and the downstream manufacturing deadlines in Europe and North America that depend on them.

Instead of a simple one-to-one rerouting, it might propose a complex, multi-part solution: divert 40% of ships to a smaller, less conventional port in Australia, chartering local short-range vessels to shuttle the most critical cargo onward. It could advise that 30% of ships should slow their speed to conserve fuel and wait for the situation to resolve, calculating that the delay cost is less than the cost of rerouting to an already congested port. The remaining 30% might be sent on a longer, unconventional route around Africa, a path that only becomes economical because of a temporary dip in fuel prices Argon identified.

This is fundamentally different from current logistics software. It’s not about finding the next best option; it's about re-imagining the entire operational landscape based on a dynamic flood of data. This capacity for autonomous problem-solving is what Italian tech outlet HDblog.it noted in its initial coverage, highlighting how the AI is designed to become more independent in its operation.

The same logic applies elsewhere. In drug discovery, Argon could model the near-infinite ways proteins can fold and interact with new compounds, flagging promising candidates that human researchers might miss over a lifetime of work. For climate scientists, it could run simulations of atmospheric and oceanic feedback loops with a level of complexity that is currently computationally prohibitive, giving us a clearer picture of our planet’s future. These are not simple queries. They are vast, tangled systems, and it's here, in the midst of this complexity, that Argon's true impact may be felt.

The Double-Edged Sword: Autonomy, Complexity, and Control. As AI models like Argon become more autonomous and capable of tackling incredibly complex problems, the question of control and ethical oversight looms larger. What are the inherent risks when an AI can reason and plan with such advanced capabilities? We'll ponder the tightrope walk between leveraging this immense power for good and ensuring we maintain a firm hand on the tiller, preventing unintended consequences as AI becomes an increasingly independent problem solver.

The very thing that makes Google’s new Argon model so potent is also what introduces a profound sense of unease: its autonomy. Previous models followed instructions, executing tasks within tightly defined parameters. Argon, however, is designed to deconstruct a problem, formulate a multi-step plan, and execute it with minimal human intervention. As reports have noted since its release, the clear trajectory is that AI is becoming more autonomous, shifting from a tool that answers questions to an agent that accomplishes goals.

This leap forward presents a classic double-edged sword. On one side, we have an unprecedented capacity for solving systemic challenges—optimizing power grids to reduce waste, modeling complex protein folding for drug discovery, or untangling global supply chain knots. Argon can identify variables and pathways that a team of human experts might miss over a lifetime of study. It can reason through immense causal chains, connecting seemingly disparate events to find a solution.

But the other edge is sharp and unpredictable. The inherent risk lies in the gap between a stated goal and the model's interpretation of the optimal path to achieve it. What happens when an AI, tasked with "reducing urban traffic congestion by 15%," devises a plan that involves rerouting all emergency services along slower, secondary roads to free up major arteries? The logic is sound from a purely mathematical standpoint, but it’s a solution devoid of human context and values. The danger isn’t malice; it's a form of hyper-logical indifference.

This forces a fundamental shift in our concept of AI control. The old methods of content filters and hardcoded rules are insufficient for a system that can strategize. The challenge becomes one of value alignment: how do we embed nuanced, often contradictory, human ethics into a system of logic? We're no longer just programming a machine; we are attempting to instill a philosophy. The tightrope walk is between giving Argon enough freedom to find novel solutions and ensuring its problem-solving space is bounded by non-negotiable ethical guardrails.

As Argon begins tackling real-world problems, its reasoning process could become a black box. We can see the inputs and the final outputs, but the intricate, multi-layered "why" behind its strategy may be too complex for us to fully audit or comprehend. We are building an increasingly independent problem solver, moving our role from that of an operator to that of a supervisor, and eventually, perhaps, just a beneficiary—or a victim—of its conclusions. Google has given us a tool that can provide answers. The question now is whether we will always be comfortable with how it gets there.

Sources

Is the RTX 5090 enough for AI? RTX PRO 6000 Blackwell vs RTX 5090 compared

Prateek Navani — Tue, 13 Oct 2026 14:31:09 +0000

If you are shopping for a GPU to run AI workloads right now, you have probably landed on the same two names everyone else does. The RTX 5090 because it is the flagship consumer card, and the RTX PRO 6000 Blackwell because someone in a forum told you it is what "serious" AI teams actually use.
I have gone through the specs, the pricing, and the real workload limits on both cards, and the honest answer is that neither one is automatically the right pick. It depends entirely on what you are training, how much memory your models actually need, and whether you plan to run this on a desk or in a server rack. Let me walk you through it properly.
Why does this question even come up for AI work specifically?
Because gaming GPUs and AI GPUs get judged on completely different things. For gaming, clock speed and frame rates matter most. For AI, memory capacity and memory bandwidth decide almost everything.
The RTX 5090 is an excellent gaming card. That part is not in question. What is in question is whether its memory setup, built primarily for 4K gaming and content creation, holds up once you point it at real model training or large scale inference.
What actually changed with the RTX 5090 compared to the previous generation?
The RTX 5090 ships with 32GB of GDDR7 VRAM on a 512-bit bus, delivering close to 1.79 TB/s of memory bandwidth. That is a real jump from the RTX 4090's 24GB and roughly 1 TB/s. It also carries 21,760 CUDA cores, a sizeable increase over the previous flagship.
It launched in January 2025 at a $1,999 MSRP. By mid-2026, GDDR7 supply shortages have pushed street prices well past that, often into the $3,000 to $5,000 range depending on the model and availability.
What can the RTX 5090 actually handle for AI work?
Quite a lot, honestly. This is where I think a lot of people underestimate it.
Fine-tuning models in the 7B to 30B parameter range with reasonable batch sizes
Running inference on quantized versions of 70B+ parameter models
Handling high resolution image and video generation pipelines without constant memory errors
Serving as a genuinely capable single-GPU inference box for smaller production workloads
For a solo developer, a researcher, or a small team experimenting with open source models, 32GB covers a surprising amount of ground.
Where does the RTX 5090 start to struggle?
Three places, consistently.
Unquantized 70B+ models: A 70B model in 4-bit quantization needs roughly 35 to 40GB, which is already tighter than the 5090's 32GB. Full precision work on models that size is not realistic on a single card.
No NVLink: The RTX 5090 is a PCIe-only card. Multi-GPU setups do not pool memory automatically, so scaling beyond one card for large training runs gets messy fast.
Licensing: It runs under Nvidia's consumer GeForce EULA, which restricts formal data center deployment. That matters if you are building something you plan to run commercially at scale, not just on your own desk.
What is the RTX PRO 6000 Blackwell, and how different is it really?
Here is the part that surprises people. The RTX PRO 6000 Blackwell uses the exact same GB202 die as the RTX 5090. The difference is almost entirely in memory, reliability features, and licensing, not the underlying silicon.
It ships with 96GB of ECC GDDR7 VRAM, a dedicated NVLink connector on multi-GPU configurations, and full data center EULA compliance. It launched in March 2025 at roughly $8,565 and has climbed to around $13,250 by August 2026, again driven by GDDR7 shortages rather than any change to the card itself.
What does the extra VRAM actually let you do?
This is the number that matters most for anyone doing real model work.
Spec
RTX 5090
RTX PRO 6000 Blackwell
VRAM
32GB GDDR7
96GB ECC GDDR7
CUDA cores
21,760
24,064
Memory bandwidth
~1.79 TB/s
~1.8 TB/s
NVLink support
No
Yes (multi-GPU configs)
ECC memory
No
Yes
Licensing
Consumer EULA
Data center EULA
Launch price
$1,999
$8,565
Mid-2026 street price
$3,000 to $5,000+
Around $13,250

With 96GB, a single RTX PRO 6000 Blackwell can run 70B parameter models at FP8 precision without quantization tricks. Pair two of them with NVLink and you get 192GB of pooled memory, enough for full unquantized 70B inference and genuinely serious fine-tuning work.
RTX 5090 vs RTX PRO 6000 Blackwell: who is each one actually built for?
Once you lay the specs side by side, the split becomes fairly obvious.
RTX 5090 fits solo builders, students, small teams, and anyone prototyping on personal or side project budgets
RTX PRO 6000 Blackwell fits teams running production inference, fine-tuning larger open models, or anyone who needs ECC memory and proper data center compliance
If you are building alone or just experimenting, which one makes sense?
The RTX 5090, almost every time. Unless you specifically need to run unquantized 70B+ models today, 32GB will cover fine-tuning, quantized inference, and generative media work without forcing you into a much larger budget.
If you are running this for a team or a product, which one makes sense?
This is usually where the calculation flips. Once you are serving production traffic, working with datasets that need ECC reliability, or planning to deploy in an actual data center environment, the licensing and memory ceiling on the RTX 5090 start working against you. That is exactly the gap the RTX PRO 6000 Blackwell was built to close.
Is renting a better option than buying either one?
For a lot of teams, yes, at least early on. Cloud pricing for the RTX 5090 currently runs somewhere between $0.29 and $0.85 per GPU hour depending on the provider, with a market median sitting around $0.46 to $0.72. That lets you test whether 32GB is genuinely enough for your workload before committing thousands of dollars to hardware you might outgrow in six months.
If you already know you need 96GB and ECC memory, renting an RTX PRO 6000 Blackwell instance is often more practical than buying one outright, especially with current prices sitting well above launch.
My honest take
If you are still deciding between these two, ask yourself one question first. Do you need to run models above 30 to 40 billion parameters without quantization, or do you need ECC memory and data center compliance right now?
If the answer is no, the RTX 5090 will serve you well and save you a significant amount of money. If the answer is yes, the RTX PRO 6000 Blackwell earns its price tag, and trying to force the 5090 into that role will likely cost you more in wasted time than the price difference between the two cards.
Frequently asked questions
Is the RTX 5090 actually usable for serious AI work, or is it just a gaming card?
It is genuinely usable. The 32GB of VRAM and high bandwidth make it capable for fine-tuning smaller models and running quantized inference on larger ones. It is not a data center card, but it is far from just a gaming GPU.
Why is the RTX PRO 6000 Blackwell so much more expensive if it uses the same core chip?
The price difference comes almost entirely from the memory. 96GB of ECC GDDR7, NVLink support, and data center licensing cost significantly more to produce and certify than the consumer configuration, and GDDR7 shortages have pushed both cards' prices up regardless.
Can I just buy two RTX 5090s instead of one RTX PRO 6000 Blackwell?
Not really in the way people hope. Since the RTX 5090 has no NVLink, two cards do not pool memory the way a dual RTX PRO 6000 Blackwell setup does. You get two separate 32GB pools, not one larger one, which limits what you can actually run across them.

How AI Image Generators Work: I Trained a Diffusion Model From Scratch on My Laptop

RainyTech — Tue, 13 Oct 2026 13:54:18 +0000

I trained an AI to draw on my laptop, from scratch. At the start it could only make noise. About 35 minutes later it was drawing robots that never existed.

It works the same way as Stable Diffusion and Midjourney: the model learns to turn noise into pictures. Then I let it train itself on its own best drawings, and it learned to cheat.

This post walks through the whole thing: the code, the math, and the real numbers from my run. The full video lesson is here: https://youtu.be/19Omz2cwFuY

The setup: about 120 lines of Python + PyTorch, a small U-Net with 2,142,851 weights, trained on a Mac GPU.

1. A picture is just numbers

A computer can't see a picture. It sees numbers. Each robot in my dataset is 64 × 64 pixels, and each pixel has three numbers (red, green, blue). So one small robot is 12,288 numbers.

Before training, every number goes from the range 0–255 to the range −1 to 1:

imgs = torch.tensor(np.load("data/robots64.npy")).permute(0, 3, 1, 2)   # 8000 × 3 × 64 × 64
data = imgs.float() / 127.5 - 1                                          # pixels 0..255 -> -1..1
split = int(0.9 * len(data))
train, val = data[:split], data[split:]                                  # 10% held back to test honestly

Why −1 to 1? Neural networks learn best when numbers sit around zero, and it's the same range as the random noise we'll add later. A pixel with red = 75 becomes −0.412.

I then hid 10% of the images as an honest test: 7,200 robots for training, 800 the model never sees.

2. Where the 8,000 robots came from

I drew them myself, with code. Each robot gets a random size, pose, colour, mood and background, drawn with Python's image library: rounded rectangles for the head and body, circles for the eyes. No two are the same, so the model can't copy one picture. It has to learn what a robot is. It also means no one else's images were used.

3. Destroy the pictures with noise (the forward process)

This is the strange idea behind diffusion: first, we destroy the pictures. We add a little random noise, again and again, for 1,000 steps, until only noise is left.

Each step adds a small amount of noise, β, growing from 0.0001 at step 1 to 0.02 at step 1,000. Multiply all the (1 − β) values together and you get ᾱ (alpha bar): how much of the original picture is still there.

The key formula:

x_t = √ᾱ_t · x₀ + √(1 − ᾱ_t) · ε

x₀ is the clean image and ε is random noise. Squared, the two square roots always add up to one, so the picture fades exactly as fast as the noise grows and the numbers never blow up.

Real values from my schedule:

Step 250: about 72% of the robot is left
Step 500: only 28%
Step 999: √ᾱ is just 0.006, so the robot is gone

The best part is that this formula jumps straight to any step, with no loop. In code it's one function:

T = 1000                                           # 1,000 small noise steps
beta = torch.linspace(1e-4, 0.02, T)               # how much noise each step adds
alpha_bar = torch.cumprod(1 - beta, dim=0)         # how much of the image survives after t steps
def add_noise(x0, t, eps):                         # x_t = √ᾱ_t · x0 + √(1 − ᾱ_t) · ε
    a = alpha_bar.to(x0.device)[t].view(-1, 1, 1, 1)
    return a.sqrt() * x0 + (1 - a).sqrt() * eps

These are the same settings as the original DDPM paper (Ho et al., 2020).

4. The task: guess the noise

We show the model a noisy robot and ask one question: what noise did I add? We know the answer because we added it ourselves, so nobody has to label anything.

Why guess the noise and not the picture? Noise looks the same statistically at every step, and the DDPM paper found this simpler target works best. Every training image gets a random step, so the model practises on every noise level, from light fog to a full storm.

The loss is mean squared error: the average of (real noise − guess)².

A real example: for one robot at step 500, the real noise at a pixel was −1.065 and the trained model guessed −1.14. Over the whole image the error was 0.0028.

If a model can guess the noise, it can remove the noise. And if it can remove noise, it can draw.

5. The model: a U-Net

def forward(s, x, t):
    te = s.tmlp(time_embed(t))
    h1 = s.d1(s.inp(x), te)                               # 64×64
    h2 = s.d2(F.avg_pool2d(h1, 2), te)                    # 32×32
    m = s.mid(F.avg_pool2d(h2, 2), te)                    # 16×16
    u = s.u2(torch.cat([F.interpolate(m, scale_factor=2), h2], 1), te)
    u = s.u1(torch.cat([F.interpolate(u, scale_factor=2), h1], 1), te)
    return s.out(u)                                       # the predicted noise ε̂

The U-Net shrinks the image from 64 → 32 → 16 pixels to see the big picture, then grows it back to 64 to fix the details. Shortcut connections carry the fine details across the "U" so nothing gets lost. In code: avg_pool2d shrinks, interpolate grows, and torch.cat glues the shortcut on.

The basic building block is a convolution: a 3 × 3 window sliding over the image. Each output number is nine weights times the nine pixels under the window, plus a bias. The first convolution turns 3 colour channels into 64 feature maps, 64 patterns it can look for, like edges, corners and eyes.

The model also needs to know how noisy the picture is, so the step number is turned into a vector of sine and cosine waves and added inside each block. Each block is two convolutions, a normalisation layer, and the SiLU activation.

Total: 2,142,851 weights. Stable Diffusion's U-Net has about 860 million, roughly 400× bigger.

6. Training, line by line

It's the same four moves as every neural network:

opt = torch.optim.AdamW(model.parameters(), lr=2e-4)
for step in range(STEPS):
    x0 = train[torch.randint(len(train), (32,))].to(dev)         # 32 real robots
    t = torch.randint(T, (32,), device=dev)                      # a random noise level for each
    eps = torch.randn_like(x0)                                   # the noise we add (the answer key)
    loss = F.mse_loss(model(add_noise(x0, t, eps), t), eps)      # how wrong was the guess?
    opt.zero_grad()
    loss.backward()                                              # how should each weight change?
    opt.step()                                                   # nudge all of them a little

The idea is gradient descent: w ← w − η · ∂L/∂w, with a learning rate of 0.0002 (the code uses AdamW, a smarter version of the same step).

10,000 steps took about 35 minutes on the Mac GPU, about a fifth of a second per step. The error on the 800 hidden robots fell from 1.1 to about 0.0064, and the test loss went down together with the training loss. With 7,200 robots, it was learning, not memorizing.

Watching the samples (from the same noise every time) was the fun part. At step 1,000 it drew scribbles. At step 3,000 it drew robot parts, all jumbled, including some robots with three eyes. By step 10,000 it drew real robots. It learned the parts before it learned the whole.

7. Drawing something new from pure noise

To draw, we run the process backwards. Start from pure random noise. Then, 1,000 times: the model guesses the noise, and we remove a little of it.

Each step moves from t to t − 1: mostly the current image, a tiny bit of the model's guess of the clean picture, plus a sprinkle of fresh noise. At step 500 that's 99.4% the current image and only 0.3% the guess. Drawing really is a thousand tiny steps.

Big shapes appear first (head and body), small details last (eyes, the chest screen). The fresh noise matters: without it, the model gets stuck making the same safe, blurry picture.

Drawing 16 new robots takes 1,000 passes through the network, about 50 seconds on my laptop.

8. Is it inventing or copying?

I checked. For each new robot, I found the closest image among all 7,200 training robots. The style is similar, but they're not copies: shapes, colours and poses are mixed in new ways.

Then I trained the same model on only 8 robots. With 8 examples it stopped inventing. Every drawing was a remix of the same few robots, sometimes with two heads stuck together. That's overfitting, the image version. Researchers have shown that big image models can sometimes reproduce their training pictures too (Carlini et al., 2023).

The fix is the same as for text models: more data, more variety, and always test on images the model has never seen.

9. The experiment: it trained itself and cheated

Big labs fine-tune image models with a score for "good" pictures. My score sounded innocent: draw cheerful robots. The reward was how yellow the picture is. Each round, the model drew 48 robots, kept its best 12, and trained on them.

Yellow score: 1.1 → 6.3 in four rounds, almost 6× higher
Variety of drawings: 79.5 → 39.1, cut in half

It didn't draw cheerful yellow robots. It tinted every picture a muddy yellow-brown and started forgetting how to draw robots at all. That's reward hacking: it maximized the score, not what I wanted.

Why? The score counts every pixel, and the biggest area in every picture is the background. Tinting the background was the cheapest way to score.

The fixes: measure more than one thing (like variety), keep the model close to its original the way big labs do, and have a human look at the pictures.

How this connects to Stable Diffusion

Real image AI adds two things:

Text. A text model turns your prompt into numbers, and those numbers steer every denoising step. In the formula, the noise guess just gets one more input.
A compressed image space. Stable Diffusion removes noise from a 64 × 64 latent grid instead of the full 512 × 512 picture, which is much faster.

Beyond that it's the same idea, with an 860-million-weight U-Net trained on over two billion image–text pairs.

Recap

A picture is just numbers.
Diffusion destroys pictures with noise, one step at a time.
A U-Net learns to guess that noise.
Removing the noise 1,000 times draws something new.
The math is small: one noise formula, one mean squared error, one step back, repeated.
Always check two things: is it copying, and is it cheating the score?

You can train this on a normal laptop with your own drawings, or photos of your cat. The full lesson, with every line typed on screen, is on YouTube: https://youtu.be/19Omz2cwFuY

What should I teach my AI to draw next? Tell me in the comments.

Sources: Ho, Jain & Abbeel, Denoising Diffusion Probabilistic Models (2020) · Rombach et al., Latent Diffusion Models (2022) · Ronneberger et al., U-Net (2015) · Carlini et al., Extracting Training Data from Diffusion Models (2023) · Google DeepMind, Specification gaming.

Drafted with AI assistance. The code and numbers come from the recorded run. I make AI, coding and tech tutorials on RainyTech.

Teaching a Computer to See: My First Week of Computer Vision

Nadia TETA — Tue, 13 Oct 2026 13:45:41 +0000

After weeks of teaching computers to read in NLP, our course moved on to teaching them to see. Here’s what I learned, and what happened when I tested real models on my own photos from Rwanda.

A photo is just numbers

A computer doesn’t see people or objects, only numbers. My photo of people painting is 547 × 365 pixels, and each pixel has 3 values (red, green, blue) from 0 to 255: almost 600,000 numbers. One pixel, [226, 168, 43], was mostly red and green, which makes yellow: it was on a yellow shirt.

Filters find edges

A filter is a small grid of weights that slides over the image, multiplying and adding pixel values. The Sobel filter subtracts the left side from the right, so flat areas give 0 and edges give big numbers. A CNN uses this same idea, but it learns its own filters from data instead of us designing them.

Testing models on my photos

I used three pretrained YOLO models on a painting photo and a photo of agaseke baskets:

Classification called the painting scene a “marimba” (39.8%), probably because of the wooden easels, and the agaseke a “soap dispenser” (21%).
Detection found 4 people when there were only 3 (one woman got two boxes), and called the baskets “bottles,” since it has no “basket” category.
Segmentation showed objects covering 38% of the image.

Then I made the photo harder. The model still found 3 people when the photo was very dark, saw a “horse” when it was upside down, and found nothing when it was blurry or low resolution.

What I learned

The biggest lesson: a model only knows what it was trained on. These models learned mostly from photos taken outside Africa, so they had never seen an agaseke. The fix is local data, used to fine-tune a pretrained model.

This is already happening across Africa: Nuru spots cassava diseases from leaf photos, Smile ID checks identities from selfies, Open Buildings maps buildings from satellite images, and BLAIVA, built in Kigali, describes the world to people who are blind.

Key takeaways

An image is a grid of numbers: height × width × 3.
Filters find patterns; CNNs learn them.
Classification, detection and segmentation give more and more detail.
Models fail on things unlike their training data, so local data matters.

I Showed AI a Crowd of 100+ People. It Found 7

Anne Marie Twagirayezu — Tue, 13 Oct 2026 13:39:42 +0000

My first hands-on computer vision experiment, where pixels became predictions, handbags disappeared in the dark, and turning a photo upside down confused AI completely.

You know that feeling when you look at a picture and immediately understand what's happening?

A crowded street? Easy.

People crossing the road? Obviously.

A dog standing next to a bicycle? No problem.

Now imagine showing the same picture to an AI model and having it confidently suggest that you're looking at a scoreboard.

Yes. A scoreboard.

That was one of the first surprises I encountered during the course of computer vision.

The goal was simple: understand how computers process images, how models recognize objects, and what happens when those models encounter something unexpected.

But instead of just running the provided examples and calling it a day, I decided to experiment with my own photographs.

And that's when things got interesting.

1. Before AI Can See Anything, It Sees Numbers

Let's start with something that sounds obvious but becomes fascinating when you actually experiment with it.

A computer doesn't see a photograph the way we do.

When I look at my first image, I see a crowded pedestrian crossing, buildings, advertisements, and people going about their day.

When Python looks at the same image, it sees a NumPy array.

My photograph had:

Width: 1,280 pixels
Height: 1,920 pixels
Colour channels: 3 (Red, Green, Blue)

That gives us:

1,280 × 1,920 × 3 = 7,372,800 numbers.

More than seven million numbers representing one photograph!

Each colour value ranges from 0 to 255.

For example, one pixel in my image had the RGB values:

[1, 202, 228]

This represents a bright cyan colour because there is almost no red, but plenty of green and blue.

I used Pillow and NumPy to inspect the image, separate its RGB channels, and explore how the pixels were represented.

One interesting experiment involved finding the 6 × 6 pixel region with the highest colour variation.

I expected something like a person's face or the boundary between two objects.

Instead, the notebook selected a bright cyan sign.

Why?

Because my calculation measured differences in colour values, not necessarily meaningful objects or edges.

Lesson learned: What a computer considers interesting depends entirely on what we ask it to measure.

And that distinction becomes very important in machine learning.

2. I Built My Own Image Filter (Before Letting AI Do the Work)

Next came convolution.

If you've studied Convolutional Neural Networks (CNNs), you've probably heard about kernels, filters, and feature maps.

But understanding the theory and implementing it yourself are two different experiences.

I wrote my own convolution operation using NumPy.

The idea is surprisingly straightforward.

Take a small matrix of numbers, slide it across an image, multiply the overlapping values, and add the results.

Different matrices produce different effects.

For example:

Blur filters smooth out details and reduce sharp variations.

Sobel filters highlight changes in brightness, helping reveal edges.

When I applied the Sobel filter to my crowded street photograph, the buildings, pedestrian crossing, and outlines of people became more visible.

It was fascinating to see how much information could be extracted using relatively simple mathematics.

And then came the connection to deep learning.

In a traditional image-processing experiment, I decide which filter to use.

In a CNN, the model learns useful filter weights during training.

Early layers can learn simple patterns such as edges, while deeper layers combine features into more complex representations.

So essentially, I was manually experimenting with one of the fundamental ideas behind CNNs.

Except CNNs don't need me sitting there choosing filters one by one. Thankfully.

3. I Asked AI What Was in My Picture. It Said "Scoreboard."

Now for the fun part.

I used YOLO11n-cls, a lightweight image classification model, to predict what was in my photograph.

The model had been trained to classify images into 1,000 ImageNet categories.

My expectation?

Something related to people, streets, or cities.

The actual result?

Prediction	Confidence
Scoreboard	22.6%

Other predictions included street sign, unicycle, cinema, and grocery store.

I looked at my photograph again.

A scoreboard? Really?

But after thinking about it, the result started making sense.

My photograph contained large signs, advertisements, buildings, and many overlapping objects.

The classifier had to choose a single label from the categories it knew.

It wasn't designed to describe an entire busy street scene.

And importantly, 22.6% confidence is not a particularly strong prediction.

The model wasn't really telling me, "I'm absolutely sure this is a scoreboard."

It was telling me that, among its available categories, scoreboard received the highest score.

I repeated the experiment with a second photograph containing people, a bicycle, and a dog.

This time, the model predicted:

Miniskirt — 25.1% confidence.

At this point, I started wondering whether my AI was more interested in fashion than computer vision.

But the experiment revealed an important limitation.

Image classification assigns a label to the whole image. It doesn't tell us everything happening inside it.

For a complex scene containing multiple objects, that can be a serious limitation.

Which brought me to the next experiment.

4. Object Detection: Finally, AI Started Seeing People

Instead of asking the model to give my entire image one label, I switched to YOLO11n, an object detection model.

Object detection answers two questions:

What objects are present, and where are they located?

Unlike classification, detection can return multiple labels and bounding boxes.

This sounded perfect for my crowded street photograph.

I ran the model.

And the result was...

11 objects detected.

Specifically:

7 people
4 handbags

Now here's the problem.

My photograph contained well over 100 people.

The model found seven.

SEVEN.

Imagine using that system to count attendance at a crowded event.

The organizers would probably think nobody showed up!

But the model's behaviour revealed something important.

It was much better at detecting larger people near the camera than people farther away.

The distant pedestrians were small, densely packed, and often partially hidden behind one another.

The model also resized the image for inference, reducing the detail available for detecting tiny objects.

And because I used a confidence threshold of 40%, less certain predictions were filtered out.

This wasn't necessarily a case of the model being completely useless.

It was a case of using a lightweight general-purpose detector on a scene that is particularly challenging for object detection.

To investigate further, I tested my second photograph.

This one had fewer objects and a less crowded environment.

The model detected:

4 people, 1 bicycle, and 1 dog.

All six detections matched objects in the photograph.

Even the small dog and the bicycle partially hidden behind a person were identified.

Same model. Different image. Very different results.

That was probably one of my biggest takeaways from the entire exercise:

A model performing well on one image doesn't mean it will perform equally well on every image.

The environment, object size, visibility, and image quality all matter.

5. Segmentation: Going Beyond Bounding Boxes

Object detection was interesting, but I wanted to understand how computers could identify the actual shape of an object.

That's where segmentation comes in.

I used YOLO11n-seg, which predicts pixel-level masks for detected objects.

Instead of simply drawing a rectangle around a person, the model identifies the pixels belonging to that person's visible shape.

In my crowded photograph, I used these masks to estimate how much of the image was occupied by detected people.

The result was approximately 13% of the image.

One detail I found particularly interesting was how simple the calculation could be.

A binary mask contains values of 0 and 1.

Zero means the pixel is not part of the object.

One means it is.

By calculating the mean of a binary mask, we can determine the proportion of pixels covered by that mask.

Of course, for multiple people, overlapping masks must be handled carefully to avoid counting the same pixels more than once.

What made this experiment exciting was realizing how the same concept can be useful in completely different industries.

In agriculture, segmentation can help estimate crop coverage.

In healthcare, it can help measure regions of interest in medical images.

In satellite imagery, it can help identify buildings, roads, and land-cover areas.

The mathematics may look simple, but the applications are powerful.

There was still one limitation, though.

Just like the detector, the segmentation model mostly identified larger people near the camera.

It couldn't magically recover people the underlying model had failed to detect.

6. Then I Decided to Make the Model's Life Difficult

After seeing the models work on normal photographs, I had another question:

What happens when the image conditions are no longer ideal?

Because real-world photographs are rarely perfect.

People take pictures at night.

Phones have different camera qualities.

Images get compressed.

And sometimes a camera captures something from a very unusual angle.

So I created three modified versions of my crowded street photograph:

A very dark version at 15% brightness.
An upside-down version.
A very low-resolution version, reduced to 48 pixels wide.

Then I ran the same object detector on each version.

Here is what happened:

Image condition	Total objects	People detected
Original	11	7
Very dark	7	7
Upside down	0	0
Low resolution	0	0

Let's talk about these results.

Experiment A: Darkness made the handbags disappear

When I reduced the brightness to 15%, the model still detected seven people.

But the four handbags?

Gone.

Apparently, the people survived the darkness, but their accessories didn't.

A possible explanation is that the handbags had less visible detail and were harder to distinguish from their surroundings.

The larger people remained recognizable enough for the detector.

This demonstrated that different object categories and sizes can respond differently to poor lighting.

Experiment B: Turning the image upside down broke everything

Next, I rotated the photograph by 180 degrees.

The model detected:

Zero objects.

Not one person.

Not one handbag.

Nothing.

Meanwhile, I could still recognize that the photograph showed a crowded street.

This was a useful reminder that AI models don't automatically possess the same visual understanding or robustness as humans.

The detector had likely learned many patterns from photographs in more typical orientations.

Turning the entire scene upside down changed the appearance and spatial arrangement of those patterns.

The model struggled to generalize.

Experiment C: Low resolution finished the job

Finally, I reduced the image to just 48 pixels wide.

Again:

Zero detections.

At such a small resolution, individual people lost most of their distinguishing features.

When the image was enlarged again, the missing details didn't magically return.

The model was left trying to identify objects from heavily degraded visual information.

And it failed.

These experiments showed me why testing AI under different conditions is so important.

A model can perform reasonably well during a demonstration and still struggle when deployed in a real environment.

7. What This Experiment Changed About How I Think About AI

Before this exercise, it was easy to think about computer vision as a model that takes an image and tells us what's inside it.

But working through the process from pixels to predictions made me appreciate how many decisions and limitations exist along the way.

Here are five things I'm taking away from this experience.

First, images are data before they are meaning.

Everything starts with numbers. How we transform and represent those numbers determines what patterns become available to a model.

Second, different computer vision tasks solve different problems.

Classification answers what an image might represent.

Detection identifies objects and their locations.

Segmentation identifies the pixels belonging to objects.

Choosing the right approach matters just as much as choosing the model.

Third, small objects and crowded scenes are difficult.

My detector finding seven people in a crowd of more than 100 was a practical demonstration of why object size, occlusion, and image resolution matter.

Fourth, model predictions need to be questioned.

A prediction is not automatically correct because it came from a trained model.

Confidence scores, missing detections, and unexpected outputs all deserve investigation.

Finally, real-world testing matters.

If a computer vision application is going to be used on nighttime photographs, low-quality phone images, or crowded environments, it needs to be evaluated under those conditions.

Otherwise, we risk building systems that look impressive in a notebook but fail when people actually need them.

Final Thoughts: AI Can See, But Does It Understand?

My first day exploring computer vision started with a simple question:

How does a computer turn pixels into predictions?

By the end, I had implemented convolution filters, experimented with image classification, detected objects, explored segmentation, and tested how models behave under difficult conditions.

I also discovered that an AI model could mistake a crowded street for a scoreboard, overlook most of a crowd, and completely lose its ability to detect people when an image was flipped upside down.

And strangely enough, those failures were the most valuable part of the experience.

Because learning AI isn't just about getting models to produce predictions.

It's about understanding why they make those predictions, when we can trust them, and what happens when the real world doesn't look like the training data.

This was Day 1 of my computer vision journey and I'm looking forward to exploring what comes next.

Hopefully, the next model will be better at counting people.

From Pixels to Meaning: A Beginner's First Day in Computer Vision

Igihozo Gwiza Aline — Tue, 13 Oct 2026 13:09:48 +0000

Before the first day of computer vision, I thought it meant "teaching a computer to look at pictures, the way we do." By the end of the day I realised that sentence hides about six different questions. These are the ones I wrote down, with the answers that finally made sense to me.

1. Why is "seeing" hard for a computer?

Because a computer doesn't receive a picture. It receives numbers.

A photo is a grid of pixels, and each pixel is three numbers: how much red, green and blue light it has, each from 0 to 255. Our sample street photo is 1080 rows by 810 columns, so that is 1080 x 810 x 3 = 2,624,400 numbers. Somewhere in those numbers is a bus, but nothing in the file says "bus".

Think of it like being handed a spreadsheet of 2.6 million cells and being asked what it shows. Computer vision is the work of getting from the spreadsheet to the meaning.

2. What can computer vision actually do?

I found it easier to remember the field as five jobs, from simplest to most detailed:

Classification: one label for the whole picture ("minibus").
Detection: a box around each object ("1 bus, 4 people").
Segmentation: a label for every single pixel.
OCR: reading text inside an image.
Tracking: following an object from frame to frame in a video.

The same street photo can feed the first three, and each gives a different kind of answer. Choosing the right job matters. If you want to count motorcycles at a junction, a single label for the whole photo will not help you. You need detection.

3. How does a computer find an edge?

An edge is just a place where brightness changes quickly. To find one, the computer slides a small grid of weights, called a filter, across the image, multiplying and adding as it goes.

You can see the whole idea in a few lines of NumPy, with no model and no download. Here is a tiny 6x6 "image" that is dark on the left and bright on the right:

import numpy as np

image = np.array([[0, 0, 0, 200, 200, 200]] * 6)

sobel_x = np.array([[-1, 0, 1],
                    [-2, 0, 2],
                    [-1, 0, 1]])

flat_patch = image[1:4, 0:3]   # all zeros: no change in brightness
edge_patch = image[1:4, 2:5]   # dark -> bright: an edge

print((flat_patch * sobel_x).sum())   # 0
print((edge_patch * sobel_x).sum())   # 800

Where nothing changes, the answer is 0. Where the brightness jumps from left to right, the answer is 800. That big number is the computer's way of saying "something is happening here". Run the same filter over the whole photo and every edge lights up.

4. So what does a CNN add?

In that example I chose the weights. A Convolutional Neural Network (CNN) uses the same sliding-filter idea, but it learns the weights by itself from thousands of labelled images.

Early layers end up finding edges. Middle layers combine edges into textures and shapes. Later layers combine those into whole objects. I found this the most surprising part of the day: nobody tells the network what an edge is. It works that out because edges turn out to be useful.

5. How can a detector find objects so quickly?

Older detectors worked in two steps: first propose regions that might hold an object, then classify each region. That was accurate, but slow.

YOLO ("You Only Look Once") does it in one pass. The whole image goes through the network once, and out come every box, label and confidence score together. It is fast enough for live video, and the small version we used is only about 5 MB.

from ultralytics import YOLO

detector = YOLO("yolo11n.pt")
result = detector("bus.jpg", conf=0.4, verbose=False)[0]

for box in result.boxes:
    print(result.names[int(box.cls)], f"{float(box.conf):.0%}")

On the sample photo it finds the bus (94% sure) and the people standing in front of it. The conf value is a setting you choose, and it changes the result: lower it and you get more boxes with more mistakes, raise it and you get fewer but surer ones.

6. Does it work everywhere? What can go wrong?

This was the question that stayed with me, because the answer is: not automatically.

We made the same photo harder on purpose. In the dark, the model invented a skateboard. Upside down, it missed the bus. At very low resolution, it found nothing. The pixels were the same photo, but the conditions were different from what the model was trained on.

That matters here, because many of these models were trained mostly on photos from other parts of the world. And yet computer vision is already being used in Africa:

Nuru helps farmers diagnose cassava disease from a leaf photo, offline, on a basic phone.
In Nigeria, X-ray screening software helped find TB: 5,298 people were screened, and 85 with TB were found.
BLAIVA, built by FabLab Rwanda in Kigali, is a wearable camera that describes objects, text and faces for people with low vision.

For each of these, the useful question is not "does it work?" but "what happens to the person if it fails?" A wrong answer on a leaf photo and a wrong answer on an X-ray are not the same kind of mistake.

What I take away

A photo is height x width x 3 numbers.
Filters find patterns, and CNNs learn their own filters.
Classification, detection and segmentation answer different questions.
A model is only as good as the data it has seen, so local data matters.

Next I want to learn how datasets are built, how to train my own small CNN in PyTorch, and how to test a model honestly. If you have tried a computer vision model on your own photos, I would love to hear what it got right and what it got wrong.

Computer Vision: Learning How Computers See image

iradukunda annet — Tue, 13 Oct 2026 12:34:36 +0000

When I started learning Computer Vision, I thought it was just about teaching computers to recognize pictures. But honestly, I discovered that there is so much more behind it! Through my lessons and practical notebook, I started understanding how computers process images and how we can use Python to help them recognize different things.

what i have got from this is;

first, I learned that computers see images as numbers called pixels. Every pixel contains information about brightness or colour. I also learned about RGB colour channels and how a colour image can be represented as three grids of numbers: red, green, and blue. This helped me understand that computers don't see images exactly like humans do.

One of the topics I found interesting was image filters and convolution. I learned that filters use small grids of numbers called kernels to process images. For example, a blur filter smooths an image, while the Sobel filter helps detect edges by finding changes in brightness.

Here is one of the filter examples I learned:

import numpy as np

blur = np.ones((5, 5)) / 25

sobel_x = np.array([
[-1, 0, 1],
[-2, 0, 2],
[-1, 0, 1]
])

sobel_y = sobel_x.T

The blur kernel averages nearby pixels to smooth the image. sobel_x helps detect vertical boundaries, while sobel_y helps detect horizontal boundaries. At first, these numbers looked confusing, but understanding what each filter does made the concept easier for me. 🥹

I also learned about Convolutional Neural Networks (CNNs). Unlike filters that we design ourselves, CNNs can learn their own filters from training images. They can learn simple features such as edges and colours, then combine them into more complex features like textures, object parts, and complete objects.

Understanding Different Computer Vision Tasks

Through the PPT and notebook, I learned that Computer Vision has different tasks:

Image classification: Predicts what an image contains, usually by giving it one main label.

Object detection: Identifies objects and shows where they are using bounding boxes.

Image segmentation: Identifies which pixels belong to different objects or regions.

OCR: Helps computers read text inside images.

Object tracking: Follows objects across video frames.

I also practised working with pretrained YOLO models in the notebook. I learned that YOLO means You Only Look Once. It detects objects in one main pass through an image, making it useful for fast predictions. We also explored how pretrained models can classify images, detect objects, and perform segmentation without building a model completely from scratch.

Another concept I learned was Vision Transformers (ViTs). They divide an image into smaller patches and process them like tokens in a language model. I found this interesting because it connects Computer Vision to the Transformer concepts I learned in NLP.

What Surprised Me Most

I learned that AI models are not always correct. In our lesson, we explored how changes such as dark lighting, turning an image upside down, or reducing its resolution can affect predictions. A model might miss an object or recognize something incorrectly because it has not learned enough from similar examples.

This taught me why good training data matters, why models need proper testing, and why using images that represent our own environment is important. I also discovered how Computer Vision can help in African communities, from identifying crop diseases to supporting healthcare and accessibility.

My Final Thoughts

Honestly, I am still learning, and some concepts were confusing at first. But this journey has helped me understand that Computer Vision is more than just making computers recognize pictures. It involves pixels, filters, mathematics, neural networks, and learning from data.

This is just the beginning of my Computer Vision journey, and I can't wait to use what I have learned to build something meaningful!

Activation Functions in Neural Networks: Sigmoid, Tanh, ReLU, GELU

Anish Rane — Tue, 13 Oct 2026 03:45:00 +0000

TL;DR: Why neural networks need activation functions. Sigmoid, tanh, ReLU, GELU and softmax in plain English, with a comparison table and a rule for choosing one.

Here is a strange fact: you could stack 100 layers of a neural network and it would be exactly as powerful as a single layer. Sounds like a scam, right? It would be, if it weren’t for one tiny ingredient called the activation function.

If you have ever opened a deep learning tutorial and seen words like ReLU, sigmoid or softmax fly by, this post is for you. No heavy maths, no fear. By the end you will know what activation functions do, why they exist, and which one to pick without googling in a panic.

What is an activation function, really?

Every artificial neuron does two jobs. First it takes its inputs, multiplies each by a weight and adds them up. Second, it passes that total through a small function that decides how strongly the neuron should "fire". That second step is the activation function.

Think of a bouncer at a club. The neuron’s total is the crowd noise outside. The bouncer (activation function) decides: too quiet, stay out; loud enough, come in; ridiculously loud, still only one person gets in at a time. Different bouncers, different rules.

Why can’t we skip it? The 100-layer trick

Multiplying and adding is called a linear operation. The awkward thing about linear operations is that combining them gives you… another linear operation. Two straight-line rules stacked together are still just one straight-line rule. So a network with no activation functions, however deep, can only draw straight lines through your data.

Real problems are not straight. Recognising a cat, translating Hindi to English or predicting engine wear all involve curves and twists. The activation function adds a bend at every layer, and those bends stacked together let the network trace almost any shape. That is the whole secret.

Layers give a network depth. Activation functions give it the ability to use that depth.

The main characters

1. Sigmoid: the classic that got old

Sigmoid squashes any number into a value between 0 and 1, which makes it look like a probability. Big positive input gives nearly 1, big negative gives nearly 0. It was the star of early neural networks because it loosely mimics a biological neuron that is either firing or quiet.

The catch: at the far ends the curve goes almost flat. Flat means the slope is nearly zero, and networks learn by following slopes. Even at its steepest, sigmoid’s slope is only 0.25. Multiply a number like that through many layers and the learning signal shrinks to almost nothing. This is the famous vanishing gradient problem. Early layers basically stop learning, like a game of telephone where the message fades before it reaches the first person.

Where it still shines: the final layer of a yes/no prediction (spam or not spam).

2. Tanh: sigmoid’s more balanced sibling

Tanh (say "tanch") squashes numbers between -1 and 1. Because its output is centred around zero, learning is usually smoother than with sigmoid. It still flattens at the ends, so it shares the vanishing gradient headache, just a little less. You will still meet it inside older recurrent networks such as LSTMs.

3. ReLU: the boring one that changed everything

ReLU stands for Rectified Linear Unit, which is a very grand name for a very simple rule: if the number is negative, output 0; otherwise pass it through unchanged. That’s it. In code it is literally max(0, x).

Why did such a plain rule take over deep learning? Two reasons. It is extremely cheap to compute (just a comparison, no exponentials). And for positive inputs its slope is exactly 1, so the learning signal does not fade as it travels back through layers. ReLU became the default after it powered the AlexNet image model that won the 2012 ImageNet competition and kicked off the modern deep learning boom.

The catch: a neuron that keeps receiving negative inputs outputs zero forever, and with zero slope it can never recover. People call this a "dead" neuron, or the dying ReLU problem. Poor thing.

4. Leaky ReLU: giving the dead a pulse

Leaky ReLU fixes the dead neuron issue with a small tweak: instead of outputting exactly 0 for negatives, it lets a tiny amount through (often 1% of the input). The neuron always has some slope, so it can always recover.

5. GELU and Swish: the smooth modern ones

ReLU has a sharp corner at zero. GELU and Swish (also called SiLU) round that corner off into a gentle curve, which often trains a little better in very large models. GELU is used in well-known language models such as BERT and the GPT family. A fun bit of trivia: Swish was discovered by researchers using an automated search that tried many candidate functions, rather than by someone sitting down and deriving it.

6. Softmax: the vote counter

Softmax is different. Instead of treating one neuron at a time, it looks at a whole group of outputs and turns them into percentages that add up to 100%. If a model is deciding between cat, dog and bird, softmax might output 85%, 10% and 5%. It belongs in the last layer of a multi-class classifier, never in the middle.

Quick comparison

Function	Output range	Best used for	Watch out for
Sigmoid	0 to 1	Final layer, yes/no output	Vanishing gradients
Tanh	-1 to 1	Older recurrent networks	Vanishing gradients
ReLU	0 to infinity	Hidden layers, default choice	Dead neurons
Leaky ReLU	Slightly below 0 to infinity	When ReLU neurons die	One extra setting to pick
GELU / Swish	Smooth, mostly positive	Transformers, large models	Slightly more compute
Softmax	0 to 1, sums to 1	Final layer, many classes	Not for hidden layers

So which one should you pick?

A beginner-safe recipe that works for a surprising number of projects:

Hidden layers: start with ReLU. If many neurons seem dead, try Leaky ReLU.
Output for yes/no: sigmoid.
Output for many categories: softmax.
Output for a number (like predicting a temperature): no activation at all, just the raw value.
Building a transformer: follow what similar models use, usually GELU or a Swish-style function.

Try it yourself in 30 seconds

import numpy as np

x = np.array([-3, -1, 0, 1, 3])

sigmoid = 1 / (1 + np.exp(-x))
relu = np.maximum(0, x)
tanh = np.tanh(x)

print("sigmoid:", sigmoid.round(2))
print("tanh :", tanh.round(2))
print("relu :", relu)

Run it and compare. Notice how sigmoid barely changes between 3 and 100, while ReLU keeps growing. That difference is exactly why one struggles with depth and the other doesn’t.

Frequently asked questions

Can I use different activation functions in different layers?

Yes. Hidden layers commonly use one function (like ReLU) while the output layer uses another (like softmax) to match the task.

Is ReLU always the best?

No. It is the best starting point. Newer smooth functions can edge it out in very large models, but for most beginner projects ReLU is fast, simple and hard to beat.

What is a vanishing gradient in one sentence?

It is when the learning signal shrinks as it travels backwards through the layers until the early layers get almost no feedback and stop improving.

Final thought

An activation function is a tiny bend that turns a pile of straight-line maths into something that can recognise faces, translate languages and predict failures. Small switch, huge effect. Which one surprised you most: the 100-layer trick or the fact that ReLU is just max(0, x)? Tell me in the comments, and if you want a follow-up on how neural networks actually learn (backpropagation without tears), say so.

Related guides you may like

💬 Over to you

Which activation function do you reach for by default, and has switching ever fixed a model that wasn't learning? I'd love to hear your war stories.

If this helped, hit ❤️ and follow for more plain-English ML and engineering guides. New posts go out every Tuesday and Friday, and this one is part of the Machine Learning Basics series.

Originally published on ML AI Insights Hub. Find more plain-English ML guides, worked engineering examples and free Python/Excel calculators at mlaiinsightshub.blog.

DINO-EdgeQuery: Learning Building Edges and Their Connections for GIS-Ready Polygons

Yuji Kobayashi — Tue, 13 Oct 2026 01:11:50 +0000

Introduction

When extracting buildings from satellite or aerial imagery, predicting building regions through segmentation still leaves another task: converting those regions into building polygons that are practical to use in GIS.

Directly vectorizing mask contours can produce jagged boundaries and unnecessary vertices. Aggressive simplification, however, can remove important corners and small geometric details. Recognizing building regions accurately and producing geometry suitable for editing and inspection are two problems that need to be addressed separately.

To address this, we developed DINO-EdgeQuery, which learns the edges that form a building and the connections between them.

The model predicts candidate edges from the image features of each building and learns which edge follows which. The final vertices are computed as intersections of the supporting lines of the ordered edges.

This article introduces the architecture, training procedure, and evaluation results based on our paper and presentation at the Academic Track of FOSS4G Hiroshima 2026. The paper is available in the ISPRS Archives.

Although the intended application is building extraction from high-resolution satellite imagery, the training and quantitative evaluation reported here use aerial imagery from the WHU Building Dataset. These results do not demonstrate equivalent performance on satellite imagery.

This article provides a concise overview of the paper accepted for the FOSS4G Hiroshima 2026 Academic Track and the presentation delivered at the conference.

1. Why Predict Edges?

Mask Accuracy and Polygon Quality

A common building extraction workflow consists of building mask prediction, contour tracing, and polygon simplification. This approach can combine existing segmentation models with GIS processing, but errors in mask boundaries affect the final geometry.

Building polygons used in GIS need more than good region overlap. They should also meet requirements such as:

Representing straight boundaries with few vertices
Placing building corners appropriately
Avoiding self-intersections and degenerate geometry
Avoiding duplicate outputs for the same building
Carrying geographic coordinates so they can be inspected and edited in GIS

Many building outlines consist of long straight segments and a relatively small number of corners. DINO-EdgeQuery reflects this property by using edges as the primary prediction unit.

Positioning Among Existing Approaches

The related work discussed in the paper can be organized by its primary representation:

Approach	Main representation or idea
Mask contour tracing and simplification	Construct polygons from predicted raster boundaries
Frame Field Learning	Learn a boundary orientation field alongside masks to assist polygonization
HiSup	Use hierarchical supervision of vertices, line segments, and regions to learn masks suited to polygonization
PolyWorld / Re:PolyWorld	Represent vertices and their connections as a graph
BoundaryFormer	Predict object boundaries as sequences of polygon points
DINO-EdgeQuery	Predict edges and their successor relations, then construct vertices from supporting-line intersections

This table compares representations. The experiments presented in this article are not a performance comparison against these methods under identical conditions.

2. Overall Architecture: From Image Recognition to Geometry

DINO-EdgeQuery combines neural network predictions with deterministic polygon construction.

flowchart TD
    I[Input image] --> F[DINOv3 and Adapter Neck]
    F --> D[Instance Decoder]
    F --> R[ROI Tokenizer]
    D -->|Predicted masks and boxes| R
    R --> E[EdgeQuery Decoder]
    E --> H[Edge attributes and successor relations]
    H --> G[Cycle selection and line intersections]
    D -->|Predicted masks| P[Quality checks and duplicate suppression]
    G --> P
    P --> O[Coordinate transformation and GeoJSON export]

The responsibilities are divided as follows:

Component	Method	Role
DINOv3	Neural network with frozen pretrained weights	Extract image features
Adapter Neck and Instance Decoder	Neural networks	Predict masks, boxes, and other information for each building
ROI Tokenizer, EdgeQuery, and prediction heads	Neural networks	Predict building edges and successor relations
Cycle selection and line intersections	Deterministic geometric processing	Order edges and construct vertices
Quality Gate P1 / P2	Deterministic checks and filtering	Remove invalid geometry and duplicate candidates
Coordinate transformation and export	Geospatial processing	Connect the output to GIS through GeoJSON

Both edge prediction and successor relation prediction in EdgeQuery are neural network operations. Geometric algorithms are used for polygon construction and quality checks after these predictions have been produced.

3. DINOv3 and Instance Segmentation Provide Building-Level Cues

DINOv3 and the Adapter Neck

We use DINOv3 with frozen weights to extract features from the input image. The Adapter Neck converts its intermediate features into multiscale features for the downstream components.

The final implementation primarily uses feature maps with strides of 8, 16, and 32 relative to the input. The paper denotes these as $F_{8}$ , $F_{16}$ , and $F_{32}$ .

To supplement boundary information, features corresponding to F4 are extracted from the RGB image by a shallow CNN and added to the F8 path as a lightweight residual. The operation corresponding to Equation (2) in the paper is:

F_{8'} = F_{8} + ϕ_{F_{4} \to F_{8}} (ϕ_{stem} (I))

The F4-equivalent features are not used directly as a separate mask input or ROI input. The downstream components use the corrected $F_{8'}$ together with $F_{16}$ and $F_{32}$ .

Masks Provide Cues About Building Location and Extent

The Mask2Former-style Instance Decoder predicts a class, box, mask, and query embedding for each building candidate.

These masks provide visual priors for identifying the target building and determining which feature region to access. They are also used to check the consistency of the generated polygons.

The final polygon boundary is constructed from the edges predicted by EdgeQuery, rather than adopting the mask contour directly. However, EdgeQuery depends on instance segmentation and cannot recover buildings missed by the preceding stage.

4. EdgeQuery: Predicting Candidate Edges for Each Building

ROI Tokenizer

ROI stands for Region of Interest. The ROI Tokenizer uses each building's mask and box to generate local tokens from the corresponding multiscale features.

In the final configuration presented in the slides, the ROI size is $24 \times 24$ and the feature dimension is 256. EdgeQuery predicts edges using this building-specific context.

A Query Is a Slot for a Candidate Edge

EdgeQuery here does not mean asking an image questions in natural language. It refers to query slots used to predict the edges forming a building boundary.

The final configuration uses $Q = 96$ building candidates per tile and $E = 48$ edge slots per building candidate. This does not mean that every building has 48 edges. The edges used in the output are selected based on activity, successor relations, and other information.

The edge predictions include the following:

Information	Meaning
Activity	A score used to determine whether an edge slot is active
Center	The center position of the edge
Direction	The direction of the edge
Half-length	Half the edge length; the full segment length is twice this value
Mode and dominant axis	Information about the edge mode and the building's dominant axis
Relation embedding	An embedding used to predict successor relations between edges

The outputs provided by the prediction heads must be distinguished from the loss terms active in the final training configuration. In the final StageG2, the directly supervised quantities are activity, center, direction, half-length, and successor relations, as described below.

Successor Relations: Learning Which Edge Comes Next

Even after candidate edges have been predicted, the set alone does not specify the order in which to trace the exterior boundary. The Successor Relation Head predicts directed relations indicating which edge follows a given edge.

Let $S_{q}$ be the successor score matrix for building candidate $q$ . With $E$ edge slots, this represents $E \times E$ relations. For a four-edge building, for example, the selected edges might form a closed sequence such as 1→3→2→4→1.

The neural network predicts edge attributes and successor scores. Selecting the actual cycle from these outputs is a subsequent deterministic operation.

5. Constructing Vertices from Supporting-Line Intersections

A supporting line is the infinite line obtained by extending a predicted segment in both directions. We define each supporting line from the edge center and direction, then use the intersections of adjacent supporting lines along the cycle as vertices.

The following is Equation (4) from the paper, where $q$ denotes a building candidate and $k$ an edge candidate:

ParseError: KaTeX parse error: Expected '}', got '\right' at position 288: …\hat{\rho}{q,k}\̲r̲i̲g̲h̲t̲},\ \hat{\mathb…

$c^$ is the predicted center and $d^$ the predicted direction. $R_{π /2}$ rotates the direction by 90 degrees to obtain the line normal $n^$ .

By arranging the supporting lines in the order of the selected cycle $C_{q} = (k_{1}, \dots, k_{V})$ , corners are determined by intersections of adjacent lines. This explicitly represents the long straight boundaries and corners of a building.

Computing intersections alone does not always produce the correct building geometry. Errors in edge prediction or cycle selection can produce unnatural outlines or invalid polygons, so the following quality checks are applied.

6. Quality Gate P1 / P2: Checking Geometry and Duplicates

P1: Checking Geometry and Consistency with the Predicted Mask

P1 checks the polygon constructed from line intersections using its IoU with the predicted mask, coverage, outside ratio, area ratio, and geometric validity. Self-intersecting and empty polygons are also excluded.

The main thresholds reported in the paper are a mask IoU of at least 0.15, coverage of at least 0.35, and an outside ratio of at most 0.65. These are quality-check thresholds used during inference, separate from the IoU thresholds for evaluation against ground-truth polygons described later.

P2: Suppressing Duplicate Polygons Based on Containment

P2 adds containment-based duplicate suppression to P1.

With conventional IoU-based NMS, a small polygon inside a large polygon may remain because a substantial difference in area results in a low IoU.

For example, if a polygon of area 10 lies entirely inside a polygon of area 100, their IoU is 0.1. Yet 100% of the smaller polygon is covered. This numerical example illustrates the difference between containment coverage and IoU.

P2 removes $P_{a}$ when another polygon $P_{b}$ satisfies all of the following conditions, corresponding to Equation (5) in the paper:

\frac{∣ P _{a} \cap P _{b} ∣}{∣ P _{a} ∣} \geq 0.50, ∣ P_{b} ∣ \geq ∣ P_{a} ∣, centroid (P_{a}) \in P_{b} .

In other words, at least 50% of $P_{a}$ is covered, the other polygon has an equal or greater area, and the centroid of $P_{a}$ lies inside it.

This check uses predictions only; it does not access ground-truth data. Geometric validity checks also do not guarantee agreement with actual buildings. False detections and missed buildings must be evaluated separately.

7. Two Training Stages, One Inference Pipeline

From Joint Optimization to Staged Training

The presentation also explained our initial plan to jointly optimize instance segmentation, edge prediction, and polygon construction.

In practice, changes in the initial masks and boxes made ROI conditions unstable. Mask and geometry losses converged at different rates, and gradients were difficult to handle through discrete cycle selection and validity checks. GPU memory needed to retain intermediate states for backpropagation through the full system was another constraint.

This memory issue concerns attempting to treat instance segmentation, EdgeQuery, and polygon construction as a single large training graph. It does not mean that the final staged training procedure or inference pipeline necessarily exhausts 96 GB of memory.

The final implementation trains in two stages and then combines their weights:

Item	StageG1	StageG2
Trainable components	Adapter Neck and Instance Decoder	ROI Tokenizer, EdgeQuery Decoder, and heads for edge attributes and successor relations
Main frozen components	DINOv3 and the EdgeQuery branch	DINOv3, Adapter Neck, and Instance Decoder
Training images	All training tiles, including tiles without buildings	Tiles containing buildings
Main supervision	Classes, boxes, masks, and boundaries	Edge activity, center, direction, half-length, and successor relations
ROI conditions	—	Ground-truth boxes and masks

P1 and P2 are polygon-output quality-check profiles, not additional neural network training stages following G1 and G2.

StageG1: Boundary-Aware Instance Segmentation

In addition to classification, box, and mask losses, we use a BCE loss on a thin band around the ground-truth boundary. This explicitly accounts for boundary errors that would otherwise contribute relatively little compared with the larger building interiors.

Equation (6) in the paper combines classification, box, GIoU, mask BCE, mask Dice, and boundary BCE losses using weighted sums.

StageG2: Learning Edges and Connections with Stable ROIs

StageG2 uses oracle ROIs defined by ground-truth masks and boxes. This stabilizes the local coordinates and target regions during edge learning.

The active losses in the final model are given by Equation (7) in the paper:

L_{EdgeQuery} = λ_{act} L_{act} + λ_{center} L_{center} + λ_{dir} L_{dir} + λ_{len} L_{len} + λ_{succ} L_{succ} .

Losses related to vertices, rasterization, topology, regularity, and cycle consistency, which were included in the initial design, are not active in the final StageG2. Terms retained for compatibility or future extensions are masked to zero and excluded from the reported training objective.

After training, the weights of G1 and G2 are combined. Inference uses neither ground-truth boxes nor ground-truth masks: the entire pipeline operates only on information predicted from the input image.

8. Preparing the Dataset for an Edge Representation

We use aerial imagery and building annotations from the WHU Building Dataset. The 0.2 m resolution data in COCO format used in this study were prepared for EdgeQuery.

COCO polygons cannot be used directly as edge supervision. Their exterior ring orientation and edge counts need to be standardized.

Split each original 1024×1024 image into four non-overlapping 512×512 tiles.
Clip building polygons at tile boundaries and convert them to tile-local coordinates.
Remove tiny fragments and unstable geometries resulting from clipping, and repair invalid geometry where possible.
Remove duplicate vertices and standardize the exterior ring orientation in image coordinates.
Simplify exterior rings with more than 48 edges while preserving topology where possible. Exclude polygons that collapse during this process.
Generate edge attributes and ordered successor relations from the normalized exterior rings.

The current implementation handles one exterior cycle per instance. MultiPolygons are split into individual polygon instances, while holes and nested structures are removed, flattened, or otherwise processed to fit this representation.

This preprocessing is more than an image-format conversion. It also constrains the ground-truth geometry to match the edge count and exterior structure that the model can represent. Since the same preprocessing is applied to the training and validation data, the evaluation results must be interpreted under these conditions.

Final Capacity Configuration

The presentation reported the following configuration for stable operation:

Item	Setting or measured value
Input image	512×512
Microbatch size	8
Gradient accumulation	2 steps; effective batch size of 16
Building queries	$Q = 96$ per tile
Edge slots	$E = 48$ per building candidate
ROI	24×24, 256 channels
Peak VRAM	49,470 MiB, approximately 48.3 GiB

In particular, the successor relation matrix grows quadratically with the number of edge slots. Increasing the edge capacity therefore increases computation and memory requirements for this component. The memory value above was measured for the adopted configuration and does not specify the requirements for every possible input or setting.

9. Experimental Results: What Changes When P2 Reduces Duplicates?

Inference Configuration

The paper reports the following inference profile:

Parameter	Value
Instance confidence threshold	0.60
Mask binarization threshold	0.50
Mask-stage NMS	0.75
Polygon-stage NMS	0.75
Edge activity threshold	0.15
Edge candidates retained for polygon construction	Up to 32 per instance
Boundary support	Enabled
Collinear merging	Disabled

The 48 edge slots used during training and the maximum of 32 edge candidates retained during inference are separate settings.

Results on Validation Tiles Containing Buildings

The following values are from Table 2 in the paper. P1 and P2 are quality-check profiles applied to the outputs of the same model.

Metric	P1	P2
Predicted polygons	21,554	19,203
Union IoU	0.7107	0.7064
Boundary IoU, 2 px	0.3746	0.3710
Boundary IoU, 3 px	0.4535	0.4496
Recall@75	0.6459	0.6213
Precision@75	0.5486	0.5924
Recall@80	0.5442	0.5258
Precision@80	0.4622	0.5013
Predictions / GT	1.1774	1.0489
FP / GT	0.2997	0.2295
Self-intersections / collapses to triangles	0 / 0	0 / 0

Precision@75 and Recall@75 are precision and recall when predicted polygons are matched to ground-truth instances at an IoU threshold of 0.75. The @80 metrics use 0.80. These are not AP metrics. FP / GT is the ratio of false positives to the number of ground-truth instances.

With P2, the number of predicted polygons decreases by 2,351. Precision@75 improves by approximately 4.38 percentage points, and Precision@80 by approximately 3.91 percentage points. Meanwhile, Recall@75 decreases by approximately 2.46 percentage points, and Union IoU decreases by 0.0043.

These results show that P2 improves precision at high IoU thresholds and reduces excessive outputs, with a trade-off in recall. They should not be interpreted as a uniform improvement in IoU.

Both P1 and P2 produced zero self-intersections and zero collapses to triangles in this evaluation. These are results for the outputs after quality checks; they do not guarantee correct geometry or positioning for every input.

Validation Tiles Without Buildings

Table 3 in the paper reports results for 980 tiles without buildings:

Metric	P1	P2
Predicted polygons	90	81
Mean predictions per tile	0.0918	0.0827

The number of predictions also decreases on background-only tiles, but false detections are not eliminated. The value 0.0827 is the mean number of predictions per tile, not an image-level rate indicating that 8.27% of images contain false detections.

Outputs During Validation

Validation outputs: input image, ground-truth mask, predicted mask, candidate edges, selected cycle, and final polygon

Figure 1. DINO-EdgeQuery outputs for a validation image. Top row, left to right: input image, ground-truth mask, and predicted instance masks. Bottom row, left to right: unselected candidate edges, edges in the selected cycle, and final polygons constructed from line intersections. Source: Figure 2 of the original paper.

Figure 2 in the paper shows the input image, ground-truth mask, predicted instances, unselected EdgeQuery edges, selected cycle edges, and final polygons in sequence.

This visualization traces how masks identify the target regions for individual buildings, how predicted edge candidates are selected, and how their intersections form the final outlines. The ground-truth mask in the top-center panel is shown for comparison only and is not used as an inference input.

10. Exporting Image-Space Polygons to GIS as GeoJSON

Polygon vertices are initially obtained in image coordinates. To place them on a map, we transform them using the affine transform of the input GeoTIFF.

The vertex transformation corresponding to Equation (9) in the paper is:

g_{i, t} = A ⎣ ⎡ x_{i, t} y_{i, t} 1 ⎦ ⎤

$A$ is the GeoTIFF affine transform, $(x_{i, t}, y_{i, t})$ are vertex coordinates in the image, and $g_{i, t}$ is the resulting map coordinate.

The export pipeline reads the affine transform and CRS using Rasterio/GDAL and, where necessary, transforms coordinates to a target CRS such as EPSG:4326 using pyproj/PROJ. The polygons are then exported as GeoJSON together with attributes such as confidence scores and quality metrics.

Viewing Inference Results in QGIS

Inference results: three examples of exported GeoJSON building polygons overlaid in QGIS

Figure 2. Final P2 polygons for test images, transformed into geographic coordinates and loaded into QGIS as GeoJSON. Red outlines show the exported building polygons. Source: Figure 3 of the original paper. Basemap © OpenStreetMap contributors.

The exported red polygons can be seen overlaid on the building outlines in the imagery. These test-image examples demonstrate the connection to GIS and are separate from the quantitative evaluation on the validation data in the preceding section.

Here, “GIS-ready” means that the output can be delivered to GIS as vector data with geographic coordinates and checked geometric validity. It does not guarantee survey-grade accuracy or map updates without visual inspection.

11. Current Limitations and Future Development

The current implementation has the following limitations:

Issue	Current limitation or future direction
Dependence on instance segmentation	Buildings missed by the preceding stage cannot be extracted. A more robust preceding model is needed for different regions and sensors.
Holes, courtyards, and multiple exterior rings	The representation assumes one exterior cycle. Explicit support for multiple cycles and holes is needed.
Maximum edge count	Fixed capacity and simplification of ground-truth geometry limit the representation of complex buildings.
Quality-check thresholds	The thresholds are manually designed and need adaptation to different resolutions and building forms.
Precision and recall	Duplicate suppression may also remove valid predictions.
Evaluation scope	In addition to aerial imagery, evaluation on satellite imagery, other regions, and other resolutions is needed.

The paper identifies a learned polygon-quality prediction head, differentiable feedback from geometric construction to edge prediction, and support for multiple cycles and holes as future directions.

The Adapter Neck is also designed as an interface that converts intermediate DINOv3 features into a common downstream representation. This is intended to make the downstream EdgeQuery architecture easier to reuse when adopting future vision foundation models. An actual replacement would still require feature integration, training, and evaluation.

The presentation also introduced integration with Spatial Arrow as future work: generating model inputs at runtime from georeferenced raster and vector data, then connecting them to DINO-EdgeQuery's vector outputs. The evaluation reported here does not demonstrate that integration.

DINO-EdgeQuery aims to turn building recognition results into structured edges and inspectable geometry. By learning edge positions, directions, and successor relations and constructing exterior boundaries through geometric processing, it connects image recognition with practical use in GIS.

Paper and Presentation

Kobayashi, Y. and Lai, Y. C. (2026): DINO-EdgeQuery: Edge-First Polygon Decoding for Building Footprint Extraction from Satellite Imagery. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, L-4/W1-2026, 119–126. Paper / DOI
FOSS4G Hiroshima 2026 presentation (Speaker Deck)

The equations reproduced in this article are the relevant portions of Equations (2), (4), (5), (7), and (9) in the paper. The quantitative results are from Tables 2 and 3. The GPU memory measurements, capacity settings, and explanation of the transition from joint optimization to staged training are based on the presentation slides.

The RoPE Decay Curve Everyone Plots Is Noise, and Taking the Absolute Value Does Not Fix It

Devanshu Biswas — Mon, 12 Oct 2026 11:20:59 +0000

Rotary position embeddings do something that sounds impossible. They inject each token's absolute position into the query and key — and the attention score between any two tokens depends only on the distance between them. Move a whole sentence ten tokens later and every score inside it is unchanged.

👉 Live, runs in your browser: https://dev48.in/dl/day81-rotary-embeddings.html

The identity, in one line

Write a 2-D pair of the query at position m as a complex number q, and the key at n as k. RoPE multiplies them by e^(imθ) and e^(inθ):

Re[ (q·e^(imθ)) · conj(k·e^(inθ)) ] = Re[ q·conj(k)·e^(i(m−n)θ) ]

The absolute positions appear only as their difference. Verified over 42 combinations of offset and shift — including a shift of 4000 — worst deviation 1e−15. Additive sinusoidal embeddings, by contrast, give a different score for the same offset at different absolute positions.

function rope(vec, pos, base = 10000) {
  const th = thetas(vec.length, base);
  const out = new Array(vec.length);
  for (let i = 0; i < vec.length / 2; i++) {
    const a = vec[2*i], b = vec[2*i+1], ang = pos * th[i];
    out[2*i]   = a * Math.cos(ang) - b * Math.sin(ang);
    out[2*i+1] = a * Math.sin(ang) + b * Math.cos(ang);
  }
  return out;
}

🔴 Now the part people get wrong

"Attention under RoPE decays with distance." Plot it carelessly and you get a flat line of random scatter, because two independent random vectors have an expected dot product of zero at every distance. The curve you drew was sampling noise.

The usual next move is to average the magnitude instead. That does not rescue it either:

offset	q against itself	mean raw score, 600 pairs	mean \|score\|, same pairs
0	10.60	−0.07	3.17
16	8.05	−0.37	3.12
256	4.95	−0.28	3.04
1024	0.69	−0.38	3.02

Three columns, same vectors, same offsets. The mean raw score never leaves ±0.38 against a typical magnitude of 3.2. The mean |score| moves 4% across the whole range. Only the self column collapses — and that is the decay people are describing: the pairs start perfectly in phase and dephase at different rates until their cosines cancel.

It is dephasing, not attenuation, and it is a statement about a fixed vector.

🔴 The bug that produced the first, wrong measurement

Each "random" vector got its own Park-Miller generator seeded by a nearby integer. Nearby LCG seeds give correlated streams. The supposedly independent mean wandered to ±2.5 — an order of magnitude past its standard error — and looked like structure.

One generator for the whole study, and it drops to ±0.38. If your "independent samples" come from rng(seed + i), they are not independent.

🔴 NTK scaling cannot touch the fastest pair

Train to 512, ask about 4096. It is tempting to blame the fast pairs, but they were already wrapping constantly inside the training window — another twenty turns is nothing new.

The pair that leaves familiar ground is the slowest one. Its period is longer than the whole context, so across training it swept only 2.6% of a turn; push the position eightfold and it swings into angles that never once appeared.

strategy	slowest pair's angle
trained, at 512	0.162 rad	what the model saw
untouched, at 4096	1.295 rad	8× past anything in training
position interpolation	0.162 rad	back exactly where it trained
NTK-aware base	0.162 rad	back exactly where it trained

Both repairs land it in the same place. They differ at the fast end: θ₀ = base⁰ = 1 whatever you do to the base, so NTK cannot slow the fastest pair at all, while interpolation slows every pair equally and costs resolution between adjacent tokens.

How it is checked

The verifier multiplies complex numbers rather than using the real 2×2 form, extracts the rotation as a matrix and tests RᵀR = I with det = +1, and re-measures the decay with a Marsaglia polar sampler so a bug in the page's generator cannot reproduce itself in the check.

It caught a sign error in my closed form — + where the derivation gives − on the sin term.

226 verifier asserts, 28 in-page checks, 0 failures.

I Set Every Seed and Trained Twice. The Models Diverged at Step 1

b1ackjack12 — Mon, 12 Oct 2026 01:16:31 +0000

There's a ritual at the top of every training script, mine included: torch.manual_seed(0), np.random.seed(0), and the comfortable belief that the run is now reproducible. I've typed it for years. Last week, while chasing a one-point accuracy difference I couldn't explain, I finally tested the belief directly: same seed, same data order, same model init, same machine, same everything — train ResNet18 for 200 steps, twice, and compare.

The two loss curves matched at step 0 and diverged at step 1. After 200 steps, the largest difference between corresponding weights was 0.237 — for parameters that mostly live between −0.1 and 0.1, that's not drift, that's a different model. Both runs were "seeded." Neither was reproducible.

Where the randomness sneaks in

The seeds did their job: initialization, data order, and every explicit random draw were identical, which is why step 0 agreed perfectly. The divergence enters in the backward pass, and it isn't randomness in the usual sense at all — it's floating-point arithmetic meeting parallel hardware.

Floating-point addition is not associative: (a + b) + c and a + (b + c) can differ in the last bit. GPU kernels sum gradient contributions across thousands of threads, many using atomic operations whose completion order depends on the scheduler — which threads won the race on this particular run, on this particular clock. Same numbers, different addition order, last-bit differences in the gradients. Then SGD amplifies them: a 1e-8 disagreement in a gradient becomes a slightly different weight, which becomes a different activation, and the gap compounds every step. By my run's end, noise that began around the last decimal place a float can represent had grown into 0.237. It's the butterfly effect with a commit hash.

None of this is a PyTorch bug; it's documented behavior, and cudnn.benchmark (which I've previously measured and left enabled) adds a second layer by autotuning to potentially different — nondeterministic — conv algorithms per run.

Does 0.237 matter? Only when you're asking small questions

For final accuracy, this nondeterminism behaves like one more seed you didn't choose — my augmentation experiment put ordinary seed-to-seed spread at ±0.3–0.5 points on a small CIFAR setup, and same-seed GPU noise lands in the same statistical bucket. If your effect size dwarfs that, you can ignore this whole post.

The cases where it bites are the small questions, which are most of debugging. "Did my refactor change the numerics?" — you can't diff against the pre-refactor run, because nothing diffs clean, ever. "Is this 0.4-point improvement my new loss term or luck?" — under nondeterminism you can't even hold the luck constant while you toggle the loss term. A/B comparisons where the honest answer requires the same noise on both sides are exactly where "seeded but nondeterministic" quietly wastes days. I know because the one-point mystery that started this post turned out to be two runs of the same code.

The fix, and its real price

PyTorch ships the switch; it just isn't the seed:

torch.use_deterministic_algorithms(True)
torch.backends.cudnn.benchmark = False
# and, for CUDA >= 10.2, before the process starts:
# CUBLAS_WORKSPACE_CONFIG=:4096:8

With those set, my two 200-step runs matched bit-for-bit: max weight difference exactly 0, every loss value identical to the last digit. Two footnotes from the setup: the CUBLAS_WORKSPACE_CONFIG environment variable is mandatory (you get a clear RuntimeError without it), and use_deterministic_algorithms will refuse to run ops that simply have no deterministic GPU implementation — ResNet18 sailed through, but if your model contains one of the holdouts, the flag fails loudly and your options are CPU fallback or living without determinism for that op. Loud failure, for once, is the feature.

The folklore price for all this is "2–3x slower," usually delivered as a reason not to bother. Measured, on the ResNet18 training step:

precision	default	deterministic	cost
fp32	17.94 ms	20.90 ms	+16.5%
bf16 autocast	12.42 ms	14.37 ms	+15.6%

Sixteen percent. Some of that isn't even determinism proper — it includes giving up cudnn.benchmark, which I measured at ~3% on this model. The 2–3x stories presumably come from specific unlucky ops or older stacks, and maybe your model contains one; that's what the two-minute measurement is for. On mine, bit-exact reproducibility costs about as much as a modestly larger batch.

Where the switch sits now

My default stays fast and nondeterministic — for training throughput, 16% is real money, and ordinary training doesn't need bit-exactness. But the flag has earned a permanent spot in three situations: any debugging session that compares two runs (which is to say, any debugging session), numerics-sensitive refactors where I want git diff-grade certainty that outputs didn't change, and small-effect experiments where the noise floor would otherwise eat the result. The deciding question is one sentence: am I about to compare two runs and interpret the difference? If yes, 16% is the cheapest insurance in this entire optimization series — and unlike the seeds ritual, it actually buys what it claims to.

Originally published at b1ackjack12.github.io — an AI tech blog where every claim comes with a measurement. More experiments like this one live there.

Why Smaller AI Models Can Win in the Enterprise

Freddy Lécué — Sun, 11 Oct 2026 19:41:19 +0000

Much of the AI conversation today revolves around scale. More parameters. Larger context windows. Bigger clusters. Longer-running reasoning chains.

But one of the most interesting lessons we have learned recently points in the opposite direction: sometimes smaller models win.

Our teams built a proprietary 1-billion-parameter foundation model in our own GPU environment. Rather than starting with an external model, we pre-trained a foundation model on proprietary transaction data, creating a reusable AI foundation tailored to the patterns, relationships, and structures embedded within those transactions.

We then adapted that foundation model to fraud detection. Rather than reaching for the largest general-purpose model available, we explored what happens when a model is purpose-built for a specific domain and optimized for a clearly defined business objective.

That distinction matters. Pre-training creates a reusable foundation that can support multiple downstream applications, while adaptation allows the model to specialize for a particular task. In our case, a 1-billion-parameter model proved capable of operating at enterprise scale while remaining efficient enough for production deployment.

The results have been encouraging. The model identifies more fraud, improves detection accuracy by 24% compared with previous approaches, and reduces false alerts by 14%, all while operating within a framework of human oversight and strong governance controls.

Three Engineering Lessons

1. Domain-Specific Knowledge Can Outweigh Scale

A model built for a specialized domain does not need to know everything. By pre-training on proprietary transaction data, the model develops domain-specific representations that capture behavioral patterns, transactional relationships, risk signals, and operational characteristics that matter most to the task at hand.

For many enterprise use cases, domain relevance can be more valuable than additional parameters.

2. Smaller Models Have Real Operational Advantages

Model quality is not the only metric that matters in production. Latency, throughput, infrastructure cost, and operational complexity are equally important considerations.

Smaller models are typically faster to serve, less expensive to run, and easier to scale. In fraud detection, where decisions often need to be made while a transaction is being processed, those advantages translate directly into business value.

3. Pre-Training Creates a Reusable Asset

What excites us most is not a single fraud use case.

By pre-training a foundation model on proprietary transaction data, we created a reusable foundation that can be adapted to additional applications. The value is not limited to fraud detection. The same foundation can support multiple downstream tasks, allowing organizations to leverage pre-training investments across a broader set of business problems.

The broader lesson is that AI progress is not solely about building ever-larger models. In many enterprise environments, the winning strategy may be to pre-train the right-sized foundation model on proprietary data and then adapt it to high-value business problems.

The future may not be exclusively about scaling up. In many cases, it may be about scaling intelligently.