DEV Community: Samuel Komfi

What Google's New Chips Mean If You Train Your Own Models

Samuel Komfi — Wed, 29 Apr 2026 16:44:44 +0000

This is a submission for the Google Cloud NEXT Writing Challenge

Google just shipped two fully custom AI chips in a single year.

Not derivatives of each other. Not a "pro" and "lite" tier of the same design. Two chips, built from the ground up, for fundamentally different jobs, one for training and one is for inference. If you're building anything on Google Cloud that touches AI workloads, this announcement deserves more than a passing glance.

Let me break down what TPU v8 actually is, why the split-chip strategy matters, and most practically what it means for developers thinking about cost and architecture decisions today.

A Quick Orientation: What Are TPUs?

Before diving into v8, some context about TPU's and what they are.

Google kicked off its Tensor Processing Unit program in 2013, four years before the transformer paper, a decade before the current AI arms race. The origin story is surprisingly pragmatic. Google realized that if every user interacted with Google Search via voice for just 30 seconds a day, running the matrix multiplications on general-purpose CPUs would require building "two or three additional complete Googles." The infrastructure cost was simply impossible.

Custom silicon was the answer. A 100x efficiency gain over waiting for CPUs to catch up.

TPU v1 shipped in 2015 as a true ASIC(a single-application chip purpose-built for inference). Since then, generation by generation, Google has added liquid cooling, custom numeric formats (bfloat16), custom network interconnects, and training support. By the time Ironwood (TPU v7, announced at Cloud Next 2025) arrived, they had the most powerful TPU supercomputer ever built, now running at massive scale across Google's own services and Google Cloud.

So the question heading into 2026 was "How do you follow that up?".

Enter TPU v8: The Year of Two Chips

The answer turned out to be: you don't ship one chip. You ship two.

TPU 8T - The Training Powerhouse

Spec	Value
Chips per pod	~9,600
FP performance vs Ironwood	~3x per pod
Scale-out network bandwidth	2x
Scale-up network bandwidth	4x
Interconnect	Custom ICI

8T is the raw horsepower play. If you are training frontier models, fine-tuning large foundation models, or running research workloads at scale, this is your chip. The 4x scale-up bandwidth improvement is particularly significant as it means faster gradient synchronization across chips, which will directly translates to shorter training runs.

TPU 8i - The Inference Engine

Spec	Value
Chips per pod	1,152
FP exaflops vs Ironwood	10x per pod
HBM capacity vs Ironwood	7x (same scale-up)
Network topology	Boardfly (novel)
Primary design goal	Minimum latency

8i is where the architectural story gets really interesting. The pod is smaller in chip count but packs a 10x leap in floating point exaflops and 7x more HBM. This chip was designed around one constraint above all else which is latency.

The Boardfly Network: Why Topology Matters for Agents

Here's the detail from the announcement that most headlines will miss.

Previous TPU generations used a network topology optimized for throughput — getting large volumes of data through the interconnect efficiently. Great for training. But for agentic inference workloads, what you actually care about is the minimum time for any one token to travel between chips — the network diameter.

Google collaborated with DeepMind to design a completely new interconnect topology for 8i called Boardfly. It dramatically reduces the diameter of the chip network, meaning the distance between any two chips — and therefore the latency floor drops significantly.

Why does this matter to you as a developer?

If you are building an AI agent that needs to orchestrate sub-agents, run tool calls, stream responses, or operate under real-time constraints, your user experience is bounded by inference latency. A 10 millisecond improvement at the chip level cascades through every layer of your stack. At scale, it's the difference between an agent that feels responsive and one that feels like it's "thinking."

The fact that Google was internally discussing agents two years ago when 8i's design began; before agents became the dominant industry conversation, speaks to how tightly DeepMind's research roadmap feeds into the hardware pipeline.

The Cost Angle: It is Not Just About AI Models

Here's the example that should make every developer paying attention.

Citadel is one of the world's leading securities trading firms is running TPUs. It is not running them for frontier model training, not even for LLM inference, they using TPUs for trading systems.

They have achieved the following:

2–4x efficiency improvements
30% cost reduction

This is important to unpack. TPUs have classically been positioned as specialized ML hardware. But across eight generations, they've become increasingly capable at any mathematically intensive workload, not just neural network ops. Matrix-heavy computation of almost any kind can benefit.

For developers, the implication is this, if your workload is dominated by dense numerical computation (financial modeling, scientific simulation, signal processing, large-scale recommendation engines), the TPU cost curve may now be favorable even if you're not training a model.

The Training to Inference Strategic Shift

In the early 2000s, Google's dominant infrastructure problem was building the web index, which is a massive upfront compute task. By 2005, the index was built. The dominant workload flipped entirely to serving it. The value was not in the index itself; it was in delivering search results to billions of users in milliseconds.

The same transition is happening with AI. The value is going to come from serving the model. And serving it is where the value is created for Gemini Enterprise and search and ads and YouTube.

Training is the index-build. Inference is the serving. And just like web search, the inference workload is going to dwarf training in the long run, both in volume and in the aggregate compute it consumes.

This is the strategic logic for 8i. It is not an afterthought. A deliberate bet that inference is the dominant workload of the next era, made two years before that became obvious to the market.

For developers building on Cloud TPUs: this means Google is investing serious silicon real estate in making inference cheaper and faster. That should flow through to pricing over time as capacity scales.

Reliability: The Problem Nobody Talks About Enough

The most candid part of the announcement was about reliability at scale.

When you're coordinating tens of thousands of chips on a single computation, the math gets uncomfortable fast. If even one chip fails, the entire computation halts — like a nervous system with a severed node. At 100,000 chips, statistically, several chips will fail every single day.

The engineering challenge isn't just detecting the failure. Here are some of the advantages:

Detecting it within seconds, not minutes
Diagnosing which chip (needle-in-haystack at scale)
Reconfiguring and resuming without human intervention

Google says they're achieving 97% "goodput" across 10,000+ chip workloads. "Goodput" is a term worth understanding, it is not theoretical peak FLOPS, it is the percentage of time the system is making actual forward progress on the computation. 97% means automated recovery is happening continuously, fast enough to maintain effective throughput.

The harder unsolved problem, acknowledged openly is silent data corruption. A chip that fails cleanly is manageable. A chip that silently produces incorrect results and propagates those errors to every chip it communicates with, 9is the nightmare scenario. It's an active area of work across the entire industry.

For developers using Cloud TPUs for long training runs, this directly affects your wall-clock training time and checkpoint strategy. Understanding goodput vs. raw FLOPS is increasingly important when evaluating real infrastructure cost.

What This Means Looking Forward

A few predictions worth noting from the announcement:

CPUs are making a comeback. Not for the ML compute itself, but for agentic orchestration — spinning up sandboxes, running tool calls, managing state between inference steps. The general-purpose compute layer of the agentic stack is going to grow substantially.

Specialization is accelerating. Two chips might become more. As specific workloads (long-context retrieval, multimodal processing, real-time RL) demand their own optimization profiles, the era of one-chip-fits-all is ending. If you're designing AI infrastructure architecture today, design for heterogeneous compute.

The performance-per-dollar curve is still steep. 10x exaflops in a single generation is a number that should recalibrate any cost model you built 12 months ago. If you're making infrastructure decisions based on 2025 TPU pricing and performance, revisit them.

TorchTPU: The Announcement That Should Have Led

If the chip specs are the hardware story, TorchTPU is the software story that makes all of it accessible to the 90% of ML teams who live in PyTorch.

The previous path to TPUs in PyTorch was torch_xlawhich is a package that bolted XLA compilation onto PyTorch via a lazy tensor execution model. It worked, but it came with real friction:

Recompilations triggered by dynamic shapes (variable sequence lengths, different batch sizes between steps)
Lazy execution semantics that didn't match PyTorch's eager-by-default mental model
Limited distributed training APIs: pure SPMD only, no DDP or FSDP compatibility
A learning curve that made GPU-trained engineers feel like they were context-switching into a different ecosystem

Using TPU's on a Pytorch(which is used by over 70% of the industry) meant a careful rewrite cost before you could even evaluate if TPU was worth it. TorchTPU enables developers to migrate existing PyTorch workloads with minimal code changes while giving them the APIs and the tools to extract every ounce of compute from the hardware. ou can run your existing PyTorch code in eager mode on a TPU VM today. No graph captures, no lazy tensor surprises. Your debugging workflows, your pdb calls, your print(tensor) statements. They all work as expected. TorchTPU integrates natively with the torch.compile interface for full-graph compilation. It uses XLA as the backend compiler.

TorchTPU natively supports custom kernels written in Pallas and JAX. Many third-party libraries that build on Pytorch's distributed API's work unchanged.

Here’s what it looks like to move an existing PyTorch training script to a TPU VM with TorchTPU:

# Before (GPU training)
import torch

device = torch.device("cuda")
model = MyModel().to(device)
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-4)

for batch in dataloader:
    inputs, labels = batch[0].to(device), batch[1].to(device)
    outputs = model(inputs)
    loss = criterion(outputs, labels)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

# After (TorchTPU — minimal changes)
import torch
import torch_tpu  # new import

device = torch.device("tpu")  # change device string
model = MyModel().to(device)
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-4)

# Optional: compile for full performance
model = torch.compile(model, backend="tpu")

for batch in dataloader:
    inputs, labels = batch[0].to(device), batch[1].to(device)
    outputs = model(inputs)
    loss = criterion(outputs, labels)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

The device string changes, the torch_tpu import is added, and torch.compile is added if you want peak performance. That’s the migration for a standard training loop.

For distributed training across a TPU slice:

# Provision a TPU VM slice
gcloud compute tpus tpu-vm create my-tpu \
  --zone=us-central2-b \
  --accelerator-type=v8t-32 \
  --version=tpu-vm-tf-2.18.0-pjrt

# Install TorchTPU on all hosts
gcloud compute tpus tpu-vm ssh my-tpu \
  --worker=all \
  --command="pip install torch torchtpu"

# Launch distributed training (standard torchrun API)
gcloud compute tpus tpu-vm ssh my-tpu \
  --worker=all \
  --command="torchrun --nproc_per_node=8 train.py"

TorchTPU is currently in preview with a road map for 2026 to add more functionality.

Cost Math: Training + Serving on TPUs

Let’s run through a concrete scenario.

Scenario: A mid-sized SaaS company fine-tuning a 7B parameter domain-specific model, serving it to ~500 concurrent users

Training side (8t)

A typical 7B fine-tuning run on TPU v7 (Ironwood) at ~$5/chip-hour across a v7-256 slice (256 chips):

Estimated run time: ~8 hours per training run
Cost per run: 256 chips × $5/hr × 8 hrs ≈ $10,240

With 8t's 2.7× training price-performance improvement, the same job at the same quality:

Same cost delivers 2.7× more throughput → run completes in ~3 hours
Or: same 8-hour budget now trains a meaningfully larger model variant
Cost-effective run: ≈ $3,800 for equivalent training quality

For teams running weekly fine-tuning cycles on fresh data, this compounds quickly.

Inference side (8i)

The 80% inference price-performance improvement is the number that hits the monthly bill hardest. If you’re serving a 7B model 24/7 to real users, your inference cost is ongoing and surprise it never actually stops.

A current TPU v7 inference budget of $8,000/month
Equivalent workload on 8i: ~$4,400/month
Annual saving: ~$43,000 on inference alone

At SMB margins, $43K/year in infrastructure savings is real money. It’s a junior engineer’s salary. It’s runway.

Developer Takeaways

If you're running inference at scale, 8i is the chip to watch. The Boardfly topology and 10x FP improvement are purpose-built for agentic, low-latency workloads. Monitor Cloud TPU availability and pricing as 8i reaches GA later in 2026.
If you're training large models, 8T's 3x pod performance and 4x scale-up bandwidth meaningfully compress training timelines. At the margin, shorter training runs = lower cost = more iteration cycles.
If your workload is numerically intensive but not ML, the Citadel case study is permission to evaluate TPUs seriously. Profile your compute bottlenecks and benchmark.
Design for reliability, not peak FLOPS. For production workloads, goodput — not theoretical throughput — is the number that determines your actual infrastructure cost. Ask your cloud provider for goodput SLAs, not just FLOPS specs.
Plan for heterogeneous compute. The agentic stack will likely involve TPUs for inference, CPUs for orchestration, and possibly additional specialized chips. Start thinking about your architecture at that level now.

Google has been building towards this for 13 years. TPU v8 with two chips, is the most ambitious step yet. The developers who understand why the split happened and what each chip was designed for will be better positioned to make infrastructure decisions that hold up as the agentic era fully arrives. TorchTPU finally makes TPUs accessible to PyTorch-native teams without a painful rewrite. The friction that kept most ML engineers defaulting to NVIDIA is being systematically removed.

The infrastructure that powers intelligence is still being defined. But it's coming into focus fast.

Photo by Alexandre Debiève on Unsplash

Sentinel Openclaw

Samuel Komfi — Sun, 26 Apr 2026 18:51:35 +0000

This is a submission for the OpenClaw Writing Challenge

What I Built

My home server used to shout at me constantly and I'd tuned it out entirely. Every time NetworkManager blinked or GNOME sneezed, I'd get a noise storm. Then one night nginx quietly died and I didn't notice for six hours.

I built Sentinel, an event-driven Linux log anomaly detector that watches your systemd journal in real time, uses a local LLM to reason about each log line, and fires a Slack alert only when something genuinely looks wrong.

The core idea: instead of rigid grep rules that miss things you didn't think to filter for, Sentinel asks a local Ollama model "is this critical or just noise?" and acts on the answer. No cloud dependencies. No API keys. Everything runs on hardware you already own.

How I Used OpenClaw

The first version of this was a standalone Python script — a while True: loop with some subprocess.Popen and requests.post calls stapled together. It worked, but adding a second output channel meant editing the loop. Testing anything in isolation meant mocking half the file.

Rebuilding it on OpenClaw fixed all of that. Here's the architecture:

journalctl -f
     │
     ▼
Keyword gate          ← pure Python, no LLM invoked
     │
     ▼
Skip-pattern gate     ← drops known-noisy services
     │
     ▼
OpenClaw fires Event  ← "system_log_error"
     │
     ▼
LogAnalyzer Skill     ← local Ollama (llama3)
     │                   prompt: "Anomaly or Noise?"
     ▼
if "Anomaly" → SlackNotifier Skill

Skills are the heart of the implementation. Each one is a focused, independently testable class:

The LogAnalyzer skill sends a zero-shot classification prompt to Ollama and returns a dict with a classification key. The prompt is intentionally constrained, respond only with Noise or Anomaly: [reason]; which keeps parsing trivial and forces the model to commit to a decision rather than hedge.

class LogAnalyzer(Skill):
    @action
    def analyze_log(self, raw_log_line: str) -> dict:
        prompt = (
            "You are a Linux sysadmin. Analyze this log line. "
            "Is it a critical anomaly or expected noise? "
            "Reply ONLY with 'Noise' or 'Anomaly: [brief reason]'.\n\n"
            f"Log: {raw_log_line}"
        )
        response = ollama.chat(model=OLLAMA_MODEL,
                               messages=[{'role': 'user', 'content': prompt}])
        return {"classification": response['message']['content'].strip()}

The SlackNotifier skill knows nothing about Ollama. The LogAnalyzer skill knows nothing about Slack. The orchestration in main.py connects them with four lines:

def process_log_event(agent: Agent, event: Event):
    log_line = event.payload['log_line']
    result = agent.call_skill(LogAnalyzer, "analyze_log", raw_log_line=log_line)
    if "Anomaly" in result['classification']:
        agent.call_skill(SlackNotifier, "send_alert",
                         analysis_text=result['classification'],
                         log_line=log_line)

One workflow. Two skills. Clean separation throughout.

The two-gate pre-filter before the LLM matters more than it sounds. On a busy Linux desktop, journalctl -f produces hundreds of lines per minute. The keyword + skip-pattern gates bring LLM invocations down to a handful per hour under normal conditions — keeping inference fast and the machine cool.

Demo

Slack alert for a real anomaly:

🛡️ Sentinel Alert
Analysis: Anomaly: OOM killer invoked — nginx process terminated unexpectedly
Raw log: Apr 25 03:14:27 kernel: Out of memory: Killed process 3821 (nginx) total-vm:512444kB

Terminal output during normal operation:

[Sentinel] Active — watching systemd journal...
[SKIP] NetworkManager line suppressed
[SKIP] gnome-shell line suppressed
[ANALYZE] kernel: Out of memory: Killed process 3821...
[ALERT] Anomaly detected — Slack notified

📎 Full source: github.com/samaras/sentinel-openclaw

What I Learned

Local LLMs are genuinely good at this task. I expected to spend time prompt-engineering my way to reliable classifications. In practice, llama3 with a tightly constrained prompt (Noise or Anomaly: [reason], nothing else) was accurate enough on the first try. The structured output constraint does most of the work, open-ended prompts produce hedged, verbose responses that are hard to act on.

The pre-filter matters more than the LLM. My first instinct was to send everything to Ollama and let it decide. That was naive. Hundreds of lines per minute, even at sub-second inference, adds up. The two-gate CPU filter — keyword match then skip-pattern check — is the real performance story. The LLM is the last resort, not the first line of defense.

Skills as a unit of composition beat functions in a script. I've written a lot of automation that became unmaintainable because it grew organically inside one file. Having a named, typed skill class with explicit inputs, outputs, and a single responsibility made the project easy to reason about from day one — and easy to explain to someone else, which is harder than it sounds.

What's next: A memory_skill that logs anomalies to SQLite so I can ask the agent "how many OOM events happened last week?", and a Slack slash command listener so I can interact with Sentinel conversationally from my phone. The OpenClaw agent architecture makes both of these additive rather than surgical changes — which is the whole point.

ClawCon Michigan

I didn't attend ClawCon Michigan. I am far away in Johannesburg, South Africa.

Amanzi - Know Your Water

Samuel Komfi — Sun, 01 Mar 2026 22:18:12 +0000

This is a submission for the DEV Weekend Challenge: Community

The Community

South Africa is in a water crisis that has taken many years to manifest into daily life.

I live in the Johannesburg area, where water is not something you take for granted. Gauteng imports nearly 98% of its water from outside the province, the Vaal Dam fluctuates dangerously, and municipal restrictions can change overnight. In townships like Alexandra and Diepsloot, flooding and water outages are a lived reality, not a news headline. Nationwide, 47% of South Africans experience regular water disruptions, and the country loses an estimated 35% of its treated water to leaking infrastructure before it ever reaches a tap.

Yet for the average resident, getting reliable, consolidated water information means digging through the Department of Water & Sanitation's PDF reports, checking three different municipal websites, and hoping the news covers it. There is no single, accessible, community-facing app that brings it all together.

The community I built Amanzi AquaSA for is every South African who has ever woken up to no water and/or tried to figure out whether they can legally fill their borehole in a Level 2 restriction area. That's a community of tens of millions of people and they deserve better tools.

What I Built

Amanzi AquaSA is a Flutter mobile app (Android & iOS) that turns South Africa's fragmented water data into one clear, beautiful, and genuinely useful experience. The tagline is simple: Know Your Water.

The app is organized into five core areas:

Dashboard - A live at-a-glance summary of national water health. Dam levels, active restrictions, rainfall this month vs. historical average, and a critical alerts banner if something urgent is happening in your area. It answers the question every Gauteng resident asks on a dry summer morning: "How bad is it today?"

Dams & Rivers - Detailed profiles for South Africa's 10 major dams (Vaal, Sterkfontein, Gariep, Theewaterskloof, and more) with historical level charts you can switch between 1-year and 10-year views. River health cards cover the Vaal, Olifants, Limpopo and other major waterways - including honest pollution and health status ratings. Catchment area maps show where the water actually comes from.

My Water Usage - A personal water calculator that lets you tally your real daily usage: glasses of water, coffee, shower minutes, toilet flushes, laundry loads, geyser, garden watering and more. It compares your total against the SA average (237L/day) and the humanitarian minimum (50L/day), estimates your monthly municipal bill, and gives you targeted tips based on where you're using the most.

Alerts - A filterable feed of water alerts: flood warnings, restriction level changes, dam level thresholds, quality advisories, and maintenance outages. Critical alerts surface across the whole app with a pulsing red banner so nothing urgent gets missed.

Explore - The bigger picture. National water infrastructure entities (Rand Water, Umgeni, Lepelle), active government water projects with budgets and timelines, dam levels by province, and an educational self-sufficiency hub covering borehole drilling (process, costs, legal requirements), DIY water treatment methods (UV, RO, ceramic filters, SODIS), and desalination techniques from emergency solar stills to municipal-scale reverse osmosis.

An interactive map ties it all together; toggling between dam markers, river health overlays, catchment area polygons, and flood-risk zones across the country.

Demo

Video demo link

APK Download

Code

Github Code

How I Built It

Amanzi AquaSA is built entirely in Flutter, targeting Android and iOS from a single codebase. Here's the technical breakdown:

Framework & Language Flutter 3.x with Dart, using a feature-first folder structure. Each feature (dams, rivers, rainfall, etc.) is fully self-contained with its own models, repository, and screen files - making the codebase easy to extend as live APIs become available.

Navigation go_router for declarative, URL-like routing with deep linking support - important for eventually sharing direct links to specific dam or river detail pages.

Maps flutter_map with OpenStreetMap tiles (no API key, no cost, works offline). Custom CustomPainter layers for catchment area polygons and color-coded river polylines. Dam markers are dynamically colored by level percentage - teal for healthy, amber for low, red for critical.

Charts fl_chart powers the historical dam level line charts, monthly rainfall bar charts, and the usage history graph. The dam detail screen supports switching between 1-year and 10-year data views with smooth animated transitions between datasets.

Usage Calculator Built with stateful widget steppers and a real-time litre accumulator. Toilet flush mode toggles between old (9L) and new (4.5L) cistern volumes. Shower time uses a Slider rather than a stepper since duration is continuous. Results are persisted locally with shared_preferences to show a 7-day usage history.

Self-Sufficiency Hub The rainwater harvesting calculator uses a simple formula: roof area (m²) × rainfall (mm) × 0.8 efficiency = litres collected. Users input their roof size and select their region's monthly rainfall to get a realistic monthly collection estimate - surprisingly powerful for planning tank sizing.

Water security isn't a future problem in South Africa - it's today's problem, and it's getting harder to ignore. Amanzi is a small piece of infrastructure for a community that needs better tools to understand, track, and protect the water they depend on.

Know your water.

Unlocking Data Insights: A Semantic Search App with MindsDB and Django

Samuel Komfi — Tue, 01 Jul 2025 10:04:50 +0000

Imagine querying your vast, disparate datasets using plain English, getting not just exact matches, but semantically relevant information. That is the power of MindsDB Knowledge Bases, and I'm excited to share how I integrated them into a Django application.

Introduction

Data is everywhere, but truly understanding it often requires complex SQL queries or specialized tools. What if you could ask your data questions naturally, like "Show me clients in high-income urban areas" or "Summarize suspicious transactions"? This is exactly what MindsDB's new Knowledge Bases (KBs) aim to enable.

This article details how a django transaction application can turn a dashboard of information. I decided to build a simple, yet powerful, Django web application that allows users to perform semantic searches on mock financial data and even get AI-powered summaries. The application only has three tables to keep it simple. It is based on the dataset from the following: Kaggle Dataset We will start by building a normal Django application with postgres as the backend database. We will later then show how to make your data available to MindsDB. Django knowledge is neccessary to follow along as I will not explain the framework.

Bridging Natural Language and Structured Data

Traditional databases excel at structured queries (e.g., SELECT * FROM clients WHERE age > 30). However, they fall short when you want to query based on meaning or context, like finding "high-risk clients" or "transactions indicative of fraud." This gap often requires complex machine learning pipelines.

MindsDB Knowledge Bases address this by:

Semantic Search: Understanding the meaning behind your queries, not just keywords.
Unified Access: Querying data across different sources (like my PostgreSQL DB).
Metadata Filtering: Combining semantic search with traditional SQL filters.

Project Overview: A Django App for Financial Data Insights

My application is a Django-based web interface designed to interact with a PostgreSQL database containing mock financial data:

Client: Information about individual clients (age, gender, address, income).
Card: Details about credit/debit cards associated with clients.
Transaction: Records of financial transactions (amount, merchant, date, location).

The core of the application's intelligence comes from MindsDB, which transforms this structured data into semantically searchable Knowledge Bases. The Django app then acts as a user-friendly frontend to query these KBs.

Django Application Setup

The Django project, named transaction_dashboard_proj or any other name, uses PostgreSQL as its database and python-dotenv for environment variable management. The name of the app will be called finance and swiftly added to a list of apps in the settings.py file.

Herer is the requirements file for the project:

Django>=5.2.3
psycopg2-binary>=2.9.9
python-dotenv>=1.0.0 
pandas>=2.0.0
mindsdb_sdk>=1.0.0

The finance/models.py defines the database schema:

# core/models.py  
from django.db import models  

class Client(models.Model):
    """Client model - separate from Django User model since clients cannot login"""
    current_age = models.IntegerField()
    retirement_age = models.IntegerField()
    birth_year = models.IntegerField()
    birth_month = models.IntegerField()
    gender = models.CharField(max_length=10)
    address = models.TextField()
    latitude = models.DecimalField(max_digits=9, decimal_places=6)
    longitude = models.DecimalField(max_digits=9, decimal_places=6)
    per_capita_income = models.DecimalField(max_digits=12, decimal_places=2)
    yearly_income = models.DecimalField(max_digits=12, decimal_places=2)
    total_debt = models.DecimalField(max_digits=12, decimal_places=2)
    credit_score = models.IntegerField(default=0)
    num_credit_cards = models.IntegerField(default=0)

    class Meta:
        db_table = 'client'
        ordering = ['id']

    def __str__(self):
        return f"Client {self.id}"


class Card(models.Model):
    """Card model linked to clients"""
    CARD_BRAND_CHOICES = [
        ('visa', 'Visa'),
        ('mastercard', 'Mastercard'),
        ('amex', 'American Express'),
        ('discover', 'Discover'),
    ]

    CARD_TYPE_CHOICES = [
        ('credit', 'Credit'),
        ('debit', 'Debit'),
        ('prepaid', 'Prepaid'),
    ]

    client = models.ForeignKey(Client, on_delete=models.CASCADE, related_name='cards')
    card_brand = models.CharField(max_length=20, choices=CARD_BRAND_CHOICES)
    card_type = models.CharField(max_length=25, choices=CARD_TYPE_CHOICES)
    card_number = models.CharField(max_length=20)  # Masked card number
    expires = models.CharField(max_length=12)
    cvv = models.CharField(max_length=4)  # Should be encrypted in production
    has_chip = models.BooleanField(default=True)
    num_cards_issued = models.IntegerField(default=1)
    credit_limit = models.DecimalField(max_digits=12, decimal_places=2, null=True, blank=True)
    acct_open_date = models.CharField(max_length=8, default="01/1970")
    year_pin_last_changed = models.IntegerField(default=2025)
    card_on_dark_web = models.CharField(default="No")


    class Meta:
        db_table = 'cards'
        ordering = ['id']

    def __str__(self):
        return f"Card {self.id} - {self.card_brand} {self.card_type}"


class Transaction(models.Model):
    """Transaction model linked to clients and cards"""

    CHIP_CHOICES = [
        ('chip', 'Chip Transaction'),
        ('swipe', 'Swipe Transaction'),
    ]

    date = models.DateTimeField()
    client = models.ForeignKey(Client, on_delete=models.CASCADE, related_name='transactions')
    card = models.ForeignKey(Card, on_delete=models.CASCADE, related_name='transactions')
    amount = models.DecimalField(max_digits=10, decimal_places=2)
    use_chip = models.CharField(default="Chip Transaction", choices=CHIP_CHOICES)
    merchant_id = models.CharField(max_length=50)
    merchant_city = models.CharField(max_length=100)
    merchant_state = models.CharField(max_length=100, null=True, blank=True)
    zip = models.CharField(max_length=10, null=True, blank=True)
    mcc = models.IntegerField(default=1000)
    errors = models.CharField(max_length=200, null=True, blank=True)

    class Meta:
        db_table = 'transactions'
        ordering = ['-date', 'id']

    def __str__(self):
        return f"Transaction {self.id} - ${self.amount} on {self.date}"

Note: Clients are not Django users and do not have login capabilities in this application, focusing purely on data querying. Some of the fields may not be exactly PCI compliant or best practice, but we working with the dataset provided for testing.

Run Django migrations to create the tables in the database:

python manage.py makemigrations
python manage.py migrate

The Kaggle Dataset

Download the dataset from the Financial Transactions Dataset: Analytics from Kaggle, specifically targeting only 3 files needed for the table models above. They are the Transaction data (transactions_data.csv), Card information( cards_dat.csv) and the Users data(users_data) which is renamed client for our Django app to avoid confusion with the django auth_user. Please note the transactions_data file is massive and will need to be split into 3 or more files for easier handling. You can find a script online to split it up.

After downloading the files we will copy them into the database by using the copy command in psql. Start with client data(users_data.csv), followed by card data, and finally transaction data(as it has card & client foreign keys).

Log into psql:

\connect fraud_mindsdb;

\copy client(id,current_age,retirement_age,birth_year,birth_month,gender,address,latitude,longitude,per_capita_income,yearly_income,total_debt,credit_score,num_credit_cards) FROM '/tmp/users_data.csv' DELIMITER ',' CSV header;

\copy cards(id, client_id, card_brand, card_type, card_number, expires, cvv, has_chip, num_cards_issued, credit_limit, acct_open_date, year_pin_last_changed, card_on_dark_web) FROM '/tmp/cards_data.csv' DELIMITER ',' CSV header;

\copy transactions(id, date,client_id, card_id, amount,use_chip,merchant_id, merchant_city, merchant_state, zip, mcc, errors) '/tmp/transaction_data.csv' DELIMITER ',' CSV header;

Connecting MindsDB to PostgreSQL

We are going to run MindsDB locally instead of using the cloud version. You will need docker installed. To run MindsDB Web Studio locally execute the following:

docker run -p 47334:47334 -p 47335:47335 mindsdb/mindsdb

If you get a 'Connection refused' error run the following to allow MindsDB to access your local Postgres instance:

docker run --network=host mindsdb/mindsdb

The first step in MindsDB is to connect it to our existing PostgreSQL database. This allows MindsDB to access the Client, Card, and Transaction tables. I ran this command directly in the MindsDB Web Studio:

CREATE DATABASE finance_db
WITH ENGINE = 'postgres',
PARAMETERS = {
  "host": "your_db_host",
  "port": 5432,
  "user": "your_user",
  "password": "your_password",
  "database": "your_postgresdb"
};

This creates a virtual database finance _db inside MindsDB, mirroring our PostgreSQL schema.

Creating MindsDB Knowledge Bases

Next, we define two Knowledge Bases: one for Client data (client_kb) and one for Transaction data (transaction_kb). The key here is defining the CONTENT_COLUMN (what MindsDB will semantically understand) and METADATA_COLUMNS (additional data to retrieve and filter on).


-- For Clients:  
CREATE KNOWLEDGE_BASE client_kb
USING 
    embedding_model = {
        "provider": "gemini",
        "model_name": 'gemini-embedding-exp-03-07',
        "api_key": "xxxxxx"
    },
    reranking_model = {
        "provider": "gemini",
        "model_name": "gemini-pro",
        "api_key": "xxxxxx"
    },
    content_column = 'address',
    metadata_columns = ['per_capita_income', 'current_age', 'gender', 'birth_year']; 

-- For Transactions:  
CREATE KNOWLEDGE_BASE transaction_kb
USING 
    embedding_model = {
        "provider": "gemini",
        "model_name": 'gemini-embedding-exp-03-07',
        "api_key": "xxxxxx"
    },
    reranking_model = {
        "provider": "gemini",
        "model_name": "gemini-pro",
        "api_key": "xxxxxx"
    },
    content_column = ['merchant_city', 'merchant_state'],
    metadata_columns = ['amount', 'use_chip', 'date', 'client_id'];

After defining the KBs, we ingest the existing data in the postgres database of the Django app:

INSERT INTO client_kb (content, metadata)
SELECT 
    address as content,
    JSON_OBJECT(
        'id', id,
        'per_capita_income', per_capita_income,
        'current_age', current_age,
        'gender', gender,
        'birth_year', birth_year
    ) as metadata
FROM finance_db.core_client;

-- Insert transaction data with merchant location as content
INSERT INTO transaction_kb (content, metadata)
SELECT 
    CONCAT(merchant_city, ', ', merchant_state) as content,
    JSON_OBJECT(
        'id', id,
        'amount', amount,
        'use_chip', use_chip,
        'date', date,
        'client_id', client_id
    ) as metadata
FROM finance_db.core_transaction;

Indexing for Performance

To ensure efficient semantic queries, it's crucial to create indexes on the Knowledge Bases:

CREATE INDEX ON client_kb;  
CREATE INDEX ON transaction_kb;

This optimizes the underlying vector search for faster retrieval.

Automated Data Ingestion with MindsDB JOBS

For a dynamic application, new data is constantly added. In our case as we inserted the data from CSV files we can simulate that by spliting the files and inserting the last CSV file to simulate new transactions or we can add them in the Django admin app in the backend. MindsDB JOBS allow for periodic updates of the Knowledge Bases. We will set up a job to automatically insert new transactions into transaction_kb every hour:

CREATE JOB update_transaction_kb_job (  
    INSERT INTO transaction_kb SELECT * FROM finance_db.transactions WHERE date > LAST;  
) EVERY 1 HOUR;

The WHERE date > LAST clause is a powerful MindsDB feature that intelligently fetches only records added since the last job run or ingestion, ensuring efficiency.

Multi-step Workflows: KB + AI Table for Summarization

One of the most exciting aspects is combining KBs with MindsDB AI Tables for multi-step workflows. We will create an AI Table using Google Gemini (you can use OpenAI if you got moola) to summarize transaction details. You need to configure your Gemini engine in MindsDB first.

First, set up your Gemini engine (if you haven't already):

CREATE ML_ENGINE google_gemini_engine
FROM google_gemini
USING
    api_key = 'AIzaSyA0for-example';

Then, define the summarization model:

CREATE ML_MODEL summarize_transactions_model
PREDICT summary
USING
    engine = 'google_gemini_engine', 
    model_name = 'gemini-pro',
    prompt_template = 'Summarize the following transaction details into a concise overview, highlighting key aspects: {{transaction_details}}.';

Now, the magic happens in a combined query, this will be called in the view of the Django application(You don't need to call it through the MindsDB web view):

SELECT  
    t.id AS transaction_id,  
    t.amount,  
    t.date,  
    t.merchant_city,  
    t.merchant_state,  
    t.use_chip,  
    t.client_id,  
    s.summary -- The predicted summary from the AI model  
FROM  
    transaction_kb AS t  
JOIN  
    summarize_transactions_model AS s ON 1=1 -- Join the KB with the AI model  
WHERE  
    t.content LIKE 'unusual large spending' AND -- Semantic search for transactions  
    s.transaction_details = CONCAT( -- Feed relevant columns into the AI model's prompt  
        'Transaction ID: ', t.id,  
        ', Amount: $', t.amount,  
        ', Date: ', t.date,  
        ', Merchant City: ', t.merchant_city,  
        ', Merchant State: ', t.merchant_state,  
        ', Used Chip: ', t.use_chip,  
        ', Client ID: ', t.client_id  
    );

This query first finds transactions semantically related to "unusual large spending" from the transaction_kb, and then for each found transaction, it sends a concatenated string of its details to the summarize_transactions_model to generate a summary.

MindsDB Integration in Django (mindsdb_util.py)

To bridge Django and MindsDB, we create a mindsdb_util.py file within my Django core app. This utility uses the mindsdb_sdk Python library to connect to and query MindsDB. All the select queries will be called from a django view and generate data to be shown in the dashboard.

import os
from typing import List, Dict, Any, Optional
from django.conf import settings
from django.db import models
import mindsdb_sdk
from .models import Transaction, Client, Card


class MindsDBUtil:
    """
    Utility class for executing MindsDB SQL queries using mindsdb_sdk
    """

    def __init__(self):
        """Initialize the MindsDB connection using mindsdb_sdk"""
        try:
            # Get MindsDB connection parameters from environment or settings
            host = getattr(settings, 'MINDSDB_HOST', 'localhost')
            port = getattr(settings, 'MINDSDB_PORT', 47334)
            username = getattr(settings, 'MINDSDB_USER', None)
            password = getattr(settings, 'MINDSDB_PASSWORD', None)

            # Initialize MindsDB connection
            if username and password:
                self.connection = mindsdb_sdk.connect(
                    host=host,
                    port=port,
                    username=username,
                    password=password
                )
            else:
                self.connection = mindsdb_sdk.connect(
                    host=host,
                    port=port
                )

            self.server = self.connection.get_server()
            print(f"Successfully connected to MindsDB at {host}:{port}")

        except Exception as e:
            print(f"Warning: Could not initialize MindsDB connection: {e}")
            self.connection = None
            self.server = None

    def execute_query(self, query: str) -> List[Dict[str, Any]]:
        """
        Execute a MindsDB SQL query and return results

        Args:
            query: SQL query to execute

        Returns:
            List of dictionaries containing query results
        """
        if not self.connection:
            raise Exception("MindsDB connection not available")

        try:
            # Execute the query
            result = self.connection.query(query)

            # Convert to list of dictionaries
            if hasattr(result, 'fetchall'):
                columns = [desc[0] for desc in result.description]
                rows = result.fetchall()
                return [dict(zip(columns, row)) for row in rows]
            else:
                # Handle different result formats
                return result.to_dict('records') if hasattr(result, 'to_dict') else []

        except Exception as e:
            print(f"Error executing query: {e}")
            raise

    def find_wealthy_clients(self, min_age: int = 40, min_income: float = 70000) -> List[Dict[str, Any]]:
        """
        Find clients in wealthy areas with age and income filtering

        Note: This assumes your client_kb has columns: id, address, current_age, per_capita_income, gender
        You may need to adjust based on your actual KB schema
        """
        query = f"""
        SELECT
            c.id AS client_id,
            c.address,
            c.current_age,
            c.per_capita_income,
            c.gender
        FROM
            client_kb AS c
        WHERE
            c.current_age > {min_age} AND                    
            c.per_capita_income > {min_income};
        """
        return self.execute_query(query)

    def find_travel_expenses(self, min_amount: float = 500, use_chip: bool = True) -> List[Dict[str, Any]]:
        """
        Find transactions related to travel with amount and chip usage filtering

        Note: This assumes your transaction_kb has appropriate columns
        """
        # Use proper boolean value for MindsDB
        chip_value = 'true' if use_chip else 'false'
        query = f"""
        SELECT
            t.id AS transaction_id,
            t.amount,
            t.date,
            t.merchant_city,
            t.merchant_state,
            t.use_chip,
            t.client_id
        FROM
            transaction_kb AS t
        WHERE
            t.amount > {min_amount} AND                   
            t.use_chip = {chip_value};
        """
        return self.execute_query(query)

    def find_online_shopping(self, state: str = 'California') -> List[Dict[str, Any]]:
        """
        Find transactions for online shopping in a specific state
        """
        query = f"""
        SELECT
            t.id AS transaction_id,
            t.amount,
            t.merchant_city,
            t.merchant_state,
            t.client_id
        FROM
            transaction_kb AS t
        WHERE
            t.merchant_state = '{state}';
        """
        return self.execute_query(query)

    def semantic_search_transactions(self, search_term: str, limit: int = 10) -> List[Dict[str, Any]]:
        """
        Perform semantic search on transaction knowledge base

        Args:
            search_term: Natural language search term (e.g., "suspicious activity", "travel expenses")
            limit: Maximum number of results to return
        """
        query = f"""
        SELECT 
            *
        FROM transaction_kb 
        WHERE 
            MATCH('{search_term}')
        LIMIT {limit};
        """
        return self.execute_query(query)

    def semantic_search_clients(self, search_term: str, limit: int = 10) -> List[Dict[str, Any]]:
        """
        Perform semantic search on client knowledge base

        Args:
            search_term: Natural language search term (e.g., "wealthy suburban areas")
            limit: Maximum number of results to return
        """
        query = f"""
        SELECT 
            *
        FROM client_kb 
        WHERE 
            MATCH('{search_term}')
        LIMIT {limit};
        """
        return self.execute_query(query)

    def get_transaction_summary(self, transaction_id: int) -> List[Dict[str, Any]]:
        """
        Get AI-generated summary for a specific transaction

        Note: This assumes you have a model named 'summarize_transactions_model'
        and that the model expects transaction_id as input
        """
        query = f"""
        SELECT 
            transaction_id,
            summary
        FROM summarize_transactions_model 
        WHERE 
            transaction_id = {transaction_id};
        """
        return self.execute_query(query)

    def analyze_suspicious_patterns(self, client_id: Optional[int] = None) -> List[Dict[str, Any]]:
        """
        Use AI model to analyze suspicious transaction patterns

        This is a more realistic approach than the original join-based method
        """
        where_clause = f"WHERE client_id = {client_id}" if client_id else ""

        query = f"""
        SELECT 
            t.id,
            t.amount,
            t.date,
            t.merchant_city,
            t.merchant_state,
            t.client_id,
            p.risk_score,
            p.risk_reason
        FROM transaction_kb t
        JOIN pattern_analysis_model p ON t.id = p.transaction_id
        {where_clause}
        ORDER BY p.risk_score DESC;
        """
        return self.execute_query(query)

    def get_knowledge_base_stats(self) -> Dict[str, int]:
        """
        Get statistics about the knowledge bases
        """
        try:
            client_count = self.execute_query("SELECT COUNT(*) as count FROM client_kb;")
            transaction_count = self.execute_query("SELECT COUNT(*) as count FROM transaction_kb;")

            return {
                'client_kb_count': client_count[0]['count'] if client_count else 0,
                'transaction_kb_count': transaction_count[0]['count'] if transaction_count else 0
            }
        except Exception as e:
            print(f"Error getting knowledge base stats: {e}")
            return {'client_kb_count': 0, 'transaction_kb_count': 0}

    def custom_semantic_search(self, 
                             search_term: str, 
                             kb_type: str = 'transaction',
                             filters: Dict[str, Any] = None,
                             limit: int = 10) -> List[Dict[str, Any]]:
        """
        Perform custom semantic search on knowledge bases with optional filters

        Args:
            search_term: Natural language search term
            kb_type: 'transaction' or 'client'
            filters: Dictionary of filters to apply
            limit: Maximum number of results

        Returns:
            List of matching results
        """
        kb_name = f"{kb_type}_kb"

        # Build the base semantic search query
        query = f"SELECT * FROM {kb_name} WHERE MATCH('{search_term}')"

        # Add filters if provided
        if filters:
            filter_conditions = []
            for key, value in filters.items():
                if isinstance(value, str):
                    # Escape single quotes in string values
                    escaped_value = value.replace("'", "''")
                    filter_conditions.append(f"{key} = '{escaped_value}'")
                elif isinstance(value, bool):
                    filter_conditions.append(f"{key} = {'true' if value else 'false'}")
                else:
                    filter_conditions.append(f"{key} = {value}")

            if filter_conditions:
                query += " AND " + " AND ".join(filter_conditions)

        query += f" LIMIT {limit};"
        return self.execute_query(query)

    def test_connection(self) -> bool:
        """
        Test if MindsDB connection is working
        """
        try:
            if not self.connection:
                return False

            # Try a simple query
            result = self.execute_query("SELECT 1 as test;")
            return len(result) > 0 and result[0].get('test') == 1

        except Exception as e:
            print(f"Connection test failed: {e}")
            return False

    def list_available_tables(self) -> List[str]:
        """
        List all available tables/knowledge bases in MindsDB
        """
        try:
            result = self.execute_query("SHOW TABLES;")
            return [row.get('table_name', row.get('Tables_in_mindsdb', '')) for row in result]
        except Exception as e:
            print(f"Error listing tables: {e}")
            return []

    def describe_table(self, table_name: str) -> List[Dict[str, Any]]:
        """
        Get schema information for a specific table
        """
        try:
            return self.execute_query(f"DESCRIBE {table_name};")
        except Exception as e:
            print(f"Error describing table {table_name}: {e}")
            return []


# Global instance for easy access
mindsdb_util = MindsDBUtil()

Django views then simply import mindsdb_client and call its query() method:

from django.shortcuts import render  
from .mindsdb_util import mindsdb_client  

def client_kb_search_view(request):  
    query_text = request.GET.get('query', '')  
    if query_text:  
        sql_query = f"SELECT id, address, current_age, per_capita_income FROM client_kb WHERE content LIKE '{query_text}' LIMIT 10;"  
        results_df = mindsdb_client.query(sql_query)  
        # Convert to list of dicts for template  
        results = results_df.to_dict('records') 
        
    return render(request, 'finance/client_kb_search.html', {'query': query_text, 'results': results})

This application demonstrates several powerful use cases for MindsDB Knowledge Bases in a financial context:

Customer Segmentation & Targeted Marketing: Semantically search for "affluent clients in suburban areas" to tailor product offerings.
Fraud Detection: Query "suspicious transactions in unusual locations" or "unusually large spending patterns" and get AI summaries for quick analysis by human analysts.
Risk Assessment: Identify clients or transaction types that semantically match "high-risk profiles."
Compliance & Audit: Quickly retrieve all data points related to specific semantic concepts for reporting.

Conclusion

MindsDB Knowledge Bases significantly lower the barrier to integrating advanced semantic search and AI capabilities into existing data infrastructure. The ability to combine natural language queries with traditional SQL filtering, and to chain these with AI Tables, opens up a world of possibilities for building intelligent data applications.

This project was a great experience in leveraging MindsDB's powerful features to transform raw data into actionable, semantically queryable insights within a familiar Django environment.

GitHub Repository: [https://github.com/samaras/django-mindsdb-transactions]
MindsDB Docs: https://mindsdb.com/docs/

Greedy algorithm

Samuel Komfi — Fri, 14 Jun 2024 19:41:26 +0000

This is a submission for DEV Computer Science Challenge v24.06.12: One Byte Explainer.

Explainer

A greedy algorithm is like a kid in a candy store: it grabs the best candy (local optimum) it sees without thinking ahead, hoping to end up with the most candy (global optimum). It’s used in problems like finding the shortest path or scheduling tasks.

Additional Context

Greedy but effective

How to search for files in Linux

Samuel Komfi — Fri, 23 Sep 2022 21:06:04 +0000

Introduction

This guide was created to help you understand how to search for files in a Linux system. The tutorial was based on a Linux Mint 18.3 installation.
Linux uses different commands to help you look for files using regular expressions. Imagine looking for a file, but don't dont remember
much except a small part of the name of the file.

Prerequisites

Before you begin this guide you'll need the following:

Desktop/laptop/server with Ubuntu/Linux Mint installed or any other Linux distribution really.

Step 1 — Using the grep command

Grep is used to search for characters inside files. It also has support for regular expressions. The family of grep commands include the
following commands: grep, egrep, fgrep.
egrep and grep are more or less the same with just minor differences. fgrep uses fixed strings to extract text from files. Grep can also take in input from the standard input.

Search for pattern in files:

$ grep [options] pattern files

The commands will read each line one at time.

\* will match any character
? or . will only match a single character
[xxx] will match a range of characters
\x will escape characters such as ? or \ which are used in other regexp
^pattern will match anything that starts with pattern
$pattern will match anything that ends with pattern

Search recursively for pattern in the home directory:

$ grep -r pattern /home

Display the find commands in your bash_history with the following:

$ grep -a find .bash_history

You'll see the following output:

[Output]

find /home -size +10000k
man find
man find > find.txt
gedit find.txt
find -type f -mtime -1 -print

Now transition to the next step by telling the reader what's next.

The find command

The find command is a slow but reliable option if you are searching for a file in your system. If the file exists, find is more likely to find it eventually. It has the following format:

$ find [path] [options]

$ find /home/tom -name 'index*'

Find can used to search for files in choosen directories. You can specify a list of directories to search in.
The find command can also search using the following criteria:

Search by size of the file
Search by name of the file
Search by permission of the files -
Search by group the file owner belongs to -gid
Search by the owner of the file -uid & -user
Search by type -type The possible arguments to -type are b{block device}, c {character device}, d {directory}, p {named pipe}, f {regular file}, l {symbolic link}

Find files names that start with "FB":

$ find /home -name 'FB*'

Find files that are larger than 10 MB in the home folder:

$ find /home -size +10000k

Find files by the type of permission allowed:

$ find /home -perm

Find files by the group of the owner of the file belongs to:

$ find -user root

<$>[warning]
Warning: The -user option will only work if the user exists and is not removed. Use the -uid argument to reference the user.
<$>

The -maxdepth levels option is used to limit how far deep the find command will search in the subdirectories.

The -exec options allows you to execute other commands on the files found. The following will remove any .tmp files found in the home directory.

$ find /home -name "*.tmp" -exec rm {} \;

To list files of a certain type which have also been accessed in the past 24 hours you can run the follwoing command:

$ find /home -mtime -1 \! -type d

The -type options specifies the type of file. In this case a directory. You add a ! to list any file type except a directory.

Locate and the whereis command

The whereis command is used to search for files in the systems path.

$ whereis elixir

The locate command is for finding files using just a simple string or filename. The command assumes you have called the updatedb. The updatedb command creates or updates a database file which has all the files attached to the linux root directory. Locate will use this database to give you the location of the file.

Find all instances of filename:

$ locate filename

ls command

The ls command is used to list the contents of the current working directory. Its one of the most used linux commands.

The most common options used with ls are the -a and -l options.
The ls -a command is used to list all files including hidden files, while the ls -l command is for listing all files with all their size, permissions and ownership information. You can also use special characters to abbreviate file names, this is called 'globbing'. It should not be confused with regular expressions which is used in the grep command.

$ ls -l filename

To list all text files in the directory run:

$ ls *.txt

The following will list any file that has a number as its name including the extension. If you omit the last star at the end, only
files ending with numbers will be listed e.g ls *[0-9].

$ ls *[0123456789]*
$ ls *[0-9]*

I got the following output on my home folder:

[digit_files Output]
buffalo_0.18.2_Linux_x86_64.tar.gz
Calculus Made Easy33283-pdf.pdf
ccenh.mp4
Cogele El Golpe - King Bongo [360p].mp4
com.keramidas.virtual.WIFI_AP_LIST-20190908-150043.tar.gz
COR14.3 (7).pdf
De Mthuda - Wamuhle ft Njelic.mp3
DesignPatternsInPython_ver0.1.pdf
design-principles4.pdf
Django 2.0 release notes _ Django documentation _ Django.pdf
djangoadvent-articles06_object-permissions.rst
DomAC6HW0Accg_6.jpg
Dxmd1ybWsAEIfc8.jpg
ejabberd 2.1.pdf
FB_IMG_1431202962739.jpg
FB_IMG_1450904477927.jpg
FB_IMG_1458068934007.jpg

$ ls *[-a-z][0-9]*

$ ls FB_IMG_1*[0-9]*.jpg

I got the following output on my home folder:

[secondary_label Output]

FB_IMG_1431202962739.jpg
FB_IMG_1450904477927.jpg
FB_IMG_1458068934007.jpg

I could also have gotten the same output using an asterik representing any zero or more characters after the 'FB_IMG_' string.

$ ls FB_IMG_*

You can escape characters with a slash "\"

$ ls *\?*

Nugget

You can search the manual pages for help in looking for a particular name and description using the apropos command. This will search the whatis database for files that have the specified string.

$ apropos search

The following output lists all the commands I can use to 'search' on my computer.

[Output]
ALTER_TEXT_SEARCH_CONFIGURATION (7) - change the definition of a text search configuration
ALTER_TEXT_SEARCH_DICTIONARY (7) - change the definition of a text search dictionary
ALTER_TEXT_SEARCH_PARSER (7) - change the definition of a text search parser
ALTER_TEXT_SEARCH_TEMPLATE (7) - change the definition of a text search template
anemotaxis (6x)      - directional search on a plane.
apropos (1)          - search the manual page names and descriptions
badblocks (8)        - search a device for bad blocks
bsearch (3)          - binary search of a sorted array
bzegrep (1)          - search possibly bzip2 compressed files for a regular expression
bzfgrep (1)          - search possibly bzip2 compressed files for a regular expression
bzgrep (1)           - search possibly bzip2 compressed files for a regular expression
CREATE_TEXT_SEARCH_CONFIGURATION (7) - define a new text search configuration
CREATE_TEXT_SEARCH_DICTIONARY (7) - define a new text search dictionary
CREATE_TEXT_SEARCH_PARSER (7) - define a new text search parser
CREATE_TEXT_SEARCH_TEMPLATE (7) - define a new text search template
docker-search (1)    - Search the Docker Hub for images
DROP_TEXT_SEARCH_CONFIGURATION (7) - remove a text search configuration
DROP_TEXT_SEARCH_DICTIONARY (7) - remove a text search dictionary
DROP_TEXT_SEARCH_PARSER (7) - remove a text search parser
DROP_TEXT_SEARCH_TEMPLATE (7) - remove a text search template
FcConfigGetCacheDirs (3) - return the list of directories searched for cache files
find (1)             - search for files in a directory hierarchy
git-bisect (1)       - Use binary search to find the commit that introduced a bug
hsearch (3)          - hash table management
hsearch_r (3)        - hash table management
lfind (3)            - linear search of an array
lsearch (3)          - linear search of an array
lzegrep (1)          - search compressed files for a regular expression
lzfgrep (1)          - search compressed files for a regular expression
lzgrep (1)           - search compressed files for a regular expression
manpath (1)          - determine search path for manual pages
res_nsearch (3)      - resolver routines
res_search (3)       - resolver routines
strpbrk (3)          - search a string for any of a set of bytes
tsearch (3)          - manage a binary tree
wcschr (3)           - search a wide character in a wide-character string
wcscspn (3)          - search a wide-character string for any of a set of wide characters
wcspbrk (3)          - search a wide-character string for any of a set of wide characters
wcsrchr (3)          - search a wide character in a wide-character string
wmemchr (3)          - search a wide character in a wide-character array
wx-config (1)        - wxWidgets configuration search and query tool
XtFindFile (3)       - search for a file using substitutions in the path list
XtResolvePathname (3) - search for a file using standard substitution
xzegrep (1)          - search compressed files for a regular expression
xzfgrep (1)          - search compressed files for a regular expression
xzgrep (1)           - search compressed files for a regular expression
zegrep (1)           - search possibly compressed files for a regular expression
zfgrep (1)           - search possibly compressed files for a regular expression
zgrep (1)            - search possibly compressed files for a regular expression
zipgrep (1)          - search files in a ZIP archive for lines matching a pattern

Conclusion

The article introduced the most frequently used command to search for files in linux. I hope you learned something new by reading this article,
and can now fully embrace the power of some the commands listed here. For more information on the commands there is the man pages which will provide more technical
information. The ls command alone has over 70 options.