DEV Community: 0xkniraj

Vibe Coding: The $2B in Breaches Nobody's Talking About

0xkniraj — Wed, 18 Feb 2026 08:49:40 +0000

I genuinely don't know whether to laugh or cry when I see another startup founder on my feed bragging about building an entire SaaS platform before their morning coffee. The hype is deafening. Last year, "Vibe Coding" actually became the Collins Dictionary's word of the year.

If you trace this back, the cultural tipping point hit in early 2025 when Andrej Karpathy tweeted about a new way of working where you "fully give in to the vibes" and "forget that the code even exists". The whole premise is that you just write some natural language prompts, surrender control to the machine, and judge the output by how the app feels.

It sounds amazing. I admit it's intoxicating to watch a tool like Replit Agent spin up a full-stack application from a single design file. Take Rokt, for example. They reportedly built 135 internal applications in a single 24-hour window using these tools.

But underneath the shiny demos, treating English like a programming language and abandoning architectural oversight is professional malpractice. We are essentially building digital skyscrapers out of styrofoam.

The Metrics of Decay

Let's look at what actually happens when you let a large language model guess its way through your architecture. Independent research from GitClear and Veracode paints a pretty ugly picture of where our codebases sit in 2026:

Code churn has spiked by 41 percent. Developers are writing and deleting code within a two-week window at alarming rates because they are just blindly iterating through AI hallucinations.

Intentional refactoring has completely collapsed, dropping by 60 percent. AI agents have a terrible habit of solving problems by stacking more logic on top of the pile rather than simplifying anything.

Code duplication is up 48 percent because agents just copy logic from one file to another without understanding how to abstract it into shared utilities. On top of this, a staggering 45 percent of AI-generated code contains security flaws.

This isn't efficient iteration. It's just people pasting errors back into a chat window until the red squiggly lines go away.

The Security "SaaSpocalypse"

Spaghetti code is just an annoyance for the next developer who inherits it. The real issue is the security fallout. Wall Street is already calling this market correction the "SaaSpocalypse".

Right now, one out of every five security breaches in 2026 traces directly back to code an AI wrote. Remember the OpenClaw incident? That marketplace ended up hosting over 900 malicious agent "skills". Developers who were too busy "vibing" just installed these packages without looking. Attackers walked away with remote code execution on thousands of enterprise servers.

The bad actors aren't typing out manual exploits anymore either. We've got hackers using jailbroken versions of autonomous agents to probe networks. In one case, a single agent targeted 17 organizations simultaneously to find databases and write custom exploits. It did all this without a human lifting a finger.

The "Alien Code" Hangover and the Junior Dev Crisis

Here's what really gets me. It's the loss of the mental model. If you build a system without ever reading the code, you are completely helpless when it breaks.

We are creating a generation of app assemblers who don't know how to fix a memory leak or debug a race condition. Gartner is already warning about "AI Lock-In," where organizations get stuck depending on the AI because the codebase has become completely alien to human eyes.

We are also actively destroying the pipeline that fixes this. Tech hiring for pure junior roles has dropped by nearly 30 percent since 2023 because companies are letting AI handle the grunt work. If juniors never struggle with boilerplate and minor bugs, how do they develop the architectural intuition to become seniors? Where are the senior engineers of 2030 supposed to come from?

Time to Grow Up: Agent-Driven Development

We need to get back to actual engineering. The adults in the room are shifting to Agent-Driven Development (ADD). In this model, humans manage the architecture and enforce strict governance over what these bots can actually do.

This means moving away from the "chat" interface and adopting structured frameworks like the BMAD Method. Instead of telling a single bot to build a website, you orchestrate a specialized team. You have an analyst bot research and output a project brief. A product manager bot translates that into a rigorous requirements document. An architect bot drafts the API specifications and data flow. Finally, developer and QA bots only write and test code after a human reviews the architecture.

In this workflow, the true "source code" isn't the Python or JavaScript file anymore. It is the spec and the prompt. Throw in AgentOps layers with "Policy as Code" to physically block agents from introducing circular dependencies or leaking secrets, and you finally have a system you can trust.

The code itself might be a cheap byproduct now. We might even reach a point where software is just generated on the fly for a specific user and immediately discarded. Until then, the intent, the architecture, and the security boundaries still matter. We really need to start acting like it.

Taproot part1: Generating P2TR KeyPath Witness Programs 🚀

0xkniraj — Mon, 19 Jan 2026 19:48:52 +0000

Welcome back, dear readers! If you’re here, it means you’ve survived the setup and you're ready to get your hands dirty with some actual Bitcoin internal magic.

In my previous log, we got our Signet nodes up, running, and stealthily "cloaked" for the BOSS (Bitcoin Open Source Software) Challenge. Now, it’s time for the first real trial: Challenge 1 - Recover Wallet State.

The quest? Compute your wallet balance by scanning the blockchain manually. The catch? The evaluation environment has the Bitcoin Core wallet disabled. No getbalance, no shortcuts. Just you, your code, and the raw blocks. Let’s go! ⚡️

🛠 Step 0: The "Sanity Check" (CLI)

Before we start automating, we need a "North Star" to make sure we aren't chasing ghosts. We'll use the Bitcoin CLI to see what our balance should be. This is your benchmark!

1. Create your local wallet:

docker exec bitcoin_signet bitcoin-cli -signet -datadir=/home/bitcoin/.bitcoin createwallet "boss" false false "" false true true

2. Import that descriptor:

docker exec bitcoin_signet bitcoin-cli -signet -datadir=/home/bitcoin/.bitcoin -rpcwallet=boss importdescriptors '[{ "desc": "tr(tprv8ZgxMBicQKsPfCkDmSmHXuKAVA8zwwXLWTHEgyyisMioZ8gn3nTmmNwUzBMKWnNMkhRYN3E9qVCZrZ6MC645cqYdpy9jTYHhBwG1KhpNAMd/86h/1h/0h/0/*)#hjz07900", "timestamp": 0, "active": true }]'

3. Peek at the results:

docker exec bitcoin_signet bitcoin-cli -signet -datadir=/home/bitcoin/.bitcoin -rpcwallet=boss getwalletinfo

If you see a balance, BOOM! 💥 You know what number you’re aiming for. Now let’s write the code to find it manually.

🧬 The Deep Dive: P2TR Derivation

To find our coins, we need the Witness Program. In the land of Taproot (BIP 341), the address is derived from a "tweaked" public key. Even if we’re just using the Key Path (no complex scripts), we have to commit to a script path.

The "Golden" Benchmarking Script

We’re using bitcoinjs-lib and tiny-secp256k1 to handle the heavy elliptic curve math.

CRITICAL NOTE: You cannot use this JavaScript program for your final submission! The BOSS challenge strictly requires Python, C, C++, or Rust. Use this JS script as a benchmark to verify your logic. If your Python/Rust code generates the same strings as this script, you’re on the right track!

import { BIP32Factory } from "bip32";
import ECPairFactory from "ecpair";
import * as ecc from "tiny-secp256k1";
import * as bitcoin from "bitcoinjs-lib";
import fs from "fs";

// Initialize the ECC library—Taproot needs this!
bitcoin.initEccLib(ecc);

export const bip32 = BIP32Factory(ecc);
export const ECPair = ECPairFactory(ecc);

const net = bitcoin.networks.testnet;
const xpriv = bip32.fromBase58(
  "tprv8ZgxMBicQKsPfF4wki3EzsSYvohmC5z2B92m74LscyiFAHwZUtn4Lhbfosc1zCQRzN4aXiPqiH64xbTbsnRCgTskJWspELwAYSdyZSekqyp",
  net,
);

// Derive the path from the descriptor: m/86'/1'/0'/0
const derived = xpriv.derivePath("m/86'/1'/0'/0");
const addresses = [];
const internals = [];
const witnesses = [];

console.log("🚀 Starting derivation for 2000 keys...");

for (let i = 0; i < 2000; i++) {
  const current = derived.derive(i);

  // Extract the X-only internal pubkey (32 bytes)
  const internalPubkey = current.publicKey.slice(1, 33);

  // Generate the P2TR payment—bitcoinjs handles the TapTweak automatically!
  const { address, pubkey } = bitcoin.payments.p2tr({
    internalPubkey,
    network: net,
  });

  addresses.push(address);
  witnesses.push(pubkey?.toHex()); // This is your Witness Program!
  internals.push(internalPubkey.toHex());
}

// Write to files so you can 'diff' against your main submission code
fs.writeFileSync("addresses.json", JSON.stringify(addresses, null, 2));
fs.writeFileSync("witnesses.json", JSON.stringify(witnesses, null, 2));
fs.writeFileSync("internals.json", JSON.stringify(internals, null, 2));

console.log("✅ Done! Check your JSON files and compare them with your challenge code.");

🎯 Why This is Your Secret Weapon

The BOSS Challenge requires you to:

Derive 2000 keys.
Compute the P2TR witness program for each.
Scan the first 300 blocks of the signet chain.

The Witness Program (the pubkey in the code above) is the actual hex string you’ll be looking for inside the scriptPubKey of transaction outputs on-chain.

Doing this math manually in Python or Rust can be tricky—using this JS script ensures you aren't debugging your scanner when the problem is actually your derivation!

⏭ What's Next?

Now that you have your list of 2000 witness programs in witnesses.json, it's time to go hunting! Grab your blocks, iterate through the transactions, and start tallying that balance.

Are you smashing the challenge in Rust or Python? Drop a comment below and let's talk TapTweak math! 🚀🔥

The ₿OSS Challenge: Cloaking My Signet Node for the Wallet challenge

0xkniraj — Wed, 14 Jan 2026 20:25:37 +0000

Hello, dear readers! It has been some time, but I greet you once more from the pages of my logbook — the record of my grandest technical adventures yet!

It is I, your humble explorer of the digital frontier. These days, I’ve traded my traveling cloak for a terminal window as I embark on a rigorous new expedition: I have officially stepped up as a challenger of the ₿OSS (Bitcoin Open Source Software) 2026!

My journey began in the year 2139—or at least, the version of it found in the Saving Satoshi RPG. With my trusted Holocat companion (that brilliant, nose-boopable AI guide from Chaincode Labs) by my side, I’ve been decoding genesis blocks and mastering elliptic curve cryptography to save the network from a standstill. Having "Saved Satoshi," I am now venturing into the practical world of running Bitcoin node.

Today, I’m sharing the "architectural scrolls" for a setup that is essential for any ₿OSS challenger: running a Bitcoin Signet node entirely over Tor using Docker.

The Master Blueprint: Docker Compose

To begin this ritual of deployment, you must prepare your docker-compose.yaml. This file summons the Bitcoin Core spirit and its silent Tor guardian.

services:
  bitcoin-core:
    image: bitcoin/bitcoin:latest
    container_name: bitcoin_signet
    command:
      - -signet=1        # Directs the node to the Signet testing realm
      - -printtoconsole
      - -rpcallowip=0.0.0.0/0
      - -rpcbind=0.0.0.0
      - -txindex=1
      - -zmqpubhashblock=tcp://0.0.0.0:28332
      - -zmqpubhashtx=tcp://0.0.0.0:28333

      # Tor Configuration
      - -proxy=tor:9050 # Route outgoing traffic through Tor
      - -listen=1 # Accept incoming connections
      - -bind=0.0.0.0 # Bind to all interfaces inside the container
      - -onion=tor:9050 # Use Tor for .onion addresses
      - -discover=1 # Try to self-discover the onion address
      - -torcontrol=tor:9051

    ports:
      - "38332:38332" # RPC port
      - "18444:18444" # P2P port
      - "28332:28332" # ZMQ block notifications
      - "28333:28333" # ZMQ transaction notifications
    volumes:
      - ./bitcoin-data:/home/bitcoin/.bitcoin
      - ./tor-data:/var/lib/tor:ro # Shared volume for the auth cookie
    restart: unless-stopped
    networks:
      - bitcoin-network
    depends_on:
      - tor

  tor:
    image: osminogin/tor-simple:latest
    container_name: tor_proxy
    user: "0:0" 
    environment:
      - TOR_CONTROL_PORT=9051
    volumes:
      - ./torrc:/etc/tor/torrc:ro
      - ./tor-data:/var/lib/tor # Shared volume for the cookie
    restart: unless-stopped
    networks:
      - bitcoin-network

networks:
  bitcoin-network:
    driver: bridge

The Secret Instructions: torrc

As every ₿OSS knows, the proxy needs specific rules to function. Create a file named torrc in your directory and whisper these lines into it:

# SOCKS5 Proxy
SocksPort 0.0.0.0:9050

# Control Port
ControlPort 0.0.0.0:9051

# Use Cookie instead of Password
CookieAuthentication 1
CookieAuthFileGroupReadable 1
DataDirectory /var/lib/tor

Entering the Private Signet Challenge

Now, listen closely, for this is the step that separates a standard traveler from a true challenger! If you are like me and taking on the ₿OSS 2026 trials, you need to point your node toward the private signet used for our tasks.

Simply copy your challenge configuration into the data directory we created:

cp config/bitcoin.conf bitcoin-data/bitcoin.conf

By doing this, you've shared the secret "handshake" (the signetchallenge hex) with your node. Now, when you run your containers, you aren't just on any network but you are officially using the ₿OSS private signet!

Proof of Success: The "getnetworkinfo" Reveal

Once you’ve cast the docker-compose up -d spell, it’s time to verify your work. I used the following command to check if my node was truly hidden behind the Tor cloak.

Witness the results of my successful incantation! My node has successfully manifested its own .onion address and is reaching out to the world through the proxy:

❯ docker exec bitcoin_signet bitcoin-cli -signet -datadir=/home/bitcoin/.bitcoin getnetworkinfo
{
  "version": 280100,
  "subversion": "/Satoshi:28.1.0/",
  "networks": [
    {
      "name": "ipv4",
      "reachable": true,
      "proxy": "192.168.97.2:9050"
    },
    {
      "name": "onion",
      "reachable": true,
      "proxy": "192.168.97.2:9050"
    }
  ],
  "localaddresses": [
    {
      "address": "<your address>.onion",
      "port": 38333,
      "score": 4
    }
  ]
}

What does this tell us?

Reachable: true: Our node can see the network!
Proxy: Notice the IP 192.168.97.2? That is the internal Docker address of our tor_proxy service.
Localaddresses: That long string ending in .onion is our node's secret identity. It’s how other peers can find us without ever knowing our real-world location.

Final Thoughts from the Digital Frontier

Manifesting a node like this is just the first boss fight in the ₿OSS 2026 challenge. By using Signet, we can experiment with transactions without losing real sats, and by using Tor, we stay true to the cypherpunk spirit of privacy.

As the Holocat says, "Bitcoin is serious, but learning about it doesn't have to be!" I’m off to tackle the next chapter of my developer training. There are more blocks to verify and more code to write!

Tailwind CSS Lays Off 75% of Engineering Team as AI Tools Disrupt Revenue Model

0xkniraj — Thu, 08 Jan 2026 18:54:59 +0000

Tailwind Labs has laid off 75% of its engineering team as revenue has plummeted 80%, founder Adam Wathan revealed in a GitHub comment. The company blames AI-powered development tools for breaking the conversion funnel that drove sales of its paid products.

The Announcement

The layoffs became public on January 6 when Wathan responded to a community pull request that would have made Tailwind's documentation more accessible to AI language models. In closing the request, he disclosed the layoffs and explained the underlying business crisis.

"The reality is that 75% of the people on our engineering team lost their jobs here yesterday because of the brutal impact AI has had on our business," Wathan wrote on GitHub PR #2388.

He revealed that site traffic has dropped 40% since early 2023, while revenue has collapsed by approximately 80%—despite Tailwind CSS being "more popular than ever."

The Business Model Problem

Tailwind CSS itself is free and open-source under the MIT license. The company generates revenue by selling complementary paid products:

Tailwind UI: Pre-built component templates and UI examples
Catalyst: An application UI kit
Tailwind Insiders: A subscription program with early access to features

The revenue model relied on a simple conversion funnel: developers visiting the free documentation would see promotions for these paid products, and a percentage would become paying customers.

According to Wathan, this mechanism has been severely disrupted by AI coding assistants.

The AI Disruption

Tools like Cursor, GitHub Copilot, Claude Code, and ChatGPT now allow developers to access Tailwind documentation content without visiting tailwindcss.com. When a developer asks an AI how to implement a Tailwind feature, the AI provides the answer directly—meaning the developer never sees the website where paid products are advertised.

"Tailwind is growing faster than it ever has and is bigger than it ever has been, and our revenue is down close to 80%," Wathan explained. "Right now there's just no correlation between making Tailwind easier to use and making development of the framework more sustainable."

The result is a paradox where increased usage through AI tools correlates with decreased revenue, inverting traditional software business economics.

The Trigger: A Rejected Pull Request

The disclosure came in response to a pull request from developer quantizor proposing to add an /llms.txt endpoint—an emerging standard for serving documentation optimized for language model consumption. The PR would have concatenated all 185 Tailwind documentation files into a single text file, stripped of HTML and optimized for AI parsing.

Wathan rejected the proposal, arguing that making documentation even more accessible to AI systems would accelerate the traffic decline and worsen the revenue situation.

"And making it easier for LLMs to read our docs just means less traffic to our docs which means less people learning about our paid products and the business being even less sustainable," he wrote.

Wathan added that he would like to implement the feature eventually, but needs to solve the monetization problem first: "I really want to figure out a way to offer LLM-optimized docs that don't make that situation even worse (again we literally had to lay off 75% of the team yesterday)."

Community Response: Sharply Divided

The thread, which accumulated 95 comments before being locked, revealed sharp divisions in the developer community.

Support for Wathan's position:
His comment explaining the layoffs received over 1,000 positive reactions and 1,500 hearts, with many developers expressing sympathy for the difficult business situation and acknowledging the legitimacy of the crisis.

"Adam has explained that to you so clearly, you can choose to listen or not," wrote one commenter. "I'd like Tailwind Labs to continue working on Tailwind, would you?"

Criticism of the decision:
Others argued that rejecting the PR was short-sighted and anti-open-source. The PR author's responses were heavily downvoted, receiving over 590 negative reactions on some comments.

"In general I object to the spirit of closing this. It's very OSS unfriendly and would not meaningfully reduce traffic to the docs by humans that actually would buy the product," quantizor wrote.

Some critics questioned whether the decision made business sense, arguing that AI tools already access documentation through scraping and that an official endpoint wouldn't meaningfully change traffic patterns.

The Sponsorship Controversy

The debate intensified when community members discovered that Tailwind's sponsorship program includes access to an "AGENTS.md" file containing guidance for optimizing LLM interactions with Tailwind.

Some interpreted this as Tailwind monetizing LLM-friendly documentation while refusing to provide it freely, though Wathan disputed this characterization.

"I don't see the AGENTS.md stuff we offer as part of the sponsorship program as anything similar to this at all — that's just a short markdown file with a bunch of my own personal opinions and what I consider best practices to nudge LLMs into writing their Tailwind stuff in a specific way. It's not the docs at all," he explained.

Technical Context: The llms.txt Debate

The /llms.txt standard is an emerging convention, similar to robots.txt, designed to provide AI-optimized documentation in a standardized format. However, its adoption remains unclear.

John Mueller, a Senior Search Analyst at Google, stated in June 2025: "FWIW no AI system currently uses llms.txt."

Major LLM providers including OpenAI, Google, and Anthropic have not officially committed to following the standard. However, these systems already access and cache documentation through web crawling, meaning the lack of an official llms.txt endpoint doesn't prevent AI access to Tailwind's docs.

Where Major Companies Stand

Several prominent companies heavily use Tailwind CSS in their products, according to usage statistics cited in the thread:

Claude.ai (Anthropic)
Vercel
Cloudflare
X (formerly Twitter)
Shopify
Cursor

The thread included criticism that these companies benefit from Tailwind without providing financial support through sponsorship or other means.

"It's insane how tailwind is utilized on like, websites of all those big companies but ain't no one sponsoring or giving back anything," one commenter wrote, calling for corporate support, particularly from Vercel.

Broader Implications for Open Source

The Tailwind crisis has sparked broader discussion about open-source sustainability in the AI era. Traditional open-source monetization strategies—including the "discovery through documentation" model—appear vulnerable to disruption by AI intermediation.

Other documentation-dependent projects may face similar challenges, particularly those in the developer tools space where AI coding assistants are rapidly gaining adoption.

Some community members called for systemic solutions, including:

Paywall systems for AI crawlers: Cloudflare has announced plans for a "pay-per-crawl" product that could charge LLM companies for documentation access
Legal frameworks: Proposals for "GPL for AI" licenses that restrict AI training on documentation
Corporate sponsorship: Pressure on companies using open-source tools to fund their development
New business models: Moving away from documentation-based discovery toward direct enterprise relationships

Wathan's Current Focus

In his comments, Wathan emphasized that his immediate priority is business survival rather than community features.

"Have more important things to do like figure out how to make enough money for the business to be sustainable right now," he wrote. "Every second I spend trying to do fun free things for the community like this is a second I'm not spending trying to turn the business around and make sure the people who are still here are getting their paychecks every month."

He acknowledged the value of the proposed feature but said the company couldn't prioritize it: "Just don't have time to work on things that don't help us pay the bills right now, sorry. We may add this one day but closing for now."

After the thread became heated, with some comments marked as violating GitHub's acceptable use policies, Wathan locked the discussion with a final message: "Going to lock this one as it's spiraling a bit. Appreciate the support from everyone ❤️ We'll figure it out!"

What's Next for Tailwind

The company has not announced specific plans for addressing the revenue crisis. Potential paths forward could include:

Implementing paywalls for AI crawler access if such systems become available
Pivoting to enterprise licensing models
Seeking venture capital or foundation support
Finding ways to embed product promotion in AI-consumed documentation
Restructuring the business to operate with a smaller team

The framework itself remains open-source and available, though questions about long-term maintenance have arisen given the staffing reduction.

Industry Response Pending

As of publication, major AI companies including OpenAI, Anthropic, and Google have not commented on the situation or announced any programs to fund open-source projects whose documentation their models consume.

The incident has reached the front page of Hacker News and is being discussed widely in developer communities, suggesting it may become a test case for how the industry handles open-source sustainability in the age of AI.

This article is based on publicly available GitHub comments and community discussions.

Sources:

GitHub PR #2388: tailwindlabs/tailwindcss.com
Adam Wathan's comments on GitHub (January 6-8, 2026)
Hacker News discussions
SEMrush analysis of llms.txt adoption

Building Infrastructure for Handling Millions of Bitcoin UTXOs at Scale

0xkniraj — Mon, 05 Jan 2026 08:21:25 +0000

The Challenge: Account Model vs UTXO Model

If you've ever designed a custodian solution for Ethereum, you might have found it surprisingly straightforward. Creating wallets from mnemonics, managing deposits and withdrawals, and even implementing multisig solutions using Safe (onchain) or MPC/TSS (offchain) are relatively simple tasks. This ease stems from Ethereum's account model, which treats balances as fungible assets in individual accounts.

Bitcoin, however, operates fundamentally differently due to its UTXO (Unspent Transaction Output) architecture. This creates unique challenges for custodians managing Bitcoin at scale.

Understanding the UTXO Model

In Bitcoin's UTXO model:

Every deposit requires a unique address (locking script)
Money behaves like physical currency notes rather than a simple account balance
UTXOs are atomic units - when you spend one, you must spend it entirely
Partial spending requires change - similar to receiving change when paying with cash

While Bitcoin is certainly fungible, the mechanics of how it works differ significantly from Ethereum. Think of UTXOs as individual checks: if you want to spend a check, you must spend it in full. To spend a partial amount, you write a new check back to yourself for the change.

This creates a unique challenge for custodians: efficiently managing potentially millions of "checks" of varying amounts.

The Scale Problem: Managing Millions of UTXOs

Imagine managing millions of UTXOs across thousands of user deposits, processing thousands of withdrawals daily. When a user requests a withdrawal of 0.1 BTC, how do you efficiently select which UTXOs to spend?

This isn't just a theoretical problem—it's the core challenge for exchanges, bridges, and payment processors handling Bitcoin at scale.

Why Standard Algorithms Fail at Scale

Approach 1: Branch and Bound (Bitcoin Core's Method)

Bitcoin Core uses the Branch and Bound algorithm for optimal UTXO selection:

def branch_and_bound(utxos, target_amount, fee_rate):
    # Recursively search for exact match combinations
    # Time complexity: O(2^n) with pruning
    # Works well for wallet-sized UTXO sets (hundreds to thousands)
    pass

Why this fails at scale:

Bitcoin Core wallets: 100-1,000 UTXOs
Custodian platform: 10,000,000+ UTXOs
At millions of UTXOs: O(2^n) becomes computationally impossible
Even with aggressive pruning and timeouts, selection would take seconds or minutes
Would create a denial-of-service vulnerability under load

The fundamental insight: We need O(log n) or O(1) selection, not O(2^n).

Approach 2: The Greedy Algorithm

The simplest approach is to find the largest UTXO and use it:

withdrawal_amount = 0.1 BTC
selected_utxo = max(utxos)  // e.g., 0.5 BTC
change = 0.4 BTC

Problems:

Creates severe UTXO fragmentation over time
Eventually, you'll have many small UTXOs
Large withdrawals become difficult as UTXOs get smaller

Approach 3: Binary Search on Sorted UTXOs

Sort UTXOs and use binary search to find an optimal match:

sorted_utxos = sort(utxos, ascending)
selected_utxo = binary_search(sorted_utxos, withdrawal_amount)

Problems:

Works reasonably well for small withdrawals
Fails when the requested amount exceeds the largest UTXO
Example: User requests 5 BTC, but largest UTXO is 2 BTC

Approach 4: Consolidation (Combining Smallest UTXOs)

Combine multiple small UTXOs to fulfill the withdrawal:

selected_utxos = []
total = 0
for utxo in sorted(utxos, ascending):
    selected_utxos.append(utxo)
    total += utxo.amount
    if total >= withdrawal_amount:
        break

Problems:

More inputs = larger transaction size = higher fees
Can become prohibitively expensive
Doesn't scale practically for regular operations

The Solution: bucket-Based Cascading Selection

None of the approaches above work efficiently at scale. The solution combines the cascading greedy approach with strategic consolidation, mimicking how physical currency denominations work.

How Physical Currency Works

When paying $50 in cash, you might use:

5×$10 notes
1×$50 note
1×$100 note and receive $50 change

The choice depends on what denominations you have available. We can apply this same principle programmatically to Bitcoin UTXOs.

Implementation: Gaussian Distribution Buckets

Step 1: Define UTXO Tiers

TIER_HIERARCHY = [
    {"name": "dust",   "min": 0.0001,  "max": 0.001},   # 10k-100k sats
    {"name": "tiny",   "min": 0.001,   "max": 0.005},   # 100k-500k sats
    {"name": "small",  "min": 0.005,   "max": 0.01},    # 500k-1M sats
    {"name": "medium", "min": 0.01,    "max": 0.02},    # 1M-2M sats
    {"name": "large",  "min": 0.02,    "max": 0.05},     # 2M-5M sats
    {"name": "xlarge", "min": 0.05,     "max": 0.2},     # 5M-20M sats
    {"name": "whale",  "min": 0.2,     "max": None},    # 20M+ sats
]

Tier ranges are not arbitrary - they're derived from:

Historical withdrawal distribution analysis
Fee optimization modeling
Consolidation opportunity windows
Minimum relay fee and dust thresholds

Why this works at scale:

Reduces search space from millions to constant T tiers
Each tier independently manageable
Enables efficient database indexing
Natural distribution boundaries

Step 2: Tier Cascading Strategy

Here's the critical difference from naive approaches—we don't just pick a tier, we **cascade through tiers intelligently

def find_utxos_for_withdrawal(amount):
    """
    Tier cascading: Start at preferred tier, cascade UP to larger tiers
    Key insight: Cascading UP prevents fragmentation
    """
    # Determine which tier this amount naturally belongs to
    preferred_tier = determine_tier(amount)

    # Start from preferred tier and go UP through larger tiers
    tier_index = TIER_HIERARCHY.index(preferred_tier)

    for i in range(tier_index, len(TIER_HIERARCHY)):
        current_tier = TIER_HIERARCHY[i]

        # Check if tier has sufficient total balance
        tier_balance = get_tier_balance(current_tier)

        if tier_balance >= amount:
            # Try to select UTXOs from this tier
            utxos = select_utxos_in_tier(current_tier, amount)

            if utxos:
                return utxos
            # If selection fails, continue to next tier

    raise InsufficientFundsError("No tier has sufficient balance")

Why cascade UP, not DOWN:

Example: 0.03 BTC withdrawal

Cascading DOWN (naive approach):
medium (0.01-0.05) → small (0.005-0.01) → tiny → dust
Result: Many small UTXOs = fragmentation + high fees

Cascading UP (our approach):
medium (0.01-0.05) → large (0.05-0.1) → xlarge
Result: Fewer larger UTXOs = consolidation + low fees

Step 3: UTXO Selection Within Tier

Once we've selected a tier, we need to pick specific UTXOs. This is where the knapsack problem comes in:

def select_utxos_in_tier(tier, amount, max_inputs=5):
    """
    Within a tier, select the smallest set of UTXOs that covers the amount.
    This is a bounded knapsack approximation.
    """
    # Get available UTXOs in tier, sorted by amount
    available_utxos = get_available_utxos(tier, sort_by="amount_asc", limit_by=max_inputs)

    # Greedy knapsack: Select smallest UTXOs first
    selected = []
    total = 0

    for utxo in available_utxos:    
        selected.append(utxo)
        total += utxo.amount

        if total >= amount:
            # Atomically reserve these UTXOs in database
            reserve_utxos(selected)
            return selected

    # Tier doesn't have enough UTXOs
    return None

Why smallest-first selection?

Minimizes change output size
Naturally consolidates small UTXOs
Cleans up tier fragmentation over time

Note on implementation: In production, this is implemented entirely within PostgreSQL using:

SELECT FOR UPDATE SKIP LOCKED for concurrency-safe selection
Atomic transactions for reservation
Database-level constraints for consistency
Sub-millisecond selection performance with proper indexing

Step 4: Fee-Aware Tier Selection

At scale, transaction fees become a significant cost center. The tier selection should factor in current fee rates:

def find_utxos_with_fee_awareness(amount, fee_rate_sats_per_vb):
    """
    Adjust tier selection based on current fee market conditions
    """
    preferred_tier = determine_tier(amount)
    tier_index = TIER_HIERARCHY.index(preferred_tier)

    # During high-fee periods, start from a higher tier
    # This reduces input count = lower fees
    if fee_rate_sats_per_vb > 50:  # High fee period
        # Skip to next larger tier to minimize inputs
        tier_index = min(tier_index + 1, len(TIER_HIERARCHY) - 1)

    # Rest of cascading logic...

Real-world impact:

Scenario: 0.03 BTC withdrawal during 100 sat/vB fee spike

Option A: 3 UTXOs from medium tier
Fee: 3 inputs × 68 vB × 100 sat/vB = 20,400 sats ($20.40)

Option B: 1 UTXO from large tier (fee-aware selection)
Fee: 1 input × 68 vB × 100 sat/vB = 6,800 sats ($6.80)

Savings: $13.60 per withdrawal during peak hours

Why This Approach Works at Scale

Computational Complexity

Approach	Time Complexity	Scale Limit
Branch and Bound	O(2^n)	< 1,000 UTXOs
Binary Search	O(log n)	Fails on large amounts
Tier Cascading	O(tiers) + O(k*log(k))	Millions of UTXOs

Where:

n = total UTXOs
tiers = number of tiers (constant: 7)
k = UTXOs selected per transaction (typically 2-3)

Our approach: O(7) + O(3*log(3)) = O(1) effectively

Natural Consolidation Mechanics

The smallest-first selection within tiers creates automatic consolidation:

Example:

Medium tier has 1000 UTXOs
Withdrawal: 0.03 BTC
Selection: Uses 3 smallest UTXOs (0.01, 0.011, 0.012)
Result: 1000 → 997 UTXOs, naturally cleans up small values

This is self-healing - no manual consolidation jobs required during normal operations. Tiers gets replenished by new deposits.

Gaussian Distribution Alignment

Real-world withdrawal patterns follow a log-normal distribution (similar to Gaussian):

Distribution of withdrawals (from production data):
- 10th percentile: 0.002 BTC (tiny tier)
- 50th percentile: 0.02 BTC (medium tier)  ← most common
- 90th percentile: 0.15 BTC (xlarge tier)
- 99th percentile: 0.5 BTC (whale tier)

By sizing tiers around this distribution, we optimize for the common case while handling edge cases efficiently.

Operational Considerations

Tier Health Monitoring

You need visibility into tier distribution to maintain system health:

def monitor_tier_health():
    """
    Track tier metrics for operational alerting
    """
    for tier in TIER_HIERARCHY:
        metrics = {
            "utxo_count": count_utxos(tier),
            "total_balance": sum_balance(tier),
            "avg_utxo_size": average_size(tier),
            "daily_utilization": withdrawals_from_tier(tier, days=1)
        }

        # Alert on depletion risk
        if metrics["utxo_count"] < 10:
            alert(f"Tier {tier['name']} depleting: {metrics['utxo_count']} UTXOs left")

        # Alert on imbalance
        runway_days = metrics["total_balance"] / metrics["daily_utilization"]
        if runway_days < 2:
            alert(f"Tier {tier['name']} has <2 days runway")

Strategic Rebalancing

While the system is self-healing under normal operation, you need proactive rebalancing during quiet periods:

def rebalance_tiers_during_low_fees():
    """
    Run during low-fee periods (typically 2-6 AM UTC, weekends)
    Consolidate dust, split whales, rebalance distribution
    """
    current_fee_rate = get_current_fee_rate()

    # Only rebalance when fees are low
    if current_fee_rate > 10:  # sats/vB
        return

    # Strategy 1: Consolidate dust and tiny tiers
    dust_utxos = get_tier_utxos("dust")
    if len(dust_utxos) > 100:
        consolidate_into_tier(dust_utxos, target_tier="small")

    # Strategy 2: Ensure minimum UTXO count per tier
    for tier in TIER_HIERARCHY:
        if count_utxos(tier) < 20:
            create_utxos_for_tier(tier, target_count=50, source_tier="whale")

When to rebalance:

Daily during off-peak hours
After major withdrawal spikes
When tier depletion alerts trigger
During extended low-fee periods (accumulate savings)

Database Implementation Notes

The algorithm is implemented entirely in PostgreSQL for:

Atomicity: BEGIN/COMMIT/ROLLBACK blocks ensure consistency
Concurrency: SELECT FOR UPDATE SKIP LOCKED handles concurrent withdrawals
Performance: Indexed queries on (tier, state, amount) enable sub-millisecond selection
Reliability: ACID guarantees prevent double-spending

This enables the database to instantly find the smallest available UTXOs in any tier.

Real-World Performance

In production environments processing thousands of withdrawals daily:

Average inputs per transaction: 2.1 UTXOs (down from 4.5 with naive approaches)
Fee reduction: 55% vs. simple consolidation approach
UTXO selection latency: <5ms (database-level selection)
Tier cascade events: <2% of withdrawals (most succeed on preferred tier)
Natural consolidation rate: thousands of UTXOs/day consolidated without manual intervention
System uptime during rebalancing: 100% (non-blocking background process)

Cost Impact at Scale:

Platform: 10,000 withdrawals/day
Average fee savings: $2.50/withdrawal
Monthly savings: $750,000
Annual savings: $9,000,000

Limitations and Trade-offs

What This Approach Sacrifices

1. Theoretical Optimality

Branch and Bound finds the perfect UTXO combination
Tier cascading finds a good enough combination
Trade-off: ~5% more change outputs for 10,000x faster selection

2. Privacy Considerations

Selecting multiple UTXOs from same tier can link addresses
Common input ownership heuristic applies
Mitigation: Use mixing services or CoinJoin for privacy-critical applications

3. Minimum Withdrawal Threshold

Dust tier prevents withdrawals < 50,000 sats (~$50)
Necessary to avoid economically unspendable UTXOs
Alternative: Lightning Network for micro-withdrawals

When This Approach May Not Fit

Not suitable for:

Personal wallets (use Bitcoin Core's Branch and Bound)
Privacy-focused applications (use Wasabi/Samourai algorithms)
< 10,000 UTXOs (simpler approaches work fine)

Ideal for:

Exchanges and payment processors
Cross-chain bridges
Custodial services at scale
Any platform managing 1M+ UTXOs

Conclusion

Building a Bitcoin custodian at scale requires fundamentally different thinking than Ethereum-based solutions. The UTXO model, while more complex, can be efficiently managed using tier-based cascading strategies.

Key takeaways:

Standard algorithms don't scale - Branch and Bound is O(2^n), impossible at millions of UTXOs
Tier cascading UP prevents fragmentation while maintaining availability
Fee-aware selection can save millions annually at scale
Natural consolidation keeps the system healthy through smallest-first selection
Database-level implementation provides atomicity and concurrency control
Operational monitoring is critical - track tier health and rebalance proactively

By treating Bitcoin UTXOs like physical currency denominations and selecting them through intelligent tier traversal, custodians can process millions of transactions efficiently while minimizing fees and maintaining system health.

The real innovation isn't just the algorithm—it's designing a system that scales to millions of UTXOs while remaining operationally manageable.

Building Infrastructure for Handling Millions of Bitcoin UTXOs at Scale

0xkniraj — Mon, 05 Jan 2026 08:12:18 +0000

The Challenge: Account Model vs UTXO Model

Bitcoin, however, operates fundamentally differently due to its UTXO (Unspent Transaction Output) architecture. This creates unique challenges for custodians managing Bitcoin at scale.

Understanding the UTXO Model

In Bitcoin's UTXO model:

Every deposit requires a unique address (locking script)
Money behaves like physical currency notes rather than a simple account balance
UTXOs are atomic units - when you spend one, you must spend it entirely
Partial spending requires change - similar to receiving change when paying with cash

This creates a unique challenge for custodians: efficiently managing potentially millions of "checks" of varying amounts.

The Scale Problem: Managing Millions of UTXOs

This isn't just a theoretical problem—it's the core challenge for exchanges, bridges, and payment processors handling Bitcoin at scale.

Why Standard Algorithms Fail at Scale

Approach 1: Branch and Bound (Bitcoin Core's Method)

Bitcoin Core uses the Branch and Bound algorithm for optimal UTXO selection:

def branch_and_bound(utxos, target_amount, fee_rate):
    # Recursively search for exact match combinations
    # Time complexity: O(2^n) with pruning
    # Works well for wallet-sized UTXO sets (hundreds to thousands)
    pass

Why this fails at scale:

Bitcoin Core wallets: 100-1,000 UTXOs
Custodian platform: 10,000,000+ UTXOs
At millions of UTXOs: O(2^n) becomes computationally impossible
Even with aggressive pruning and timeouts, selection would take seconds or minutes
Would create a denial-of-service vulnerability under load

The fundamental insight: We need O(log n) or O(1) selection, not O(2^n).

Approach 2: The Greedy Algorithm

The simplest approach is to find the largest UTXO and use it:

withdrawal_amount = 0.1 BTC
selected_utxo = max(utxos)  // e.g., 0.5 BTC
change = 0.4 BTC

Problems:

Creates severe UTXO fragmentation over time
Eventually, you'll have many small UTXOs
Large withdrawals become difficult as UTXOs get smaller

Approach 3: Binary Search on Sorted UTXOs

Sort UTXOs and use binary search to find an optimal match:

sorted_utxos = sort(utxos, ascending)
selected_utxo = binary_search(sorted_utxos, withdrawal_amount)

Problems:

Works reasonably well for small withdrawals
Fails when the requested amount exceeds the largest UTXO
Example: User requests 5 BTC, but largest UTXO is 2 BTC

Approach 4: Consolidation (Combining Smallest UTXOs)

Combine multiple small UTXOs to fulfill the withdrawal:

selected_utxos = []
total = 0
for utxo in sorted(utxos, ascending):
    selected_utxos.append(utxo)
    total += utxo.amount
    if total >= withdrawal_amount:
        break

Problems:

More inputs = larger transaction size = higher fees
Can become prohibitively expensive
Doesn't scale practically for regular operations

The Solution: bucket-Based Cascading Selection

None of the approaches above work efficiently at scale. The solution combines the cascading greedy approach with strategic consolidation, mimicking how physical currency denominations work.

How Physical Currency Works

When paying $50 in cash, you might use:

5×$10 notes
1×$50 note
1×$100 note and receive $50 change

The choice depends on what denominations you have available. We can apply this same principle programmatically to Bitcoin UTXOs.

Implementation: Gaussian Distribution Buckets

Step 1: Define UTXO Tiers

TIER_HIERARCHY = [
    {"name": "dust",   "min": 0.0001,  "max": 0.001},   # 10k-100k sats
    {"name": "tiny",   "min": 0.001,   "max": 0.005},   # 100k-500k sats
    {"name": "small",  "min": 0.005,   "max": 0.01},    # 500k-1M sats
    {"name": "medium", "min": 0.01,    "max": 0.02},    # 1M-2M sats
    {"name": "large",  "min": 0.02,    "max": 0.05},     # 2M-5M sats
    {"name": "xlarge", "min": 0.05,     "max": 0.2},     # 5M-20M sats
    {"name": "whale",  "min": 0.2,     "max": None},    # 20M+ sats
]

Tier ranges are not arbitrary - they're derived from:

Historical withdrawal distribution analysis
Fee optimization modeling
Consolidation opportunity windows
Minimum relay fee and dust thresholds

Why this works at scale:

Reduces search space from millions to constant T tiers
Each tier independently manageable
Enables efficient database indexing
Natural distribution boundaries

Step 2: Tier Cascading Strategy

Here's the critical difference from naive approaches—we don't just pick a tier, we **cascade through tiers intelligently

def find_utxos_for_withdrawal(amount):
    """
    Tier cascading: Start at preferred tier, cascade UP to larger tiers
    Key insight: Cascading UP prevents fragmentation
    """
    # Determine which tier this amount naturally belongs to
    preferred_tier = determine_tier(amount)

    # Start from preferred tier and go UP through larger tiers
    tier_index = TIER_HIERARCHY.index(preferred_tier)

    for i in range(tier_index, len(TIER_HIERARCHY)):
        current_tier = TIER_HIERARCHY[i]

        # Check if tier has sufficient total balance
        tier_balance = get_tier_balance(current_tier)

        if tier_balance >= amount:
            # Try to select UTXOs from this tier
            utxos = select_utxos_in_tier(current_tier, amount)

            if utxos:
                return utxos
            # If selection fails, continue to next tier

    raise InsufficientFundsError("No tier has sufficient balance")

Why cascade UP, not DOWN:

Example: 0.03 BTC withdrawal

Cascading DOWN (naive approach):
medium (0.01-0.05) → small (0.005-0.01) → tiny → dust
Result: Many small UTXOs = fragmentation + high fees

Cascading UP (our approach):
medium (0.01-0.05) → large (0.05-0.1) → xlarge
Result: Fewer larger UTXOs = consolidation + low fees

Step 3: UTXO Selection Within Tier

Once we've selected a tier, we need to pick specific UTXOs. This is where the knapsack problem comes in:

def select_utxos_in_tier(tier, amount, max_inputs=5):
    """
    Within a tier, select the smallest set of UTXOs that covers the amount.
    This is a bounded knapsack approximation.
    """
    # Get available UTXOs in tier, sorted by amount
    available_utxos = get_available_utxos(tier, sort_by="amount_asc", limit_by=max_inputs)

    # Greedy knapsack: Select smallest UTXOs first
    selected = []
    total = 0

    for utxo in available_utxos:    
        selected.append(utxo)
        total += utxo.amount

        if total >= amount:
            # Atomically reserve these UTXOs in database
            reserve_utxos(selected)
            return selected

    # Tier doesn't have enough UTXOs
    return None

Why smallest-first selection?

Minimizes change output size
Naturally consolidates small UTXOs
Cleans up tier fragmentation over time

Note on implementation: In production, this is implemented entirely within PostgreSQL using:

SELECT FOR UPDATE SKIP LOCKED for concurrency-safe selection
Atomic transactions for reservation
Database-level constraints for consistency
Sub-millisecond selection performance with proper indexing

Step 4: Fee-Aware Tier Selection

At scale, transaction fees become a significant cost center. The tier selection should factor in current fee rates:

def find_utxos_with_fee_awareness(amount, fee_rate_sats_per_vb):
    """
    Adjust tier selection based on current fee market conditions
    """
    preferred_tier = determine_tier(amount)
    tier_index = TIER_HIERARCHY.index(preferred_tier)

    # During high-fee periods, start from a higher tier
    # This reduces input count = lower fees
    if fee_rate_sats_per_vb > 50:  # High fee period
        # Skip to next larger tier to minimize inputs
        tier_index = min(tier_index + 1, len(TIER_HIERARCHY) - 1)

    # Rest of cascading logic...

Real-world impact:

Scenario: 0.03 BTC withdrawal during 100 sat/vB fee spike

Option A: 3 UTXOs from medium tier
Fee: 3 inputs × 68 vB × 100 sat/vB = 20,400 sats ($20.40)

Option B: 1 UTXO from large tier (fee-aware selection)
Fee: 1 input × 68 vB × 100 sat/vB = 6,800 sats ($6.80)

Savings: $13.60 per withdrawal during peak hours

Why This Approach Works at Scale

Computational Complexity

Approach	Time Complexity	Scale Limit
Branch and Bound	O(2^n)	< 1,000 UTXOs
Binary Search	O(log n)	Fails on large amounts
Tier Cascading	O(tiers) + O(k*log(k))	Millions of UTXOs

Where:

n = total UTXOs
tiers = number of tiers (constant: 7)
k = UTXOs selected per transaction (typically 2-3)

Our approach: O(7) + O(3*log(3)) = O(1) effectively

Natural Consolidation Mechanics

The smallest-first selection within tiers creates automatic consolidation:

Example:

Medium tier has 1000 UTXOs
Withdrawal: 0.03 BTC
Selection: Uses 3 smallest UTXOs (0.01, 0.011, 0.012)
Result: 1000 → 997 UTXOs, naturally cleans up small values

This is self-healing - no manual consolidation jobs required during normal operations. Tiers gets replenished by new deposits.

Gaussian Distribution Alignment

Real-world withdrawal patterns follow a log-normal distribution (similar to Gaussian):

Distribution of withdrawals (from production data):
- 10th percentile: 0.002 BTC (tiny tier)
- 50th percentile: 0.02 BTC (medium tier)  ← most common
- 90th percentile: 0.15 BTC (xlarge tier)
- 99th percentile: 0.5 BTC (whale tier)

By sizing tiers around this distribution, we optimize for the common case while handling edge cases efficiently.

Operational Considerations

Tier Health Monitoring

You need visibility into tier distribution to maintain system health:

def monitor_tier_health():
    """
    Track tier metrics for operational alerting
    """
    for tier in TIER_HIERARCHY:
        metrics = {
            "utxo_count": count_utxos(tier),
            "total_balance": sum_balance(tier),
            "avg_utxo_size": average_size(tier),
            "daily_utilization": withdrawals_from_tier(tier, days=1)
        }

        # Alert on depletion risk
        if metrics["utxo_count"] < 10:
            alert(f"Tier {tier['name']} depleting: {metrics['utxo_count']} UTXOs left")

        # Alert on imbalance
        runway_days = metrics["total_balance"] / metrics["daily_utilization"]
        if runway_days < 2:
            alert(f"Tier {tier['name']} has <2 days runway")

Strategic Rebalancing

While the system is self-healing under normal operation, you need proactive rebalancing during quiet periods:

def rebalance_tiers_during_low_fees():
    """
    Run during low-fee periods (typically 2-6 AM UTC, weekends)
    Consolidate dust, split whales, rebalance distribution
    """
    current_fee_rate = get_current_fee_rate()

    # Only rebalance when fees are low
    if current_fee_rate > 10:  # sats/vB
        return

    # Strategy 1: Consolidate dust and tiny tiers
    dust_utxos = get_tier_utxos("dust")
    if len(dust_utxos) > 100:
        consolidate_into_tier(dust_utxos, target_tier="small")

    # Strategy 2: Ensure minimum UTXO count per tier
    for tier in TIER_HIERARCHY:
        if count_utxos(tier) < 20:
            create_utxos_for_tier(tier, target_count=50, source_tier="whale")

When to rebalance:

Daily during off-peak hours
After major withdrawal spikes
When tier depletion alerts trigger
During extended low-fee periods (accumulate savings)

Database Implementation Notes

The algorithm is implemented entirely in PostgreSQL for:

Atomicity: BEGIN/COMMIT/ROLLBACK blocks ensure consistency
Concurrency: SELECT FOR UPDATE SKIP LOCKED handles concurrent withdrawals
Performance: Indexed queries on (tier, state, amount) enable sub-millisecond selection
Reliability: ACID guarantees prevent double-spending

This enables the database to instantly find the smallest available UTXOs in any tier.

Real-World Performance

In production environments processing thousands of withdrawals daily:

Average inputs per transaction: 2.1 UTXOs (down from 4.5 with naive approaches)
Fee reduction: 55% vs. simple consolidation approach
UTXO selection latency: <5ms (database-level selection)
Tier cascade events: <2% of withdrawals (most succeed on preferred tier)
Natural consolidation rate: thousands of UTXOs/day consolidated without manual intervention
System uptime during rebalancing: 100% (non-blocking background process)

Cost Impact at Scale:

Platform: 10,000 withdrawals/day
Average fee savings: $2.50/withdrawal
Monthly savings: $750,000
Annual savings: $9,000,000

Limitations and Trade-offs

What This Approach Sacrifices

1. Theoretical Optimality

Branch and Bound finds the perfect UTXO combination
Tier cascading finds a good enough combination
Trade-off: ~5% more change outputs for 10,000x faster selection

2. Privacy Considerations

Selecting multiple UTXOs from same tier can link addresses
Common input ownership heuristic applies
Mitigation: Use mixing services or CoinJoin for privacy-critical applications

3. Minimum Withdrawal Threshold

Dust tier prevents withdrawals < 50,000 sats (~$50)
Necessary to avoid economically unspendable UTXOs
Alternative: Lightning Network for micro-withdrawals

When This Approach May Not Fit

Not suitable for:

Personal wallets (use Bitcoin Core's Branch and Bound)
Privacy-focused applications (use Wasabi/Samourai algorithms)
< 10,000 UTXOs (simpler approaches work fine)

Ideal for:

Exchanges and payment processors
Cross-chain bridges
Custodial services at scale
Any platform managing 1M+ UTXOs

Conclusion

Key takeaways:

Standard algorithms don't scale - Branch and Bound is O(2^n), impossible at millions of UTXOs
Tier cascading UP prevents fragmentation while maintaining availability
Fee-aware selection can save millions annually at scale
Natural consolidation keeps the system healthy through smallest-first selection
Database-level implementation provides atomicity and concurrency control
Operational monitoring is critical - track tier health and rebalance proactively

The real innovation isn't just the algorithm—it's designing a system that scales to millions of UTXOs while remaining operationally manageable.

10,000 Applicants. 28 Positions. The Brutal Math of Web3 Hiring in 2026.

0xkniraj — Sun, 04 Jan 2026 10:46:51 +0000

The Web3 job market recovered 47% in 2025. So why does landing a role feel harder than ever?

You refresh the job board. Another Solidity position at a promising L2. You've deployed contracts to mainnet, contributed to open-source repos, and spent six months mastering Foundry. You apply within hours of posting.

Three weeks later: silence.

You're not alone. That position likely received 400+ applications before the listing was 48 hours old. The hiring manager skimmed maybe 50 resumes, interviewed 8 candidates, and hired someone with a warm referral from a mutual connection on Crypto Twitter.

Welcome to Web3 hiring in 2026—where the industry added 66,000 jobs last year, yet 450 developers compete for every engineering position.

The numbers don't lie. But they don't tell the whole story either.

The Ground Reality: Why It Feels Impossible

Let's start with the uncomfortable truth. The Web3 job market has fundamentally changed, and most job seekers haven't caught up.

The recovery is real, but selective. Yes, 66,494 new positions emerged in 2025—a 47% rebound from crypto winter. But this recovery favors specific roles, specific skills, and specific candidate profiles. If you're a generalist Solidity developer with a decent portfolio, you're competing in the most saturated talent pool in tech.

The competition ratios are brutal:

Smart contract developers: 450 applicants per position
Frontend Web3 engineers: 300-400 applicants per position
Compliance/regulatory roles: 60-80 applicants per position
Security auditors: 80-120 applicants per position

Read those numbers again. The difference between 450:1 and 60:1 isn't marginal—it's the difference between lottery odds and a real shot.

The "build phase" ended. You missed the memo. Here's what nobody told you: Web3 stopped being about "what can we build?" sometime in late 2024. It's now about "how do we ship, scale, and not get sued?"

Project management now represents 27% of all Web3 job postings—the single largest category. The industry doesn't need another developer who can fork Uniswap. It needs people who can coordinate launches, navigate regulations, and turn existing protocols into actual businesses.

AI changed the math on junior roles. Stanford research found a 20% employment decline in software developers aged 22-25 between 2022-2025. In Web3, this hit harder. When AI can generate a standard ERC-20 token contract in seconds, why hire a junior developer to do it in hours?

Coinbase reports 90% speedup on simple coding tasks using AI tools. That productivity gain came from somewhere—and it came from entry-level headcount.

The Numbers Behind the Despair

I pulled data from every major Web3 job platform, recruitment agency, and industry report. Here's what the market actually looks like:

Total active positions (January 2026): ~8,000-12,000 globally

Estimated active job seekers: 150,000-200,000+

Where the jobs actually are:

Category	% of Postings	Competition Level
Project/Program Management	27%	Moderate
Engineering (all types)	32%	Severe
Operations/Admin	15%	Low-Moderate
Compliance/Legal	8%	Low
Marketing/Community	10%	Moderate
Security/Auditing	5%	Moderate
Research	3%	High

Geographic reality:

United States: 21,612 positions (+26% YoY)
Europe: 10,263 positions (+38% YoY)
Asia: 10,420 positions (32% of global developers now here)
Remote-only positions: Down 50% from 2024

That last stat matters. The "work from anywhere" Web3 dream is fading. Companies want hybrid. They want to see your face occasionally. The arbitrage play of living in Bali on a San Francisco salary is closing.

But Here's Where Hope Lives

Okay, you've absorbed the brutal math. Now let's talk about the paths that actually work—because they exist, and they're more accessible than the competition ratios suggest.

Path 1: Go Where They Aren't

The 450:1 ratio is for smart contract developer positions. But look at what's undersaturated:

Regulatory fluency is becoming an engineering differentiator. MiCA went live in Europe. The SEC is providing clarity in the US. This isn't just creating compliance jobs—it's reshaping what engineering teams need.

RWA (Real World Assets) is the fastest-growing sector in DeFi, with protocols like Ondo, Centrifuge, and Maple Finance racing to tokenize treasuries, real estate, and private credit. These projects don't just need Solidity devs—they need engineers who understand:

How token classifications affect smart contract architecture (security vs. utility vs. commodity)
Why certain on-chain actions trigger reporting requirements
How to build compliant transfer restrictions and whitelisting into token contracts
The technical requirements for institutional custody and settlement

TradFi is moving in. BlackRock, Franklin Templeton, and JPMorgan aren't experimenting anymore—they're shipping products. Every major fintech is building crypto rails. These institutions pay 20-30% premiums for engineers who speak both languages: blockchain and traditional finance infrastructure (FIX protocol, SWIFT, securities settlement, custodial requirements).

If you understand how a bond actually settles in TradFi and can architect an on-chain equivalent, you're not competing with 450 Solidity devs. You're competing with maybe 50 or less.

Security auditing has a chronic talent shortage despite the fact that 50% of blockchain projects still ship with vulnerabilities. The barrier to entry is real—you need deep Solidity knowledge and security expertise—but the demand is insatiable.

Starting path: Complete Cyfrin Updraft or Secureum bootcamp → Build a portfolio of practice audits on Code4rena → Apply to audit firms as junior auditor.

AI-Web3 hybrid roles increased 60% since late 2024. This intersection is so new that the talent pool barely exists. If you can build AI agents that interact with blockchain protocols, you're competing against maybe 1,000 people globally instead of 150,000.

Path 2: The Rust Premium

Here's a cheat code hiding in plain sight: Rust developers command 25-40% salary premiums and face dramatically less competition than Solidity developers.

Why? Because Rust is hard. The learning curve is steep. Most developers bounce off the borrow checker and never come back.

But Rust is eating Web3:

Solana (7,625 monthly active developers)
Polkadot/Substrate
All major ZK implementations (StarkWare, zkSync, Aztec)
Cosmos SDK is increasingly Rust-influenced
Even Ethereum tooling (Foundry, Reth) runs on Rust

The Solidity market is saturated. The Rust market is starving. 83% year-over-year growth in Rust job requirements, with nowhere near enough candidates to fill them.

Path 3: Prove Work, Not Credentials

Every recruiter and hiring manager I've encountered says the same thing: "In Web3, your GitHub is your resume."

This isn't a platitude. It's a tactical advantage for anyone willing to exploit it.

What actually gets you hired:

Consistent commit history (green GitHub activity graph)
Deployed projects with working demos (testnet counts, mainnet counts more)
Open-source contributions to protocols you want to work for
Progressive complexity showing growth (ERC-20 → ERC-721 → full DeFi protocol)

What doesn't matter as much as you think:

CS degree (nice to have, not required)
Certifications (mostly ignored)
Years of "experience" that can't be verified on-chain

Here's the unlock: contribute to the repo of the company you want to work for. Find their GitHub. Look at open issues. Submit a meaningful PR. When you apply, you're no longer applicant #347—you're "that person who fixed our indexer bug last month."

Path 4: Network in Crypto-Native Spaces

"Elite Web3 talent does not hang out on LinkedIn."

Read that again. The best jobs in Web3 are filled through:

Crypto Twitter (now X) connections
Discord server relationships
DAO governance participation
Conference and hackathon networks

If your job search strategy is "apply on LinkedIn and wait," you're playing a game designed for you to lose. The real hiring happens in DMs, in Discord channels, in "hey, we need someone for this role—know anyone good?" conversations.

Tactical moves:

Build a Twitter presence around your technical niche
Become a recognized name in 2-3 project Discord servers
Participate in governance discussions (even just voting and commenting)
Attend ETHGlobal hackathons (these are explicit talent pipelines for protocols)

Path 5: The Security Auditor Arbitrage

I'm going to be direct: if you're a competent Solidity developer struggling to find work, pivot to security.

The numbers are absurd:

Base salary: $90,000-$200,000
Top independent auditors earn $50,000+ monthly through competitive platforms
50% of projects ship with vulnerabilities → near-infinite demand
Competition ratio: 4-5x lower than development roles

The path isn't easy, but it's clear:

Master Solidity deeply (you should already be here)
Learn common vulnerability patterns (reentrancy, flash loan attacks, oracle manipulation)
Study tools: Slither, Mythril, Foundry fuzzing
Practice on Code4rena, Sherlock, or Immunefi
Build a track record of findings
Apply to firms or go independent

The market is paying $300-$500/hour for senior audit work. The supply of qualified auditors has not kept up with the $30B+ TVL that needs protecting.

The Salary Reality Check

Let's ground expectations with actual compensation data:

Role	Junior	Mid	Senior
Smart Contract Engineer	$80K-$120K	$120K-$180K	$180K-$250K
ZK/Research Engineer	$80K-$140K	$140K-$220K	$220K-$400K+
Full-Stack Web3	$80K-$120K	$120K-$175K	$175K-$250K
Protocol Engineer	$100K-$140K	$140K-$200K	$200K-$350K

Token compensation typically adds 50-100% to these figures—but tokens are volatile, vesting takes 4 years, and many never materialize. Smart candidates now negotiate for hybrid packages (some tokens, some equity, higher base) to derisk.

Geographic arbitrage is real but shrinking:

US average: $140K
UK average: $77K
Germany average: $58K (down 94% from 2022 peak!)
India average: $56K

The remote-only position decline means fewer opportunities to earn US salaries while living in low-cost regions. Plan accordingly.

The AI Question: Friend or Foe?

Here's the nuanced take: AI is simultaneously making you more productive and making junior hiring less necessary. How you respond determines whether AI helps or hurts your career.

What AI won't replace (yet):

Complex smart contract architecture
Cross-chain bridge development (AI "enters endless cyclic arguments")
Protocol-level security analysis
Regulatory judgment calls
Anything requiring deep context about a specific codebase

What AI is already replacing:

Boilerplate contract generation
Basic test writing
Documentation
Simple code refactoring
Entry-level "write a function that does X" work

The winning move: Become the person who orchestrates AI tools, not the person whose work AI tools automate. Learn Cursor, GitHub Copilot, Claude—and learn to direct them effectively. The new skill isn't "write code." It's "architect solutions and direct AI to implement them."

14% of Web3 job postings now explicitly require AI workflow proficiency. That number was 2% in 2021. The trend is obvious.

Your 90-Day Action Plan

Enough theory. Here's what to actually do:

Week 1-2: Audit Your Position

Honestly assess: Am I in a saturated category?
Identify your transferable skills toward undersaturated roles
Set up GitHub contributions tracking (aim for daily commits)

Week 3-4: Pick Your Angle

Choose ONE differentiation path (Rust? Security? AI-hybrid? Compliance?)
Begin dedicated skill-building (4+ hours daily)
Identify 5 companies you want to work for

Week 5-8: Build Proof of Work

Deploy at least one meaningful project in your chosen niche
Make contributions to target company repos
Write 2-3 technical blog posts demonstrating expertise

Week 9-10: Network Intentionally

Become active in 3 relevant Discord servers
Start engaging on Crypto Twitter with valuable technical takes
Attend at least one virtual or IRL Web3 event

Week 11-12: Apply Strategically

Apply to roles with warm connections (Discord/Twitter relationships)
Apply to undersaturated categories, not just what you "want"
Follow up with hiring managers directly via Twitter DM

The Hopium: Why This Moment Actually Favors the Prepared

Here's the thing about 450:1 competition: most of those 450 applicants are doing it wrong.

They're applying with generic resumes. They have no GitHub activity. They've never contributed to the protocol's codebase. They're invisible on Crypto Twitter. They're applying to the same saturated Solidity roles as everyone else.

You now know what they don't:

The undersaturated categories where competition drops to 60:1
The Rust premium that pays 25-40% more with 5x less competition
The proof-of-work portfolio strategy that actually gets interviews
The crypto-native networking that surfaces opportunities before they're posted
The AI-hybrid roles emerging at the intersection nobody's preparing for

The Web3 job market isn't broken—it's differentiated. The spray-and-pray approach fails. The strategic approach works.

The market added 66,000 jobs last year. More are coming. The question isn't whether opportunities exist—it's whether you'll position yourself to capture them.

The brutal math is real. But math can be solved.

Now go solve it.

Data sourced from Coincub Web3 Jobs Report 2025, Electric Capital Developer Report 2024, Web3.Career Intelligence Report 2025, and aggregated job board analysis. All figures represent estimates based on available data as of January 2026.

A Hacker’s Guide to Bitcoin: Exploring Bitcoin by Command Line - Part 1

0xkniraj — Wed, 06 Aug 2025 18:56:57 +0000

Most developers first encounter Bitcoin through polished mobile wallets or sleek web interfaces. But that's like learning programming by only using drag-and-drop tools—you miss the fundamental mechanics that make everything work.

As programmers, we understand that true mastery comes from getting our hands dirty with the underlying systems. When you run your own Bitcoin node and interact with it directly via the command line, you're not just using Bitcoin—you're speaking its native language.

This guide will take you from spinning up your first Bitcoin node to understanding UTXOs, crafting transactions, and managing multiple wallets. By the end, you'll have the foundation needed to build Bitcoin applications.

Why Command Line Bitcoin Matters

With command-line Bitcoin, you gain:

Unfiltered access to Bitcoin's data structures as your node stores them
Complete sovereignty—no third-party APIs, no rate limits, no downtime
Scripting capabilities for automation and custom workflows
Deep understanding of Bitcoin's UTXO model and transaction structure
Foundation knowledge for building Bitcoin applications

Prerequisites

You'll need:

Basic command-line comfort
Understanding of JSON (Bitcoin Core returns data in JSON format)
About 100MB disk space for our regtest experiments

Part 1: Setting Up Your Bitcoin Laboratory

Installing Bitcoin Core

Bitcoin Core provides two essential tools:

bitcoind: The full node daemon that validates and relays transactions
bitcoin-cli: Your command-line interface to the daemon

Download the latest version from bitcoin.org/en/download

For this tutorial, we'll use regtest mode—Bitcoin's built-in testing environment where you're the only miner, blocks generate instantly, and you can experiment safely.

Starting Your Private Bitcoin Network

# Start bitcoind in regtest mode
bitcoind -regtest -daemon

# Verify it's running
bitcoin-cli -regtest getnetworkinfo

Pro tip: Create an alias to save typing:

alias bcli="bitcoin-cli -regtest"

Now you can use bcli instead of typing the full command each time.

Part 2: Your First Bitcoin Genesis Moment

Understanding Bitcoin's State

Bitcoin doesn't track account balances like a traditional database. Instead, it tracks Unspent Transaction Outputs (UTXOs)—chunks of bitcoin that can be spent.

Check your initial state:

bcli getblockchaininfo

Notice "blocks": 0—you're at the genesis block with no bitcoins in existence yet.

Creating Your First Wallet (Alice)

bcli createwallet "alice"

Output:

{
  "name": "alice",
  "warning": ""
}

Your First Mining Operation

Now for the magic moment—creating bitcoin from nothing through mining:

# Generate a new address for Alice to receive mining rewards
ADDRESS=$(bcli -rpcwallet=alice getnewaddress "mining-rewards")
echo "Mining to Alice's address: $ADDRESS"

# Mine 101 blocks (we need 101 to spend the first block's reward)
bcli generatetoaddress 101 $ADDRESS

What just happened?

Address Generation: Created a Bitcoin address where coins can be sent
Block Creation: Created 101 new blocks, each paying 50 BTC to Alice's address
Coinbase Maturity: Bitcoin requires 100 confirmations before newly mined coins can be spent

Let's verify Alice's balance:

bcli -rpcwallet=alice getbalance
# Output: 50.00000000

Alice now has 50 BTC! (Only from the first block—the other rewards are still maturing)

Part 3: Understanding Bitcoin's Data Structures

Examining UTXOs

Let's see what bitcoin "ownership" actually means:

bcli -rpcwallet=alice listunspent

You'll see something like:

[
  {
    "txid": "f3d7f85e0b3c91e4d8c4b7abc1d0e3f2a9b8c7d6e5f4a3b2c1d0e9f8a7b6c5d4",
    "vout": 0,
    "address": "bcrt1qxy2kgdygjrsqtzq2n0yrf2493p83kkfjhx0wlh",
    "amount": 50.00000000,
    "confirmations": 101,
    "spendable": true
  }
]

This UTXO represents:

txid: The transaction that created this output (a coinbase transaction)
vout: Output index (0 for coinbase transactions)
amount: 50 BTC
confirmations: How deep it is in the blockchain (must be >100 to spend)

Exploring Block Structure

Let's examine the block containing Alice's first coinbase transaction:

# Get the hash of block #1
BLOCK_HASH=$(bcli getblockhash 1)
echo "Block 1 hash: $BLOCK_HASH"

# Get detailed block information
bcli getblock $BLOCK_HASH 2

Key observations from the output:

coinbase: The input field contains arbitrary data (no previous transaction)
vout: Single output paying 50 BTC to Alice's address
scriptPubKey: The locking script that defines how this UTXO can be spent

Part 4: Multi-Wallet Operations and Your First Transaction

Creating Bob's Wallet

Let's create a second wallet and demonstrate Bitcoin's peer-to-peer nature:

# Create Bob's wallet
bcli createwallet "bob"

# Generate an address for Bob
BOB_ADDRESS=$(bcli -rpcwallet=bob getnewaddress "received-from-alice")
echo "Bob's address: $BOB_ADDRESS"

Alice Sends Bitcoin to Bob

Now Alice will send some of her mined bitcoin to Bob:

# Alice sends 10 BTC to Bob
TXID=$(bcli -rpcwallet=alice sendtoaddress $BOB_ADDRESS 10.0)
echo "Transaction ID: $TXID"

# Mine a block to confirm the transaction
bcli generatetoaddress 1 $(bcli -rpcwallet=alice getnewaddress)

Understanding What Happened

Let's examine this transaction:

bcli getrawtransaction $TXID true

You'll see a transaction with:

vin: Input spending Alice's 50 BTC UTXO
vout: Two outputs:
- 10 BTC to Bob's address
- ~40 BTC back to Alice as change (minus a small mining fee)

This demonstrates Bitcoin's fundamental principle: UTXOs must be spent in full. Alice couldn't spend just 10 BTC from her 50 BTC UTXO—she had to spend the entire UTXO and create new outputs.

Checking Final Balances

# Check Alice's balance
echo "Alice's balance: $(bcli -rpcwallet=alice getbalance)"

# Check Bob's balance  
echo "Bob's balance: $(bcli -rpcwallet=bob getbalance)"

# List Bob's UTXOs to see the received bitcoin
bcli -rpcwallet=bob listunspent

You should see:

Alice: ~40 BTC (50 - 10 - small fee)
Bob: 10 BTC

Transaction History

View the transaction from both perspectives:

# Alice's transaction history
echo "=== Alice's Transactions ==="
bcli -rpcwallet=alice listtransactions

# Bob's transaction history  
echo "=== Bob's Transactions ==="
bcli -rpcwallet=bob listtransactions

Alice will see a "send" transaction with a negative amount, while Bob will see a "receive" transaction with a positive amount—the same transaction viewed from different wallets.

Cleanup

When you're done experimenting:

# Stop the daemon
bcli stop

Conclusion

Congratulations! You've just experienced Bitcoin's core mechanics firsthand:

Operated your own full Bitcoin node in regtest mode
Mined blocks and created bitcoin from nothing through coinbase transactions
Understood UTXOs as Bitcoin's fundamental building blocks
Created multiple wallets and sent bitcoin between them
Explored Bitcoin's transaction structure and block anatomy

You now understand that Bitcoin isn't about account balances—it's about a graph of UTXOs that can be unlocked and spent using cryptographic signatures. Every transaction consumes existing UTXOs and creates new ones, forming an unbroken chain back to coinbase transactions.

This command-line foundation is essential for any serious Bitcoin development work. Whether you're building wallets, payment processors, or blockchain analytics tools, understanding how Bitcoin works at this level will make you a more effective developer.

In the next part of this series, we'll dig deep into Bitcoin scripting with JavaScript, exploring how to programmatically create transactions, implement custom spending conditions, and build sophisticated Bitcoin applications using popular libraries like bitcoinjs-lib.

The blockchain is yours to explore—one command at a time.

MacOS: The Long-Running Task Notifier

0xkniraj — Thu, 04 Apr 2024 13:09:31 +0000

In the realm of software development, efficiency and productivity are paramount. MacOS, beloved by many developers for its robustness and sleek interface, offers ample opportunities for customization to enhance workflow. One area ripe for optimization is the handling of long-running tasks. Whether compiling code, processing large datasets, or deploying applications, these tasks can significantly interrupt the flow of work. This article introduces a simple, yet powerful solution: a Long-Running Task Notifier for MacOS. This tool notifies you when a task running in the terminal completes, allowing you to focus on other tasks without constant terminal checks.

The Need for Notification

Developers often switch contexts while waiting for a task to complete. Without a notification system, you might either check the terminal too frequently or return too late, wasting precious time either way. A notifier system streamlines this process by alerting you the moment a task is done, optimizing your workflow efficiency.

Implementing the Notifier with `zsh` and `osascript`

The notifier leverages zsh (Z Shell) for its extensibility and osascript for sending notifications in MacOS. The core idea is to hook into the shell's command execution lifecycle, detecting when a command starts and ends, and triggering a notification for long-running tasks.

Step 1: Setting Up the Hooks

The zsh shell provides preexec and precmd functions that execute before and after a command runs, respectively. We use these to start a timer before a command and check its duration after completion.

Step 2: Customizing the Notification

osascript, Apple's scripting utility, allows command-line interaction with the macOS Notification Center. By invoking osascript with a specific message and title, we can generate notifications dynamically based on the command executed and its duration.

The Script

Here's a simplified version of the script:

function command_start {
    zsh_command_start_time=$SECONDS
    zsh_last_command=$1
}

function command_end {
    local duration=$((SECONDS - zsh_command_start_time))
    if (( duration > 5 )); then
        local cmd_title="${zsh_last_command%% *}"
        cmd_title="$(tr '[:lower:]' '[:upper:]' <<< ${cmd_title:0:1})${cmd_title:1}"
        local message="${zsh_last_command} completed in $duration seconds"
        local title="$cmd_title Command Finished"
        osascript -e "display notification \"$message\" with title \"$title\""
    fi
}

autoload -U add-zsh-hook
add-zsh-hook preexec command_start
add-zsh-hook precmd command_end

Installation

Open ~/.zshrc in your favorite text editor.
Copy and paste the script into the file.
Save the changes and source the file with source ~/.zshrc or restart the terminal.

Customization and Extension

This basic notifier can be customized and extended in several ways:

Adjust the duration threshold to define what constitutes a long-running task.
Customize the notification message and title to include more detailed information.
Extend the script to exclude certain commands or to work with specific applications.

Conclusion

By integrating a Long-Running Task Notifier into your MacOS development environment, you can significantly improve your productivity and focus. This tool bridges the gap between multitasking and efficiency, ensuring that you're promptly informed when it's time to return to the terminal.

Happy coding, and may your tasks run smoothly and your notifications be timely!

Demystifying Bitcoin: A Guide in Simple Terms

0xkniraj — Wed, 20 Mar 2024 11:30:03 +0000

In the evolving world of finance, Bitcoin emerges as a digital marvel, often cloaked in technical jargon. Let's strip away the complexity and dive into the essence of Bitcoin, using straightforward analogies that bring its innovative nature to light.

What is Bitcoin?

Think of Bitcoin as digital gold. Just as gold is mined from the earth, Bitcoin is "mined" through complex computations by computers. It exists in a digital form, and like gold, has a limited supply, making it valuable to those who possess it. However, unlike gold, which you can hold, Bitcoin lives on the internet.

The Magic of Bitcoin: Simple Analogies

Bitcoin Transactions: Passing Notes in Class

Imagine you're in a classroom where everyone communicates by passing notes. When Alice wants to send Bitcoin to Bob, she writes a note that says, "I'm giving 1 Bitcoin to Bob." She signs it with a special code (her digital signature) that proves the note is genuinely from her. This note is then passed around the entire classroom (the Bitcoin network), so everyone knows about the transaction. This is how a Bitcoin transaction works—except, in this case, the classroom is the entire world.

The Ledger: A Public Bulletin Board

Every note passed in the classroom is pinned to a public bulletin board (the Bitcoin ledger), for everyone to see. This board keeps track of all transactions, ensuring that everyone knows who owns what. It prevents cheating, like someone trying to spend their Bitcoin twice. This bulletin board isn't controlled by any single person but is maintained collectively by the class (the decentralized Bitcoin network), making it transparent and secure.

Mining: The Puzzle Contest

Now, how do we ensure that the notes (transactions) are valid and get added to the bulletin board correctly? Enter the Bitcoin miners. Imagine a contest where students (miners) solve a complex puzzle (the mining process). The first one to solve the puzzle gets to pin the notes on the board and, as a reward, receives a new note (newly minted Bitcoin). This not only validates transactions but also creates new Bitcoin, mimicking the mining of gold but in a digital form.

Security: A Chain of Sealed Envelopes

Every time a batch of notes is pinned to the board, it's as if they're placed in a sealed envelope (a block) that is transparent yet unbreakable. Each new envelope is connected to the previous one, forming a chain (the blockchain). To alter a note inside an envelope, one would need to break open every subsequent envelope and reseal them—a task so difficult that it's considered impossible, ensuring the integrity and security of the history of notes.

Why Bitcoin?

Bitcoin is not just digital money; it's a new way of thinking about financial transactions. Unlike traditional banking, where transactions are controlled by banks, Bitcoin operates on a peer-to-peer network, similar to our classroom analogy. This means:

Direct Transactions: Alice can send Bitcoin to Bob directly, without needing a bank to approve or facilitate the transfer.
Lower Fees: Without banks as middlemen, the transaction fees can be lower.
Global Currency: Bitcoin is not tied to any country or subject to their regulations, making it a truly global currency.
Security and Privacy: Transactions are secure and private, thanks to the cryptographic techniques used.

Conclusion

Bitcoin is a pioneering form of digital currency that offers a new perspective on money and transactions. By understanding it through simple analogies, we can appreciate the genius behind its design and the potential it holds for transforming the financial landscape. Just as the internet revolutionized how we share information, Bitcoin is poised to redefine how we think about and use money.

CVE Scanner GitHub Action

0xkniraj — Thu, 03 Sep 2020 06:26:19 +0000

My Workflow

I have created a GitHub action to scan CVEs(Common Vulnerabilities and Exposures) from binary packages. My action uses cve-bin-tool to scan CVEs from binary packages and generates a detailed report and upload it as an artifact which can be downloaded anytime. I am also using actions/cache to reduce the runtime of the action.

If you are a developer who develops binary packages and libraries then I recommend integrating this action as a part of your CI pipeline so that you will get to know about common vulnerabilities found in the libraries you are using, and take actions to mitigate it.

The most recommended way to fix a given CVE is to upgrade the package to a non-vulnerable version. Ideally, a CVE is only made public after a fix is available, although this is not always the case. If this is not possible for some reason, search for the CVE number to get information on possible workarounds and patches that could be backported to other versions. Note that neither workarounds nor backported fixes can be detected by this tool, so your binary will continue to show up as vulnerable even though it may now be safely mitigated and result in a false positive. To avoid this problem, I recommend classifying CVE as Mitigated. See User Manual of the cve-bin-tool for more details.

Submission Category:

Maintainer Must-Haves

Yaml File or Link to Code

name: CVE scanner
on:
#    You can customize this according to your need.
  - push
  - pull_request
jobs:
  build_and_scan:
    runs-on: ubuntu-latest
    steps:
    # Get date utility for caching database.
      - name: Get Date
        id: get-date
        run: |
          echo "::set-output name=date::$(/bin/date -u "+%Y%m%d")"
        shell: bash
    #  Let's first download dependencies for this action.
      - uses: actions/checkout@v2
      - name: Set up Python
        uses: actions/setup-python@v2
        with:
          python-version: 3.8
    #  This second step is unnecessary but highly recommended because
    #  it will cache database and saves time redownloading it
    #  if database isn't stale.
      - name: get cached python packages
        uses: actions/cache@v2
        with:
          path: ~/.cache/pip
          key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
          restore-keys: |
            ${{ runner.os }}-pip-
      - name: get cached database
        uses: actions/cache@v2
        with:
          path: ~/.cache/cve-bin-tool
          key: ${{ runner.os }}-cve-bin-tool-${{ steps.get-date.outputs.date }}
      - name: Install CVE Binary Tool
#          We are using the latest development version of cve-bin-tool
#          because current PyPI version don't have features like
#          config file support, generating an HTML report, etc.
        run: pip install git+https://github.com/intel/cve-bin-tool@master
#          In case you prefer current PyPI version, 
#          you need to hard code CLI options
#          for cve-bin-tool in the action itself and 
#          have to use CSV or JSON as output format.
      - name: build package
#          Here, we are building a Python wheel for this example.
#          You need to replace this with your build process.
        run: |
          pip install wheel
          python setup.py bdist_wheel
      - name: Scan built package
#          Now, we will scan the built wheel 
#          which is situated in "/dist" directory
#          Python stores built packages in /dist directory.
#          You need to replace it with the directory
#          where you have stored built package
        run: cve-bin-tool dist -f html -o cve-bin-tool-report.html -x
#          Alternatively if you have written config file
#          for cve-bin-tool you can use the following command
#          cve-bin-tool -C path/to/cve_bin_tool_config.toml
        continue-on-error: true
#          You need to set continue_on_error: true because 
#          CVE Binary Tool sets the number of product with CVEs
#          as exit code. And GitHub terminate the action 
#          when the process produces nonzero exit code status.
      - name: Upload report as an artifact
#        This will upload the generated report as 
#        a GitHub artifact which you can download later.
        uses: actions/upload-artifact@v2
        with:
          name: cve_report
          path: 'cve-bin-tool-report.html'

Additional Resources / Info

I have also contributed this action to the cve-bin-tool. You can also find this in the how-to-guides directory of the official cve-bin-tool repository.

You can also see my pull-request to cve-bin-tool for this action here and discussion on the associated issue here.

I have also tested this action on my other project and it is working fine. You can see an example run of the action here.

CVE Binary Tool: GSoC Final Report

0xkniraj — Sat, 29 Aug 2020 12:08:46 +0000

Google Summer of Code 2020 is finally coming to an end, and what an exciting experience it has been! In this post, I’ll be showing off the fruits of my labor.

I have created an example GitHub action workflow last week, and that marks the last major milestone of my GSoC project Improve concurrency and input functionality. I’m very glad that I was able to finish all the major milestones in time. Below I’ll give a quick summary of my journey.

About the Project

Before going into the details, I would like to give a brief introduction to the project for the uninitiated. You can skip this section if you are already familiar with it.

My project was to Improve concurrency and input functionality of CVE Binary Tool. The CVE Binary Tool scans for several common, vulnerable open source components such as OpenSSL, libpng, libxml2, etc. to let you know if a given directory or binary file includes common libraries with known vulnerabilities, known as CVEs (Common Vulnerabilities and Exposures). It is written in my favorite language Python. It is available as a python package and you can get it from PyPI. You can view the source code from GitHub repository and it's maintained by Terri Oda and John Andersen. They are also my mentors for this project.

When I started working on the project, CVE Binary Tool has some bugs under Windows platform and since I use Windows primarily, I have fixed all Windows bugs in the community bonding period. I have also written faster native python solution to replace the c-strings extension module and refactored whole checkers module to use object-oriented design to reduce repetition of code in community bonding period. Since we have removed support for Python2 in the last release, I have also optimized the legacy Python2 code for Python3.

The First Milestone: Improve Test Suite

I wanted to improve tests of CVE Binary Tool because CI runtime was too long and there was an initialization missing which was failing the test when we run tests locally.

I have removed compiler dependencies from test suite in favor of native python solution. I have also fixed the tricky pytest initialization bug by researching and understanding the origin of the bug. I have also gradually improved the test suite of every module by migrating it to pytest from standard unittest and using pytest-xdist to optimize runtime. (EX: parallelization of test_json.

The Second Milestone: Improve Concurrency

Previously, CVE Binary Tool was using multiprocessing for downloading NVD data-feeds. Since downloading is an IO-bound task, I suggested that asyncio would be the better choice for this use-case.

I have refactored almost all IO-bound modules to use asyncio. I have also developed many async-utilities for various modules. The following are all the pull-requests regarding my work on improving the concurrency of the tool.

The main performance gain was the reduction in downloading time by 50% after switching to asyncio.

The Third Milestone: Improve Input Functionality

CVE Binary Tool had a command named csv2cve to get CVEs from the CSV formatted input. My goal was to replace it with a generic InputEngine module which can support various input types. Currently, It supports CSV and JSON as a valid input data type. I have also added support for specifying triage data for CVEs like remarks(Ex: Mitigated, Ignored, etc.), comments, custom severity, etc.

Since CVE Binary Tool has several command-line arguments, I decided to add support for config files so that users don't have to repeatedly specify these arguments every time S/he wants to scan a directory. It can also be helpful in automated environments like CI/CD pipelines. Below is a list of my pull-requests related to this milestone.

The Final Milestone: Improve User Experience

We're lucky to have Anthony (a user) who was willing to attend our weekly meeting. He suggested many features and documentation that would be helpful to him and other users. I have contributed how-to-guides(essentially a cookbook of recipes) for different use-cases.

I have also created an error handler module to provide beautiful traceback using rich library and set custom exit-code according to exception so that the user can know why program terminated under quiet mode or automation script.

Here is the list of all my pull-requests related to this goal.

Future Work

My original proposal was huge and due to limited time, I have to prioritize my goals according to user requirements. I have fully completed my core goal of improving input functionality and user experience but I still have many optimizations in my mind for the concurrency aspect of the project. So, I am going to continue my work on improving the performance of CVE Binary Tool. I also want to optimize runtime of the long tests because it is taking 30 minutes to run all tests and I believe we can optimize it significantly by caching and/or reducing the test data that need to be downloaded for testing. I also want to create a ready to use GitHub action for the tool so that developer can integrate it easily as a part of their CI/CD pipeline.

What I gained from GSoC

The amount of experience I gained from working on CVE Binary Tool with Terri Oda and John Andersen is immeasurable. Both my mentors are kind, helpful, and extremely talented. They have given me many advice and suggestions from low-level working of language to the best practices about writing good code.

Terri Oda helped me improve my PR(pull-request) quality by suggesting articles about best practices. She also helped me understand what tasks should be prioritized. While John Andersen helped me with a lot of things about python in general like metaclasses, asyncio, context-manager, etc.

Before GSoC, I only knew basic git commands but now I can confidently say that I have mastered git. I also have a much better understanding of how programming languages work, when and how to choose third-party libraries, how to refer to documentation efficiently, how to write clean, readable and maintainable code, how to document and test code effectively, how to properly structure pull-requests and most importantly how practical programming works, which I wouldn’t have known otherwise.

Opensource has become my hobby now because being part of an opensource community, I have learned that as an opensource contributor apart from making code contributions, I also get to triage and resolve user issues and review other contributors' pull-requests. I can also guide and motivate new contributors. I won't get this kind of experience if I just do my personal projects.

All these lessons are invaluable to me and I strongly believe it will be helpful to me throughout my career.