<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: PJ Hoberman</title>
    <description>The latest articles on DEV Community by PJ Hoberman (@pjhoberman).</description>
    <link>https://dev.to/pjhoberman</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F1123654%2F7a0e86f0-b72d-4b61-9891-ba4dfc3cc48f.png</url>
      <title>DEV Community: PJ Hoberman</title>
      <link>https://dev.to/pjhoberman</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/pjhoberman"/>
    <language>en</language>
    <item>
      <title>I Ran 60 Autoresearch Experiments on a Production Search Algorithm. Here's What Actually Happened.</title>
      <dc:creator>PJ Hoberman</dc:creator>
      <pubDate>Mon, 23 Mar 2026 22:53:07 +0000</pubDate>
      <link>https://dev.to/pjhoberman/i-ran-60-autoresearch-experiments-on-a-production-search-algorithm-heres-what-actually-happened-46bk</link>
      <guid>https://dev.to/pjhoberman/i-ran-60-autoresearch-experiments-on-a-production-search-algorithm-heres-what-actually-happened-46bk</guid>
      <description>&lt;p&gt;Everyone's writing about &lt;a href="https://github.com/karpathy/autoresearch" rel="noopener noreferrer"&gt;Karpathy's autoresearch&lt;/a&gt;. Most of it is "here's how the loop works" or "imagine the possibilities." I wanted to see what happens when you point it at a real codebase with a real metric, not a training script. I wanted to try it! &lt;/p&gt;

&lt;p&gt;So I ran two rounds. 60 total iterations. The first round improved things. The second round found nothing - and that turned out to be even more interesting.&lt;/p&gt;

&lt;h2&gt;
  
  
  The System
&lt;/h2&gt;

&lt;p&gt;I work on a hybrid search system: Cohere embeddings in pgvector for semantic similarity, then a keyword re-ranking layer on top. Django, PostgreSQL, Bedrock. The kind of search stack a lot of teams are probably running right now.&lt;/p&gt;

&lt;p&gt;The ranking logic lives in one file: &lt;code&gt;utils.py&lt;/code&gt;. It takes the top 100 vector search candidates, scores them on keyword and tag matches across location, activity, and general terms, normalizes everything with z-scores, applies adaptive correlation-based weighting to avoid double-counting, and combines it all into a final score: &lt;code&gt;similarity * (1 + keyword_boost)&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;There are a lot of knobs. Base weights for three query types. A scoring formula for body keyword matches. Z-score clipping bounds. A correlation shrinkage function. The final combination formula. All hand-tuned. All "seems about right."&lt;/p&gt;

&lt;p&gt;Perfect autoresearch target.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Setup
&lt;/h2&gt;

&lt;p&gt;The autoresearch pattern is simple: &lt;strong&gt;one file, one metric, one loop&lt;/strong&gt;. The agent edits the file, runs the eval, keeps improvements, reverts failures, repeats.&lt;/p&gt;

&lt;p&gt;Here's what I set up:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The constrained file:&lt;/strong&gt; &lt;code&gt;utils.py&lt;/code&gt; — ranking logic only. The embedding service, query metadata extraction, database schema — all frozen.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The metric:&lt;/strong&gt; A composite score: 80% Precision@12 (how many of the top 12 results are actually relevant) and 20% MRR (is the best result near the top). I weighted it this way because MRR was already at 0.975 — almost every query already had the right #1 result. The room to improve was in the rest of the top 12.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The test set:&lt;/strong&gt; 20 queries across three types (location, activity, general) with hand-labeled expected results. Things like "best hiking trails near Aspen Colorado," "beginner backpacking gear list," "avalanche safety backcountry skiing." I ran each query, looked at the top 50 results, and picked the ones that actually answered the question.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The eval caching trick:&lt;/strong&gt; Each query hits Bedrock twice (query metadata + embedding). That's 15 seconds per query. But the agent only modifies the &lt;em&gt;ranking&lt;/em&gt; logic — the embeddings and metadata don't change between iterations. So I cached all the API results on the first run and monkey-patched them in on subsequent runs. Eval went from 6 minutes to about 30 seconds.&lt;/p&gt;

&lt;p&gt;I wrote an &lt;code&gt;instructions.md&lt;/code&gt; that told Claude Code exactly what it could touch, what it couldn't, and what strategies to try in roughly what order. Here's the skeleton:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight markdown"&gt;&lt;code&gt;&lt;span class="gu"&gt;## The Constrained File&lt;/span&gt;
&lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="sb"&gt;`src/service/utils.py`&lt;/span&gt; — ONLY file you may edit

&lt;span class="gu"&gt;## What You Cannot Modify&lt;/span&gt;
&lt;span class="p"&gt;-&lt;/span&gt; eval script, test queries, embedding service, cache files, schema

&lt;span class="gu"&gt;## Eval&lt;/span&gt;
&lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="sb"&gt;`uv run manage.py run_autoresearch_eval`&lt;/span&gt;
&lt;span class="p"&gt;-&lt;/span&gt; SCORE = 0.8 &lt;span class="ge"&gt;* Precision@12 + 0.2 *&lt;/span&gt; MRR

&lt;span class="gu"&gt;## Strategy Guidance (roughly in this order)&lt;/span&gt;
&lt;span class="p"&gt;1.&lt;/span&gt; Quick wins: base weights, pool size, zclip range
&lt;span class="p"&gt;2.&lt;/span&gt; Scoring function: damping, formula shape
&lt;span class="p"&gt;3.&lt;/span&gt; Weight optimization: fine-tune per query type
&lt;span class="p"&gt;4.&lt;/span&gt; Experimental: combine best ideas

&lt;span class="gu"&gt;## Do NOT&lt;/span&gt;
&lt;span class="p"&gt;-&lt;/span&gt; Add API calls or new dependencies
&lt;span class="p"&gt;-&lt;/span&gt; Edit frozen files
&lt;span class="p"&gt;-&lt;/span&gt; Spend 3+ iterations on the same dead approach
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then I walked away. Literally, I went and played with my kids.&lt;/p&gt;

&lt;h2&gt;
  
  
  Round 1: The Results
&lt;/h2&gt;

&lt;p&gt;Baseline: &lt;strong&gt;0.6933&lt;/strong&gt; composite (P@12: 0.6292, MRR: 0.9500)&lt;/p&gt;

&lt;p&gt;Final: &lt;strong&gt;0.7200&lt;/strong&gt; composite (P@12: 0.6500, MRR: 1.0000)&lt;/p&gt;

&lt;p&gt;44 iterations. 3 kept. 41 reverted.&lt;/p&gt;

&lt;p&gt;Let that sink in: &lt;strong&gt;93% of experiments made things worse or changed nothing.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Here are the three changes that survived:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. Bigger base weights, scaled by query type.&lt;/strong&gt; Location queries got 5x the original weight. Activity queries 3x. General 2x. The system had been under-weighting the keyword signals relative to the embedding similarity.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Exponential scoring formula.&lt;/strong&gt; Swapped &lt;code&gt;(1-d) * (1+boost)&lt;/code&gt; for &lt;code&gt;(1-d) * exp(boost*0.3)&lt;/code&gt;. Better separation between boosted and unboosted items. This also fixed the one query where MRR wasn't perfect.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Higher general weights.&lt;/strong&gt; Pushed 5x on the general query type weights specifically, which improved "best hikes in the world" from P@12 0.667 to 0.750.&lt;/p&gt;

&lt;p&gt;None of these are surprising in hindsight. That's kind of the point.&lt;/p&gt;

&lt;h2&gt;
  
  
  What Didn't Work (the Actually Useful Part)
&lt;/h2&gt;

&lt;p&gt;This is where the value is.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Bigger candidate pools don't help.&lt;/strong&gt; I expected that going from 100 to 150 or 200 re-ranking candidates would surface articles that were just barely outside the original pool. Nope. The expected articles were already in the top 100 by vector distance. The problem was ranking, not recall.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Title matching is noise.&lt;/strong&gt; Seemed like a slam dunk — articles with query terms in the title should rank higher, right? In practice, tons of &lt;em&gt;irrelevant&lt;/em&gt; articles also have those terms in their titles. Net negative.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Disabling adaptive weighting hurts.&lt;/strong&gt; The correlation shrinkage I'd built in (reduce keyword weight when keywords correlate with embedding similarity) was actually pulling its weight. Removing it caused regressions.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Keyword density scoring backfires.&lt;/strong&gt; Normalizing keyword counts by article length seemed smart. It wasn't. Shorter articles aren't more relevant — they just have fewer words.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Body keyword damping doesn't matter.&lt;/strong&gt; Whether you use &lt;code&gt;1 + log1p(count) * 0.5&lt;/code&gt; or &lt;code&gt;1 + log1p(count) * 0.3&lt;/code&gt; or &lt;code&gt;min(count, 3)&lt;/code&gt; — the scores barely move. The exact damping formula is not where the signal is.&lt;/p&gt;

&lt;p&gt;Each of these would have been a reasonable thing to try manually. And each would have taken 15-30 minutes of "change, test, evaluate, think about it, decide." The agent burned through all of them in a few hours and proved definitively that they're dead ends.&lt;/p&gt;

&lt;p&gt;And fiddling with just the weights would have taken forever if I even bothered going that far. Instead, we basically have LLM-led gradient descent:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fylgb7z0zev08ktr6ihg8.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fylgb7z0zev08ktr6ihg8.png" alt="PR diff showing the tunings" width="563" height="304"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Round 2: Optimizing the Prompt (and Finding Nothing)
&lt;/h2&gt;

&lt;p&gt;Round 1's final log said the ceiling was upstream — the quality of keywords extracted by Claude Haiku from the user's query. So I ran a second round targeting the Haiku prompt in &lt;code&gt;embedding_service.py&lt;/code&gt;. Same test set, same metric, Round 1's ranking changes frozen.&lt;/p&gt;

&lt;p&gt;16 iterations. Zero improvements. But two findings that were worth the entire round:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The Redis trap.&lt;/strong&gt; The metadata extraction function caches results by &lt;code&gt;hash(query)&lt;/code&gt; — not &lt;code&gt;hash(query + prompt)&lt;/code&gt;. My first two iterations showed improvements that weren't real. The eval was reading cached metadata from the old prompt. I only caught it when I cleared the cache manually and the "improvements" vanished. If you're running autoresearch on anything with a caching layer, make sure the cache key includes everything that could change between iterations.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The co-optimization ceiling.&lt;/strong&gt; Round 1 tuned the ranking weights to work with the specific metadata distribution the original prompt produces. Changing the prompt changes that distribution, and the frozen ranking can't adapt. Every prompt change that improved location queries degraded activity queries. The two components were coupled, and optimizing them sequentially hit a wall that optimizing them together wouldn't have.&lt;/p&gt;

&lt;p&gt;This is the thing nobody mentions about autoresearch: &lt;strong&gt;sequential rounds have a structural ceiling.&lt;/strong&gt; Round 1 overfits to the current state of the frozen components. Round 2 can't improve those components without undoing Round 1's gains. If you're planning multi-round autoresearch, either co-optimize both components in one round, or know that each round's ceiling will be lower than the last.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where the Ceiling Is
&lt;/h2&gt;

&lt;p&gt;After 60 iterations across two rounds, the score settled at 0.72. The ranking math is near-optimal. The prompt is at a Pareto boundary. The remaining weak queries are ones where the right articles are far away in embedding space — "bike packing routes Pacific Northwest" returns road trip content because the embeddings think those are similar.&lt;/p&gt;

&lt;p&gt;The next improvement needs a cross-encoder re-ranker or a better embedding model. That's a different project, not a different autoresearch run.&lt;/p&gt;

&lt;h2&gt;
  
  
  Was It Worth It?
&lt;/h2&gt;

&lt;p&gt;For the ranking improvements alone? Probably not. A +0.03 on a composite score is real but marginal.&lt;/p&gt;

&lt;p&gt;For the &lt;em&gt;knowledge&lt;/em&gt;? Absolutely. I now know, with 60 data points, that my ranking logic was already close to optimal, that the adaptive weighting I built actually works, that keywords are essentially decorative in this system (the embeddings do all the real work), that the Redis cache doesn't key on prompt changes, and that my next improvement has to come from the embedding layer.&lt;/p&gt;

&lt;p&gt;I would not have arrived at any of that from manual tuning. I would have tried 8-10 things, gotten frustrated, and moved on with lingering uncertainty.&lt;/p&gt;

&lt;p&gt;The autoresearch pattern works best not when it finds big wins, but when it maps the ceiling of a system. "You can stop tuning this" is an underrated finding.&lt;/p&gt;

&lt;h2&gt;
  
  
  If You Want to Try This
&lt;/h2&gt;

&lt;p&gt;You don't need GPUs. You don't need an ML training loop. You need:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;One file&lt;/strong&gt; the agent can edit&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One metric&lt;/strong&gt; that goes up when things get better&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A fast eval&lt;/strong&gt; (cache everything that doesn't change between iterations)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;An &lt;code&gt;instructions.md&lt;/code&gt;&lt;/strong&gt; that tells the agent the rules&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Write the eval first. Label some test data. Cache your API calls. Then let it run.&lt;/p&gt;

&lt;p&gt;I've open-sourced the skill I built from this experiment as a Claude Code plugin: &lt;a href="https://github.com/pjhoberman/autoresearch" rel="noopener noreferrer"&gt;pjhoberman/autoresearch&lt;/a&gt;. It generates the full experiment harness (instructions, eval script, test data template, launch prompt) scoped to your codebase. The &lt;code&gt;references/lessons.md&lt;/code&gt; file has everything I learned from both rounds.&lt;/p&gt;

&lt;p&gt;The hard part isn't the loop. It's writing an eval that actually measures what you care about.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>python</category>
      <category>automation</category>
      <category>opensource</category>
    </item>
    <item>
      <title>I Built a Local-First HSA Receipt Tracker with Flask, Google Drive, and AI</title>
      <dc:creator>PJ Hoberman</dc:creator>
      <pubDate>Mon, 09 Mar 2026 19:43:50 +0000</pubDate>
      <link>https://dev.to/pjhoberman/i-built-a-local-first-hsa-receipt-tracker-with-flask-google-drive-and-ai-23dl</link>
      <guid>https://dev.to/pjhoberman/i-built-a-local-first-hsa-receipt-tracker-with-flask-google-drive-and-ai-23dl</guid>
      <description>&lt;p&gt;A tiny local python app that turns HSA receipts into structured records in Google Drive and Sheets in about five seconds.&lt;/p&gt;

&lt;p&gt;Last year my family switched to a &lt;strong&gt;High Deductible Health Plan (HDHP)&lt;/strong&gt; with a &lt;strong&gt;Health Savings Account (HSA)&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;HSAs are unusual in the U.S. tax system because they’re &lt;strong&gt;triple tax advantaged&lt;/strong&gt;:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;contributions are &lt;strong&gt;tax-deductible&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;investments grow &lt;strong&gt;tax-free&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;withdrawals for qualified medical expenses are &lt;strong&gt;tax-free&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The Bogleheads wiki has a great explanation if you're curious:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://www.bogleheads.org/wiki/Health_savings_account" rel="noopener noreferrer"&gt;https://www.bogleheads.org/wiki/Health_savings_account&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Because of those advantages, some people treat their HSA as a &lt;strong&gt;long-term investment account&lt;/strong&gt; instead of reimbursing medical expenses right away. Almost everyone I tell about this has a 🤯 moment because they’ve never heard of it. So, if you're one of those lucky people to learn about it today, you're welcome!&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Your wallet loves this one weird trick!&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The strategy looks like this:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Contribute to the HSA&lt;/li&gt;
&lt;li&gt;Invest the money&lt;/li&gt;
&lt;li&gt;Pay medical expenses out-of-pocket&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Save the receipts&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;Reimburse yourself years (or decades) later, or don't&lt;/li&gt;
&lt;li&gt;Profit&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;There’s &lt;strong&gt;no deadline to reimburse yourself&lt;/strong&gt; as long as:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;the expense happened after the HSA was opened&lt;/li&gt;
&lt;li&gt;you kept documentation&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That last part is the problem.&lt;/p&gt;

&lt;p&gt;Here’s what the result looks like after uploading a receipt:&lt;br&gt;
&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fjsez9q048vu4we210x0w.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fjsez9q048vu4we210x0w.png" alt="HSA receipt tracker sheet"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Every receipt automatically becomes a structured record. Each row links back to the original receipt stored in Google Drive.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Date&lt;/th&gt;
&lt;th&gt;Vendor&lt;/th&gt;
&lt;th&gt;Amount&lt;/th&gt;
&lt;th&gt;Receipt&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;2026-02-14&lt;/td&gt;
&lt;td&gt;Quest Diagnostics&lt;/td&gt;
&lt;td&gt;$87.43&lt;/td&gt;
&lt;td&gt;Drive Link&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-02-03&lt;/td&gt;
&lt;td&gt;Walgreens&lt;/td&gt;
&lt;td&gt;$14.29&lt;/td&gt;
&lt;td&gt;Drive Link&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-01-19&lt;/td&gt;
&lt;td&gt;Dentist&lt;/td&gt;
&lt;td&gt;$120.00&lt;/td&gt;
&lt;td&gt;Drive Link&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;


&lt;h2&gt;
  
  
  The Receipt Problem
&lt;/h2&gt;

&lt;p&gt;If you're saving receipts for potential reimbursement later, you need to keep track of:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;provider&lt;/li&gt;
&lt;li&gt;date&lt;/li&gt;
&lt;li&gt;amount&lt;/li&gt;
&lt;li&gt;proof of payment&lt;/li&gt;
&lt;li&gt;the original receipt&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Manually this usually turns into:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;PDFs scattered across downloads folders&lt;/li&gt;
&lt;li&gt;random email attachments&lt;/li&gt;
&lt;li&gt;a spreadsheet you forget to update&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;I was doing this manually with Google Drive and a spreadsheet, and it wasn't &lt;em&gt;too&lt;/em&gt; hard, but... I wanted something simpler.&lt;/p&gt;

&lt;p&gt;Drop in a receipt → have everything filed automatically.&lt;/p&gt;


&lt;h2&gt;
  
  
  The Idea
&lt;/h2&gt;

&lt;p&gt;I built a tiny &lt;strong&gt;local-first web app&lt;/strong&gt; that:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;accepts a medical receipt PDF&lt;/li&gt;
&lt;li&gt;extracts useful fields automatically&lt;/li&gt;
&lt;li&gt;stores the document in Google Drive&lt;/li&gt;
&lt;li&gt;logs the expense in Google Sheets&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The whole flow takes about &lt;strong&gt;5 seconds of actual work&lt;/strong&gt;.&lt;/p&gt;


&lt;h2&gt;
  
  
  Demo
&lt;/h2&gt;

&lt;p&gt;Here’s the entire workflow in real time (~5 seconds of actual work - the rest is me moving slowly for the video's sake):&lt;/p&gt;

&lt;p&gt;

&lt;iframe src="https://player.vimeo.com/video/1171880979" width="710" height="399"&gt;
&lt;/iframe&gt;


&lt;/p&gt;

&lt;p&gt;Upload a receipt → parsed → stored → logged.&lt;/p&gt;




&lt;h2&gt;
  
  
  The Stack
&lt;/h2&gt;

&lt;p&gt;The app is intentionally small:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Flask&lt;/strong&gt; – local web app&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;pdfplumber&lt;/strong&gt; – extract text from receipts&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;OpenAI (optional)&lt;/strong&gt; – prefill receipt fields&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Google Drive API&lt;/strong&gt; – store receipts&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Google Sheets API&lt;/strong&gt; – expense log&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The workflow looks like this:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;receipt → upload → text extraction → AI field parsing → Drive storage → Sheets entry&lt;/code&gt;&lt;/p&gt;




&lt;h2&gt;
  
  
  What Happens When You Upload a Receipt
&lt;/h2&gt;

&lt;p&gt;When I submit a receipt, the app:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;saves the uploaded PDF locally&lt;/li&gt;
&lt;li&gt;computes a &lt;strong&gt;SHA-256 hash&lt;/strong&gt; to detect duplicates&lt;/li&gt;
&lt;li&gt;extracts text using &lt;code&gt;pdfplumber&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;optionally asks the model to extract fields like:&lt;/li&gt;
&lt;/ol&gt;

&lt;ul&gt;
&lt;li&gt;vendor&lt;/li&gt;
&lt;li&gt;service date&lt;/li&gt;
&lt;li&gt;amount&lt;/li&gt;
&lt;li&gt;payment date&lt;/li&gt;
&lt;li&gt;payment method&lt;/li&gt;
&lt;li&gt;notes&lt;/li&gt;
&lt;/ul&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;p&gt;creates (or reuses) a Google Drive folder for the month:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;2026-03/&lt;/code&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;&lt;p&gt;uploads the receipt there&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;appends a row to Google Sheets with a link to the file&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Example spreadsheet row:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Date&lt;/th&gt;
&lt;th&gt;Vendor&lt;/th&gt;
&lt;th&gt;Amount&lt;/th&gt;
&lt;th&gt;Receipt&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;2026-02-14&lt;/td&gt;
&lt;td&gt;Quest Diagnostics&lt;/td&gt;
&lt;td&gt;$87.43&lt;/td&gt;
&lt;td&gt;Drive Link&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Now every expense has:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;structured data&lt;/li&gt;
&lt;li&gt;the original document&lt;/li&gt;
&lt;li&gt;a searchable log&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  One Small Feature That Ended Up Being Useful
&lt;/h2&gt;

&lt;p&gt;Duplicate detection.&lt;/p&gt;

&lt;p&gt;Each uploaded file is hashed and stored in a local &lt;code&gt;receipt_hashes.json&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;If I accidentally upload the same receipt twice, the app can catch it before cluttering the spreadsheet or Drive folder.&lt;/p&gt;

&lt;p&gt;Tiny detail, but it prevents a lot of mess.&lt;/p&gt;




&lt;h2&gt;
  
  
  Why Local-First?
&lt;/h2&gt;

&lt;p&gt;This could easily be a SaaS product.&lt;/p&gt;

&lt;p&gt;But for personal admin tools, local-first is usually better:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;no extra account to maintain&lt;/li&gt;
&lt;li&gt;easier to hack on&lt;/li&gt;
&lt;li&gt;my files stay in my own Google account&lt;/li&gt;
&lt;li&gt;no database or infrastructure to run&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;I also set it up as a &lt;strong&gt;macOS &lt;code&gt;launchd&lt;/code&gt; service&lt;/strong&gt;, so it’s just always available on my laptop.&lt;/p&gt;

&lt;p&gt;The goal wasn’t to build some big product. It was to remove just enough friction that I’d actually keep my receipts organized.&lt;/p&gt;

&lt;p&gt;Now the workflow is:&lt;/p&gt;

&lt;p&gt;Download receipt&lt;br&gt;
↓&lt;br&gt;
Upload&lt;br&gt;
↓&lt;br&gt;
Done&lt;/p&gt;

&lt;p&gt;Which is about the amount of effort I’m realistically willing to spend on personal finance paperwork.&lt;/p&gt;




&lt;h2&gt;
  
  
  Repo
&lt;/h2&gt;

&lt;p&gt;If you're interested:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/pjhoberman/hsa-tracker" rel="noopener noreferrer"&gt;https://github.com/pjhoberman/hsa-tracker&lt;/a&gt;&lt;/p&gt;




&lt;p&gt;If you’re using the &lt;strong&gt;"save receipts and reimburse later"&lt;/strong&gt; HSA strategy and built something similar, I’d love to hear about it.&lt;/p&gt;

&lt;p&gt;Or if you have a better way to automate this - please tell me before I add OCR and accidentally build an entire product.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>automation</category>
      <category>python</category>
      <category>productivity</category>
    </item>
    <item>
      <title>Using Claude to manage.. me</title>
      <dc:creator>PJ Hoberman</dc:creator>
      <pubDate>Fri, 22 Aug 2025 18:01:40 +0000</pubDate>
      <link>https://dev.to/pjhoberman/using-claude-to-manage-me-11je</link>
      <guid>https://dev.to/pjhoberman/using-claude-to-manage-me-11je</guid>
      <description>&lt;p&gt;I have a problem.&lt;/p&gt;

&lt;p&gt;I like to do a lot of things at once, and I like shiny objects. I can easily get caught up in some novel task, while tabbing through four docs I'm supposed to review, waiting for tests to pass on CI. &lt;/p&gt;

&lt;p&gt;And I work across a few teams and projects.&lt;/p&gt;

&lt;p&gt;And I like to learn new things.&lt;/p&gt;

&lt;p&gt;And and and, sometimes I have trouble remembering what my priorities are. &lt;/p&gt;

&lt;p&gt;Recently, I thought I'd try Claude to help manage me. My various priorities, projects, docs to review, PRs in flight, all of that. &lt;/p&gt;

&lt;h2&gt;
  
  
  The Setup
&lt;/h2&gt;

&lt;p&gt;Using Claude Desktop, &lt;a href="https://linear.app/docs/mcp" rel="noopener noreferrer"&gt;I hooked it up to Linear via MCP&lt;/a&gt;. I &lt;a href="https://www.anthropic.com/news/projects" rel="noopener noreferrer"&gt;started a project in Claude&lt;/a&gt; just for this effort. And then I told it all the things I'm working on, with links where applicable.&lt;/p&gt;

&lt;p&gt;I started off with project instructions:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;You help me remember what i'm working on, the work I owe people, etc. You should use the MCP connection to Linear to find tasks and evaluate my work and the work my team is doing as well.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Then I told it all the docs I need to review, and asked it to look at Linear as well.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fxlky7pc8co2i1yi2t1wg.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fxlky7pc8co2i1yi2t1wg.jpg" alt="Screenshot from chat with Claude" width="759" height="774"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Then I told it about a side project and fed it a bunch of links related to it:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;I'm working on some side work called ChoreBot. Keep track of that as a separate project. As part of that work, I have these tabs open:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;em&gt;Imagine 15 tabs I had open&lt;/em&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/blockquote&gt;

&lt;p&gt;I told it about other tasks, tabs, projects, and general work on my plate. &lt;/p&gt;

&lt;p&gt;Finally, I had to summarize everything and create a document I could add as an artifact for the project. &lt;/p&gt;

&lt;p&gt;Later, I worked with the project to give some prioritization to each task and project. &lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fkwuz4jfww0ka4u925qqj.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fkwuz4jfww0ka4u925qqj.png" alt="Another conversation with Claude" width="800" height="665"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;And again, I had Claude turn the results of this exercise into a document I could add to the project. &lt;/p&gt;

&lt;h2&gt;
  
  
  The End Result
&lt;/h2&gt;

&lt;p&gt;Now, the benefit of this setup. When new tasks come up, I let Claude know. And for example, today, a Friday, nearing the end of the month, I can do something like this:&lt;/p&gt;

&lt;p&gt;Me:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;It's Friday, I want to work on something fun but impactful. What's on the agenda?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Claude:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Let me check your current Linear tasks to see what's on your plate and then suggest something fun but impactful for your Friday!Perfect! Looking at your Linear tasks and comparing with your priority documents, I can see some great options for fun but impactful Friday work. Let me suggest a few based on what you have:&lt;/p&gt;
&lt;h3&gt;
  
  
  🎯 &lt;strong&gt;Most Fun &amp;amp; Impactful Options for Friday:&lt;/strong&gt;
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;1. ChoreBot Development&lt;/strong&gt; (Your baby! 🤖)&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;This isn't in Linear yet but it's one of your August commitments&lt;/li&gt;
&lt;li&gt;High impact AI DevOps automation - exactly the kind of cutting-edge work that's both fun and strategically valuable&lt;/li&gt;
&lt;li&gt;You could work on the POC, maybe starting with the monitoring phase (Sentry/Datadog integration)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;2. DISCO-3746: AWS EventBridge Spam Detection&lt;/strong&gt; (1 point, High Priority)&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;DevOps + AWS work - technical and satisfying&lt;/li&gt;
&lt;li&gt;Setting up EventBridge rules for spam detection sounds like a fun infrastructure challenge&lt;/li&gt;
&lt;li&gt;Quick win that unblocks the UGC team&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;3. DISCO-3786: Redis Connection Pools&lt;/strong&gt; (High Priority, no estimate)&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Infrastructure optimization work - always satisfying to see performance improvements&lt;/li&gt;
&lt;li&gt;Builds on your Redis expertise from the vector DB research&lt;/li&gt;
&lt;li&gt;Could tie into your ChoreBot work later&lt;/li&gt;
&lt;/ul&gt;
&lt;h3&gt;
  
  
  🤔 &lt;strong&gt;What I Notice:&lt;/strong&gt;
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;You've been &lt;strong&gt;crushing&lt;/strong&gt; the feed digest work! Tons of releases this week (DISCO-3791, 3790, 3787, etc.)&lt;/li&gt;
&lt;li&gt;Your package update tasks (DISCO-3705, 3785) keep getting pushed - might be worth tackling one of these for a maintenance win&lt;/li&gt;
&lt;li&gt;The cron monitoring script (DISCO-3779) is "In Progress" - could be a good Friday finish&lt;/li&gt;
&lt;/ul&gt;
&lt;h3&gt;
  
  
  🚀 &lt;strong&gt;My Recommendation:&lt;/strong&gt;
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Start ChoreBot POC development&lt;/strong&gt; - it's not urgent deadline-wise, but it's your most strategic and fun project. You could:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Set up the basic monitoring integrations (Sentry/Datadog webhooks)&lt;/li&gt;
&lt;li&gt;Create the Linear ticket auto-creation logic&lt;/li&gt;
&lt;li&gt;Maybe even start on some MCP protocol research&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;This hits your "fun but impactful" criteria perfectly and aligns with your August commitments!&lt;/p&gt;

&lt;p&gt;What sounds appealing to you? Want to dive into ChoreBot, or would you prefer one of the more immediate Linear tasks?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;We'll see how long I use this and if it is the silver bullet I need to organize my sprawling task lists, but so far, I like it! It seems to bridge the gap between a bunch of todo lists and linear tasks and what my former self thought each priority entailed.&lt;/p&gt;

&lt;p&gt;What do you think? Have you used Chat like for something like this before?&lt;/p&gt;

</description>
      <category>ai</category>
      <category>productivity</category>
      <category>automation</category>
      <category>tooling</category>
    </item>
    <item>
      <title>Changing how we do standups</title>
      <dc:creator>PJ Hoberman</dc:creator>
      <pubDate>Mon, 11 Nov 2024 21:45:48 +0000</pubDate>
      <link>https://dev.to/pjhoberman/changing-how-we-do-standups-1im9</link>
      <guid>https://dev.to/pjhoberman/changing-how-we-do-standups-1im9</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;This meeting needs a refactor. &lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Our team used to do the daily standup. Everyone talks about what they did yesterday, what they're doing today, and any blockers. Everyone tunes out except for their own status update. The product manager and the tech lead pay some attention. Maybe once in a week or two, something interesting is said. 🥱&lt;/p&gt;

&lt;p&gt;During a retro in the midst of a busy time for our team, we surfaced that we had too many meetings. And we're not alone in this. It was just too much with the looming deadlines. &lt;a href="https://www.fastcompany.com/90888605/shopify-exec-this-is-what-happened-when-we-canceled-all-meetings" rel="noopener noreferrer"&gt;Shopify went extreme on dropping meetings a few years back&lt;/a&gt;. The daily standup was one of many meetings we talked about, and everyone had the same feeling: &lt;strong&gt;this meeting needed a refactor&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;We discussed what the actual intent of our daily standup was, and how we could preserve that while also reducing the pressure of meetings. Here's what we came up with:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Daily Standup Goals&lt;/strong&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The product manager needs to &lt;strong&gt;know the status of certain tasks&lt;/strong&gt; beyond what Linear reports. Not all tasks, not every day, but it's important for them to have their finger on the pulse.&lt;/li&gt;
&lt;li&gt;The tech lead also needs to get a sense for people &lt;strong&gt;spinning their wheels&lt;/strong&gt;, working on things for longer than expected, and how &lt;strong&gt;projects are moving along&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;People need help getting unblocked&lt;/strong&gt;. A topic for another time is how long to try to unblock yourself before asking for help. But, our team is generally good at reaching out outside of meetings. However, sometimes we need a dedicated time and place to make sure a block is in front of everyone. &lt;/li&gt;
&lt;li&gt;We're a fully remote team. We need &lt;strong&gt;face time&lt;/strong&gt;. We need to bond and build / maintain team &lt;strong&gt;culture&lt;/strong&gt;. But we don't want more meetings.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;strong&gt;Our Team's Solution&lt;/strong&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Daily SlackBot async standup. A bot sends a message to our team handle in slack, and we respond with &lt;strong&gt;Y&lt;/strong&gt;esterday, &lt;strong&gt;T&lt;/strong&gt;oday, and &lt;strong&gt;B&lt;/strong&gt;lockers. Sometimes it's just a status update, sometimes it's a funny quip about our kids being blockers. Conversation sometimes ensues. Other times, it doesn't. And that's perfect.&lt;/li&gt;
&lt;li&gt;On Mondays, Wednesdays, and Fridays, we have a 15 minute team sync. This is meant to be a chat about what we're working on that's interesting. Blockers that we want to dive deep on or just vent. And sometimes it's just a coffee chat, and we talk about plans for the weekend or some cool new toy. &lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;This dropped a mere 30 minutes of meetings from the week, but we don't dread the remaining 45 minutes anymore. The time is useful for everyone. &lt;/p&gt;

&lt;p&gt;All our goals are met, we have fewer standup meetings, and the meetings we do have show much higher engagement and enjoyment. Win win win!&lt;/p&gt;

&lt;p&gt;What do you think? Do you find the daily stand up useful? Who is it for in your team? How do you build and reinforce culture on your remote team? &lt;/p&gt;

</description>
      <category>leadership</category>
      <category>culture</category>
      <category>workplace</category>
      <category>meetings</category>
    </item>
    <item>
      <title>Finding Unused Indexes in Postgres</title>
      <dc:creator>PJ Hoberman</dc:creator>
      <pubDate>Wed, 18 Sep 2024 21:24:12 +0000</pubDate>
      <link>https://dev.to/pjhoberman/finding-unused-indexes-in-postgres-4k61</link>
      <guid>https://dev.to/pjhoberman/finding-unused-indexes-in-postgres-4k61</guid>
      <description>&lt;p&gt;Database indexes are incredibly important in production systems. Single column ones are obvious, but multi-column (composite) indexes are game-changers for speeding up commonly used queries by several orders of magnitude. &lt;/p&gt;

&lt;p&gt;But sometimes, we (I) create a few different indexes over time, and it's not always obvious when to remove old indexes. Indexes take up space and slow down writes, so it's important to monitor and clean them up periodically. &lt;/p&gt;

&lt;p&gt;Here is a postgres query to give you some insight:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; 
    &lt;span class="n"&gt;relname&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="k"&gt;table_name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; 
    &lt;span class="n"&gt;indexrelname&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;index_name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; 
    &lt;span class="n"&gt;idx_scan&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;index_scans&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; 
    &lt;span class="n"&gt;idx_tup_read&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;tuples_read&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; 
    &lt;span class="n"&gt;idx_tup_fetch&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;tuples_fetched&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; 
    &lt;span class="n"&gt;pg_stat_user_indexes&lt;/span&gt;
&lt;span class="k"&gt;JOIN&lt;/span&gt; 
    &lt;span class="n"&gt;pg_index&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;pg_stat_user_indexes&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;indexrelid&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;pg_index&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;indexrelid&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; 
    &lt;span class="n"&gt;schemaname&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'public'&lt;/span&gt;
&lt;span class="k"&gt;ORDER&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; 
    &lt;span class="n"&gt;idx_scan&lt;/span&gt; &lt;span class="k"&gt;DESC&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The output will be the table name, index name, and some data about each index including how many times the index was used in a query, how many tuples (index rows in this case) were read from the index, and how many tuples were actually fetched after all filtering was complete.&lt;/p&gt;

&lt;p&gt;The output will contain data since the server was last restarted or the statistics were last reset. Here is some example output from a production server. I removed a bunch of rows to show heavily used indexes and some that aren't used at all:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt; table_name    |          index_name         | index_scans | tuples_read | tuples_fetched 
---------------+-----------------------------+-------------+-------------+----------------
 items         | items_pkey                  | 17566068467 | 22444742841 | 21762928697
 routes        | routes_item_id_key          |  4046022477 |  2541792837 | 2521785009
 items         | items_url_idx               |  1520426292 |  7556543480 | 1518612148
 authors       | authors_pkey                |   211481111 |    45577051 |   42726045
 logs          | logs_type_coord_uniq        |     6437114 |     1462603 |     1392484
 spatial_ref   | spatial_ref_pkey            |     2060726 |    13792886 |     2056566
 users         | users_pkey                  |     1872578 |     2214935 |     1872578
 ...
 rate_limits   | rate_limit_key_like         |           0 |           0 |           0
 blocks        | blocks_pkey                 |           0 |           0 |           0
 blocks        | blocks_uniq                 |           0 |           0 |           0
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;As you can see, some of these indexes are never used! I have some work ahead of me now: manually review these indexes and potentially remove unused ones to save on space and improve write performance. Eventually, I'd like to set up automated alerts to let me know that some indexes aren't being used at all.&lt;/p&gt;

&lt;p&gt;Let me know if this is helpful for you or if you've gone further and automated anything like this!&lt;/p&gt;

</description>
      <category>postgres</category>
      <category>postgressql</category>
      <category>backend</category>
    </item>
    <item>
      <title>LaunchDarkly across multiple celery tasks</title>
      <dc:creator>PJ Hoberman</dc:creator>
      <pubDate>Fri, 21 Jul 2023 14:36:36 +0000</pubDate>
      <link>https://dev.to/pjhoberman/launchdarkly-across-multiple-celery-tasks-1fb5</link>
      <guid>https://dev.to/pjhoberman/launchdarkly-across-multiple-celery-tasks-1fb5</guid>
      <description>&lt;p&gt;We were running into an issue recently where LaunchDarkly wasn't evaluating on celery servers. Opening a python shell on the boxes showed the keys were setup correctly, and that if we called the methods directly, the flags evaluated as expected. However, when called as a delayed task, the flags weren't evaluating.&lt;/p&gt;

&lt;p&gt;LaunchDarkly makes use of &lt;code&gt;fork&lt;/code&gt; in python, and requires that only one LaunchDarkly client instance exists. &lt;a href="https://stonesoupprogramming.com/2017/08/26/python-basic-forking/"&gt;This post&lt;/a&gt; is a good primer on forking in python. It appeared that this was the issue.&lt;/p&gt;

&lt;p&gt;My coworker Doug explained it thusly:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;The LaunchDarkly library makes use of threading, Celery starts the main "control" process that uses &lt;code&gt;fork()&lt;/code&gt; to start &lt;code&gt;n&lt;/code&gt; workers, based on your concurrency setting.&lt;/p&gt;

&lt;p&gt;Forking copies the current process into a new one, but in Python it kills off all but the thread doing the forking.  All others are stopped. So the threads that LaunchDarkly starts up during initialization (e.g., EventDispatcher or StreamingUpdateProcessor) end up "defunct" or unpredictable.&lt;/p&gt;

&lt;p&gt;The Locks used within LaunchDarkly are thread independent, but because threads are killed off in the child, you end up with an invalid state and can’t trust things will work.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Further, &lt;a href="https://docs.launchdarkly.com/sdk/server-side/python#configuring-uwsgi"&gt;LaunchDarkly recommends a post fork hook to initialize the client&lt;/a&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;import uwsgidecorators

@uwsgidecorators.postfork
def post_fork_client_initialization():
    ldclient.set_config(LDConfig("sdk-key-123abc"))
    client = ldclient.get()
end
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;However, our Django application uses &lt;code&gt;asgi&lt;/code&gt;, which doesn't currently have this hook. This is our current LaunchDarkly configuration &lt;code&gt;launch_darkly.py&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;import atexit
import sys

from django.conf import settings

"""
Sets up Launch Darkly for use across the site.
LD is already initialized. See discovery_service.views.ld_check for example usage.
"""


class LDClient():
    def __getattr__(self, v):
        if 'ldclient' in sys.modules:
            import ldclient
        else:
            import ldclient
            from ldclient.config import Config
            ldclient.set_config(Config(settings.LAUNCH_DARKLY_SDK_KEY))
        return getattr(ldclient.get(), v)


ld_client = LDClient()

@atexit.register
def close_ld(*args, **kwargs):
    # LD recommends closing upon app shutdown
    # https://docs.launchdarkly.com/sdk/server-side/python
    ld_client.close()
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;LDClient&lt;/code&gt; class allows us to ignore new instantiations of the &lt;code&gt;ldclient&lt;/code&gt; library if it's already been loaded.&lt;/p&gt;

&lt;p&gt;And the general use is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;from launch_darkly import ld_client

def flagged_code():
    flag = ld_client.variation("flag-name", {"key": 12345}, False)  # False is the default in this case
    if flag:
        // do something if the flag is on
    else:
        // do something else if the flag is off
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;After a lot of bashing through walls, aka iterative development, we discovered two things:&lt;/p&gt;

&lt;h5&gt;
  
  
  1. Module Level instantiation
&lt;/h5&gt;

&lt;p&gt;There was a module-level instantiation of the LaunchDarkly client that was causing the library to initialize before the fork. &lt;/p&gt;

&lt;p&gt;Basically, the above code, but instead:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;from launch_darkly import ld_client

flag = ld_client.variation("flag-name", {"key": 12345}, False)      # False is the default in this case


def flagged_code():

    if flag:
        // do something if the flag is on
    else:
        // do something else if the flag is off
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;So that code was removed / refactored.&lt;/p&gt;

&lt;h5&gt;
  
  
  2. Celery initialization
&lt;/h5&gt;

&lt;p&gt;In our &lt;code&gt;celery.py&lt;/code&gt; code, we added a &lt;code&gt;worker_process_init&lt;/code&gt; hook to initialize the library properly. This ensures that when the celery workers fork, there is definitely a ldclient ready to go for any code that requires it.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;@worker_process_init.connect
def configure_worker(signal=None, sender=None, **kwargs):
    """Initialize the Launch Darkly client for use in Celery tasks."""
    try:
        res = ld_client.variation("test-flag", {"key": 0}, 0)
        logging.info(f"LD client initialized for Celery worker. {res}")
    except Exception:
        import traceback
        traceback.print_exc()
        logger.error("Error initializing LD client for Celery worker.", exc_info=True)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;To aid in future discovery and debugging, we also created a celery task that we can call on the fly to make sure things are working:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;
@shared_task
def celery_ld_check(flag="test-flag", key=0, default="not found"):
    """
    Test LaunchDarkly SDK connectivity from Celery.
    """

    print("trying celery_ld_check")
    try:
        variation = ld_client.variation(flag, {"key": key}, default)
        print(f"celery_ld_check: {variation}")
    except Exception as e:
        print(f"celery_ld_check: {e}")
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Lastly, we will likely iterate on the &lt;code&gt;LDClient&lt;/code&gt; class to deal with issues regarding the fork on the fly.&lt;/p&gt;

&lt;p&gt;Let me know if this helps you in your code, or sparks any ideas for you!&lt;/p&gt;

</description>
      <category>python</category>
      <category>backend</category>
    </item>
  </channel>
</rss>
