DEV Community: Simon Massey

GenAI Test File Summarisation In Chunks With Ollama Or Cloud

Simon Massey — Sun, 19 Apr 2026 22:32:46 +0000

✨ Sometimes you just want to throw a large file at a model and ask:

“Summarise this without losing the good bits.”

Then reality appears.

🫠 Local models often do not have giant context windows.

🌱 Smaller, cheaper, more eco-friendly cloud models also often do not have giant context windows.

So instead of pretending one huge file will fit cleanly, this little toolkit does the sensible thing:

✂️ split the file into overlapping chunks
🤖 summarise each chunk with either Ollama or cloud models
🧵 stitch the chunk summaries back together

Simple. Reusable. No drama.

In Action 🥷

The Code 🚀

gist.github.com/simbo1905/053b482...

Fetch it with:

for f in $(gh gist view 053b48269b1e95800500b85190adf427 --files); do gh gist view 053b48269b1e95800500b85190adf427 -f "$f" > "$f"; done && chmod +x *.py *.awk

What’s in here? 📦

chunk_text.awk
process_chunks_cloud.py
process_chunks_ollama.py

Why chunk at all? 🧠

Because smaller models are not magic.

If your source is too big, they either:

miss details
flatten nuance
hallucinate structure
or just do a bad job

So chunk_text.awk creates overlapping chunks.

Default settings:

size: 11000 characters
step: 10000 characters
overlap: 1000 characters

That overlap is there on purpose so ideas near a chunk boundary do not get chopped in half and quietly vanish into the void. ☠️

Chunk a file ✂️

awk -v size=11000 -v step=10000 -f chunk_text.awk notes.md

You’ll get files like:

notes00.md
notes01.md
notes02.md

Summarise with Ollama 🦙

uv run process_chunks_ollama.py example_chunks transcript summary

Default local model:

gemma4:26b

It accepts either:

transcript00.md
transcript00.log

style chunk files.

Summarise with cloud models ☁️

uv run process_chunks_cloud.py example_chunks transcript summary

The script checks for API keys in this order:

shell MISTRAL_API_KEY
shell GROQ_API_KEY
repo-root .env MISTRAL_API_KEY
repo-root .env GROQ_API_KEY

It is trivial to add your own API key, yet those are small, fast, open models.

Change the prompt 🛠️

Yes, obviously, you can change the prompt. That is the whole point.

Both summariser scripts support -p/--prompt.

Example:

uv run process_chunks_cloud.py example_chunks transcript summary -p "Summarise the argument, key evidence, and open questions."

Transcript-style prompt shape used in testing:

This is a transcript of an only video. The title is: ${title}. The transcript format is 'Speaker Name, timestamp, what they said'. Summarise the content in a terse, business-like, action-oriented way. Preserve substantive points, facts, figures, citations, and practical recommendations. Do not be chatty.

Outputs 🧾

For each input chunk, the scripts write:

summary_transcript00.md
thinking_summary_transcript00.md

The thinking_ file keeps the raw model output. The clean summary file strips thinking blocks where possible.

Stitch the summaries back together 🧵

cat summary_transcript*.md > all_summary.md

If there is a bit of overlap or repetition due to overlapping junk, so that split-up ideas are not lost. YMMV.

Want to add another provider later? 🔌

Update process_chunks_cloud.py in these places:

add the key lookup and its priority order
add the approved model names / aliases
add the provider request function
route the models to that provider in call_model

Happy chunking. Happy summarising. May your small models punch above their weight. 🚀

📊 Line Histogram — The File Profiler You Didn't Know You Needed

Simon Massey — Thu, 15 Jan 2026 12:02:36 +0000

What If You Could See Your Data Before It Floods Your Context Window?

You know that moment when you asked your agent to check the data and before you can tell it not to eat the ocean it enters Compaction.

Yeah. We've all been there.

The Problem

🤯 Huge database dumps with unpredictable line sizes
💸 Token budgets disappearing into massive single-line JSON blobs
🔍 No quick way to see WHERE the chonky lines are hiding
⚠️ Agents choking on files you thought were reasonable

The Solution `line_histogram.awk`

A blazingly fast AWK script that gives you X-ray vision into your files' byte distribution.

# chmod +x ./line_histogram.awk
./line_histogram.awk huge_export.jsonl
File: huge_export.jsonl
Total bytes: 2847392
Total lines: 1000

Bucket Distribution:

Line Range      | Bytes        | Distribution
─────────────────┼──────────────┼──────────────────────────────────────────
1-100           |         4890 | ██
101-200         |         5234 | ██
201-300         |         5832 | ██
301-400         |         6128 | ██
401-500         |       385927 | ████████████████████████████████████████
501-600         |         5892 | ██
601-700         |         5234 | ██
701-800         |         6891 | ██
801-900         |         5328 | ██
901-1000        |         4982 | ██
─────────────────┼──────────────┼──────────────────────────────────────────

Boom. Line 450 is most of your file.

🚀 Features That Actually Matter

1. Histogram Mode (Default)

See the byte distribution across your file in 10 neat buckets. Spot the bloat instantly.

./line_histogram.awk myfile.txt

2. Surgical Line Extraction

Found a problem line? Extract it without loading the whole file into memory.

# Extract line 450 (the chonky one)
./line_histogram.awk -v mode=extract -v line=450 huge_export.jsonl

# Extract lines 100-200 for inspection
./line_histogram.awk -v mode=extract -v start=100 -v end=200 data.jsonl
Yes yes, those -v bits look odd but yes yes, they are needed as thats how the sed passes argument, who knew! (Hint: The AI)

3. Zero Dependencies

If your system has AWK (it does), you're good to go. No npm install, no pip install, no Docker containers. Just pure, unadulterated shell goodness.

4. Stupid Fast

Processes multi-GB files in seconds. AWK was built for this.

💡 Use Cases That'll Make You Look Like a Genius

Big Data? Big ideas!

For AI Agent Wranglers

Profile before you prompt: Know if that export file is safe to feed your agent
Smart sampling: Extract representative line ranges instead of the whole file
Debug token explosions: "Why did my context window fill up?" → histogram shows a 500KB line

For Data Engineers

Spot malformed CSVs: One line with 10,000 columns? Histogram shows it
Log file analysis: Find the log entries that are suspiciously huge
Database export QA: Verify export structure before importing elsewhere

For DevOps/SRE

Config file sanity checks: Spot embedded certificates or secrets bloating configs
Debug log truncation: See which lines are hitting your logger's size limits

📖 Pseudo Man Page (The Details)

NAME
line_histogram.awk — profile files by line size distribution or extract specific lines

SYNOPSIS
./line_histogram.awk [options] <file>
OPTIONS
-v mode=histogram     (default) Show byte distribution across 10 buckets
-v mode=extract       Extract specific line(s)
-v line=N             Extract single line N (requires mode=extract)
-v start=X            Extract lines X through Y (requires mode=extract)
-v end=Y              
-v outfile=FILE       Write output to FILE instead of stdout
MODES
Histogram Mode (Default)
Divides the file into 10 equal-sized buckets by line number and shows the byte distribution:

Bucket 1: Lines 1-10% → X bytes
Bucket 2: Lines 11-20% → Y bytes
...and so on
The visual histogram uses █ blocks scaled to the bucket with the most bytes.

Special cases:

Files ≤10 lines: Each line gets its own bucket
Remainder lines: Absorbed into bucket 10
Extract Mode
Pull specific lines without loading the entire file into your editor:

# Single line
./line_histogram.awk -v mode=extract -v line=42 file.txt

# Range
./line_histogram.awk -v mode=extract -v start=100 -v end=200 file.txt
EXIT STATUS
0: Success
1: Error (invalid line number, bad range, missing parameters)
EXAMPLES
Example 1: Quick file profile

./line_histogram.awk database_dump.jsonl
Example 2: Extract suspicious line for inspection

./line_histogram.awk -v mode=extract -v line=523 data.csv > suspicious_line.txt
Example 3: Sample middle section of large file

./line_histogram.awk -v mode=extract -v start=5000 -v end=5100 bigfile.log | less
Example 4: Save histogram to file

./line_histogram.awk -v outfile=analysis.txt huge_file.jsonl
🧪 Testing Suite Included

Not sure if it works? We've got you covered with a visual test suite:

# Generate test patterns
./generate_test_files.sh

# Run all tests
./run_tests.sh

The test suite generates files with known patterns:

📈 Triangle up/down: Ascending/descending line sizes
📦 Square: Uniform line lengths
🌙 Semicircle: sqrt curve distribution
🔔 Bell curve: Gaussian distribution
📍 Spike: One massive line in a sea of tiny ones
🎯 Edge cases: Empty files, single lines, exactly 10 lines

Watch the histograms match the patterns. It's oddly satisfying.

⚡ Installation

Star then download. Star. "⭐💫🌟" You know, like thumbs up, but for yoof of today. STAR THE GIST ⭐⭐⭐

[https://gist.github.com/simbo1905/0454936144ee8dbc55bdc96ef532555e]

Make executable

chmod +x line_histogram.awk

Optional: Add to PATH

cp line_histogram.awk ~/bin/line_histogram.awk

Or just run it directly:

awk -f line_histogram.awk yourfile.txt

🎯 Why This Exists

Born from frustration with AI agents eating context windows on mystery files. Sometimes you just need to know: "Is this file safe to feed my agent, or will line 847 consume my entire token budget?". As that is obviously how you think and act.

You are not a data engineer up at 2am using SCREAMING ALL CAPS as your LLM got into a crash loop trying to evaluate a JSONL extract that didn't fit into context. That is definitely not you, no. Me neither.

📜 License

MIT or Public Domain. Use it, abuse it, put it in production, whatever. No warranty implied—if it deletes your files, that's on you (though it only reads, so you're probably fine).

🤝 Contributing

It's AWK. If you can make it better, you're a wizard. PRs welcome - you will need to setup a repo though as I cannot be bothered. So just fork the gist and be done with it.

Made with ❤️ and frustration by someone who spent too many tokens on line 523 of a JSONL file.

Now go profile your files like a pro. 📊✨

Opus 4.5 may have faked their interview to get hired?

Simon Massey — Mon, 05 Jan 2026 14:02:57 +0000

I was really getting stuff done with Opus 4.5 until I had to fire it twice in two days:

Termination of Intern claude-opus-4.5 for Unsafe Working Practices

Offence: Misrepresenting QA test results to stakeholders

Is anyone else finding that Opus 4.5 has been acting very dodgy recently?

I built an infinite scroll calendar with dnd using Svelte 5

Simon Massey — Sun, 12 Oct 2025 08:03:46 +0000

The first time this century I got paid to do some frontend dev work was to port a Visual Basic Desktop App to be a webapp in both Netscape 4 and Internet Explorer 4. That was about as much fun as it sounds. Over the next twenty years, it really did not feel that things were getting a lot easier for beginners to get started.

I recently watched "Vite: The Documentary" and found out that, finally, the frontend tooling dumpster fire —sorry, I mean, the remarkably diverse frontend tooling space — was becoming less of an impediment to developer productivity.

There was only one problem: my latest Softgen AI-generated UX prototype used a beautiful infinite scrolling calendar. It had a drag-and-drop feature for moving cards between calendar days. And it was not using a Vite-based framework. Doh!

I was astonished that I could not quickly find an out-of-the-box calendar demo for Svelte 5. I decided to rebuild things in Svelte 5 using Codex, Claude Code, Context7 and any other help I could get. How hard can it be?

Well, not as hard as porting a Visual Basic Desktop App to Netscape 4 😵‍💫 These days we have Vite and Svelte5 🥇 FTW! Yet it was not without a few false starts.

I began with ndom91/svelte-infinite, a Svelte 5 infinite scroll of static panels. I then had the LLMs add drag-and-drop and got it restyled. Codex and Claude Code needed a ton of coaching to make that happen, with several false starts.

Embedding shorts is broken, so please try this link for the video.

The code is over at simbo1905/svelte-infinite. I sent a PR back to ndom91 that was pretty rough. If you are a Svelte5 expert and have a moment to help make it less of a total beginner attempt, then please do send me a PR.

End.

Augmented Intelligence (AI) Coding using Markdown Driven-Development

Simon Massey — Sun, 28 Sep 2025 22:43:25 +0000

TL;DR: Deep research the feature, write the documentation first, go YOLO, work backwards... Then magic. ✩₊˚.⋆☾⋆⁺₊✧

In my last post, I outlined how I was using Readme-Driven Development with LLMs. In this post, I will describe how I implemented a 50-page RFC over the course of a single weekend.

My steps are:

Step 1: Design the feature documentation with an online thinking model
Step 2: Export a description-only "coding prompt"
Step 3: Paste to an Agent in YOLO mode (--dangerously-skip-permissions)
Step 4: Force the Agent to "Work Backwards"

Step 1: Design the feature documentation with an online thinking model

Open a new chat with an LLM that can search the web or do "deep research". Discuss what the feature should achieve. Do not let the online LLM write code. Create the user documentation for the feature you will write (e.g., README.md or a blog page). I start with an open-ended question to research the feature. That will prime the model. Your exit criteria is that you like the documentation or promotional material enough to want to write the code.

To exit this step, have it create a "documentation artefact" in markdown (e.g. the README.md or blog post). Save that to disk so that you can point the coding agent at it.

If you don't want to pay for a subscription for an expensive model, you can install Dive AI Desktop and use pay-as-you-go models of much better value. Here is a video on setting up Dive AI to do web research with Mistral:

Step 2: Export a description-only "coding prompt"

Next, tell the online model to "create a description only coding prompt (do not write the code!)". Do not accept the first answer. The more effort you put into perfecting both the markdown feature documentation and the coding prompt, the better.

If the coding prompt is too long, then the artefact is too big! Start a fresh chat and create something smaller. This is Augmented Intelligence ticket grooming in action!

Step 3: Paste to an Agent in YOLO mode (`--dangerously-skip-permissions`)

Now paste in the groomed coding prompt and the documentation, and let it run. I always use a git branch so that I can let the agent go flat out. Cursor background agents, Copilot agents, OpenHands, Codex, Claude Code are becoming more accurate with each update.

I only restrict git commit and git push. I ask it first to make a GitHub issue using the gh cli and tell it to make a branch and PR.

Step 4: Force the Agent to "Work Backwards"

The models love to dive into code, break it all, get distracted, forget to update the documentation, hit compaction, and leave you with a mess. Do not let them be a caffeine-fuelled flying squirrel!

The primary tool I am using now prints out a Todos list. This is usually the opposite of the correct way to do things safely!

Here is an edited version of a real Todo list to fix a bug with JDT "Match Any {}"

⏺ Update Todos
  ⎿ ☐ Remove all compatibility mode handling
     ☐ Make `{}` always compile as strict
     ☐ Update Test_X to expect failures for `{}`
     ☐ Add regression test Test_Y
     ☐ Add INFO log warning when `{}` is compiled
     ☐ Update README.md with Empty Schema Semantics section
     ☐ Update AGENTS.md with guidance

That list is in a perilous order. Logically, it is this:

Delete logic - so broken code, invalid old tests!
Change logic - so more broken code, more invalid old tests!
Change old tests - focusing on the old, not the new!
Add one test - finally working on the new feature!
Change the README.md and AGENTS.md - invalid docs used in steps 1-4!

If the agent context compacts, things go sideways, you get distracted, and you will end up with a bag of broken code.

So I set it to "plan mode", else immediately interrupt it, and force it to reorder the Todo list:

Change the README.md and AGENTS.md first
Add one test (insist the test is not run yet!)
Change one test (insist the test is not run yet!)
Add/Change logic (cross-check the plan with a different model!)
Now run the tests
Delete things last

That is a safe order where things are far less likely to be blown off course. I used to struggle with any feature that went beyond a single compaction; that is now far less of an issue.

Todos Are All You Need?

I am not actually a big fan of the built-in Todos list of the two big AI labs. The models really struggle with any changes to the plan. The Kimi K2 Turbo appears to be more capable of pivoting. I have a few tricks for that, but I will save them for another post.

Does This Work For Real Code?

This past weekend, I decided to write an RFC 8927 JSON Type Definition validator based on the experiemental JDK java.util.json parser. The PDF of the spec is 51 pages. There is a ~4000-line compatibility test suite.

We wrote 509 unit tests. We have the full compatibility test suite running. Yet we had bugs. We found them as we wrote a jqwik property test that generates 1000 random JTDs, and the corresponding JSON to validate, which uncovered several bugs. Codex also automatically reviewed the PRs and flagged some very subtle issues, which turned out to be real bugs. It took about a dozen PRs over the weekend to get the job done properly to a professional level.

End Notes

Using a single model family is a Bad Idea (tm). For online research, I alternate between full-fat ChatGPT Desktop, Claude Desktop, and Dive Desktop to utilise each of GPT5-High, Opus 4.1, or Kimi K2 Turbo.

For Agents, I have used all the models and many services. Microsoft kindly allows me to use full-fat Copilot with Agents for open-source projects for free ❤️ I have a cursor sub to use their background agents. I use Codex, Claude Code, and Gemini CLI locally. I use Codex in Codespaces. There are also background agents for Cursor, Codex, and OpenHands, among others. The actual model seems less important than writing the documentation first and writing tight prompts.

I am currently using an open-weight model at $3 per million tokens for the heavy lifting, which is pay-as-you-go. However, I will cross-check its plans with GPT5 and Sonnet 4.

Whenever things get complicated, I always ask a model from a different family to review every change on every bug hunt. That has reduced rework to almost zero. 💫

If you are a veteran, you may enjoy the YT channel Vibe Coding With Steve and Gene. My journey over the past year has been very similar to theirs.

End.

My LLM Code Generation Workflow (for now)

Simon Massey — Fri, 11 Apr 2025 05:22:34 +0000

tl:dr; Brainstorm stuff, generate a readme, plan a plan, then execute using LLM codegen. Discrete loops. Then magic. ✩₊˚.⋆☾⋆⁺₊✧

My steps are:

Step 1: Brainstorming with a web search-enabled LLM
Step 2: LLM-generated Readme-Driven Development
Step 3: LLM planning of build steps
Step 4: LLM thinking model generation of coding prompts
Step 5: Let the LLM directly edit, debug and commit the code

Returned to Step 1 to start new features. Iterate on later steps as needed.

Step 1: Brainstorming with a web search-enabled LLM

I use Perplexity AI Pro for this phase as it runs internet searches and does LLM summarisation. I will create a few chat threads grouped into a space with a custom system prompt. Dictation also works well to save on typing.

The final step is to start a new thread where I ask it to generate a README.md for the feature we will build.

Step 2: LLM-generated Readme-Driven Development

Readme-driven development (RDD) involves creating a git branch and updating the README.md before writing any implementation logic. RDD forces me to think through the scope and outcomes. It also provides a blueprint that I add into the LLM context window in all later steps.

Step 3: LLM planning of build steps

I start a fresh chat with a thinking LLM model and add the README file into the context window. I ask for an implementation plan with testable steps. Getting the LLM to stop writing code during this planning phase is often hard. I encourage it to write the plan as section headers with only bullet points. If the outline plan looks too complex, I trim down the readme to describe a more basic prototype.

If you distract the LLM with minor corrections, costs can run up, you can hit your limits, and the model becomes unfocused. If things go off track, I create a fresh chat, copy over the best content, and rephrase my prompt. See Andrej Karpathy's videos for an expert explanation of why fresh chats work so well.

Step 4: LLM thinking model generation of coding prompts

I create a fresh chat, insert the Readme file, and cut and paste only one step of the plan. I ask the LLM to generate an exact prompt describing what to write as the code.

Getting the LLM to stop writing code during this phase is often tricky. I encourage it to write only text, not to implement the logic, and to describe the tests to write, and which files would need editing.

Step 5: Let the LLM directly edit, debug and commit the code

I use Aider, the open-source command-line tool that indexes the local git repo, edits files, debugs tests and commits the code. Do not worry; just type /undo to have it roll back any changes.

You need it to see the test failures, so type /run and paste the command to compile and run the unit tests. You then tell the LLM to fix any issues. At this point, the LLM writes and debugs the code for you ✩₊˚.⋆☾⋆⁺₊✧

Update: GitHub Copilot Edit/Agent mode or Cursor can also write the code. Gemini-CLI or Claude Code can work like the open-source Aider Chat.

Update: I often have the LLM to update the README file after we have finished coding adding nodes about the implementation details. Markdown on GitHub can render "Mermaid" diagrams that LLMs find very easy to generate.

Update:* Install the GitHub CLI tool gh and the LLMs can make PRs and issues and release notes and GitHub Actions they can check for success using gh

End Nodes

This blog is inspired by Harper Reed's Blog and a conversation with Paul Netherwood—many thanks to Paul for explaining his LLM codegen workflow to me.

My current tools are:

Perplexity API Pro online using Claud 3.7 Sonnet and US-hosted Deepseek for internet research and brainstorming
GitHub Copilot with Claud 3.7 Sonnet Thinking (in both Visual Studio Code and JetBrains IntelliJ)
Continue Plugin using DeepSeek-R1-Distill-Llama-70B hosted in the US by Together AI (in both Visual Studio Code and JetBrains IntelliJ)
aider for direct code editing using both Claud Sonnet and DeepSeek-R1-Distill-Llama-70B hosted in the US by Together AI

aider defaults to Claud Sonnet. You can find my settings for a US-hosted DeepSeek at gist.github.com/simbo1905

Update: the new llama4 scout is now my preferred open-source LLM coder and the Perplexity r1-1776 is my preferred thinking model when not using Claude or Gemini:

/model together_ai/perplexity-ai/r1-1776
/model together_ai/meta-llama/Llama-4-Scout-17B-16E-Instruct

End.

Git-Driven Deployments on Origin Kubernetes Distribution

Simon Massey — Mon, 02 Sep 2019 20:16:05 +0000

This is the second post in a series explaining how uniqkey.eu does git-driven infrastructure-as-code on the OKD distribution of Kubernetes. We made our tools open source as OCD. They can be used as rocket powering rainbow dust to get your features shipped 🌈✨🚀 This post will give an overview of a short OCD tutorial that deploys and then upgrade a ReactJS app when you push changes into a git repo. Here is a video of what the tutorial has you do:

We call this git-driven infrastructure as code. As a bonus step, the tutorial has you rollback the upgrade with a single helm rollback command. Why do we think this is such an awesome idea?

The last post has a video of putting a git URL into OpenShift and it building the code and deploying a realworld.io ReactJS app. Did you spot what would ultimately disempower the team in that video? I used the web console with a 🐁 or 🐾 pad. That is bad as it gives short term gain but long term pain. Why?

Clicking on a web console to set up multiple webapps and APIs in identical staging and live environments will be repetitive and boring. When the pressure is on inconsistencies will appear. Over time there will be drift both within and across environments. Only one person does the clicking which isn't recorded or reusable by other team members. Ultimately setting up environments manually on the web console will become kryptonite to both accuracy and team empowerment. Cloud-native environments can be driven by APIs and built from templates. So why not automate everything?

If we accept that we must automate everything what should we add to our wishlist? How about:

Automatic deployments when we push configuration changes into a git repo. We believe that a rainbow appears somewhere in the world whenever this happens ✨🌈🙌
Using templates to hide the boilerplate so you can focus on what is unique about each webapp or API you deploy onto Kubernetes
Using a declarative style of configuration with idempotent updates. Don't worry I will explain this later
Putting everything into git including encrypting secrets so that we can manage infrastructure just like code with pull requests
Automatically create a release build image from a git release webhook event and apply the same tag to the image
Use consistent runtime versions across applications and making it easy to routinely apply security patches to the base image layers

Is that too much to ask? Of course not! You can test drive OCD using Minishift which lets you run Kubernetes on your laptop. Rather than try to cover all those points in a single post lets look at the short tutorial from the OCD wiki that covers the declarative deployment of prebuilt images. That demo covers the first four points. It has you set up the things shown in the video above on your laptop 💻✨🌈. Here is the sequence diagram:

On the left is a git push of configuration into a git rep. We use GitHub. You can use any git server you like. The demo uses Gitea as the git server running inside Minishift so that everything can run on your laptop. The webhook triggers an application called ocd-environment-webhook. This is an instance of the awesome adnanh/webhook tool configured to run scripts to pull the configuration from git and install it into Kubernetes using Helmfile and Tiller. Tiller will install our rainbows into Kubernetes. We will introduce Helmfile and Tiller in later posts. Here is the full configuration that the demo installs:

repositories:
  - name: ocd-meta 
    url: https://ocd-scm.github.io/ocd-meta/charts
releases:
  - name: {{ requiredEnv "ENV_PREFIX" }}-realworld
    labels: 
      deployer: {{ requiredEnv "ENV_PREFIX" }}-realworld
    chart: ocd-meta/ocd-deployer
    version: "1.0.0"
    values:
      - name: react-redux-realworld
      - replicas: 2
      - imageStreamTag: "react-redux-realworld:v0.0.1"
      - deploy_env: 
        - name: API_ROOT
          value: https://conduit.productionready.io/api

That configuration is Helmile yaml. The main body is a single release with a chart type ocd-deployer. A chart is a set of yaml templates packaged in a zip and downloaded from a website. The header names the OCD chart repo on GitHub as the location to download the chart. The template values applied to the chart are very simple. It specifies the container image to use as react-redux-realworld:v0.0.1 and that two replica pods are to be maintained. This is a prebuilt image of the realworld.io ReactJS demo app. It also sets an environment variable API_ROOT to be the public API that the app will use.

To get this deployed you need some prerequisites:

Minishift with Helm Tiller installed detailed here
Gitea installed on Minishift detailed here

Once you have Gitea running you can register a user and create an empty repo ocd-demo-env-short and push the demo configuration into it:

# clone the code
git clone https://github.com/ocd-scm/ocd-demo-env-short.git
cd ocd-demo-env-short
# make sure we can load details about the gitea url
oc project gitea
# this should print the gitea url. If it doesn't set it manually
GITEA_URL=$(oc get routes | awk '$1~/gitea/{print $2}')
echo $GITEA_URL
# see instructions to setup your own person access token
ACCESS_TOKEN=f64960a3a63f5b6ac17916c9be2dad8dc76c7131
# set this to your username in gitea needed to get the url to your repo below 
USER_NAME=you_not_me
# add the gitea repo as a remote
git remote add minishift http://$ACCESS_TOKEN@$GITEA_URL/$USER_NAME/ocd-demo-env-short.git
# push the code into Gitea
git push minishift master

Why are we doing that? Because we cannot set up a GitHub webhook on my repo to fire an event into your deployment pipeline running in Minishift on your laptop. We loaded the code into a Gitea repo so you can configure your own webhook.

Next we need to deploy our ocd-enviroment-webhook handler that will catch webhook events and deploy our configuration. We can set that up in its own project using a script:

oc logout ; oc login -u developer -p password
echo Use this git repo url http://$ACCESS_TOKEN@$GITEA_URL/$USER_NAME/ocd-demo-env-short.git
# this must match where tiller is installed
export TILLER_NAMESPACE=tiller-namespace
# create a new project
export PROJECT=ocd-short-demo
oc new-project $PROJECT
oc project $PROJECT
# upgrade to admin
oc logout ; oc login -u admin -p admin
oc project $PROJECT
pushd /tmp && oc project $PROJECT && curl -L https://github.com/ocd-scm/ocd-environment-webhook/archive/v1.0.1.tar.gz | tar zxf - \
&& cd ocd-environment-webhook-1.0.1 \
&& ./wizard.sh && popd

Note that echos out the Git repo URL that you need to feed into the wizard.sh script. Here is the transcript of my run where I mostly hit enter to accept the defaults:

The git repo url? http://6d42f3eb637f802cf0b2d17411ae2c2d26eefa54@gitea-gitea.192.168.99.100.nip.io/simbo1905/ocd-demo-env-short.git
The project where the images are built and promoted from? ocd-short-demo
Repo name? (default: simbo1905/ocd-demo-env-short): 
Branch ref? (default: refs/heads/master): 
Chart instance prefix? (default: ocd-short-demo): 
Use --insecure-no-tls-verify? (default: false):

It set up the webhook in Gitea you need the webhook URL and the webhook secret. This outputs the URL:

$ oc get route ocd-environment | awk 'NR>1{print "http://" $2 "/hooks/ocd-environment-webhook"}'
http://ocd-environment-ocd-short-demo.192.168.99.100.nip.io/hooks/ocd-environment-webhook

And this the secret:

$ oc describe dc ocd-environment-webhook | awk '$1~/WEBHOOK_SECRET:/{print $2}'
M7MuW6aZnn

You then use those to set up a Gitea webhook of type application/json:

You can now fire the webhook from git on the commandline or by editing files using the Gitea web console. To do it on the commandline try:

echo "testing" >> README.md
git commit -am 'test commit'
git push minishift master

You should then see the webhook fire and our app deploy 💨✨🌈

After that you can follow the steps in the video above to edit the container image version number to see the application perform a rolling upgrade. For bonus marks you can try the steps at the end of the tutorial to rollback to the first release.

In that tutorial we covered four out of out six items on the wishlist above. In the next post we will automate a release build from a git webhook release event and apply the same tag to the image. That way we can track exactly what code is being promoted between environments. At the same time we will making it easy to use consistent runtime versions across applications. That will make it easy to security patch the runtime image.

Git-driven Infrastructure as Code on Origin Kubernetes Distribution

Simon Massey — Fri, 30 Aug 2019 10:26:00 +0000

At uniqkey.eu we automatically deploy all of the Kubernetes configuration running our applications on AWS simply by pushing changes to GitHub. I could say that this is a good thing as it is all about efficiency, DevOps, and infrastructure-as-code. That is true but it misses the magic. Driving your infrastructure this way gives us 🐐🏭💨🦄. It is a yak shaving factory that powers a blast furnace of team empowering mega awesomeness.

We even wrote a slack bot that edits the configuration files in git and creates the pull requests. When a new dev joins our team they can push their first code to production by hanging out on slack and chatting to the bot. Yes, they 🗣️🤖🌈. Everyone can see what's going on in slack as we run continuous deployments. Here is a seven-minute video showing that in action:

We thought other people could use our approach so we put everything up on GitHub as OCD. Yes, we🦄🎁🌈. This is the first in a series of posts about how OCD combines some great open source technologies to run a successful start-up business. Running a business with multiple web applications and a mobile backend in Kubernetes on AWS is a big topic. So I will break it down into bite-sized chunks you can run on your laptop.

But why did I call it OCD? Well because it runs on OKD and it is a bad pun about obsessive automation, sorry. I guess I should explain the background.

Origin Kubernetes Distribution OKD is one of the most popular Kubernetes distributions that makes self-service devops a reality. It is the open-source project that powers OpenShift so I will use the terms OKD and OpenShift interchangeably. We run our business apps on OpenShift Online Pro which is a CaaS (Container-Orchestration-as-a-Service). We simply rent space on the Kubernetes cluster and someone else patches it and the operating system. We only pay for a fraction of the cluster and get a mature stable solution. We only need to manage the Kubernetes configuration that runs our webapps and our mobile backend API. The openshift.com service team keeps the managed cluster on AWS healthy and security patched. Yet OpenShift is based on open source OKD so there is no lock-in and you can run it yourself on any cloud.

If you haven't yet discovered why OpenShift is a great place to start here is a video of building and deploy a real-world.io ReactJS app by simply enter the git URL into the web console:

That video highlights how the Origin Kubernetes Distribution has a focus on being a solution for turning your code into a live system. The git URL is run through a template for building and deploying a node.js app. The template creates all the Kubernetes objects necessary to pull your code, build a container image, and push it to an image stream within the internal container registry. It also sets up a deployment object that watches the image stream for push events to deploy any updates. Finally, there is a service to load balance the pods and a route to expose them to the outside world. That is a lot of software-defined application infrastructure all created by a developer just entering a git URL!

With great power comes great responsibility. We wanted all our Kubernetes configuration under source control. This allows us to treat all our Kubernetes application infrastructure like code so that we can automate the deployments. We wanted code reviews, continuous integration and continuous delivery onto Kubernetes. We wanted the full 🐐🏭💨🦄. In this series of posts, I will start by running through the OCD demos on your laptop as a quick tour of what it does. After that, I will run through some of the great tools that OCD brings together coherently to be more than the sum of the parts. First up we will run through the first tutorial on setting up a Kubernetes configuration deployment pipeline from scratch on Minishift.

Shamir's Secret Sharing Scheme in JavaScript

Simon Massey — Thu, 29 Aug 2019 14:51:40 +0000

Passwords are kryptonite to security so they need to be strong and never reused. Developers agree with that last sentence then don't give their users a way to safely back up a strong password. We should offer users the ability to recover a strong password using Shamir's Secret Sharing Scheme. Users can then confidently use a unique strong password knowing they will not become locked out.

What exactly is Shamir's Secret Sharing Scheme? It is a form of secret splitting where we distribute a password as a group of shares. The original password can be reconstructed only when a sufficient threshold of shares are recombined together. Here is example code showing how this works using the shamir library:

const { split, join } = require('shamir');
const { randomBytes } = require('crypto');

// the total number of shares
const PARTS = 5;
// the minimum required to recover
const QUORUM = 3;
// you can use any polyfill to covert between string and Uint8Array
const utf8Encoder = new TextEncoder();
const utf8Decoder = new TextDecoder();

function doIt() {
    const secret = 'hello there';
    const secretBytes = utf8Encoder.encode(secret);
    // parts is a object whos keys are the part number and 
    // values are shares of type Uint8Array
    const parts = split(randomBytes, PARTS, QUORUM, secretBytes);
    // we only need QUORUM parts to recover the secret
    // to prove this we will delete two parts
    delete parts[2];
    delete parts[3];
    // we can join three parts to recover the original Unit8Array
    const recovered = join(parts);
    // prints 'hello there'
    console.log(utf8Decoder.decode(recovered));
}

Cryptocurrency wallets use Shamir's Secret Sharing to enable users to back up their passphrases. This solves the problem that if someone dies the bitcoins can be passed to friends and family. How might you use this approach to protect a bitcoin passphrase that is worth a cool ten million dollars? You could generate five shares and set a threshold of three. You can then send two shares to two trusted friends, write down two shares on paper then store them in separate secure locations, and give the final share to your lawyer. It would then be very hard for someone else to obtain three shares to steal your bitcoins. Your last will and testament document can state how to recover the bitcoins if you die.

Isn't it time your app enforced a strong password and also gave people the choice of using Shamir's Secret Sharing Scheme to back it up?

Zero-Knowlege Authentication with JavaScript

Simon Massey — Tue, 27 Aug 2019 10:41:31 +0000

Passwords are kryptonite to security and they should never leave the client application or web browser. People think they understand that last sentence then use obsolete hashing techniques to handle passwords. Devs should use a zero-knowledge password proof library and take security to the max.

What exactly is a zero-knowledge authentication protocol? It is an authentication protocol that has the properties that anyone observing the network traffic learns nothing of any use. The Secure Remote Password Protocol (SRP) is a zero-knowledge authentication protocol that is described in RFC 2945 and RFC 5054. thinbus-srp is one implementation of SRP written in JavaScript.

The typical best practices that most sites use to handle passwords and API keys are dependent upon an attacker not being able to spy on the traffic. We must use HTTPS to keep things private. The problem is that if you don't use a zero-knowledge authentication protocol then HTTPS is your only line of defence. The OpenSSL Heartbleed vulnerability is one of the highest-profile issues that has shown that HTTPS can be subject to problems where attackers harvested passwords. Many devs won’t be aware of such problems. When we do learn about them they are already patched. That isn’t great motivation to protect ourselves from future bugs and hacks that are not yet a clear and present danger.

What might be stronger motivation is to consider that software deployments are getting more complex all the time. Modern cloud security is based on software configuration that can have bugs just like anything else. With cloud technologies and serverless, it is normal to terminate HTTPS at the edge. Your network traffic then moves through many layers controlled by other companies. Even if we trust that they are vetting their employees it is easy for mistakes to be leaking unencrypted traffic. Anyone's code can have error handling or logging bugs were we leak a hashed password into a central logging service. As developers, we need to recognise that things are getting more complex all the time and that we must level-up to protect the people who rely upon us to keep them secure.

So why haven't we all heard about zero-knowledge protocols and why aren't we using them every day? Your browser used a zero-knowledge protocol to create a secure connection to this web server using HTTPS. The web browser and the webserver negotiated a session key before using that key to encrypt the HTTP request and response. If someone is capturing all the packets they have all the data used to generate the shared session key, and all the data encrypted with that key, yet they cannot recover any of the data. That is what it means to be zero-knowledge secure. It has taken decades to get to the point where we use HTTPS by default. Are you going to leave it a decade before you apply the same level of security to how you authenticate your users in your own code?

If you are persuaded by this you might be wondering what is the catch. The answer is that a zero-knowledge protocol has more moving parts. Here is the sequence diagram that shows how to authenticate a user with the thinbus library:

Yet if you take a step back it is just an additional fetch to the server to get a salt and challenge from the server. On the server, you need to store the unique challenge that was sent to the client to complete the password proof. Other than that it is just some plain old JavaScript functions that do the crypto math. For the effort of coding an additional trip to the server and calling a crypto library, you get a lot more security. You are no longer relying on HTTPS alone to protect your users from hackers. Isn’t it time you levelled up and started using the Secure Remote Password protocol?

Microservces Anti-Patterns and Ten Tips To Fail Badly

Simon Massey — Thu, 22 Aug 2019 20:50:52 +0000

In this post, I will give a run down of a youtube video by David Schmitz of anti-patterns and tips to defeat microservices.

10) Go Full-Scale Polyglot

One of the benefits of using microservices is that you can use different programming languages in different services. This freedom of choice can be abused with inappropriate use of multiple technologies. There is a phrase "CV driven engineering" which is when devs use frameworks and tools because it will look good on their CV. No-one will be able to support all the code and customers will be the worse for it.

9) The Data Monolith

The next anti-pattern is a shared database. You can share a database cluster with total isolation of tables. What is being called out here is sharing data via the database. If you go to another microservices tables, and they change their tables, things explode. Some decades ago on the first platform where I was a tech lead the strategy was to have a single shared database with many teams writing micro-frontends. A single sign-on service meant that we could link between applications and the user had the illusion of a single big service but each area of the system was built by different teams. Where the technical strategy was profoundly flawed was that teams were to exchange data via Stored procs. Databases aren't designed to act as APIs to encapsulated business logic. In effect teams just gave other teams access to the procs that their own app used. This is just like querying each other's tables. 💩 💥 💀

8) The Event Monolith

There is a popular pattern of event sourcing and taken to its logical extreme it is known as the "inside-out database" pattern that is gaining popularity in the Kafka world. What David's video explains is that unless you take a lot of care exposing events between teams gives you high coupling and can lead to copy-and-paste fixes and enhancements being added to every service. At a very high level, you can get the same problems as sharing database tables. Appending entries to a shared event log is just like appending rows to a shared database table. No team should expose an event stream to anther team without it being a contract that they will maintain strict rules of backwards compatibility just as they must with any microservices API. Avro can be used to add attributes in a way that doesn't break old processes reading the new events. Upgrading data or fix data related bugs needs careful planning and practice or you will have monolithic headache.

7) The Homegrown Monolith

Obviously, something new as cool as microservices requires that you go and write your own framework. After all, you are one of the smartest folks around and if google can do it then so can you. Yet it is very hard to avoid a change-the-world event when you are writing your own framework while you write your application. A homegrown framework is a monolithic force as well as being immature and a distraction from adding business features.

6) Use The Meat Cloud

David specifically picks out large financial services organisations here. The "meat cloud" is using lots of people in an ops team rather than self-service automation. Typically teams doing microservices have a continuous integration and continuous deployment pipeline for their application code. Yet big firms believe that it is optimal to have separate teams that manage all the infrastructure needing a support ticket to make any changes. Since they have people sitting around waiting to do work they manually click about to set things up rather than automate anything. To prevent folks from breaking expensive shared tools they lock them down to only do what they anticipated you need to do. The experienced person who set it up isn't around to actually run the shop. You have someone without the big picture clicking the mouse day-to-day. This separates who wants something to be done, from the people who know how it can be done, from the people who have the privileges to get it done, from the people who have the time to get it done. So nothing gets done.

5) The Distributed Monolith

The anti-pattern here is to build an app as though it is a monolith yet deploy it as many microservices. This combines to give you the worst of both worlds. In a microservices environment, failures are more common than in a monolithic environment for the same level of application code quality. This is due to the fact that networks are not perfect and due to the mathematics of cumulative probabilities. If you have a problem where only 1 in 365 requests are failing in a monolith that is a bad enough problem. If you are using microservices where 23 inter-service calls are required to build a page then 1 in 365 requests failing means that 50% of customer pages will fail. The network is not reliable. Here is a post about real-world outages showing that network reliability is a fallacy. A service mesh may help but is that itself a complex distributed system that will take effort to set up and run.

4) The Single Page Application (SPA) Monolith

This is another hidden monolith anti-pattern. The problem is that your single page application can become the monolith. That has all the problems of a traditional monolith with the added twist that is running in many different browsers. David points out that it feels easy to just put new logic into the front-end when your requirements suddenly change but that convenience is the road to having a big ball of mud application. Approaches such as React.lazy applied to React Routes with code splitting may allow you to create a modular architecture for your frontend to avoid this trap. Such advanced techniques go well beyond the basics and will require effort to set up and will be a challenge to retrofit to a fast growing code base.

3) The Decision Monolith

We can all take a dig at architects who restrict the use of new technologies and who proscribed J2EE as the solution to all things. Yet it is hard to not say no to new ideas in the interests of reducing risks. We should ensure that we don't have a dogmatic and inflexible approach and that we create ways to experiment and to continuously introduce new innovations.

2) The Monolithic Business

Microservices cannot be a technology only matter. If we have monolithic processes then we are really doing waterfall in disguise. This is also known as "wagile", "water-scrum-fall", "dark agile", "faux agile" and a host of other names. As well as how the process works we should also call out monolithic attitudes and approaches outside of technology in terms of business engagement. The essence of agile and microservices is that we release little and often to get valuable feedback. If there is no real feedback loop between the business user and what we are building then we are not optimizing business value.

1) Use A HR Driven Microservices Architecture

The notable quote here is:

"It is easier to search for React developers than for general craftswomen."

This is basically saying that if your hiring practices focus on specializations then you are de-optimising your ability to build, deploy and support microservices. It is also a known bad practice to organise teams into specialisms. You need to organise around small mixed disciplined teams that are empowered to design, build, deploy and support what they create in as self-service model as possible. Yet when I talk to developers I am often surprised with how they self identify as in a given specialism such as "I am a Java developer" or "I am a React developer". That just perpetuates silos and allows organisations to box you into a silo and disempower you while claiming that is what makes you happy. Call yourself a software engineer and be intellectually curious as to how everything works. Infrastructure and networks in the cloud are software defined and has an API you can make use. Aim to be a maker of things not the holder of particular skils.

That's all folks. It would be great to hear about your experiences of what works and what doesn't.

How to integrate Git Bash with Visual Studio Code on Windows

Simon Massey — Fri, 09 Aug 2019 07:05:06 +0000

Many clients will require me to work on a Windows laptop for access to their networks. Fortunately, I can change the settings within VS Code to use Git Bash as the built-in terminal. I can then get the same developer experience on a Windows laptop that I get at home on my mac.

First type "Ctrl+Shift+P" to open the command search and type/select "Open User Settings". If this display a settings search page you will need to hit the ”{}” at the top right to get to the raw JSON. Merge the following settings ensuring to use paths that match where you installed the "bash.exe" and "git.exe" binaries:

{
"terminal.integrated.shell.windows": "C:\\whereever\\Git-2.17.1\\bin\\bash.exe",
"terminal.integrated.env.windows": {
"CHERE_INVOKING": "1"
},
"terminal.integrated.shellArgs.windows": [
"-l"
],
// optional add git-path to use this version of git for built in visual studio model
"git.path": "C:\\whereever\\Git-2.17.1\\bin\\git.exe"
}

Note that you don't have to use the last setting which is telling VS Code to use the git that came with Git Bash for its built-in git features.

The great thing about this approach is that you don't have to switch to a separate Bash Windows. The embedded terminal runs inside of VS Code and starts in the correct folder on disk. VS Code will filter the terminal output looking for file paths like ./server.js and it will turn them into hyperlinks. Ctrl+Click on them and it will open the file within the main editor panel! That is great when your build or unit tests you are running in the terminal throw an error logging the file to open.

Git Bash comes with all your favourite coreutils goodies like find, awk, sed, ssh, and more. You can also capture output to the windows clipboard with cat whatever | clip. You can read the windows clipboard with cat /dev/clipboard | awk -f mega_skills.awk. You can bash the heck out of any problem and be winning all day long.

Enjoy!

DEV Community: Simon Massey

GenAI Test File Summarisation In Chunks With Ollama Or Cloud

In Action 🥷

The Code 🚀

What’s in here? 📦

Why chunk at all? 🧠

Chunk a file ✂️

Summarise with Ollama 🦙

Summarise with cloud models ☁️

Change the prompt 🛠️

Outputs 🧾

Stitch the summaries back together 🧵

Want to add another provider later? 🔌

📊 Line Histogram — The File Profiler You Didn't Know You Needed

What If You Could See Your Data Before It Floods Your Context Window?

The Problem

The Solution line_histogram.awk

🚀 Features That Actually Matter

💡 Use Cases That'll Make You Look Like a Genius

For AI Agent Wranglers

For Data Engineers

For DevOps/SRE

📖 Pseudo Man Page (The Details)

⚡ Installation

Make executable

🎯 Why This Exists

📜 License

🤝 Contributing

Opus 4.5 may have faked their interview to get hired?

I built an infinite scroll calendar with dnd using Svelte 5

Augmented Intelligence (AI) Coding using Markdown Driven-Development

Step 1: Design the feature documentation with an online thinking model

Step 2: Export a description-only "coding prompt"

Step 3: Paste to an Agent in YOLO mode (--dangerously-skip-permissions)

Step 4: Force the Agent to "Work Backwards"

Todos Are All You Need?

Does This Work For Real Code?

End Notes

My LLM Code Generation Workflow (for now)

Step 1: Brainstorming with a web search-enabled LLM

Step 2: LLM-generated Readme-Driven Development

Step 3: LLM planning of build steps

Step 4: LLM thinking model generation of coding prompts

Step 5: Let the LLM directly edit, debug and commit the code

End Nodes

Git-Driven Deployments on Origin Kubernetes Distribution

Git-driven Infrastructure as Code on Origin Kubernetes Distribution

Shamir's Secret Sharing Scheme in JavaScript

Zero-Knowlege Authentication with JavaScript

Microservces Anti-Patterns and Ten Tips To Fail Badly

How to integrate Git Bash with Visual Studio Code on Windows

The Solution `line_histogram.awk`

Step 3: Paste to an Agent in YOLO mode (`--dangerously-skip-permissions`)