DEV Community: jldec

It's early days for Agents

jldec — Mon, 13 Oct 2025 22:52:18 +0000

Reasoning and function calling both marked significant breakthroughs in the road to making agents more capable.

Developer coding agents like Amp and Claude Code now leverage those capabilities to generate real value, for which users are willing to pay. They are also pushing the boundaries of what is possible with agents, e.g. by delegating tasks to specialized subagents and carefully managing the context over extended periods.

But coding agents are not general purpose. If we continue to rely on hand-coded agentic apps for specialized domains, that means that we haven't fully grokked the bitter lesson.

To deliver value at scale, Agents need to learn on the job.

Agents should get smarter over time, constantly refining their knowledge based on what they learn from their interactions at inference-time.

Two approaches

AI researchers tend to approach the problem from the perspective of reinforcement learning or other forms post-training. This makes sense coming from the "models-learn" perspective.
AI engineers are getting better behavior out of LLMs with context engineering. Capturing specialized knowledge, and including relevant details in the context used for inference is a natural way for prior conversations to shape the generated output, without additional training of the model.

A combination of both approaches will be required eventually, but in the short term there is lots of room for agent framework providers to innovate without pushing the problem back into the labs

Remember all conversations.
Provide search interfaces to map new tasks onto existing knowledge.
Refine existing knowledge by subjecting it to feedback.

If this is done well, it will also facilitate sharing and learning amongst teams of agents and humans.

Agents FTW

I am optimistic that even with these modest first steps, we can start shipping agentic systems that are capable of learning and improving over time.

In the meantime, innovations in how users interact with agents (like solveit), and how agents interact with their environment (like codemode) are exciting evidence of how fertile this space is.

🚀🚀🚀

Quick overview of Model Context Protocol (MCP)

jldec — Sat, 15 Mar 2025 22:36:20 +0000

MCP is a protocol for integrating AI applications with tools and external services. It was created by Anthropic as a way for Claude to access resources on the desktop, taking inspiration from LSP for editor extensions.

MCP servers can be coded do almost anything, including network calls to other services. This has led to MCP being talked about as a foundation for building AI agents, even though the original desktop-centric transport is far from the Web protocols you might expect.

What follows is very a quick overview of MCP. For deeper dives, I suggest:

How MCP was born interview with @dsp_ and @jspahrsummers - April 3 '25
Workshop from the AI engineer summit - Feb '25
MCP User guide
MCP Spec
MCP TypeScript SDK
Example MCP servers
Swyx's spicy take

How it works

To make a tool call, the MCP client exchanges JSON-RPC messages with the MCP server.

A desktop client will spawn servers as subprocesses so that it can communicate with them over stdio.

Besides tool calls, the protocol also supports:

Server-to-client notifications
URL-based resources with subscriptions for change notifications
Server-provided Prompt templates
Nested LLM calls or sampling

There are debugging tools and SDKs for Python, Typescript, Java, and Kotlin.

Why are people excited?

Talking to your own tools feels like magic.

Using MCP, AI appplications (clients) can connect to tools (servers) without custom integration.

Here is another example from Cloudflare.

What's next?

The buzz is real, but it's early days and AI agent platforms are just getting started.

I'm optimistic for HTTP to supplant stdio, especially with the recent announcement of remote servers with auth. This is particularly important for scaling agents on the network and supporting multiple clients per server.
A registry should accelerate the network effects of MCP, making servers and their capabilities more discoverable e.g. by other agents.
.well-known/mcp.json provides a way for agents to find AI interfaces on websites without the need for a central registry.
Stateless requests, streaming, and namespacing - all good things adopted from Web protocols.

🚀

3 essential elements for Web publishing

jldec — Mon, 13 Jan 2025 22:40:58 +0000

Content editing tools with reusable themes enable creators to publish professional-looking websites without the need for custom development.

This is the long tail of web publishing dominated by WordPress and platforms like Wix and Shopify.

Web frameworks like Next.js start with developers first. Even when they are integrated with a CMS, the dependency on developers remains.

An ideal web publishing architecture incorporates 3 key elements:

A theme that ensures content looks great on all devices.

A usable content editor with built-in theme awareness.

Fast, reliable web hosting for the rendered content.

Publishers need efficient editing tools. If you're building tools for web publishing, don't forget the editor experience.

Astro v5 Blog starter

jldec — Sun, 29 Dec 2024 21:16:22 +0000

Astro v5 blog starter

I recently moved a friend's blog to Astro v5. The motivation behind selecting Astro was its first-class support for markdown content.

With just a minimal amount of boilerplate, Astro will validate markdown frontmatter, generate static pages for each post, and optimize all the images in your posts.

https://astro-v5-blog-starter.jldec.me/blog/first-post

Markdown

---
date: '2024-12-01'
title: 'My First Blog Post'
image:
  src: '../images/birch-trees.webp'
---

## Markdown is

- a lightweight markup language
- for decorating plain text
- with things like headings, lists, links, and blockquotes
- making minimal assumptions about formatting.

#### Here is an inline image:

![sunset](../images/sunset-cambridge.jpg)

HTML

<img
  src="/_astro/birch-trees.DrRha0ED_9bfrU.webp"
  alt=""
  width="2016"
  height="955"
  loading="lazy"
  decoding="async"
  class="w-full h-60 object-cover object-bottom"
/>
<article class="prose mx-auto p-4">
  <h1>My First Blog Post</h1>
  <a href="/">&lt;&lt; Back</a>
  <h2 id="markdown-is">Markdown is</h2>
  <ul>
    <li>a lightweight markup language</li>
    <li>for decorating plain text</li>
    <li>with things like headings, lists, links, and blockquotes</li>
    <li>making minimal assumptions about formatting.</li>
  </ul>
  <h4 id="here-is-an-inline-image">Here is an inline image:</h4>
  <p>
    <img
      alt="sunset"
      width="1343"
      height="683"
      loading="lazy"
      decoding="async"
      src="/_astro/sunset-cambridge.7ZAluiBF_15WAVg.webp"
    />
  </p>
</article>

Starter Repo

Here is the result of extracting these key features into a new blog starter. The repo does not include a lot of design - just the configuration and a minimal amount of code.

https://github.com/jldec/astro-v5-blog-starter

The project includes:

Tailwind CSS with typography
Image optimization with the <Image> component
Static builds (SSG) for Cloudflare Pages.

File Structure

├── LICENSE
├── README.md
├── astro.config.mjs
├── package.json
├── public
│   ├── _headers
│   └── favicon.svg
├── src
│   ├── assets
│   │   └── astro.svg
│   ├── components
│   │   └── AstroLogo.astro
│   ├── content
│   │   ├── blog
│   │   │   ├── 2nd-post.md
│   │   │   └── first-post.md
│   │   └── images
│   │       ├── birch-trees.webp
│   │       └── sunset-cambridge.jpg
│   ├── content.config.ts
│   ├── layouts
│   │   └── Layout.astro
│   ├── pages
│   │   ├── 404.astro
│   │   ├── blog
│   │   │   └── [id].astro
│   │   └── index.astro
│   └── styles
│       └── global.css
├── tailwind.config.mjs
├── tsconfig.json
└── wrangler.toml

Markdown driven blog

Markdown blog posts live in src/content/blog.

The schema for the blog collection is defined in content.config.ts. This file also includes utility functions for sorting and filtering posts. E.g. if the draft flag is set, a post will only be included during dev, but not published with the production build.

The home page is defined in src/pages/index.astro which lists posts in date order.

Posts are rendered by the dynamic route src/pages/blog/[id].astro. In order for Astro to pre-render static pages, dynamic routes export a getStaticPaths function which returns a list of params and props for each rendered route.

src/pages/blog/[id].astro

---
import { Image } from 'astro:assets';
import { getCollection, render } from 'astro:content';
import { filterAllPosts } from '~/content.config';
import Layout from '~/layouts/Layout.astro';

export async function getStaticPaths() {
  const posts = await getCollection('blog', filterAllPosts);
  return posts.map((post) => ({
    params: { id: post.id },
    props: { post },
  }));
}

const { post } = Astro.props;
const { Content } = await render(post);
---

<Layout title={post.data.title}>
  <Image
    src={post.data.image.src}
    alt={post.data.image.alt || ''}
    class="w-full h-60 object-cover object-bottom"
  />
  <article class="prose mx-auto p-4">
    <h1>{post.data.title}</h1>
    <a href="/">&lt;&lt; Back</a>
    <Content />
  </article>
</Layout>

Image optimization

A src path and alt text can be declared for images in markdown frontmatter, or inline in markdown.

These are passed into the <Image> component which inspects each image, and generates <img> tags with width and height attributes, thereby reducing layout shifts in the browser.

Images are converted to webp format, and stored in dist/_astro with unique (cacheable) names during the build process.

Publishing on Cloudflare Pages

The @astrojs/cloudflare adapter is not needed for static sites.

Cloudflare Pages matches routes with .html files. To avoid trailing slashes, configure the build to generate <route>.html files instead of <route>/index.html.

The _headers file adds cache control headers for immutable content.

Conclusion

Astro v5 is a great choice for a markdown driven blog, as long as you're fine with doing occasional maintenance to update dependencies.

Here are some ideas for future improvements to this starter:

Sitemap
Menu component for desktop and mobile
Nicer fonts
Icons for social links

I hope you find this starter useful. Please reach out on X if you have any feedback or suggestions.

GitHub Flavored Callouts

jldec — Mon, 18 Dec 2023 00:09:07 +0000

I enjoyed adding support for callouts in zaui today.

Here's an example:

And here's the markdown source,

> [!TIP]
> The [zaui guide](https://zaui.zeroasic.com/guide/markdown-extensions) has more details.

The markdown format matches the recently introduced GitHub Markdown Alerts extension.

In addition to the now GitHub standard [!NOTE|TIP|IMPORTANT|CAUTION|WARNING] you can also follow the [!{something}] pattern, to make up your own callouts, perhaps with an emoji.

> [!💥 zaui]
> Getting a little better every day

Careful, this 👆 only works with zaui, not on GitHub.

Along with this change, zaui also gained a next button in the Post layout.

This should make it easier to navigate to the next page in the guide, and to the next post in the blog.

You can try it out at the top of this page...

💥

Getting started with Python Packaging

jldec — Wed, 22 Feb 2023 16:43:30 +0000

Let's do some Python

In preparation for using a lot more Python, I decided to refresh my Python knowedge and publish my first Python module at https://pypi.org/project/shortscale/.

Some readers may recognize shortscale from earlier explorations in JavaScript, Rust, and Go.

This post covers the following steps:

Install Python on macOS
Write the skeleton code, with just a one-line function.
Build and publish the incomplete v0.1 module.
Complete the logic v1.0.0.
Benchmarks
Python in the browser
Jupyter notebooks

Install python v3.10 (the hard way)

Installing Python on macOS is easiest with the official installer or with homebrew.

I wanted a way to switch between Python versions, so I followed the instructions for pyenv.

NOTE: This does a full local build of CPython, and requires dependencies different from the macOS command line tools.

# 1. Install pyenv 
# from https://github.com/pyenv/pyenv#set-up-your-shell-environment-for-pyenv
git clone https://github.com/pyenv/pyenv.git $HOME/.pyenv
export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init --path)"

# 2. Fix dependencies for macOS 
# from https://github.com/pyenv/pyenv/wiki#suggested-build-environment
brew install openssl readline sqlite3 xz zlib tcl-tk

# 3. After the brew install, fix LDFLAGS, CPPFLAGS and add tcl-tk/bin onto PATH 
export LDFLAGS="$LDFLAGS -L$HOME/homebrew/opt/openssl@3/lib -L$HOME/homebrew/opt/readline/lib -L$HOME/homebrew/opt/sqlite/lib -L$HOME/homebrew/opt/zlib/lib -L$HOME/homebrew/opt/tcl-tk/lib -L$HOME/homebrew/opt/openssl@3/lib -L$HOME/homebrew/opt/readline/lib -L$HOME/homebrew/opt/sqlite/lib -L$HOME/homebrew/opt/zlib/lib -L$HOME/homebrew/opt/tcl-tk/lib"
export CPPFLAGS="$CPPFLAGS -I$HOME/homebrew/opt/openssl@3/include -I$HOME/homebrew/opt/readline/include -I$HOME/homebrew/opt/sqlite/include -I$HOME/homebrew/opt/zlib/include -I$HOME/homebrew/opt/tcl-tk/include -I$HOME/homebrew/opt/openssl@3/include -I$HOME/homebrew/opt/readline/include -I$HOME/homebrew/opt/sqlite/include -I$HOME/homebrew/opt/zlib/include -I$HOME/homebrew/opt/tcl-tk/include"
export PATH=$HOME/homebrew/opt/tcl-tk/bin:$PATH

# 4. Use pyenv to build and install python v3.10 and make it the global default
pyenv install 3.10
pyenv global 3.10

# Point to the installed version in  .bash_profile (instead of depending on the pyenv shim)
export PATH=$HOME/.pyenv/versions/3.10.9/bin:$PATH

Virtual environments and pip

Python modules and their dependencies can be installed from pypi.org using pip install.

Configuring a virtual environment will isolate modules under a .venv directory, which is easy to clean up, rather than installing everything globally.

I created a venv under my home directory.

python3 -m venv ~/.venv

Instead of "activating" the venv, which changes the prompt, I prepended the .venv/bin directory onto my PATH.

export PATH=$HOME/.venv/bin:$PATH

Create a new module called shortscale

First I wrote a skeleton shortscale function which just returns a string with the input.

The rest of the code is boilerplate, to make the function callable on the command line. Passing base=0 to int() enables numeric literal input with different bases.

shortscale.py

"""English conversion from number to string"""
import sys

__version__ = "0.1.0"

def shortscale(num: int) -> str:
  return '{} ({} bits)'.format(num, num.bit_length())

def main():
  if len(sys.argv) < 2:
    print ('Usage: shortscale num')
    sys.exit(1)

  print(shortscale(int(sys.argv[1],0)))
  sys.exit(0)

if __name__ == '__main__':
  main()

The output looks like this:

$ python shortscale.py 0x42
66 (7 bits)

Next, I built and published this incomplete v0.1 shortscale module.

Unlike the npm JavaScript ecosystem, you can't just use pip to publish a module to the pypi repository. There are different build tools to choose from.

I chose setuptools because it appears to be the recommended tool, and shows what it's doing. This meant installing build and twine.

Python packages are described in a pyproject.toml. Note that project.scripts points to the CLI entrypoint at main().

pyproject.toml

[project]
name = "shortscale"
description = "English conversion from number to string"
authors = [{name = "Jürgen Leschner", email = "jldec@users.noreply.github.com"}]
readme = "README.md"
license = {file = "LICENSE"}
classifiers = ["License :: OSI Approved :: MIT License"]
dynamic = ["version"]

[project.urls]
Home = "https://github.com/jldec/shortscale-py"

[project.scripts]
shortscale = "shortscale:main"

[build-system]
requires = ["setuptools>=61.0"]
build-backend = "setuptools.build_meta"

[tool.setuptools.dynamic]
version = {attr = "shortscale.__version__"}

Build the module

The build tool creates 2 module bundles (source and runnable code) in the ./dist directory.

$ python -m build
...
Successfully built shortscale-0.1.0.tar.gz and shortscale-0.1.0-py3-none-any.whl

Publish to pypi.org

$ python -m twine upload dist/*
Uploading distributions to https://upload.pypi.org/legacy/
Uploading shortscale-0.1.0-py3-none-any.whl
Uploading shortscale-0.1.0.tar.gz
...
View at:
https://pypi.org/project/shortscale/0.1.0/

Install and run in a venv

The moment of truth. Install the module in a new venv, and invoke it.

$ mkdir test
$ cd test
$ python -m venv .venv
$ source .venv/bin/activate

(.venv) $ pip install shortscale
...
Successfully installed shortscale-0.1.0

(.venv) $ shortscale 0xffffffffffff
281474976710655 (48 bits)

$ deactivate

Complete the logic

Python still amazes me with its terseness and readability.

The first iteration had 3 functions, of which the longest had 30 lines with generous spacing.

One of those functions decomposes a number into powers of 1000.

def powers_of_1000(n: int):
    """
    Return list of (n, exponent) for each power of 1000.
    List is ordered highest exponent first.
    n = 0 - 999.
    exponent = 0,1,2,3...
    """
    p_list = []
    exponent = 0
    while n > 0:
        p_list.insert(0, (n % 1000, exponent))
        n = n // 1000
        exponent += 1

    return p_list

Playing aound in a Jupyter notebook, I was able to eliminate the extra function (and the list which it returns), simply by reversing the order of building the shortscale output.

Using a Jupyter environment in VS Code is a clear win. The result was simpler and faster.

Testing

There is nice support for Python testing and debugging in VS Code.

The function to run unit tests took just 3 lines.

Benchmarks

I was pleased with the benchmarks as well. For this string manipulation micro-benchmark, CPython 3.11 is only 1.5x slower than V8 JavaScript!

Compiled languages like Go and Rust will outperform that, but again, not by a huge amount.

The results below are from my personal M1 arm64 running macOS.

Python

Python v3.11.2

$ python tests/bench_shortscale.py

 50000 calls,    5000000 bytes,     1264 ns/call
100000 calls,   10000000 bytes,     1216 ns/call
200000 calls,   20000000 bytes,     1216 ns/call

Python v3.10.9

$ python tests/bench_shortscale.py

 50000 calls,    5000000 bytes,     1811 ns/call
100000 calls,   10000000 bytes,     1808 ns/call
200000 calls,   20000000 bytes,     1809 ns/call

Javascript

$ node test/bench.js

20000 calls, 2000000 bytes, 796 ns/call
20000 calls, 2000000 bytes, 790 ns/call
20000 calls, 2000000 bytes, 797 ns/call

Go

$ go test -bench . -benchmem

BenchmarkShortscale-8        4227788           252.0 ns/op       248 B/op          5 allocs/op

Rust

$ cargo bench

running 2 tests
test a_shortscale                        ... bench:         182 ns/iter (+/- 3)
test b_shortscale_string_writer_no_alloc ... bench:          63 ns/iter (+/- 2)

Let's run shortscale in the browser

Open your browser on https://pyodide.org/en/stable/console.html and paste the following python commands into the python REPL, line by line.

import micropip
await micropip.install("shortscale")
import shortscale

shortscale.shortscale(0xffff0000)

shortscale.bench_shortscale()

Looks like Python in WASM in the browser is about 2 to 3 times slower than native CPython.

GitHub and Google Colaboratory

Open the shortscale Jupyter notebook from GitHub in a Google Colaboratory environment

https://colab.research.google.com/github/jldec/shortscale-py/blob/main/shortscale.ipynb

Keep on learning 🚀

What is git LFS?

jldec — Wed, 09 Feb 2022 18:23:45 +0000

What is git LFS?

Git large file storage or LFS is a way to store binaries outside your git repo, but still work with them as if they were part of the repo.

It works by replacing the actual file with a small text file which contains a pointer to the real thing. Every problem in computer science can be solved by another level of indirection.

$ cat screenshot.png 
version https://git-lfs.github.com/spec/v1
oid sha256:8ded39c045d36d4f745e46525b5c8b6a29baceb89006a3e85148ce66db9c187d
size 280654

For a brief introduction to LFS see https://git-lfs.github.com/ or watch this video.

Installation

Installation requires 2 steps.

Place git-lfs on the path.

I pulled the latest binary for macOS arm64 from GitHub releases.
Configure git to use LFS with git lfs install.

Both of these steps are included in the install.sh script, in the GitHub release.

Why is the binary called 'git-lfs'?

A git extension is a binary prefixed with git-.

Running git lfs invokes git-lfs, which gives you a nice summary of the available commands.

To get started with Git LFS, the following commands can be used.

 1. Setup Git LFS on your system. You only have to do this once per
    repository per machine:

        git lfs install

 2. Choose the type of files you want to track, for examples all ISO
    images, with git lfs track:

        git lfs track "*.iso"

 3. The above stores this information in gitattributes(5) files, so
    that file need to be added to the repository:

        git add .gitattributes

 4. Commit, push and work with the files normally:

        git add file.iso
        git commit -m "Add disk image"
        git push

What does 'git lfs install' do?

The command git lfs install configures a git filter, which calls git-lfs to convert between pointer files and the actual blobs.
This filter configuration lives in your git config (e.g. at ~/.gitconfig) and applies to all your repos.

[filter "lfs"]
    clean = git-lfs clean -- %f
    smudge = git-lfs smudge -- %f
    process = git-lfs filter-process
    required = true

If you prefer not to run these hooks all the time, you can uninstall them with git lfs uninstall after finishing your work on an LFS git repo.

How is a repo configured to use LFS?

The command git lfs track '*.png' adds the following line to the .gitattributes file in the repo.

*.png filter=lfs diff=lfs merge=lfs -text

This associates git-lfs with .png files.
Once the association is made, you can work with those files normally, and git-lfs will handle the LFS interactions with GitHub automatically, in the background.

Example repo

This repo lives at github.com/jldec/lfs-test

$ git init
Initialized empty Git repository in /Users/jldec/lfs-test/.git/

$ git lfs track '*.png'
Tracking "*.png"

$ git lfs track
Listing tracked patterns
    *.png (.gitattributes)
Listing excluded patterns

$ git add -A
$ git commit -m 'initial commit'
[main (root-commit) c25b950] initial commit
 2 files changed, 4 insertions(+)
 create mode 100644 .gitattributes
 create mode 100644 screenshot.png

$ git lfs ls-files
ba2a8c5285 * screenshot.png

$ git remote add origin git@github.com:jldec/lfs-test.git
$ git push -u origin main
Uploading LFS objects: 100% (1/1), 146 KB | 0 B/s, done.
Enumerating objects: 4, done.
Counting objects: 100% (4/4), done.
Delta compression using up to 8 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (4/4), 408 bytes | 408.00 KiB/s, done.
Total 4 (delta 0), reused 0 (delta 0), pack-reused 0
To github.com:jldec/lfs-test.git
 * [new branch]      main -> main
Branch 'main' set up to track remote branch 'main' from 'origin'.

$ git status
On branch main
Your branch is up to date with 'origin/main'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   screenshot.png

no changes added to commit (use "git add" and/or "git commit -a")

$ git commit -a -m 'update lfs file screenshot.png'
[main e695f5f] update lfs file screenshot.png
 1 file changed, 2 insertions(+), 2 deletions(-)

$ git push
Uploading LFS objects: 100% (1/1), 281 KB | 121 KB/s, done.
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 8 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 405 bytes | 405.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
To github.com:jldec/lfs-test.git
   c25b950..e695f5f  main -> main

First impressions of the new GitHub Projects Beta

jldec — Sat, 06 Nov 2021 17:19:33 +0000

Issues with GitHub issues

GitHub issues have historically provided a simple yet powerful way to track work in your GitHub repositories. Each issue includes a description, assignee(s), and a timeline with a comment thread and automatically-generated references to related issues and PRs. Issues can be categorized using labels, and the issues list can be searched or filtered in many ways.

But as projects get larger and more complex, working with issues also has its challenges.

How to turn large "Epic" issues into smaller issues?
How to prioritize and organize issues into iterations?
How to track issues across multiple repositories?
How best to incorporate community contributions and other feedback?

GitHub has incrementally tried to address some of these challenges.

Task lists add convenient checkboxes to markdown lists in issue descriptions.
Milestones provide a simple way to collect and prioritize issues within a repo.
Projects started as single-repo kanban boards, with issues or notes moving vertically or sideways. They later acquired cross-repo and limited automation capabilities.
Issue templates help contributors to include specific information in an issue.

The new GitHub Issues (Beta)

With their Issues announcement in June, GitHub signalled a less-incremental approach.

The announcement included two parts

A powerful new way to create and group issues into projects.
Ways to grow from ideas expressed as text, into collections of issues.

More announcements have followed, and last week the Beta opened up to all users.

Beta Projects

Project tables are spreadsheet-like views where each row is a real or draft issue.

Rows (issues) can be grouped by field value. This includes custom fields whose values are maintained in the project instead of on issues in a repo. Maintaining custom field data inside projects is key to their power.

Rather than polluting your issues with all possible categories of tags, new categories can be scoped inside a project.

Since Beta Projects also support kanban views, I expect them to replace the existing Projects once they reach feature parity.

From text to issues

Auto-creating issues from task lists makes it easier to break down "epic" issues into smaller sub-issues. The task list item is checked off when the issue is resolved, and the child-issue links to the parent-issue with the task list.

Similarly, draft issues which are simply rows entered as text in a project table, can also be converted into issues.

This makes task lists and project tables two convenient ways to brainstorm ideas, and break them down into smaller issues.

What's next

The GitHub public roadmap features a planning view specific to issues.

A number of informative talks at the recent GitHub Universe 2021 also provided hints about what the team is planning.

Remember that the new projects and issues features are still in Beta.

Using Gitpod to create a PR

jldec — Sun, 24 Oct 2021 23:02:21 +0000

Gitpod Workspaces

Gitpod hosts workspaces for developers.

Think of each workspace as your own Linux container in the cloud, with a fully functional development environment, including:

A clone of your git repo and your git working tree - the files you're working on.
The tools you need while coding - compilers, SDKs, runtimes.
Your editor - the default is VS Code + extensions - reachable through a browser.
Shell access to run commands in the container.

Instead of laboriously maintaining your local development environment with everything you need, you simply open a new workspace every time you start working on a new project or branch.

PR for the Gitpod website

Here's how I created a PR for the Gitpod website in just a few minutes.

Starting from the issue, I opened a new workspace in Gitpod by prefixing the GitHub url like so: https://gitpod.io/#https://github.com/gitpod-io/website/issues/1139.

That url opened the workspace in my browser.

Running git branch -vv in a workspace terminal showed that git was already set to a new branch, conveniently named with my username and the description and id of the GitHub issue.

The workspace started up with npm modules already installed by a prebuild.

The repo was also configured to start a dev server listening on port 3000 in the workspace container. The open ports can be seen in the Remote Explorer sidebar on the left.

I used the Open Browser icon to open the website in another browser window, and watched my changes taking effect each time I modified the code.

Finally, I pushed a commit with my changes on the new branch to GitHub, and proceeded to create my PR as usual. No localhost interaction other than running my browser was required.

This blogpost was written from a Gitpod workspace.
🚀

Preventing concurrent GitHub Actions

jldec — Sun, 13 Jun 2021 22:49:22 +0000

GitHub Actions Workflows

What happens when you trigger a GitHub Actions workflow which is already running? Workflows which depend on being run one-at-a-time might fail.

I recently encountered this with a workflow for publishing a static website. This workflow generates HTML files which are pushed to another git repo for publishing by GitHub Pages.

When two workflows try to push to a checked-out repo at the same time, one will fail because it is missing the last commit from the other.

This is just one example where concurrent workflows are problematic. Workflows which automate deployments have the same problem.

A number of 3rd party solutions exist, but these introduce additional waiting costs or other issues. For one of my projects, I host a lock-service, just to force concurrent workflows to exit quietly, and then auto-trigger re-runs.

Finally, on April 19, 2021, this appeared in the GitHub Blog.

In the case of using actions to generate a GitHub Pages website, the feature works exactly as required.

The first workflow will run to completion.
Subsequent concurrent workflows will either be delayed or cancelled.
In the end only the first and last of the overlapping workflows will be run.

And all you need is 2 lines of yaml.
This is from the workflow which generates jldec.uk.

The group can be any string - workflows in the same group are effectively serialized.

Thank you GitHub!

Getting started with Goroutines and channels

jldec — Sun, 25 Apr 2021 15:01:05 +0000

Golang

This is part 3 of my experience as a new user of Go, focusing on concurrency with Goroutines and channels.

For installation, testing, and packages, see Getting started with Go, and for pointers see Getting started with Go pointers.

Counting HTTP requests

The server below counts HTTP requests, and returns the latest count on each request.

To follow along, clone https://github.com/jldec/racey-go, and start the server with 'go run .'

package main

import (
    "fmt"
    "net/http"
)

func main() {
    var count uint64 = 0

    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        count++
        fmt.Fprintln(w, count)
    })

    fmt.Println("Go listening on port 3000")
    http.ListenAndServe(":3000", nil)
}

$ curl localhost:3000
1
$ curl localhost:3000
2

Let's try sending multiple requests at the same time. This command invokes curl with urls from a file using xargs to spawn 4 processes at once.

$ cat urls.txt | xargs -P 4 -n 1 curl

The file contains 100 lines, but instead of ending on a nice round number, on systems with more than 1 core you may see something like this (e.g. after 3 runs)

289
292
291

Replace the Go server with 'node server.js' to compare the results (e.g. after 3 runs again)

298
299
300

Now repeat the experiment with the race detector turned on. The detector will report a problem on line 12 of main.go which is count++.

$ go run -race .
Go listening on port 3000
==================
WARNING: DATA RACE
Read at 0x00c000138280 by goroutine 7:
  main.main.func1()
      /Users/jleschner/pub/racey-go/main.go:12 +0x4a
  net/http.HandlerFunc.ServeHTTP()
      /Users/jleschner/go1.16.3/src/net/http/server.go:2069 +0x51
  net/http.(*ServeMux).ServeHTTP()
      /Users/jleschner/go1.16.3/src/net/http/server.go:2448 +0xaf
  net/http.serverHandler.ServeHTTP()
      /Users/jleschner/go1.16.3/src/net/http/server.go:2887 +0xca
  net/http.(*conn).serve()
      /Users/jleschner/go1.16.3/src/net/http/server.go:1952 +0x87d

Previous write at 0x00c000138280 by goroutine 9:
  main.main.func1()
      /Users/jleschner/pub/racey-go/main.go:12 +0x64
  net/http.HandlerFunc.ServeHTTP()
      /Users/jleschner/go1.16.3/src/net/http/server.go:2069 +0x51
  net/http.(*ServeMux).ServeHTTP()
      /Users/jleschner/go1.16.3/src/net/http/server.go:2448 +0xaf
  net/http.serverHandler.ServeHTTP()
      /Users/jleschner/go1.16.3/src/net/http/server.go:2887 +0xca
  net/http.(*conn).serve()
      /Users/jleschner/go1.16.3/src/net/http/server.go:1952 +0x87d

Data races

From the race detector docs:

A data race occurs when two goroutines access the same variable concurrently and at least one of the accesses is a write.

It's clear that 'count++' modifies the count, but what are goroutines and where are they in this case?

Goroutines

Goroutines provide low-overhead threading. They are easy to create, and scale well on multi-core processors.

The Go runtime can schedule many concurrent goroutines across a small number of OS threads. Under the covers, this is how the http library handles concurrent web requests.

Let's start with an example. You can run it in the Go Playground.

package main

import (
    "fmt"
    "time"
)

func main() {
    ch := make(chan string)

    // start 2 countdowns in parallel goroutines
    go countdown("crew-1", ch)
    go countdown("crew-2", ch)

    fmt.Println(<-ch) // block waiting to receive 1st string
    fmt.Println(<-ch) // block waiting to receive 2nd string
}

func countdown(name string, ch chan<- string) {
    for i := 10; i > 0; i-- {
        fmt.Println(name, i)
        time.Sleep(1 * time.Second)
    }
    ch <- "blastoff " + name
}

Each 'go countdown()' starts a new goroutine. Notice how the countdowns are interleaved in the output.

...
crew-1 3
crew-2 3
crew-2 2
crew-1 2
crew-1 1
crew-2 1
blastoff crew-2
blastoff crew-1

Channels

Channels allow goroutines to communicate and coordinate.

In the example above, <-ch (receive) will block until another goroutine uses ch <- to send a string to the channel. This happens at the end of each countdown.

Sends will also block if there are no receivers, but that is not the case here.

There are many other variations for how to use channels, including buffered channels which only block sends when the buffer is full.

Atomicity

Given that net/http requests are handled by goroutines, can we explain why there is a data race when the function which handles a request increments a shared counter?

The reason is that count++ requires a read followed by write, and these are not automatically synchronized. One goroutine may overwrite the increment of another, resulting in lost writes.

To fix this, the counter has be protected to make the increment operation atomic.

Counter-go

github.com/jldec/counter-go demonstrates 3 different implementations of a threadsafe global counter.

CounterAtomic uses atomic.AddUint64 and atomic.LoadUint64.
CounterMutex uses sync.RWMutex.
CounterChannel serializes all reads and writes inside 1 goroutine with 2 channels.

All 3 types implement a Counter interface:

type Counter interface {
    Get() uint32 // get current counter value
    Inc()        // increment by 1
}

The modified server will work with any of the 3 implementations, and no data race should be detected.

package main

import (
    "fmt"
    "net/http"

    counter "github.com/jldec/counter-go"
)

func main() {
    count := new(counter.CounterAtomic)
    // count := new(counter.CounterMutex)
    // count := counter.NewCounterChannel()

    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        count.Inc()
        fmt.Fprintln(w, count.Get())
    })

    fmt.Println("Go listening on port 3000")
    http.ListenAndServe(":3000", nil)
}

Coordination with channels

Of the 3 implementations, CounterChannel is the most interesting. All access to the counter goes through 1 goroutine which uses a select to wait for either a read or a write on one of two channels.

Can you tell why neither Inc() nor Get() should block?


package counter

// Thread-safe counter
// Uses 2 Channels to coordinate reads and writes.
// Must be initialized with NewCounterChannel().
type CounterChannel struct {
    readCh  chan uint64
    writeCh chan int
}

// NewCounterChannel() is required to initialize a Counter.
func NewCounterChannel() *CounterChannel {
    c := &CounterChannel{
        readCh:  make(chan uint64),
        writeCh: make(chan int),
    }

    // The actual counter value lives inside this goroutine.
    // It can only be accessed for R/W via one of the channels.
    go func() {
        var count uint64 = 0
        for {
            select {
            // Reading from readCh is equivalent to reading count.
            case c.readCh <- count:
            // Writing to the writeCh increments count.
            case <-c.writeCh:
                count++
            }
        }
    }()

    return c
}

// Increment counter by pushing an arbitrary int to the write channel.
func (c *CounterChannel) Inc() {
    c.check()
    c.writeCh <- 1
}

// Get current counter value from the read channel.
func (c *CounterChannel) Get() uint64 {
    c.check()
    return <-c.readCh
}

func (c *CounterChannel) check() {
    if c.readCh == nil {
        panic("Uninitialized Counter, requires NewCounterChannel()")
    }
}

Benchmarks

All 3 implementations are fast. Serializing everything through a goroutine with channels, costs only a few hundred ns for a single read or write. When constrained to a single OS thread, the cost of goroutines is even lower.

$ go test -bench .
goos: darwin
goarch: amd64
pkg: github.com/jldec/counter-go
cpu: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz

Simple: 1 op = 1 Inc() in same thread

BenchmarkCounter_1/Atomic-12                 195965660          6 ns/op
BenchmarkCounter_1/Mutex-12                   54177086         22 ns/op
BenchmarkCounter_1/Channel-12                  4499144        286 ns/op

Concurrent: 1 op = 1 Inc() across each of 10 goroutines

BenchmarkCounter_2/Atomic_no_reads-12          7298484        191 ns/op
BenchmarkCounter_2/Mutex_no_reads-12           1966656        621 ns/op
BenchmarkCounter_2/Channel_no_reads-12          256842       4771 ns/op

Concurrent: 1 op = [ 1 Inc() + 10 Get() ] across each of 10 goroutines

BenchmarkCounter_2/Atomic_10_reads-12          3922029        286 ns/op
BenchmarkCounter_2/Mutex_10_reads-12            416354       2844 ns/op
BenchmarkCounter_2/Channel_10_reads-12           21506      55733 ns/op

Constrained to single thread

$ GOMAXPROCS=1 go test -bench .

BenchmarkCounter_1/Atomic                    197135869          6 ns/op
BenchmarkCounter_1/Mutex                      55698454         22 ns/op
BenchmarkCounter_1/Channel                     5689788        214 ns/op

BenchmarkCounter_2/Atomic_no_reads            19519166         60 ns/op
BenchmarkCounter_2/Mutex_no_reads              4702759        254 ns/op
BenchmarkCounter_2/Channel_no_reads             530554       2197 ns/op

BenchmarkCounter_2/Atomic_10_reads             6269979        189 ns/op
BenchmarkCounter_2/Mutex_10_reads               927439       1354 ns/op
BenchmarkCounter_2/Channel_10_reads              47889      25054 ns/op

🚀 - code safe - 🚀

Getting started with Go pointers

jldec — Sun, 18 Apr 2021 19:24:42 +0000

Golang

This is part 2 of my experience as a new user of Go, focusing on the quirks and gotchas of pointers. For installation, testing, and packages, see Getting started with Go.

If you'd like to follow along, and try out out the code in this article, all you need is the Go playground to run the examples.

Pointers

The shortscale package which I covered last time, uses a string Builder. Here is the example from the Builder docs.

package main

import (
    "fmt"
    "strings"
)

func main() {
    var b strings.Builder
    for i := 3; i >= 1; i-- {
        fmt.Fprintf(&b, "%d...", i)
    }
    b.WriteString("ignition")
    fmt.Println(b.String())
}

Notice that var b is an instance of the Builder. When you run the code, it will output: 3...2...1...ignition.

Pointer receiver methods and interfaces

The first argument to fmt.Fprintf is &b, a pointer to b. This is necessary, because fmt.Fprintf expects an io.Writer interface.

type Writer interface {
    Write(p []byte) (n int, err error)
}

The Builder.Write method matches the io.Writer interface. Notice the pointer syntax in the method receiver after the func keyword.

func (b *Builder) Write(p []byte) (int, error)

I was tempted to replace Fprintf(&b, ...) with Fprintf(b, ...), to make it more consistent with the b.WriteString() and b.String() further down, but doing this causes the compiler to complain:

"cannot use b (type strings.Builder) as type io.Writer in argument to fmt.Fprintf: strings.Builder does not implement io.Writer (Write method has pointer receiver)"

Value vs. pointer function arguments

What if, instead of depending on the Writer interface, we called our own write() function?

func main() {
    var b strings.Builder
    for i := 3; i >= 1; i-- {
        write(b, fmt.Sprintf("%d...", i))
    }
    b.WriteString("ignition")
    fmt.Println(b.String())
}

func write(b strings.Builder, s string) {
    b.WriteString(s)
}

Running the code above in the example sandbox outputs just the word ignition.

The 3 calls to write(b) do not modify the builder declared at the top.

This makes sense, because passing a struct to a function copies the struct value.

To fix this, we have to use a pointer to pass the struct by reference, and we have to invoke the function with write(&b, ...). This works, but it doesn't make the code any more consistent.

func main() {
    var b strings.Builder
    for i := 3; i >= 1; i-- {
        write(&b, fmt.Sprintf("%d...", i))
    }
    b.WriteString("ignition")
    fmt.Println(b.String())
}

func write(b *strings.Builder, s string) {
    b.WriteString(s)
}

Why do the method calls work?

Why are we allowed to use b instead of &b in front of b.WriteString and b.String? This is explained in the tour as well.

"...even though v is a value and not a pointer, the method with the pointer receiver is called automatically. That is, as a convenience, Go interprets the statement v.Scale(5) as (&v).Scale(5) since the Scale method has a pointer receiver."

Start with a pointer

If all this mixing of values and pointers feels inconsistent, why not start with a pointer from the beginning?

The following code will compile just fine, but can you tell what's wrong with it?

func main() {
    var b *strings.Builder
    for i := 3; i >= 1; i-- {
        fmt.Fprintf(b, "%d...", i)
    }
    b.WriteString("ignition")
    fmt.Println(b.String())
}

The declaration above results in a nil pointer panic at run time, because b is uninitialized.

panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x4991c7]

Create the Builder with new()

Here is one way to initialize a pointer so that it references a new Builder.

func main() {
    b := new(strings.Builder)
    for i := 3; i >= 1; i-- {
        fmt.Fprintf(b, "%d...", i)
    }
    b.WriteString("ignition")
    fmt.Println(b.String())
}

new(strings.Builder) returns a pointer to a freshly allocated Builder, which we can use for both functions and pointer receiver methods. This is the pattern which I now use in shortscale-go.

An alternative, which does the same thing, is the more explicit struct literal shown below.

func main() {
    b := &strings.Builder{}
    ...

There's no avoiding pointers in Go.
Learn the quirks and the gotchas today.
✨ Keep learning! ✨