Sending your git diffs to an HTTP endpoint from GitHub Actions, and the four things that broke it

Sending your own diffs somewhere is a surprisingly common need. Post them to an internal service, run them through a review bot, feed them to a summarizer, archive them for compliance. The GitHub Actions job that does it looks like five lines until you run it on a real repo, at which point four separate things break.

Here is a workflow that works, followed by why each awkward bit is there. Everything is plain git, jq and curl, all present on ubuntu-latest.

name: Send diffs

on:
  push:
    branches: [main]

jobs:
  send:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Send changes
        env:
          TOKEN: ${{ secrets.MY_TOKEN }}
        run: |
          BEFORE="${{ github.event.before }}"
          AFTER="${{ github.sha }}"

          # First push to a new branch has no usable "before" ref.
          if ! git cat-file -e "$BEFORE^{commit}" 2>/dev/null; then
            BEFORE="$(git rev-parse "$AFTER~1" 2>/dev/null || echo "$AFTER")"
          fi

          git diff "$BEFORE" "$AFTER" -- . ':(exclude).env*' ':(exclude)*.lock' > /tmp/full.diff
          head -c 200000 /tmp/full.diff > /tmp/send.diff
          TRUNCATED=false
          [ "$(wc -c < /tmp/full.diff)" -gt 200000 ] && TRUNCATED=true

          git log --format='%H%x09%s%x09%an%x09%aI' "$BEFORE".."$AFTER" \
            | jq -Rn '[inputs | split("\t") | {sha:.[0], message:.[1], author:.[2], date:.[3]}]' \
            > /tmp/commits.json

          jq -n \
            --arg repository "${{ github.repository }}" \
            --arg before "$BEFORE" \
            --arg after "$AFTER" \
            --rawfile diff /tmp/send.diff \
            --slurpfile commits /tmp/commits.json \
            --argjson truncated "$TRUNCATED" \
            '{repository:$repository, before:$before, after:$after,
              commits:$commits[0], diff:$diff, truncated:$truncated}' \
            > /tmp/payload.json

          curl -sS -X POST "https://example.com/api/push" \
            -H "Authorization: Bearer $TOKEN" \
            -H "Content-Type: application/json" \
            -H "Accept: application/json" \
            --retry 8 --retry-delay 15 --retry-max-time 300 --retry-all-errors \
            --fail-with-body \
            --data-binary @/tmp/payload.json

Now the parts that are not obvious.

fetch-depth: 0, because the default checkout cannot diff

actions/checkout does a shallow clone by default. Shallow means git diff BEFORE..AFTER fails on anything but the most recent commit, because the older side of the range is not in the local history at all.

This is the failure that looks like a bug in your endpoint. The workflow goes green on the push where you tested it and starts failing on real ones.

github.event.before is not always a commit

On the first push to a new branch, github.event.before is forty zeros. On a force push it can point at a commit that no longer exists in the remote history.

git cat-file -e "$BEFORE^{commit}" asks a simple question: does that object exist here and is it a commit? When it does not, the fallback diffs against the parent of the current commit, and if even that fails (an initial commit with no parent), it diffs the commit against itself and sends an empty diff. An empty payload is fine. A crashed workflow on somebody's first push is not.

Write to files, do not put diffs in shell variables

This is the one that cost me the most time, and it fails in a way that reads as nonsense:

jq: error: Argument list too long

A week of changes can be hundreds of kilobytes. ARG_MAX on Linux is a couple of megabytes for the whole argument list and environment, but you hit trouble well before that, and the error blames jq when the kernel refused the exec.

The fix is to stop passing large content as arguments. jq --rawfile name path reads a file in as a JSON string, correctly escaped, and --slurpfile reads a file of JSON in as a value. The diff never becomes an argument, so it never counts against ARG_MAX.

The same trick applies to the request body: --data-binary @file streams from disk instead of expanding on the command line.

While you are there, use jq -n to build the whole payload rather than string concatenation. A diff contains quotes, backslashes and newlines, which is exactly the set of characters that turns hand-rolled JSON into an unparseable mess at 2am.

Cap the size, and say that you capped it

head -c 200000 bounds what you send. The interesting half is the truncated flag next to it.

A consumer that receives a silently truncated diff has no way to know the input was incomplete, so it will report confidently on half a change. Sending the flag lets the other side adapt (in my case, lean on commit messages where the diff runs out). A boolean is cheap. Silent truncation is a correctness bug wearing a performance costume.

One caveat: head -c cuts on a byte boundary, so it can split a multibyte character in half. Depending on what reads the file next, that stray tail byte turns into a replacement character or an encoding error further down the pipe. If your repo has non-ASCII content, piping through iconv -c -f utf-8 -t utf-8 after the head drops the broken tail and costs nothing.

Exclude paths in CI, not on the server

git diff -- . ':(exclude).env*' uses pathspec magic to drop files before the diff is built. It is worth understanding why that placement matters.

Filtering on the receiving end means the sensitive file was transmitted, arrived, and was then deleted by code that has to be correct. Filtering in CI means it never left the machine. Those are very different promises to make to whoever owns the repo, and only one of them survives a bug on the server.

Accept: application/json and --fail-with-body

Without an Accept header, a lot of server frameworks answer a validation failure with an HTML error page or a 302 back to a login screen. Your workflow then either goes green on a redirect it never followed, or logs a wall of HTML.

--fail-with-body is the flag people miss. Plain -f fails the step on a 4xx or 5xx but throws the response body away, which is where the reason lives. --fail-with-body fails the step and prints the body, so the Actions log tells you what was wrong.

Retry, because your endpoint will be deploying

--retry 8 --retry-delay 15 --retry-max-time 300 --retry-all-errors covers about five minutes of downtime.

--retry-all-errors is the important one. By default curl retries only on a small set of transient conditions and does not retry on connection refused, which is exactly what you get while the receiving app restarts. Deploys are the single most likely reason a push arrives at an endpoint that is not there, and a push is not repeatable: nobody is going to git push --force just to trigger your workflow again.

Make the receiving endpoint idempotent on the commit sha as well. A unique index on the after sha turns a duplicate delivery into a no-op, which is what you want when a retry succeeds after the first attempt actually landed.

The general shape

Four rules, none of them specific to diffs:

1. Assume the range you were handed might not exist, and degrade instead of crashing.
2. Keep large content out of argument lists. Files and --rawfile are not a style preference.
3. Truncate loudly and let the consumer know.
4. Make failures readable in the Actions log, and retry the network like the other side might be restarting, because it might be.

I build Patchlog, and this workflow is the one it hands you for turning pushes into draft changelog entries, which is why it has been beaten on by real repos. The traps above are yours to use with any endpoint at all.