DEV Community

Cover image for A Gemini 500 on a resume upload turned into a .docx bug, a frozen DigitalOcean droplet, a timed-out deploy, and a corrupted zip file. Here's every fix.
Vicente G. Reyes
Vicente G. Reyes

Posted on Originally published at vicentereyes.org AI-assisted

A Gemini 500 on a resume upload turned into a .docx bug, a frozen DigitalOcean droplet, a timed-out deploy, and a corrupted zip file. Here's every fix.

#django #python #devops #debugging

It started with one Sentry alert on a Tuesday night:

ServerError: An internal error has occurred. Please retry or report in
https://developers.generativeai.google/guide/troubleshooting
Transaction: job_board.seekers.tasks.parse_resume
Enter fullscreen mode Exit fullscreen mode

GiggleGigs is a job board I run. When a job seeker uploads a resume, a Celery task sends it to Google's Gemini API, which pulls out their name, headline, skills, links and years of experience and fills in their profile. One seeker's resume had failed to parse.

Easy fix, I thought. Gemini had a hiccup, so add a retry.

Two days later I had fixed four separate bugs, frozen my production server, restarted it from the DigitalOcean dashboard, and learned more about the .docx format than I ever wanted to. Here's how it went.

Bug #1: One transient error, permanent failure

The task looked roughly like this:

@shared_task
def parse_resume(profile_id):
    ...
    try:
        response = client.models.generate_content(...)
        # parse JSON, fill in the profile
        profile.parsing_status = SeekerProfile.ParsingStatus.DONE
        profile.save()
    except Exception:
        logger.exception("Resume parsing failed for profile %s", profile_id)
        profile.parsing_status = SeekerProfile.ParsingStatus.FAILED
        profile.save(update_fields=["parsing_status"])
Enter fullscreen mode Exit fullscreen mode

Every exception was final. A 500 INTERNAL from Gemini, which is Google's problem and usually gone a minute later, permanently marked the resume as failed and sent an error to Sentry.

The fix was to make the task bound, retry the error codes that are worth retrying, and back off exponentially:

# Gemini errors worth retrying: quota exhausted (429), internal error (500),
# model overloaded (503). Retried after 1, 2, then 4 minutes.
TRANSIENT_GEMINI_CODES = (429, 500, 503)
PARSE_RESUME_MAX_RETRIES = 3


@shared_task(bind=True, max_retries=PARSE_RESUME_MAX_RETRIES)
def parse_resume(self, profile_id):
    from google.genai import errors as genai_errors
    ...
    try:
        ...
    except genai_errors.APIError as exc:
        if exc.code not in TRANSIENT_GEMINI_CODES:
            logger.exception("Resume parsing failed for profile %s", profile_id)
        elif self.request.retries < self.max_retries:
            raise self.retry(countdown=60 * 2**self.request.retries)
        else:
            logger.warning(
                "Gemini unavailable (%s); gave up parsing resume for profile %s: %s",
                exc.code, profile_id, exc.message,
            )
        profile.parsing_status = SeekerProfile.ParsingStatus.FAILED
        profile.save(update_fields=["parsing_status"])
    except Exception:
        ...
Enter fullscreen mode Exit fullscreen mode

Two details are worth pointing out.

  • Order matters. self.retry() raises celery.exceptions.Retry, which is a subclass of Exception. If the generic except Exception came first, it would swallow the retry. Raising from inside one except clause skips the sibling clauses, so this layout is safe.
  • Warning vs. error. If Gemini is still down after three retries, that's an outage, not a bug, so it gets logged as a warning. Real bugs, like a 400, still go to Sentry as errors.

I deployed it and re-ran the failed resume. It failed again.

Bug #2: Gemini doesn't read Word files

I opened the profile in the Django admin and saw the file name: ..._CV_Updated.docx.

The task sent every resume to Gemini as raw bytes with whatever MIME type matched the file extension:

MIME_MAP = {
    ".pdf": "application/pdf",
    ".docx": "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
    ".doc": "application/msword",
}
Enter fullscreen mode Exit fullscreen mode

Gemini reads PDFs as inline documents. Word files aren't supported, and instead of a clear 400 Unsupported MIME type, the request came back as a 500 INTERNAL. So the "transient" error was actually permanent for this file, and my new retries would have tried it three more times and failed every time.

The fix: for .docx, extract the text locally and send it to Gemini as part of the prompt. PDFs still go as bytes. Legacy .doc files (the old binary format) get skipped with a warning.

My first version used python-docx. That turned out to be the wrong choice, as you'll see in Bug #4.

Bug #3: The deploy that froze the server

Adding python-docx meant a new line in requirements/base.txt. My deploy is a GitHub Action that SSHes into a DigitalOcean droplet and runs:

git pull origin main
docker compose -f production.yml build
docker compose -f production.yml run --rm django python manage.py collectstatic --noinput
docker compose -f production.yml run --rm django python manage.py migrate
docker compose -f production.yml up -d
Enter fullscreen mode Exit fullscreen mode

Changing the requirements file invalidates Docker's cached pip wheel layer, so every package is rebuilt from scratch. One of those packages was psycopg[c], the C-accelerated PostgreSQL driver, which compiles from source.

Here's what went wrong:

  1. The build was still compiling psycopg-c when appleboy/ssh-action hit its default 10-minute command_timeout and killed the session.
  2. The droplet has 2 GB of RAM and no swap. Compiling C while serving the site pushed it into memory exhaustion. The dashboard showed 90% memory and 83% CPU.
  3. The site stopped responding, and so did SSH.

I had to click Actions → Restart in the DigitalOcean dashboard. Docker's restart: unless-stopped policy brought the old containers back, so the site came back up on the previous image.

Three fixes came out of this.

Add swap so a heavy build slows down instead of freezing the machine:

sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
Enter fullscreen mode Exit fullscreen mode

During the next build, top showed about 920 MB of swap in use. Without it, the box would have frozen again.

Stop compiling on the server. psycopg[binary] ships precompiled wheels:

-psycopg[c]==3.2.2
+psycopg[binary]==3.2.2
Enter fullscreen mode Exit fullscreen mode

The next full rebuild went from timing out to about 3 minutes.

Make the deploy job tolerant and fail-fast:

- name: Deploy via SSH
  uses: appleboy/ssh-action@v1.0.3
  with:
    # ...
    # A requirements change rebuilds every wheel, which can
    # outlast the action's 10m default.
    command_timeout: 30m
    script_stop: true
Enter fullscreen mode Exit fullscreen mode

script_stop: true matters as much as the timeout. Without it, a failed build still continues on to up -d.

Bonus bug: ghost containers

While I was recovering, I ran the deploy by hand in tmux, and the GitHub Action ran at the same time. The two docker compose up -d runs collided and left behind containers with names like 430e938fc82c_gigglegigs-celerybeat-1. After that, every deploy failed with:

Error response from daemon: Conflict. The container name "/gigglegigs-celerybeat-1"
is already in use by container "430e938fc82c...".
Enter fullscreen mode Exit fullscreen mode

The fix is to remove the stateless app containers (never Postgres or Redis) and bring them back up:

docker ps -a --format '{{.Names}}' \
  | grep -E 'gigglegigs-(django|celeryworker|celerybeat|flower)-1$' \
  | xargs -r docker rm -f
docker compose -f production.yml up -d
Enter fullscreen mode Exit fullscreen mode

The lesson: never run two deploys against the same Compose project at the same time.

Bug #4: A corrupted image inside a Word file

With everything deployed, I re-ran the resume. FAILED, again. The worker logs had the answer:

File "/usr/local/lib/python3.11/site-packages/docx/opc/phys_pkg.py", line 83, in blob_for
    return self._zipf.read(pack_uri.membername)
...
zipfile.BadZipFile: Bad CRC-32 for file 'word/media/image1.png'
Enter fullscreen mode Exit fullscreen mode

A .docx is a zip archive. The text lives in word/document.xml, and images live under word/media/. This seeker's file had an embedded image (probably a profile photo) whose checksum didn't match. Word doesn't care and opens the file fine. python-docx, however, loads every part of the package when you open it, including images it never needs, so one bad PNG took down the whole parse.

I only needed the text, so I dropped python-docx and read the one XML file directly with the standard library:

import io
import zipfile
from xml.etree import ElementTree

WORD_NS = "{http://schemas.openxmlformats.org/wordprocessingml/2006/main}"


def _docx_text(file_content):
    """Plain text of a .docx, one line per paragraph (table cells included).
    Only word/document.xml is read, so a corrupt embedded image (which Word
    tolerates) doesn't fail it."""
    with zipfile.ZipFile(io.BytesIO(file_content)) as archive:
        root = ElementTree.fromstring(archive.read("word/document.xml"))
    lines = [
        "".join(node.text or "" for node in paragraph.iter(f"{WORD_NS}t"))
        for paragraph in root.iter(f"{WORD_NS}p")
    ]
    return "\n".join(line for line in lines if line.strip())
Enter fullscreen mode Exit fullscreen mode

Every paragraph is a <w:p> and every run of text is a <w:t>. Table cells contain paragraphs too, so resumes laid out in tables come through with no extra code. And I had one fewer dependency.

To make sure this stays fixed, the test builds a .docx by hand and deliberately corrupts the image. In ZIP_STORED mode the bytes sit raw in the archive, so you can change them without updating the CRC:

def _docx_bytes(corrupt_image=False):
    buffer = io.BytesIO()
    with zipfile.ZipFile(buffer, "w") as archive:
        archive.writestr("word/document.xml", DOCUMENT_XML)
        archive.writestr("word/media/image1.png", IMAGE_BYTES)
    content = buffer.getvalue()
    if corrupt_image:
        # Flip the stored image bytes without updating the zip's CRC-32.
        content = content.replace(IMAGE_BYTES, IMAGE_BYTES.upper())
    return content


@pytest.mark.parametrize("corrupt_image", [False, True])
def test_docx_resume_is_sent_as_text(profile, corrupt_image):
    ...
Enter fullscreen mode Exit fullscreen mode

I checked that this fixture raises exactly the same Bad CRC-32 for file 'word/media/image1.png' as the real resume before trusting the test.

One more deploy, one more parse_resume.delay(...), and then:

DONE
Enter fullscreen mode Exit fullscreen mode

What I took away

  • Not every 5xx is transient. Gemini returned 500 INTERNAL for an unsupported file type. Retries handle real outages, but read the error and what you sent before assuming it's the provider's fault.
  • Only parse what you need. A full-document library is the wrong tool when all you want is text. zipfile plus ElementTree handled a file that python-docx couldn't.
  • Small servers need swap. On a 2 GB droplet, one heavy build can take down the site and SSH with it. Swap is cheap insurance.
  • Don't compile in production. Prefer prebuilt wheels like psycopg[binary]. If you must compile, build the image in CI and pull it, rather than building on the box that serves traffic.
  • Deploy scripts should fail fast and never overlap. script_stop: true, a realistic timeout, and one deploy at a time.
  • Run long server commands in tmux. If your SSH connection drops halfway through a build, the build keeps going.

A single user's resume exposed four weaknesses in the stack. All of them are fixed now, and the next person who uploads a slightly broken Word file won't notice anything happened.

I'm building GiggleGigs, a job board for gigs and contract work, with Django, Celery and React. If you've been bitten by a .docx file too, I'd love to hear about it in the comments.

Top comments (0)