Fair question quotas across PDFs and pages: notes from building a multiple-choice generator

I'm Manu, a freelance developer in Málaga, Spain. I'm building OppTests, a web tool in beta that turns PDFs or notes into multiple-choice practice tests, each question with one correct answer. This post is about the pipeline behind it: problems that come up when you generate questions from documents with a language model, and what I chose to do about them.

The stack

• Backend: Django 4.2 and Django REST Framework on PostgreSQL 16.
• Frontend: Angular 18, built to static files served by nginx.
• Containers: Docker Compose runs the backend, PostgreSQL and Redis.
• Jobs: the queue is a PostgreSQL table. A worker claims tasks atomically with SELECT ... FOR UPDATE SKIP LOCKED. Redis pub/sub nudges the worker when there's work (polling is the fallback), and Redis also serves as a cache.
• Model calls: a routing gateway with one model combination per plan (free, basic, premium) and another for a quality judge.

Pipeline order

1. Extract native text per page. If heuristics judge a page's native text unusable, OCR replaces it; OCR engines are configurable, with a fallback.
2. Split into 750-character chunks with a 20-character overlap (RecursiveCharacterTextSplitter), keeping each chunk's source page.
3. Build the allocation plan before generating anything.
4. Generate with a margin: ask the model for a few more questions than needed.
5. Validate the JSON contract, drop questions that make no sense without the source text (a fast deterministic filter plus an LLM judge), and deduplicate.
6. Trim to the required count within coverage limits.
7. Fill any deficit.
8. Shuffle the answer options.
9. Save.

The margin lets the filters reject questions without leaving an immediate hole, and shuffling last means no later step can move the correct answer.

Two levels of quotas

Across files. When there are at least as many questions as files, every valid file gets one question and the rest are split in proportion to each file's useful chunks with the largest remainder method; ties on the remainder go to the larger weight, then to whichever file id sorts first. If a file can't hold its share, the excess moves to files that still have room. When there are fewer questions than files, a rotation keyed on the exam id picks which files get one.

Within a file. Questions are placed on units: pages in a PDF, or sections or three-chunk blocks in documents without pages. The file's units are laid out in order and its quota is spread over them in equal-width bands, one per question; each question lands on the unit at the midpoint of its band, with no weighting by text length. Density matters at one point: a unit with fewer than 200 useful characters is excluded. Above that line, a dense page and a sparse page count the same.

This is a deliberate trade-off: I put an even spread ahead of generation speed and accept that exams take longer.

Largest remainder, sketched

Ten questions split three ways is 3.33 each, and you can't generate a third of a question. Largest remainder (Hamilton's method) gives each part the integer floor of its ideal share, then hands leftover questions one at a time to the parts with the largest fractional remainders. The parts always add up to the total. Here's a sketch of the file level (simplified, not the production code):

# Simplified, not the production code.
from fractions import Fraction


def largest_remainder(total, weights, ids=None):
    """Split `total` into integers proportional to `weights`.

    Ties on the fractional part go to the larger weight, then the smaller id.
    """
    if ids is None:
        ids = list(range(len(weights)))
    s = sum(weights)
    ideal = [Fraction(total * w, s) for w in weights]
    quotas = [int(x) for x in ideal]  # floor, since values are non-negative
    leftover = total - sum(quotas)
    order = sorted(
        range(len(weights)),
        key=lambda i: (-(ideal[i] - quotas[i]), -weights[i], ids[i]),
    )
    for i in order[:leftover]:
        quotas[i] += 1
    return quotas


def file_quotas(total, files, exam_id):
    """`files` is a list of (file_id, useful_chunks) for valid files."""
    files = sorted(files)
    n = len(files)
    if total < n:
        # Fewer questions than files: rotate which files get one.
        start = exam_id % n
        chosen = {files[(start + k) % n][0] for k in range(total)}
        return {fid: int(fid in chosen) for fid, _ in files}
    ids = [fid for fid, _ in files]
    weights = [w for _, w in files]
    extra = largest_remainder(total - n, weights, ids)
    return {fid: 1 + e for fid, e in zip(ids, extra)}


if __name__ == "__main__":
    print(largest_remainder(10, [1, 1, 1]))  # [4, 3, 3]
    print(largest_remainder(2, [1, 3]))      # [0, 2]
    files = [(1, 6), (2, 3), (3, 1)]
    print(file_quotas(10, files, exam_id=42))  # {1: 5, 2: 3, 3: 2}
    print(file_quotas(2, files, exam_id=42))   # {1: 1, 2: 1, 3: 0}
    print(file_quotas(2, files, exam_id=43))   # {1: 0, 2: 1, 3: 1}

The sketch uses Fraction so remainders that should tie really do tie; floats can break a tie with a rounding error. The production code gets the same effect with integer arithmetic. It also leaves out the capacity redistribution.

When a unit comes up short

When a unit ends up short, deficit filling retries it at most twice; after two consecutive failures, the unit is excluded. If every unit is exhausted, the loop stops and the exam may end up with fewer questions than requested. The loop is also capped by rounds and by time. Without those limits, a page that lacks material could keep a job retrying indefinitely.

Shuffle indices, not text

Shuffling is the last step before saving. The worker permutes the option indices with random.shuffle, reorders the list by that permutation, and recomputes the correct answer with perm.index(correct_index). Because it tracks positions instead of text, two options with identical wording can't point the answer key at the wrong one.

Language: a prompt rule, not a validator

Each file's language is detected. If they all match, the prompt names that language; if they don't, a generic rule applies: same language as the document. There's no validator afterward, just the prompt rule plus the unanimity check, and it has caused problems more than once. With mixed sources, decide what should happen when they disagree; the model will pick something either way.

Don't render what isn't there

In the Angular app, explanations and source citations render when they exist. When they don't, the UI distinguishes between pending generation and not available, so a missing explanation doesn't look like a broken one. The button that opens the source needs both a file and a page.

Wrapping up

None of this is exotic: an old apportionment method, a PostgreSQL table as a queue, capped retries, an index-based shuffle. The harder part is deciding what "fair" means for a set of documents and how much generation time you'll spend on it.

If you've handled per-page allocation, PostgreSQL-backed queues or language drift differently, I'd like to compare approaches in the comments.

Disclosure: I used an AI model to help draft this post, and I checked the technical claims against the code.