DEV Community

Cover image for Deep Dive: Implementing RFC 9110 HTTP Content Negotiation for Semantic Web in GitLab Pages
Manish Kumar
Manish Kumar

Posted on Edited on

Deep Dive: Implementing RFC 9110 HTTP Content Negotiation for Semantic Web in GitLab Pages

#architecture #backend #softwaredevelopment #webdev

Engineering HTTP Content Negotiation in GitLab Pages: RFC 9110, Semantic Web, and Production-Grade Request Resolution

How I implemented media-type negotiation for Linked Data resources in GitLab Pages using Go, bounded parsing, MIME resolution, virtual filesystem lookups, and HTTP cache semantics.

Static hosting appears straightforward: receive a request, resolve a path, read a file, and return a response.

However, that model becomes limiting when a URL represents a logical resource rather than a particular file.

Consider:

GET /alice HTTP/1.1
Host: example.gitlab.io
Accept: text/turtle
Enter fullscreen mode Exit fullscreen mode

The client is not necessarily asking for an HTML document. It may be requesting an RDF representation of a person, an ontology, or another Linked Data resource.

A browser might request the same URL with:

Accept: text/html
Enter fullscreen mode Exit fullscreen mode

A semantic crawler might request:

Accept: application/rdf+xml
Enter fullscreen mode Exit fullscreen mode

And an application consuming structured linked data might request:

Accept: application/ld+json
Enter fullscreen mode Exit fullscreen mode

All three requests refer to the same URI, but they express different representation preferences.

This is the problem addressed by HTTP content negotiation.

I implemented this capability in GitLab Pages through Merge Request !1384, addressing the long-standing Issue #1135.

The implementation introduces media-type-aware resolution for extensionless paths, allowing GitLab Pages to serve Semantic Web representations while preserving its existing HTML fallback behavior.

This article explores the engineering decisions behind the implementation, including HTTP semantics, parsing strategy, resource bounds, MIME-to-extension resolution, virtual filesystem integration, cache correctness, and testing.

1. Understanding the Architectural Problem

1.1 Traditional static-file resolution

A conventional static hosting system generally follows a predictable mapping:

Request URL          Resolved file

/about               /about.html
/docs                /docs.html
/                     /index.html
Enter fullscreen mode Exit fullscreen mode

The URL-to-file mapping is primarily path-driven.

This works well for traditional websites, but it introduces a limitation when multiple representations of the same logical resource need to be exposed through one URI.

For example, a Semantic Web publisher might maintain:

public/
├── alice.html
├── alice.rdf
├── alice.ttl
├── alice.jsonld
├── alice.nt
└── index.html
Enter fullscreen mode Exit fullscreen mode

The existing extensionless resolution mechanism has no reason to distinguish these representations based on the request headers.

Consequently, a request for /alice may resolve to alice.html, even when the client explicitly prefers RDF or Turtle.

1.2 The desired behavior

The objective was to extend the existing resolution mechanism without changing the established behavior of ordinary static websites.

The expected mapping becomes:

Request Accept header Preferred representation
/alice text/html alice.html
/alice application/rdf+xml alice.rdf
/alice text/turtle alice.ttl
/alice application/ld+json alice.jsonld
/alice application/n-triples alice.nt

The same principle must work for directory index resources.

For example:

GET /
Accept: application/rdf+xml
Enter fullscreen mode Exit fullscreen mode

should be able to resolve:

/index.rdf
Enter fullscreen mode Exit fullscreen mode

when that representation exists.

The important architectural constraint was that content negotiation should extend the existing file resolution pipeline rather than introduce an independent serving mechanism.

This keeps the behavior integrated with GitLab Pages' existing virtual filesystem and content-type detection infrastructure.

2. HTTP Content Negotiation and RFC 9110

The implementation is based on the HTTP Accept request header, specified in RFC 9110, Section 12.5.1.

The header allows a client to communicate which response media types it can accept and how it prioritizes them.

Consider:

Accept: text/turtle;q=0.9,
        application/rdf+xml;q=0.8,
        text/html;q=0.5
Enter fullscreen mode Exit fullscreen mode

The client expresses three preferences:

  1. Turtle is preferred.
  2. RDF/XML is an acceptable alternative.
  3. HTML is less preferred.

The q parameter is called a quality value.

RFC 9110 defines its valid range as:

0 <= q <= 1
Enter fullscreen mode Exit fullscreen mode

A missing quality value defaults to 1.

A value of 0 means that the corresponding media range is not acceptable.

2.1 Media-range specificity

HTTP also supports wildcard media ranges.

There are three relevant forms:

Media range Specificity
text/html 2
text/* 1
*/* 0

For equal quality values, the implementation uses specificity as a secondary ordering criterion.

For example:

Accept: */*;q=0.8,
        text/*;q=0.8,
        text/html;q=0.8
Enter fullscreen mode Exit fullscreen mode

The resulting candidate ordering is:

text/html
text/*
*/*
Enter fullscreen mode Exit fullscreen mode

The original header position is retained as a further tie-breaker.

2.2 An important RFC distinction

There is a subtle difference between sorting media-range candidates and calculating the effective quality of a concrete representation.

RFC 9110 specifies that the most specific matching media range determines the quality associated with a representation.

For example:

Accept: text/*;q=0.3,
        text/html;q=0.9
Enter fullscreen mode Exit fullscreen mode

The effective quality for text/html is determined by the explicit text/html range, not by treating both entries as independent preferences for that representation.

This distinction matters when implementing a general-purpose HTTP negotiation engine.

In this implementation, the candidate parser establishes an ordered list of supported media-type preferences, using quality, specificity, and original order. The filesystem resolver then attempts those candidates in order.

The implementation is deliberately scoped to the supported media types and file-extension resolution required by the GitLab Pages feature.

3. Designing a Bounded Accept Header Parser

The parser is one of the most important parts of this implementation.

An HTTP header is external input. It must be handled defensively, especially in infrastructure that processes requests at scale.

A parser that works correctly for ordinary browser requests can still behave poorly when presented with malformed, oversized, or adversarial input.

The implementation introduces a dedicated parser in:

internal/serving/disk/negotiate.go
Enter fullscreen mode Exit fullscreen mode

Instead of mixing header parsing directly into filesystem resolution, the logic is separated into smaller responsibilities:

parseAcceptHeader()
    |
    +-- parseCandidate()
    |       |
    |       +-- mime.ParseMediaType()
    |       +-- parseQuality()
    |       +-- mimeSpecificity()
    |
    +-- sortCandidates()
    |
    +-- return ordered candidates
Enter fullscreen mode Exit fullscreen mode

This separation makes the parser easier to reason about and allows it to be tested independently.

3.1 Candidate representation

Each parsed media range is represented by a small structure:

type mimeCandidate struct {
    mediaType   string
    quality     float64
    specificity int
    order       int
}
Enter fullscreen mode Exit fullscreen mode

Each field serves a specific purpose:

  • mediaType: Normalized media type used for MIME mapping.
  • quality: Client-provided preference weight.
  • specificity: Specificity classification.
  • order: Original position in the request header.

Keeping the original position is important because sorting must remain deterministic when multiple candidates have identical quality and specificity.

3.2 Parsing quality values safely

Go's strconv.ParseFloat is useful for converting numeric strings into floating-point values.

However, successful parsing alone does not guarantee a valid HTTP quality value.

For example, floating-point parsing can accept special values such as:

NaN
+Inf
-Inf
Enter fullscreen mode Exit fullscreen mode

These values must not enter a sorting algorithm that expects ordinary comparable quality values.

In particular, NaN has unusual comparison behavior:

math.NaN() == math.NaN() // false
Enter fullscreen mode Exit fullscreen mode

A sorting comparator operating on such values can violate the assumptions expected by the sorting implementation.

The parser therefore validates the result explicitly:

func parseQValue(qStr string) (float64, bool) {
    qStr = strings.TrimSpace(qStr)

    q, err := strconv.ParseFloat(qStr, 64)

    if err != nil ||
        math.IsNaN(q) ||
        math.IsInf(q, 0) ||
        q < 0.0 ||
        q > 1.0 {
        return 0, false
    }

    return q, true
}
Enter fullscreen mode Exit fullscreen mode

This establishes several useful invariants:

  • Parsing errors are rejected.
  • NaN is rejected.
  • Positive and negative infinity are rejected.
  • Negative values are rejected.
  • Values greater than one are rejected.
  • Only finite values within the expected range are accepted.

One standards detail is worth distinguishing: RFC 9110's qvalue grammar permits at most three decimal digits. The range validation above enforces the numeric bounds but is not, by itself, a complete lexical validator for that grammar.

The key engineering principle is to validate untrusted numeric input before it participates in ordering decisions.

3.3 Handling malformed media-type parameters

The implementation uses Go's standard library:

mime.ParseMediaType(part)
Enter fullscreen mode Exit fullscreen mode

This provides structured parsing of media types and parameters.

However, malformed input creates an important edge case.

Consider:

Accept: text/html; malformed; q=0
Enter fullscreen mode Exit fullscreen mode

If parsing fails and the implementation blindly assigns the default quality of 1, it could accidentally reintroduce a representation the client explicitly excluded.

To avoid that behavior, the implementation has a fallback path that extracts the q parameter from the raw entry when structured parsing fails.

The relevant logic is:

func parseQuality(
    params map[string]string,
    rawPart string,
) (float64, bool) {

    if params != nil {
        if qStr, ok := params["q"]; ok {
            return parseQValue(qStr)
        }

        return 1.0, true
    }

    if idx := strings.Index(rawPart, ";"); idx != -1 {
        for _, param := range strings.Split(
            rawPart[idx+1:],
            ";",
        ) {
            param = strings.TrimSpace(param)

            if strings.HasPrefix(
                strings.ToLower(param),
                "q=",
            ) {
                return parseQValue(param[2:])
            }
        }
    }

    return 1.0, true
}
Enter fullscreen mode Exit fullscreen mode

This is a defensive parsing decision: malformed syntax should not cause an explicitly excluded media type to be treated as fully acceptable.

It also demonstrates why error handling should consider the semantics of the input, rather than simply returning a default value whenever parsing fails.

3.4 Bounding parser work

Another important concern is resource consumption.

An attacker could send an Accept header containing a very large number of comma-separated entries.

Even if each individual entry is inexpensive to parse, processing an unbounded number of entries creates unnecessary CPU and allocation pressure.

The implementation introduces:

const maxAcceptParts = 16
Enter fullscreen mode Exit fullscreen mode

The parsing loop is bounded:

for parsedParts := 0;
    len(header) > 0 && parsedParts < maxAcceptParts;
    parsedParts++ {

    var part string

    if idx := strings.IndexByte(header, ','); idx != -1 {
        part = header[:idx]
        header = header[idx+1:]
    } else {
        part = header
        header = ""
    }

    // Parse and validate the candidate.
}
Enter fullscreen mode Exit fullscreen mode

An important detail is that the counter tracks entries inspected, not just valid candidates appended.

Why does this matter?

Consider:

Accept: text/html,
        text/html,
        text/html,
        text/html,
        ...
Enter fullscreen mode Exit fullscreen mode

If the limit only applied to successfully appended candidates, repeated duplicate entries could still force the parser to inspect a large amount of input.

By counting every inspected entry, the implementation establishes a predictable upper bound on parsing iterations.

This is a small implementation detail with meaningful operational consequences.

It also keeps the candidate slice bounded, which limits subsequent sorting and allocation work.

3.5 Deduplicating media types

The parser maintains a set:

seenTypes := make(map[string]struct{})
Enter fullscreen mode Exit fullscreen mode

After parsing and validation:

if _, seen := seenTypes[candidate.mediaType]; seen {
    continue
}

seenTypes[candidate.mediaType] = struct{}{}
candidates = append(candidates, candidate)
Enter fullscreen mode Exit fullscreen mode

This avoids retaining repeated entries for the same normalized media type.

Without deduplication, duplicate entries could trigger redundant resolution attempts and complicate deterministic behavior.

The implementation uses a map for efficient membership checks.

3.6 Deterministic candidate ordering

The candidates are sorted using:

func sortCandidates(candidates []mimeCandidate) {
    sort.SliceStable(candidates, func(i, j int) bool {
        if candidates[i].quality != candidates[j].quality {
            return candidates[i].quality > candidates[j].quality
        }

        if candidates[i].specificity != candidates[j].specificity {
            return candidates[i].specificity >
                candidates[j].specificity
        }

        return candidates[i].order < candidates[j].order
    })
}
Enter fullscreen mode Exit fullscreen mode

The ordering criteria are:

  1. Higher quality first.
  2. Higher specificity first.
  3. Earlier position in the original header first.

Stable sorting provides an additional guarantee that equal candidates do not acquire arbitrary ordering.

This makes the parser's output deterministic and easier to test.

4. MIME-to-Extension Resolution

Once the request preferences have been parsed, the next problem is translating media types into filesystem candidates.

HTTP clients communicate using MIME types.

Static hosting systems store resources using filenames and extensions.

The implementation needs a mapping between these two representations.

4.1 Supported media types

The implementation defines explicit mappings for the supported Semantic Web formats:

Media type File extension
application/rdf+xml .rdf
text/turtle .ttl
application/x-turtle .ttl
text/n3 .n3
text/rdf+n3 .n3
application/n-triples .nt
application/ld+json .jsonld
application/owl+xml .owx, .owl
text/html .html
application/xhtml+xml .html

The mapping is maintained centrally:

var knownMimeExtensions = map[string][]string{
    mimeRdfXML:   {extRDF},
    mimeTurtle:   {extTTL},
    mimeXTurtle:  {extTTL},
    mimeN3:       {extN3},
    mimeRdfN3:    {extN3},
    mimeNTriples: {extNT},
    mimeLdJSON:   {extJSONLD},
    mimeOwlXML:   {extOWX, extOwl},
    mimeTextHTML: {extHTML},
    mimeXhtmlXML: {extHTML},
}
Enter fullscreen mode Exit fullscreen mode

The important design decision is that MIME support is explicit.

The resolver does not attempt to derive arbitrary file extensions from arbitrary client-supplied media types.

That keeps the supported surface predictable and avoids turning request headers into unrestricted filesystem path construction.

4.2 Multiple extensions for one representation

Some media types can correspond to more than one established filename convention.

For example:

mimeOwlXML: {extOWX, extOwl}
Enter fullscreen mode Exit fullscreen mode

The resolver checks the supported extensions in the configured order.

This provides compatibility with existing naming conventions without duplicating the negotiation logic.

4.3 MIME type detection must match the selected file

Selecting the correct file is only half of the problem.

The HTTP response must also communicate the correct media type.

GitLab Pages already contains extension-based content-type detection.

The implementation extends that mapping to include Semantic Web extensions such as:

.rdf
.ttl
.n3
.nt
.jsonld
.owl
.owx
Enter fullscreen mode Exit fullscreen mode

This keeps file resolution and response metadata aligned.

A server should not return a Turtle file with an unrelated HTML content type simply because the original URL did not contain an extension.

5. Integrating Negotiation with the Virtual Filesystem

The resolver should not bypass the existing storage abstraction.

GitLab Pages uses a virtual filesystem layer, which allows the serving implementation to work with its supported storage backends without embedding backend-specific file access logic into content negotiation.

The negotiation implementation therefore operates through the existing VFS interfaces.

The central function is:

func (reader *Reader) negotiatePath(
    ctx context.Context,
    root vfs.Root,
    basePath string,
    r *http.Request,
) (string, bool, error)
Enter fullscreen mode Exit fullscreen mode

Its responsibilities are:

  1. Read the request's Accept header.
  2. Parse the supported media-type candidates.
  3. Translate candidates into file extensions.
  4. Probe the VFS for existing representations.
  5. Return the first matching path.
  6. Fall back to HTML when no preferred representation exists.

5.1 Resolution pipeline

The high-level flow looks like this:

                 HTTP Request
                      |
                      v
              Read Accept Header
                      |
                      v
             Parse Media Ranges
                      |
                      v
          Validate Quality and Specificity
                      |
                      v
             Sort Candidates
                      |
                      v
           Map MIME Types to Extensions
                      |
                      v
             Probe Virtual Filesystem
                      |
             +--------+--------+
             |                 |
          Found              Not Found
             |                 |
             v                 v
        Return Path       Try Next Candidate
                               |
                               v
                         HTML Fallback
                               |
                       +-------+-------+
                       |               |
                     Found          Missing
                       |               |
                       v               v
                   Return Path       Error
Enter fullscreen mode Exit fullscreen mode

This architecture separates negotiation policy from the actual storage implementation.

5.2 Probing candidate extensions

The resolver uses a dedicated helper:

func (reader *Reader) tryExtensions(
    ctx context.Context,
    root vfs.Root,
    basePath string,
    exts []string,
    seenExts map[string]struct{},
) (string, error)
Enter fullscreen mode Exit fullscreen mode

The helper iterates through candidate extensions and resolves each path using the existing path resolution mechanism.

Conceptually:

for _, ext := range exts {
    if _, seen := seenExts[ext]; seen {
        continue
    }

    seenExts[ext] = struct{}{}

    fullPath, err := reader.resolvePath(
        ctx,
        root,
        basePath+ext,
    )

    if err == nil {
        return fullPath, nil
    }
}
Enter fullscreen mode Exit fullscreen mode

There are two important decisions here.

First: reuse the existing resolver.

The implementation does not introduce a separate filesystem access path just for semantic resources.

This preserves existing VFS behavior and keeps storage-specific concerns out of the negotiation code.

Second: deduplicate extensions across candidates.

Different MIME types can map to the same extension.

For example, both text/turtle and application/x-turtle map to .ttl.

Without deduplication, the same path could be probed more than once.

The seenExts map prevents redundant resolution attempts.

5.3 Complexity and operational cost

Let:

  • (n) be the maximum number of inspected Accept entries.
  • (e) be the maximum number of extensions associated with a supported media type.
  • (p) be the number of unique candidate paths actually probed.

With the parser bound set to 16, the sorting cost is:

[
O(n \log n)
]

Candidate deduplication and extension lookup are expected constant-time map operations.

The filesystem resolution cost depends on the number of unique candidates and the underlying VFS backend.

The important operational property is that parsing and candidate generation are bounded, and duplicate path probes are avoided.

This does not imply a universal latency guarantee: actual lookup performance remains dependent on the storage implementation.

6. Backward Compatibility and HTML Fallback

Introducing content negotiation into a mature static hosting platform creates a compatibility requirement.

Existing projects should not need to rename files or change their URLs.

The original behavior must remain available.

The resolver therefore uses HTML as its final fallback.

Consider:

public/
├── alice.html
└── alice.ttl
Enter fullscreen mode Exit fullscreen mode

A client requests:

GET /alice
Accept: application/rdf+xml
Enter fullscreen mode Exit fullscreen mode

There is no alice.rdf.

The resolver checks the preferred representation, finds that it does not exist, and falls back to:

alice.html
Enter fullscreen mode Exit fullscreen mode

If the requested Turtle representation exists, it can be selected instead.

This behavior is important because it allows semantic resources to be introduced incrementally.

A project can publish HTML today and add RDF, Turtle, or JSON-LD representations without redesigning its URL structure.

6.1 Requests without a usable Accept header

When the header is absent or produces no usable candidates, the implementation resolves the HTML representation.

This preserves the traditional behavior of extensionless static URLs.

It also has an important caching implication: the URL is still negotiable even when the current request does not provide a usable preference.

That distinction becomes relevant in the next section.

7. HTTP Caching and the Vary: Accept Header

Caching is one of the easiest parts of content negotiation to overlook.

Suppose a client requests:

GET /alice
Accept: text/html
Enter fullscreen mode Exit fullscreen mode

The server returns:

Content-Type: text/html
Enter fullscreen mode Exit fullscreen mode

A shared cache stores the response under /alice.

Now another client requests:

GET /alice
Accept: text/turtle
Enter fullscreen mode Exit fullscreen mode

If the cache does not know that the response depends on the Accept header, it might reuse the previously cached HTML representation.

The result is incorrect content delivery.

7.1 Why Vary matters

HTTP defines the Vary response header to communicate which request headers influenced representation selection.

For content-negotiated responses:

Vary: Accept
Enter fullscreen mode Exit fullscreen mode

This tells compliant caches that the Accept request header is part of the cache key for the response variant.

Conceptually:

/alice + Accept: text/html
    |
    v
alice.html

/alice + Accept: text/turtle
    |
    v
alice.ttl

/alice + Accept: application/rdf+xml
    |
    v
alice.rdf
Enter fullscreen mode Exit fullscreen mode

The cache must distinguish these variants.

7.2 The important case: missing Accept headers

A subtle but important implementation detail is that Vary: Accept is relevant even when the current request has no usable Accept header.

For example:

  1. A client requests /alice without an Accept header.
  2. GitLab Pages serves alice.html.
  3. Another client requests /alice with Accept: text/turtle.

If the first response is cached without accounting for Accept, the cache could incorrectly reuse HTML for the second request.

The implementation therefore retains the negotiation state even when it falls back to HTML because there are no usable candidates.

7.3 Direct file requests

There is also an important distinction between negotiated and explicit file requests.

For example:

GET /alice.ttl
Enter fullscreen mode Exit fullscreen mode

This request already identifies a specific representation through its URL.

It does not need content negotiation to determine the extension.

The implementation keeps direct extension requests outside the negotiation path, avoiding unnecessary Vary: Accept behavior.

This is a useful example of applying HTTP caching semantics only where the representation actually depends on request negotiation.

8. Feature-Flagged Rollout

A production infrastructure change should not unexpectedly alter the behavior of existing deployments.

The implementation introduces a dedicated feature flag:

var ContentNegotiation = Feature{
    EnvVariable:    "FF_ENABLE_CONTENT_NEGOTIATION",
    defaultEnabled: false,
}
Enter fullscreen mode Exit fullscreen mode

The flag provides an explicit control over the new functionality.

When disabled:

  • Existing extensionless URL resolution remains unchanged.
  • Traditional HTML resolution continues to work.
  • Existing static sites do not automatically enter the new negotiation flow.

When enabled:

  • The request's Accept header is considered.
  • Supported Semantic Web media types can be resolved.
  • Existing HTML fallback behavior is retained.

The default-disabled configuration provides a controlled introduction of the feature.

This is particularly important in infrastructure software, where even a seemingly small change to path resolution can affect a large number of unrelated applications.

9. Testing Strategy: Validate Behavior, Not Just Parsing

A parser that passes a few unit tests is not sufficient for a feature that changes request resolution.

The implementation includes both parser-level and integration-level tests.

9.1 Parser unit tests

The dedicated parser tests cover cases such as:

Test case Expected behavior
Empty Accept header No candidates
Single media type Default quality of 1
Multiple media types Quality-based ordering
Equal quality values Specificity-based ordering
Equal quality and specificity Original order preserved
q=0 Candidate excluded
q=NaN Candidate rejected
q=Inf Candidate rejected
Quality above 1 Candidate rejected
Negative quality Candidate rejected
Malformed entry containing q=0 Explicit exclusion preserved
Duplicate media types Duplicate candidates removed
Excessive header entries Parsing remains bounded

These tests are particularly valuable because they validate invariants rather than only testing ordinary inputs.

9.2 Integration tests

The integration tests exercise the actual content-resolution path.

Important scenarios include:

Semantic file resolution

/alice + application/rdf+xml -> alice.rdf
/alice + text/turtle         -> alice.ttl
/alice + application/ld+json -> alice.jsonld
Enter fullscreen mode Exit fullscreen mode

Directory index resolution

/ + application/rdf+xml -> index.rdf
Enter fullscreen mode Exit fullscreen mode

Quality-weighted selection

Accept: text/turtle;q=0.9,
        application/rdf+xml;q=0.8
Enter fullscreen mode Exit fullscreen mode

The resolver should prefer the Turtle representation when both corresponding files are available.

Fallback behavior

If the requested semantic representation does not exist, the resolver should attempt the HTML file.

Default behavior

A request without a usable Accept header should retain HTML resolution.

Explicit extension requests

Direct requests for files such as /alice.ttl should bypass negotiation.

9.3 Why both layers matter

Unit tests answer:

Does the parser correctly interpret and order supported media-type preferences?

Integration tests answer:

Does the complete serving pipeline select the correct file and preserve the expected HTTP behavior?

Both are necessary.

A parser can be correct while the resolver is wrong. Likewise, filesystem resolution can work while response metadata or cache headers are incorrect.

Testing the complete behavior is what makes this type of infrastructure change reliable.

10. Engineering Trade-offs and Lessons

Several decisions in this implementation are worth highlighting.

10.1 Explicit MIME mappings over arbitrary extension derivation

Mapping supported MIME types explicitly is more predictable than deriving filesystem extensions from arbitrary request input.

It limits the resolution surface and makes supported behavior easier to review.

10.2 Bounded parsing over unbounded flexibility

HTTP headers are client-controlled.

A fixed maximum number of inspected entries gives the parser a predictable processing budget.

This is especially useful in high-throughput request-serving systems.

10.3 Reusing the VFS abstraction

Filesystem resolution should remain the responsibility of the existing storage layer.

Integrating negotiation through the VFS avoids introducing storage-specific assumptions into HTTP parsing logic.

10.4 Cache correctness is part of protocol correctness

Serving the correct file is not sufficient if an intermediary cache can deliver the wrong representation to another client.

Vary: Accept is part of the feature's correctness, not an optional performance optimization.

10.5 Backward compatibility must be intentional

A new resolution strategy should not silently redefine existing URLs.

Feature flags and HTML fallback behavior provide a controlled path toward introducing the new capability.

11. Final Architecture

The complete implementation can be understood as five cooperating layers:

                  HTTP Request
                       |
                       v
             Accept Header Parser
                       |
                       v
             Preference Ordering
                       |
                       v
              MIME Extension Map
                       |
                       v
             VFS Path Resolution
                       |
                       v
             Content-Type Detection
                       |
                       v
                HTTP Response
                       |
                       v
                Vary: Accept
Enter fullscreen mode Exit fullscreen mode

Each layer has a clear responsibility:

Layer Responsibility
HTTP parsing Interpret client media-type preferences
Candidate ordering Establish deterministic preference order
MIME mapping Translate supported media types to extensions
VFS resolution Locate the first available representation
HTTP response Return the selected resource with correct metadata and cache variation

The result is an extension to GitLab Pages' existing static serving architecture rather than a separate content-serving subsystem.

12. Conclusion

Implementing HTTP content negotiation in GitLab Pages was an interesting exercise in connecting protocol semantics with an existing production serving architecture.

The challenge was not simply recognizing Accept: text/turtle.

The implementation needed to account for:

  • HTTP quality values and media-range specificity.
  • Malformed and non-finite numeric inputs.
  • Bounded parsing of untrusted request headers.
  • Duplicate candidate and filesystem lookup elimination.
  • MIME-to-extension mapping.
  • Virtual filesystem integration.
  • HTML fallback and backward compatibility.
  • Correct cache variation through Vary: Accept.
  • Controlled rollout through feature flags.
  • Unit and integration-level validation.

These details illustrate an important engineering principle:

A production-ready protocol feature is not just about implementing the happy path. It is about preserving correctness across parsing, resolution, storage, caching, and compatibility boundaries.

This contribution gave me an opportunity to work deeper into Go-based infrastructure, HTTP semantics, and the design considerations involved in extending an established open-source platform.

The work is part of my contribution to GitLab Pages through the community merge request below.

References and Implementation

Author: Manish Kumar

GitLab: @manishpatel953249

Project: GitLab Pages

Building, understanding, and contributing to production-grade open-source infrastructure.

Top comments (0)