The Semantic Web and Linked Open Data (LOD) initiatives rely on a foundational principle: a single persistent URI (e.g., https://example.com/alice or https://example.com/ontology) should be accessible to both humans in their web browsers and machines consuming semantic data models.
Through HTTP Content Negotiation, a browser requesting Accept: text/html receives a styled HTML page, while an RDF crawler or knowledge-graph ingestion pipeline requesting Accept: text/turtle or Accept: application/ld+json receives machine-readable triples from the exact same URI.
In my recent contribution to GitLab Pages (MR !1384, closing Issue #1135), I designed and implemented RFC 9110-compliant content negotiation for static sites hosted on GitLab Pages.
Here is an engineering deep-dive into how we implemented media-type negotiation, specificity weighting, and cache safety in Go.
1. The Problem: Extensionless URLs in Static Hosting
Historically, GitLab Pages mapped extensionless HTTP requests (GET /research-paper or GET /) exclusively to .html (or index.html).
For scientific organizations, academic publishers, and open-data projects hosting ontologies on GitLab Pages:
- A browser requesting
/schemareceivedschema.html. - A machine or graph database sending
Accept: application/rdf+xmlorAccept: text/turtlestill received HTML or a 404 error.
Because static sites couldn't resolve semantic file variants without forcing users to type explicit extensions like schema.ttl, GitLab Pages could not serve Linked Data in compliance with W3C best practices.
2. Designing an RFC 9110 Compliant Accept Parser in Go
HTTP content negotiation is defined in RFC 9110 (Section 12.5.1). A robust content negotiation parser must handle:
-
Quality Values (
q): Entries with higher weights (q=1.0vsq=0.5) take priority. Zero quality (q=0) rejects the type. -
Specificity Weighting:
- Exact type/subtype:
application/ld+json(highest precedence) - Subtype wildcards:
text/*(medium precedence) - Global wildcards:
*/*(lowest precedence)
- Exact type/subtype:
We built a parser that parses candidate media types and orders them by quality and specificity:
type MediaRange struct {
Type string
Subtype string
Quality float64
Specificity int
}
When a client sends:
Accept: text/html;q=0.8, text/turtle;q=0.9, */*;q=0.1
The parser evaluates text/turtle ($q=0.9$) first, followed by text/html ($q=0.8$), and finally the wildcard fallback.
3. Mapping Media Types to Semantic File Extensions
GitLab Pages now recognizes common Linked Data and Semantic Web formats:
| MIME Media Type | File Extension | Format Description |
|---|---|---|
text/turtle |
.ttl |
Terse RDF Triple Language |
application/ld+json |
.jsonld |
JSON for Linked Data |
application/rdf+xml |
.rdf |
Resource Description Framework XML |
application/n-triples |
.nt |
N-Triples |
text/n3 |
.n3 |
Notation3 |
application/owl+xml |
.owl |
Web Ontology Language |
text/html, application/xhtml+xml
|
.html |
HyperText Markup Language |
Resolution Algorithm:
- When receiving a request for
/dataset: - Inspect the client's
Acceptheader. - For each candidate media type in descending preference order, test if the corresponding file exists on disk (e.g.,
dataset.ttl,dataset.jsonld,dataset.rdf). - If a match is found, serve that file with the matching
Content-Typeheader. - If no semantic file matches or no
Acceptheader is provided, safely fall back to the defaultdataset.html.
4. Cache Correctness: Why Vary: Accept is Crucial
In static web hosting and CDNs, caching responses correctly is just as important as serving them.
If a CDN caches dataset.html for GET /dataset, a subsequent request from an RDF crawler with Accept: text/turtle would mistakenly receive the cached HTML from the CDN edge server!
To prevent cache poisoning:
- Whenever content negotiation takes place, GitLab Pages appends:
Vary: Accept
- This instructs CDNs, reverse proxies, and browsers that the cached representation depends strictly on the incoming
Acceptheader. - Direct extension requests (e.g.,
GET /dataset.ttl) bypass negotiation and do not emit unnecessaryVaryheaders.
5. Testing & Verification
The feature is covered by unit tests and end-to-end integration tests in TestReader_ContentNegotiation:
-
Directory Indexes:
GET /withAccept: application/rdf+xmlservesindex.rdf. -
Quality Fallbacks: Verifying that
q=0.9wins overq=0.7. - Graceful HTML Fallback: If an unsupported media type is requested, standard HTML is delivered without breaking normal user navigation.
go test -v ./internal/vfs/zip -run '^TestReader_ContentNegotiation$'
Conclusion
With this contribution, thousands of static websites hosted on GitLab Pages can now serve both human-friendly documentation and Linked Open Data from identical canonical URLs.
Special thanks to the GitLab Pages maintainers for their thorough reviews and guidance throughout this implementation!
Check out the merged implementation on GitLab:
👉 GitLab Pages MR !1384
Author: Manish Kumar
GitLab: @manishpatel00

Top comments (0)