Kazu

Posted on Nov 13

nginx proxy_cache shows another user's name

#nginx #webdev #devops #caching

Product pages are cached with nginx's proxy_cache. With proxy_cache, nginx stores a response from the application server and, until it expires, returns that response to requests for the same URL. A product page has the same content no matter who opens it, so serving it from the cache cuts the requests that reach the application server.

Then a change goes in that shows the logged-in user's name on the page. Now a product page with one user's name in it is returned to everyone until it expires, including users who are not logged in. Below, a page that shows a user's name other than the viewer's own is called "someone else's page." If the stored response was for a user who is not logged in, logged-in users get a page with no name. That one is called "the logged-out page."

When a request carries the cookie issued at login, the application server identifies the user from its value and shows that user's name. nginx does not look at the request's cookies, so it returns the stored response no matter whom it was made for.

sequenceDiagram
    participant A as Logged-in user A
    participant B as User B, not logged in
    participant N as nginx
    participant S as Application server
    A->>N: Product page (with cookie)
    N->>S: The cache entry has expired, so forward it with the cookie
    S-->>N: Page with A's name
    Note over N: Store it in the cache (cookie not checked)
    N-->>A: Page with A's name
    B->>N: Same product page (no cookie)
    Note over N: Same URL, so return it from the cache
    N-->>B: Page with A's name (someone else's page)

I reproduced this on nginx 1.30.6 and measured how often someone else's page is returned and what each fix does.

What is happening

nginx decides whether to store a response in the cache from the response headers, and looks up the stored response to return by URL. Whom the stored response is for is decided by the first request after the cache entry expires.

Headers that decide whether a response is cached

Product page responses with a name in them get cached too. nginx decides whether to store a response mainly from the following response headers¹. Set-Cookie is the response header that makes the browser store a cookie. private is a value saying the response is meant for one user, asking shared caches not to store it.

flowchart LR
    Q1["Set-Cookie"] -- present --> N["Not stored"]
    Q1 -- absent --> Q2["Cache-Control has<br>private, no-cache,<br>or no-store"]
    Q2 -- no --> Y["Stored"]
    Q2 -- yes --> N

If only the login response issues the cookie, product page responses carry no Set-Cookie. If the change that adds the name leaves Cache-Control as it was, the chart ends at "Stored."

The cache key that looks up stored responses

nginx builds a string called the cache key from the request and uses it to look up a stored response. The default cache key is $scheme$proxy_host$request_uri, which is made from the URL². So as long as the URL is the same, requests with different cookies get the same response.

Whose response gets stored

The stored response goes to everyone, so whether users see someone else's page or the logged-out page depends on whose response is stored. What gets stored is the response to the first request that arrives after the entry expires.

Measurement

I measured how often someone else's page is returned without a fix, with a client, nginx, and the application server connected as follows.

flowchart LR
    L["Client"] -- "500 requests/s<br>10% logged in" --> N["nginx<br>caches for 10 s"]
    N -- "only when the cache can't serve it" --> S["Application server<br>/items/1 to /items/100"]

For each request, the client picks one product page from /items/1 to /items/100, and it sends 500 requests per second for 120 seconds. 10% of the requests come from logged-in users and the rest from users who are not logged in
With the config below, nginx caches product page (/items/) responses for 10 seconds
The application server shows the user's name when a cookie is present, and a login link when it is not. It sets neither Set-Cookie nor Cache-Control on its responses

proxy_cache_path /var/cache/nginx/pages keys_zone=pages:10m;

server {
    listen 80;

    location /items/ {
        proxy_pass http://app:8080;
        proxy_cache pages;
        proxy_cache_valid 200 10s;
    }
}

The application server embeds in each page the user it built the page for. The client compares that user with the one who sent the request and counts whom each returned page was for³. The correct page is one with your own name if you are logged in, and one with a login link if you are not.

Without a fix, about 1 in 10 pages is someone else's page, whether or not the user is logged in.

Check whether your setup is affected

Send two requests in a row to the same URL: one as a logged-in user, then one as a user who is not logged in. If the second page shows the first user's name, your setup returns someone else's page.

To the end of the URL, append a parameter that the application server does not read, set to a value not used before. The default cache key is made from the URL including the query string, so requests to this URL never get a response stored by other users' requests.

curl -s -H 'Cookie: <name>=<value>' '<product page URL>?check=<unused value>'
curl -s '<product page URL>?check=<unused value>'

For Cookie, use the name and value of the cookie the browser received at login.

If the second response shows the first user's name, the first response was stored in the cache and returned to the second request for the same URL.

Fixes

Each option below removes both someone else's page and the logged-out page, and sends almost all requests from logged-in users to the application server. "Wrong page" is the share of returned pages that are not the correct page (someone else's page and the logged-out page). "Forwarded" is the number of requests nginx forwarded to the application server.

Option	Wrong page	Forwarded (/s)
Do nothing	about 20%	9
Skip the cache when logged in	0%	60
Add `Vary: Cookie`	0%	58
Cookie in the cache key	0%	58

Skip the cache when logged in

Add these two lines to the location where proxy_cache is set⁴.

proxy_cache_bypass $cookie_session;
proxy_no_cache $cookie_session;

$cookie_session is the value of the session cookie. Replace session with the name of your own cookie that is first issued at login. If you use the name of a cookie that is issued before login, nobody uses the cache.

Requests from logged-in users are forwarded instead of being served from the cache, and their responses are not stored. Only responses for users who are not logged in remain in the cache.

Add Vary: Cookie

The application server adds Vary: Cookie to every response. Vary is a response header that names the request headers that change the content for the same URL. nginx stores and returns a separate response for each cookie value⁵.

Put the cookie in the cache key

Append the cookie value to the end of nginx's cache key.

proxy_cache_key $scheme$proxy_host$request_uri$cookie_session;

Requests from logged-in users get a separate cache key for each user, and requests from users who are not logged in keep the URL-only cache key.

Which to choose

The wrong-page rate and the forwarded count are about the same for all three options, so choose by what you can change and what ends up stored in the cache.

If you can change the nginx config, skip the cache when logged in. Putting the cookie in the cache key also takes only an nginx config change, but the forwarded count stays the same while a page for each logged-in user piles up in the cache.

If you can change only the application server, add Vary: Cookie. nginx splits on the value of the whole Cookie header, so on a site that issues other cookies, such as for analytics, pages for users who are not logged in are also stored separately for each value of those cookies.

Someone else's page is returned even when neither the nginx config nor the application server has a mistake in it. Before a change that adds per-user content to a page goes in, I want to check whether nginx caches that page.

ngx_http_upstream_process_set_cookie() and ngx_http_upstream_process_cache_control() in the nginx 1.30.6 source. ↩
The nginx docs on proxy_cache_key. ↩
Run on a 4-core GitHub Actions runner. The code is in show-your-work, and the results are in the CI run. ↩
The nginx docs on proxy_cache_bypass and proxy_no_cache. ↩
The nginx docs on proxy_cache_valid. They state that a response with Vary is cached taking the corresponding request header fields into account. ↩

DEV Community