DEV Community

Harish
Harish

Posted on

Two quote marks took down the datacenter proxy

A proxy URL in our secret store was wrapped in typographic quotes. Nothing could parse it, one service crash-looped on startup, and two others quietly fell back to direct HTTP. The command that wrote it looked correct, and when we copied it out of chat it worked fine.

This is how we found it, and what to do so it doesn't happen to you.

What we saw

An alert fired: a queue backlog had aged past two hours. Every pod of the download worker was dying at startup with the same line:

failed to create HTTP client: failed to parse proxy URL (invalid)
Enter fullscreen mode Exit fullscreen mode

Two other services that read the same secret didn't crash. They logged a warning about an invalid proxy URL and fell back to direct HTTP, so they kept running without their datacenter proxy tier. The queue grew to about 12,000 messages while the worker restarted in a loop.

What "invalid" meant

Our startup log prints a safe description of each proxy URL: host, port and zone, with no credentials. For a URL that fails to parse, it prints the word invalid.

Two proxy URLs are configured, a datacenter one and an unlocker one. They come from separate secret properties and are parsed by the same code. The unlocker logged a normal host and zone. Only the datacenter one logged invalid. So only one value was bad.

What was actually stored

Our team could write the secret but not read it or its history, which is why this took a while. Once we had read access, the version history of the property showed it. One version was 96 characters long instead of 94, and it began and ended with these:

Character Name Code point Who understands it
' straight quote U+0027 the shell strips it
‘ left curly quote U+2018 nothing, it's just text
’ right curly quote U+2019 nothing, it's just text

A shell only treats ' and " as quoting. The curly versions are ordinary characters, so the secret store kept them.

You can see it in zsh:

$ a='http://u:p@h:1'
$ echo "$a"
http://u:p@h:1

$ b=‘http://u:p@h:1’
$ echo "$b"
‘http://u:p@h:1’
Enter fullscreen mode Exit fullscreen mode

The stored value was never a URL. It was a URL wrapped in two extra characters.

Why the parser rejected it

Go's net/url gives different errors depending on where the stray character sits:

package main

import (
    "fmt"
    "net/url"
)

func main() {
    for _, c := range []string{
        "http://user:pw@proxy.example.com:33335",
        "‘http://user:pw@proxy.example.com:33335’",
        "'http://user:pw@proxy.example.com:33335'",
        "http://user:pw@proxy.example.com:33335'",
        "http://user:pw@proxy.example.com:33335\n",
        "user:pw@proxy.example.com:33335",
    } {
        u, err := url.Parse(c)
        switch {
        case err != nil:
            fmt.Printf("%q -> error: %v\n", c, err)
        case u.Host == "":
            fmt.Printf("%q -> no error, but empty host\n", c)
        default:
            fmt.Printf("%q -> ok, host %s\n", c, u.Host)
        }
    }
}
Enter fullscreen mode Exit fullscreen mode
Value Result
clean URL ok
wrapped in curly or straight quotes error: first path segment in URL cannot contain colon
trailing quote only error: invalid port ":33335'" after host
trailing newline error: invalid control character in URL
no http:// scheme no error, but the host comes back empty

If you only check err != nil, the last row slips through, so check the host too.

Where the curly quotes came from

The commands were kept in Apple Notes. Notes turns on Smart Quotes by default, and it converts quotes as you type, not when you paste.

Two lines in the same note can therefore differ:

Two lines ending in :33335. The first ends in a straight quote, the second in a right curly quote.

The first line was pasted into the note. The second was typed by hand. The closing quote on the second line is the right curly quote, U+2019, the same character we found at the end of the stored secret.

That also explains why only one of the two commands failed. The unlocker command was pasted and stayed straight. The datacenter command had been edited by hand in the note.

Why nobody could reproduce it from chat

The command was shared in a chat app, and chat apps tend to normalise quotes back to straight ones. Copying the command out of the message gave one that worked. When someone ran the original command again to show it was broken, it broke again, with the same bytes. Only the raw stored value showed the difference.

A short sequence of events:

  1. The command is edited in Notes. A quote is typed, so it's curly.
  2. It's run in a terminal. The shell keeps the curly quotes as data.
  3. The secret store saves a 96-character value.
  4. Pods restart and read the new value. Parsing fails, so one service crashes and two fall back.
  5. The command is shared over chat. The quotes are normalised, so nobody can reproduce it.

What to do

  • Turn off Smart Quotes in Notes (Edit → Substitutions → Smart Quotes), and in macOS under System Settings → Keyboard → Text Input. Better, don't keep secrets or commands in a rich-text app. Use a plain-text file.
  • Pass secrets on stdin so quoting stops mattering, and so the value stays out of shell history:
  printf '%s' 'http://user:pw@proxy.example.com:33335' | vault kv patch -mount=secret myapp/proxy PROXY_URL=-
Enter fullscreen mode Exit fullscreen mode
  • Read the value back after writing it and check the first and last characters.
  • Look for non-ASCII in a pasted command before you run it:
  pbpaste | grep -nP '[^\x00-\x7F]'
Enter fullscreen mode Exit fullscreen mode
  • Validate at startup. A bad proxy URL should fail with a message that names the property and the reason, not a generic invalid.
  • Give the people who write secrets the ability to read their history. We found the cause in minutes once we could see the versions. Before that we were guessing.

The takeaway

The shell, the secret store and the URL parser all did exactly what they were told. The only thing that changed was two characters that look almost the same as the ones you meant. If a value that "looks right" won't parse, look at its bytes.

Top comments (0)