How I blundered with a private key and newlines

A short story about wasting a day on a single newline at the end of a private key.

I was trying to set up a CI/CD pipeline with GitHub Actions, more specifically — configure a deployment via SSH. Generally, to connect to a server with SSH one needs a key pair: the public key goes to the server’s .ssh/authorized_keys and the private key is used to connect to the server.

So, I’ve generated a key pair with a command like ssh-keygen -b 4096.

I’ve added the public key to the target server and tried to connect with a new key — everything was working on my machine.

Then I had to put the private key somewhere on GitHub so I could use it in a workflow. GitHub doesn’t support files as secrets, only strings. So, I’ve copied my private key and pasted it as a secret.

Like that:

Note that cursor at the end of the last line of the secret.

Then I used some black magic to save this secret to a file:

- run: printf %s "$DEPLOYMENT_KEY" > ~/.ssh/deployment_key
  shell: bash
  env:
    DEPLOYMENT_KEY: ${{ secrets.DEPLOYMENT_KEY }}
- run: chmod 600 ~/.ssh/deployment_key

…spending like an hour learning the importance of double quotes…

Finally, I got my key in ~/.ssh/deployment_key and it looked fine. To check that, I’ve stored it as an artifact, downloaded that artifact at the end of the workflow, and checked the key manually.

But when I tried to use that key with Ansible to connect to a server I got a really strange error message:

fatal: [42.42.42.42]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Warning: Permanently added '42.42.42.42' (ECDSA) to the list of known hosts.\r\nLoad key \"/home/runner/.ssh/deployment_key\": invalid format\r\nubuntu@42.42.42.42: Permission denied (publickey).", "unreachable": true}

I was really surprised to see that "invalid format" and "permission denied". And the shit hit the fan.

I’ve increased Ansible’s logging to the most verbose level, but it wasn’t informative, just more text with the same message at the end.

I’ve googled those errors and double-checked the number of hyphens in "BEGIN OPENSSH PRIVATE KEY" line.

I’ve generated a new key a few times.

I’ve tried RSA, ECDSA, and ED25519 algorithms.

I’ve ensured the permissions on the key are 0600, and even tried 0400.

No luck.

At the very bottom of my despair, I ended up reading an article about PEM file format on Wikipedia, trying to find the answer. From there I’ve jumped to the RFC 7468 and tried to understand its BNF description.

I’ve checked the RFC 1421 and it mentioned CRLF at the end of the -----END BLA-BLA-BLA----- line.

What a surprise!

I’ve updated the secret on GitHub one last time, making sure that there is a newline and the end of it.

And it fixed the error, the deployment proceeded. 😌