Securely Logging & Tracing HTTP Requests in Go

Securely Logging & Tracing HTTP Requests in Go

Table of Contents

  1. Introduction
  2. Why not PGP?
  3. Why not cryptopasta?
  4. NaCL
  5. Implementation
  6. Conclusion

 

1 | Introduction

I was recently debugging a nasty issue in one of our backend services and needed to view the exact HTTP request & response being sent to an authentication server. Fortunately, Go’s standard library provides http.RoundTripper, httputil.DumpRequestOut & httputil.DumpResponse, which are great for dumping the exact out-bound request & the response. But since an authentication request contains credentials and a response contains a security token, it would have been insecure to record credentials & tokens in our logging systems. How could I securely exfiltrate the information I needed, while maintaining security and not requiring a whole lot of changes to my codebase or deployment environment?

My first thought was encryption: after all, that’s the right way to ensure confidentiality of information. I definitely didn’t want to hand-roll my own crypto code, so I looked for a way to do it out of the box. I also wanted it to be pretty simple code: a single line (or three, with error-handling) at each call site, without any initialization. In a production system this would be less important, but in debugging code simplicity of removal is almost as important as simplicity of implementation.

2 | Why not PGP?

Go does have an OpenPGP library, but it is extremely complex; there doesn’t appear to be a one-line way to use the functionality it exposes. And PGP keys tend to be long, not amenable to drop-in use for debugging purposes.

3 | Why not cryptopasta?

cryptopasta provides some nice ready-rolled options which don’t require anything more than the standard library. Unfortunately, it only provides secret-key encryption1, and that’s bad — it’d mean that I’d have to put the secret key into the deployment configuration or, even worse, hard-code it. Neither of those options was really secure, and security was the whole point!

It’d be much better to use public-key encryption for this kind of thing: there’s no security risk configuring deployments with a public key, or even (when debugging) hard-coding a public key.

4 | NaCL

Fortunately, the Go project has an experimental crypto package which contains an implementation of the well-regarded NaCL library’s protocol, and it does have a public-key box API. Encrypting (‘sealing’) a message is as simple as

encryptedMessage = box.Seal(nil, message, nonce, recipientPublicKey, myPrivateKey)

Except … what’s that myPrivateKey at the end there? It turns out that NaCL’s API assumes that one wants to sign as well as encrypt messages. Well, I didn’t need it for our use case, so I decided to just generate a new ephemeral keypair for each message, and prepend the ephemeral public key to the encrypted message itself.

And what about the nonce? Well, according to the docs the nonce must be unique for each sender-recipient pair. Since I’m generating a new ephemeral sender key for each message, I could just use the same nonce over and over, but I decided to generate a random nonce for each message, prepending it as well.

My final encryption function looks like this:

And it can be used like this:

Decryption is very simple, too:

As you might imagine, DecryptWithPrivKey looks a lot like EncryptToPubKey:

5 | Implementation

Armed with the above pico-library2, I only needed a very little bit of glue code to get this working with the rest of our system. I implemented a custom http.RoundTripper which wrapped the one in the http.Client my code was using. First it printed out the encrypted request, then it made the request using the wrapped http.RoundTripper, the it printed out the encrypted response, and finally it returned the request as normal.

Here’s how simple it was:

To generate a key, I just called nacl.GenerateKey and printed the results, encoded in Base64.

6 | Conclusion

This is far from a complete crypto-system, and indeed production code would need quite a bit more. But for debugging purposes, this did the trick: it preserved the confidentiality of authentication information and it was easy to use; I was able to decrypt the requests & responses locally and do the debugging I needed, without exposing plaintext credentials in logs or traces. Most importantly, I didn’t roll my own crypto, but rather relied on time-tested protocols & libraries.

Footnotes:

  1. It also provides public-key signatures — but no public-key encryption.
  2. A pico-library is like a micro-library, but even smaller!