Simplifying Serverless Secrets

By JP Robinson

Illustration by Jon Han

Over the past year, several New York Times engineering teams have been busy transitioning systems to run on either Google App Engine or Google Kubernetes Engine. We use these platforms to host systems like our games, email and core news applications. As we’ve transitioned to these technologies, we’ve also started adopting HashiCorp’s Vault as a unified solution to securely store, share and access application secrets like database passwords and API keys.

Our applications running on Google’s Kubernetes Engine use the sidecar container pattern to interact with Vault. Instead of requiring developers to add additional code to get secrets from Vault, the sidecar logs into the service, fetches the configured secrets and writes the secrets onto a dedicated and secure shared memory volume for the application to read on startup.

Unfortunately, applications on Google’s serverless solutions, which includes the App Engine Standard and Flexible environments, don’t have the option of configuring sidecar containers or shared volumes. This has forced us to fall back to injecting our secrets as environment variables at deployment time. A recent change to the App Engine console has enabled users to see these secrets in plain text, which is not ideal. We needed a more secure way of managing serverless secrets and we needed to unify our secrets solutions across different styles of infrastructure.

Today, we’re happy to introduce gcp-vault, a new Go library that eases the use of Vault in the Google Cloud’s serverless solutions by reducing a four step interaction with Vault and GCP to a single function call.

Both gcp-vault and our sidecar solution rely on a Vault plugin that allows applications to authenticate against Vault using Google credentials. Once the plugin is added to a Vault installation, developers must give Vault access to their GCP project so it has the ability to verify Google-signed JSON web tokens (JWT). After this configuration is completed, an application must look up its default credentials, sign a JWT using Google’s IAM service and then login to Vault using that JWT. Our new library reduces all of that application-side work to a single GetSecrets() function.

Beyond simplifying the interaction required for accessing Vault from GCP, the library also includes a method for logging into Vault while developing locally and a small utility package to help mock out Vault and Google’s IAM and metadata services in unit tests.

To safely initialize secrets on application startup, we recommend users of App Engine’s legacy Standard Environment call the gcpvault.GetSecrets function within a sync.Once block and hook it into their application’s middleware. The process of logging into Vault is somewhat expensive and can take upwards of one second. To deal with this, we also recommend using Warmup or Startup requests to ensure the secrets are fetched before exposing the service to clients.

Users of the App Engine Flexible Environment and the new Go 1.11 beta runtime in the Standard Environment don’t have the same restrictions on network access as App Engine’s legacy Standard Environment (Go ≤1.9) so users can make the GetSecrets call on application startup inside their main() or init() functions.

We’ve included more information and a handful of examples of using the tool in our GitHub repository. Here are some links to help developers get started:

We’re open sourcing this project not only to aid the community, but also to ask the community to help make gcp-vault better. If you are interested in helping out, please feel free to contribute or reach out on the #gcp-vault channel in the Gopher Slack Community if you have any questions.