Safely storing passwords and secrets in your git repo

Managing secrets for a code base is a tricky proposition. For too long, traditional DevOps approaches have seen teams build elaborate out-of-band workflows to manage system secrets, API keys and tokens, and traditional passwords. This tends to be an error-prone and painful process. This is never a good scenario to be faced with when considering options around security.

At the same time, storing these secrets inside your DVCS is also a really easy way to get onto the MyBroadband or IT Web front page with a “YourCompany: HACKED!” headline.

As an illustration, a quick GitRob spree on your local GitHub organisation can reveal some serious vulnerabilities that are literally just waiting there for someone to exploit. I ran this on an open source project whose name I will not disclose, and got some very interesting results that could lead to a lot of mischief.

Recently, some new software has hit the scene: BlackBox, a utility built by the smart people at StackExchange. Blackbox is a suite of scripts that aims to marry the ever-warring security and convenience into something that should satisfy both the development and security teams.

Blackbox uses GPG to encrypt files before they’re sent to your DVCS (assuming GitHub for the remainder of this article), and decrypts them when needed, either by you, your team, or an automated system. Blackbox is so promising, that in fact ThoughtWorks have identified it on their tools to assess for the January 2015 Tech Radar

There really is a lot to cover in terms of workflow, but for today we’ll keep it simple, and walk through the most typical steps of starting a blackboxed project, adding files, editing files, and adding collaborators.

Dependencies

The following instructions assume some basic groundwork has been laid down.

• You are working on a project hosted on GitHub or similar online source code management tool.
• You and your whole team have access to this project
• As all examples are run from the shell, you are comfortable running shell commands
  • If you’re copying and pasting the commands below from ZSH instead of BASH, please run this command first setopt interactivecomments
• You have blackbox installed as per one of these installation methods.
• You have GPG installed
• You have GPG keys on your machine (Zero One uses, and loves Keybase.io)
  • Use Keybase and install your keys via keybase login, or
  • Simply generate your own keys via the GPG command line. You can use this tutorial to get started

Prep your project for Blackbox

Start up your gpg-agent so your passphrased keys are stored in memory:

$ eval $(gpg-agent --daemon)

Let Blackbox initialize some housekeeping files:

$ blackbox_initialize

The next command is provided by blackbox, just copy and paste your version:

$ git commit -m'INITIALIZE BLACKBOX' keyrings .gitignore

Now add yourself as an admin

Blackbox requires admins to be managed via a simple text list file, and a keyring of GPG keys. The keyring is used so that all admins can have their own keys, and still decrypt files. As we’ve said earlier, we’re fans of Keybase, a crypto service that helps manage GPG transmissions. I’ll be encrypting files using my Keybase ID.

To see a list of GPG keys on your machine, and to get the UID you need to provide blackbox with, run:

$ gpg --list-keys

Choose your key, and give the UID to blackbox:

$ blackbox_addadmin keybase.io/gee_forr

Time to get some files blackboxed

Let’s get Blackbox to manage a couple of secretive files for us. To do this, we register a new file with a simple command. In this demo, I’ll be encrypting a Rails app’s secrets.yml and database.yml files. You can add as many, or as few files as you wish. Really paranoid people might want to consider even blackboxing the db/schema.rb file.

$ blackbox_register_new_file config/secrets.yml
$ blackbox_register_new_file config/database.yml
$ blackbox_register_new_file db/schema.rb

Blackbox does a couple of clever things here. Firstly, it adds the GPG version of the file to the git repo, and secondly, it adds the unencrypted file’s name to the repo’s .gitignore file so that you won’t accidentally add the decrypted file back into source control.

What about editing blackboxed files?

To edit a file, you have a number of options. You can decrypt, then edit, then encrypt, or you can do that all in one command.

$ blackbox_edit config/secrets.yml.gpg

This will prompt for your passphrase, decrypt the file, and open up $EDITOR. Writing and quitting the editing session will re-encrypt the file.

Whilst you’re editing secrets.yml, why not remove the production reference to an env var, and put the secret key right in there? Don’t forget to keep on committing changes as you go along.

If you want to have a file decrypted so that you can run a development environment, just run blackbox_edit_start on the file name. This will decrypt it and do nothing else. Running blackbox_edit_end on the file name will re-encrypt.

I work on a team, how do I get other devs access to these files?

This is a slightly finicky process. It involves all the prerequisites outlined above, and then implements a “vouchsafe” process to allow new people onto the project. Here’s another reason we love Keybase. Keybase provides a way for someone to verify another person using their social media logins. If you know someone’s Twitter or GitHub handles, you can find their Keybase fingerprint from there, and rest assured that the person you are “indoctrinating” (Blackbox’s term, not ours) is who they claim to be.

On your new colleagues machine, the following commands are run from within the project’s repo:

$ blackbox_addadmin keybase.io/ridhwana

Here, Ridhwana adds her Keybase UID. This outputs a git command suggestion for committing the files.

Push the application for the new admin back to the central git repo:

$ git push

Once the application for admin rights is back on a central git repo, a current admin can continue the indoctrination process on their machine.

Fetch the latest code with the new proposed admin:

$ git pull

Add the new public keyring to GPG

$ gpg --import keyrings/live/pubring.gpg

Now decrypt and re-encrypt all the files using the new keyring:

$ blackbox_update_all_files

And finally push all these changes back to your central git repo

$ git commit -a
$ git push

From this point on, both admins should be able to work with these files securely.

Other points to consider

What happens when someone leaves the team?

Standard practice in these cases, with or without a tool like Blackbox, should be to regenerate any keys or tokens, and change any passwords. Once that’s done, running the blackbox_removeadmin command will help to remove their key from the keyring, and re-encrypt all files.

What about automated systems that need to access these files?

Your CI server, or orchestration servers can have a GPG key without a passphrase added. This is done by creating a sub-key from another key. These servers should be locked down extremely tight. If someone breaks into a server like this, you’ve already lost, and fretting over a machine-use-only key will be the least of your worries.