Secrets management

Secrets management is the practise of handling sensitive information such as API keys, passwords, and tokens in a secure and efficient manner. This involves storing, accessing, and distributing these secrets in a way that minimises the risk of unauthorised access and data breaches.

Effective secrets management ensures that secrets are encrypted both at rest and in transit, access is controlled and monitored, and secrets are rotated regularly to reduce the risk of exposure. By implementing robust secrets management practices, organisations can protect their critical assets. This article will focus mostly on secrets management in the context of software development and source code security. For a more in-depth discussion on Secrets Management, you can check out the OWASP Secrets Management Cheat Sheet.

What are secrets?

In the context of software development, secrets refer to sensitive information that should be kept confidential and hidden from others. These can include:

API keys: Used to authenticate and authorise access to APIs.
Passwords: For user/service accounts, databases and many other systems.
Connection strings: Such as databases and storage accounts.
Private keys: For cryptographic operations such as SSH keys or TLS certificates
Tokens: Such as OAuth for authentication.

Such information is considered valuable and needs to be protected from unauthorised access or use. When these secrets are inadvertently committed to git repositories, they can pose a significant security risk if exposed.

Why is secrets management important?

In GitHub’s 2024 Octoverse report, 39 million secret leaks were detected in GitHub repositories. According to the IBM Cost of Data Breach Report (2024), breaches involving compromised credentials cost organisations an average of $4.88 million per incident. This is a 10% increase from the previous year.

Good secrets management is important as it helps organisations prevent security threats that may arise through exposed credentials. If exposed, secrets can grant adversaries unauthorised access to an organisation’s code base, databases and other sensitive data. This is of particular concern in public repositories where attackers can, and do, regularly scan for leaked secrets. Managing secrets securely can help protect organisations in multiple ways, including:

Preventing data breaches: Exposed secrets can grant attackers unauthorised access to critical business systems. These business systems may include personally identifiable information (PII) such as medical information, identification numbers and address details. Managing secrets correctly can help prevent them from being leaked and exploited by attackers.
Protecting reputations: Data breaches and other security incidents can have a significant impact on an organisation’s reputation. This kind of damage can lead to a loss of confidence in the company, harm to its brand, and financial losses.
Reducing costs: Data breaches can result in significant costs and sometimes fines if adequate measures were not taken to protect the data. The later in the software development lifecycle the secrets are discovered, the more expensive it can be to remediate.

Several high-profile security incidents have occurred due to secrets leaked in git repositories:

Uber data breach (2016): Attackers were able to gain access to a private GitHub account of an Uber developer. Within the source code stored in the GitHub account was plain text login credentials to one of Uber’s AWS servers. From this, the attackers managed to access the personal information of 57 million users.
Stack Overflow breach (2019): A hacker was able to use secrets stored in cleartext in git repositories to escalate their privileges and access PII of 184 users. The subsequent investigation identified bad secrets hygiene as one of the key shortcomings.
LastPass breach (2022): Attackers were able to use cleartext embedded credentials stored in source code repositories to move undetected through the system and escalate their privileges, accessing sensitive information.

Manging secrets securely can help prevent breaches such as the above, protecting the organisation from reputational and financial loss. By managing secrets securely, the risks of unauthorised access can be mitigated.

Best practises for managing secrets

Never store unencrypted secrets in git repositories

Unfortunately, it’s common for users to hardcode credentials in plaintext within their code. This is usually done under the false assumption that because the repository is private, it’s safe to store them there. This is not true. Private repositories are not appropriate places to store secrets. There are multiple reasons for this:

Version control systems (particularly git) are designed to be duplicated and distributed across multiple locations. Therefore, you never know where the source code could end up. It could be cloned/forked to a compromised workstation, intentionally or unintentionally made public, or pasted in collaboration tools such as Confluence, Slack or Teams.
Private repositories typically contain the source code for more sensitive, business critical systems. A leaked secret for one of these systems could have a significant impact on the business.
Research by GitGuardian identified that 4.6% of public repositories contain a secret. This increases to 35% for private repositories. This makes private repositories a common target for attackers.

If a secret is pushed to a remote repository, public or private, it should be considered compromised and rotated asap.

There’s a misconception that removing the secret from the git history will resolve the issue. This sounds plausible in theory but in practise it’s much more difficult. Rewriting the git history requires careful coordination with all collaborators to successful execute, and there’s numerous side effects that must be managed. Of particular concern is the fact that all commit hashes since the secret was introduced will change. This can affect any tooling or automation that relies on these commit hashes. You can view a full list of the side effects at Side effects of rewriting history.

In summary, rewriting the git history is difficult. It’s usually easier to rotate the secret and remove it from the code base, leaving the old secret in the git history.

Use gitignore for sensitive files

Developers will typically use a .env file (or other configuration file) to store variables and secrets they require for the project to run locally. These files should be excluded from commits by adding them to a gitignore file. GitHub have a collection of useful gitignore templates you can use to get started.

Use GitHub Actions secrets for CI/CD

If secrets are required as part of a CI/CD pipeline, you can use GitHub Actions secrets to store them securely. They can be associated with a repository, environment or organisation. Typically, you’d want to store them at the repository or environment level. You can then use them within your workflow file as an input or environment variable.

steps:
  - name: My action
    with: # Set the secret as an input
      my_secret: $
    env: # Or as an environment variable
      my_secret: $

Use automated secrets scanning tools

There are numerous open-source secret scanning tools you can use to identify leaked secrets in source code. These tools typically search for patterns and keywords associated with known secret types using regular expressions. These tools can be used to:

Scan files locally using the CLI to identify secrets.
Scan the whole git repository locally, including git history, using the CLI to identify current and historical secrets.
Integrate with pre-commit hooks to block commits that contain secrets.
Scan remote git repositories for secrets. Secrets identified in remote repositories should be considered compromised and need to be rotated.

As a best practise, secrets scanning should be implemented as part of a pre-commit hook to block commits that contain secrets. Using pre-commit hooks is preferred as it prevents secrets being committed and pushed to the remote repository where it’s harder to remediate them. Once pushed to the remote repository, the secret needs to be rotated which can require additional resources.

If you have a public repository hosted on GitHub, you can use GitHub secret scanning for free to scan your repository for leaked secrets. This also includes push protection to prevent secrets being pushed to the remote repository.

If you have a private or internal repository, you can use trufflehog.