In software development, the most important topic these days is security. Security of all kinds like security of the system, security of data, pipelines, or application secrets. Today, we will talk about one such issue that we, as developers or engineers, face in day-to-day business, i.e., security of the secrets like ClientID, ApplicationID, etc. These secrets are usually very critical to the system and can affect the overall integrity of it or be a threat to software security if it gets leaked.
Now, we all know the best practice to secure the secrets is to use KeyVaults. Whether that will be Azure or AWS or GCP, every cloud has its own KeyVault. Even the mobile devices we use, such as iPhone or Android, have KeyVaults where all the secrets are saved and can be saved to. By secrets I mean your credentials or passwords, not the deep dark ones, so don’t be alarmed ;). To prevent unwanted access to these secrets and their values, these KeyVaults are access-restricted. So far so good, but that also does not ensure with 100% guarantee that these secrets are safe. Now, what to do about that?
To add security and to be fully secure, these secret values can be rolled, means that they can be changed from time to time and replaced with new ones. Now you know the reason why you are asked to change your password so often ;). This time to time can be every day, every week, every two weeks, every month, every quarter, and what not. Now, imagine you are asked to change these secrets at the desired schedule, but you forget to do one of them. Or you forget it entirely because of other priorities. Or you change the one that was not supposed to be changed instead of the ones that were supposed to be changed. What to do now? So much for human errors, which can happen to anyone. Let me remind you once again that we are talking about the secrets for your software application that are deployed on cloud systems like AWS, Azure, etc.
I know that the answer to this question is to automate it. But how exactly do you want to do this? Re-deploy the complete KeyVault? Doesn’t that scare you to re-deploy? If yes, then what to do next? Simply just update the key that you want to update with a new secret value. But how to do that in a safe manner to ensure nothing is hampered.
Further in this post, I will talk about the approach that I adopted, and you can use it too, to automate the rolling of one secret or many in a safe environment. Firstly, I have provided an initial flow in Fig. 1. I created a pipeline to automate the secret. The steps would be as follows:
- Create a pipeline with a trigger or cron job. The trigger can be used to trigger the pipeline in special cases like pull requests, and the cron job can be used to schedule it, e.g., one month, three months, etc. You can choose either or both ways, depending on what you are doing.
- Create a nested task in the pipeline to call a script. You can directly call the script from the pipeline, but a nested task will help you provide environment variables to access the KeyVault if you have some special security feature. I like to make environment execution independent of the script, so in the pipeline, I provide an environment and in the nested task, I provide variables to access that environment and run the script.
- Next, a script that will do that rolling for you based on the conditions. The conditions can be that the key has expired or if there is a soft delete state or if there is an old key there, etc.
- In the script, you can implement the business logic of generating the secret values like a combination of random hexadecimal characters with the first 10 as numbers or the last 5 as numbers, etc. This business logic generates the key for you in the desired format that your application can accept.
In Fig. 1, you can see that the script accesses the KeyVault and not the pipeline directly. This ensures there is no leak from the pipeline and only the script, a PowerShell script, is the one responsible for creating the connection to the KeyVault and then closing the connection post execution. In my case, I used az commands from Azure CLI to access the KeyVault and generate new secrets.
Another two cents to take from this would be to not just delete the secret but purge it as well. Depending on how you implement your script, if you are required to delete the secret, do not forget to purge the key before adding it again. What you can do is just add a new secret value and not delete the key to prevent the script from the need to purge the keys in the KeyVault as it is not good to purge the keys time and again, and by deleting it entirely, you will lose the old keys as well.
For reference, a simple flow for the secret can be as follows:
- Start with checking if the key exists.
- If it does, check for the datetime of the schedule to which it should be rolled.
- If it is due to delete, delete it and generate a new one.
- Use the last version of the secret or the new secret in your application and that’s it.
- The schedule will automatically trigger the pipeline time and again or it can be run manually as per convenience.
And that’s it, you have successfully automated rolling of the keys from the KeyVault in a secure manner, with minimal downtime, as the new key will be available for use as soon as the pipeline finished successfully.
This approach is based on Azure but can be adapted to other clouds based on the DevOps process your team follows.