Release pipelines: STOP DOING THIS - Use the Deploy Script Pattern instead

Do your release pipelines look like this? Stop over depending on your deployment tool. You're making life harder for yourself. Use the Deploy Script Pattern.

Deployment tools such as Azure DevOps and Github Actions allow us to create release pipelines for our applications quickly. It's a fantastic advantage over manual deployment. However, how most people write release pipelines slows development and reduces maintainability.

Over the last ten years, I've seen and written many release pipelines. The most common pattern I've seen is the heavy use of built-in tasks. Built-in tasks let us provide a few values and then run deployment actions without much effort from developers. On the surface, they are an excellent way to get going fast. However, cracks start to show as you progress through the development lifecycle.

The issues with built-in tasks

Issue 1: Changes are SLOW

Changes to a release pipeline have a slow feedback loop. For example, a typical change would require the following steps:

1. Edit the release pipeline development stage task in a YAML file or UI.
2. If the pipeline uses a YAML file, check it in and wait for the build to complete. 
3. Create a new release and deploy it to the development stage.
4. Wait for all tasks before the task that is modified runs.
5. Once the task runs, check the log output.
6. If successful, make the exact change to other stages. If not successful, repeat this process.

Depending on the project size, the time it takes between steps 2 to 6 generally takes five minutes to one hour. Developers are used to near real-time confirmation. An hour is far too long.

If your pipeline requires you to make changes for each environment (Dev, Test, UAT, Prod), you can multiply the time taken to make a simple change by four.

A small change that requires a few loops around the process may not be worth the effort. Bad practices will creep into your project to avoid the long feedback loop. Bug fixes and version updates will be ignored. No one will want to improve the release pipeline. Even if they did, it would cost too much to make improvements.

I have seen this issue cripple the development of release pipelines. Eventually, becoming the reason for moving to manual deployment.

We need to decrease the feedback loop time to seconds.

Issue 2: Releases are SLOW

At the beginning of every task in a release pipeline, the task's resources will be downloaded and initialised on the agent. This loading time can be from two seconds to one minute for some tasks. A release pipeline with several tasks can spend significant time in this task-loading state.

The fewer tasks, the better.

However, some deployment tools enable multiple tasks to run in parallel (asynchronously). Running tasks in a release pipeline is useful asynchronously. Tools like Terraform and Pulumi will run their actions asynchronously where they can. If you combine tasks into a single task to save task-loading time, ensure that you make the task perform actions asynchronously. Most scripting languages can be written asynchronously.

Issue 3: Vendor LOCK-IN

It shouldn't be surprising that using built-in tasks will lock you into the deployment tool you are using. Although this may not seem like an issue when creating the release pipeline, deployment tools have shifted significantly over the last ten years, and changes are still occurring. For instance, GitHub Actions, now a popular deployment tool, was only released to the public in 2019.

The solution - Deploy Script Pattern

Instead, we aim for the least number of tasks in our release pipelines. Only use tasks that set up the agent and then a single task to run a deployment script. Your deployment script should do everything except install software and compile code (compiling should be done by the build pipeline). It can run Terraform/Pulumi, deploy code to web apps, seed databases, deploy batch jobs, and execute smoke tests. Split the different actions into child scripts which a parent deployment script can invoke. Make use of script parameters to pass in environment variables.

 This pattern has several benefits:

• More control over release orchestration.
• Developers can make changes to scripts and test them instantly on their machines.
• Every environment calls the same script, keeping environments aligned.
• Your release pipelines are versioned in source control.
• Hand-written scripts are more flexible than built-in tasks.
• The deployment tool loads fewer tasks, speeding up your release.
• Moving to a different deployment tool requires less effort.

In a few weeks, I will follow up with an article showing an example of implementing the deploy script pattern using Azure DevOps/GitHub and PowerShell. I hope you come back to take a look.