I would like to discuss a practice that I recommend for CI/CD design: separation of CI and CD.

It always depends

One important not that this post must start with is that there is not 1 way to implement a CI/CD pipeline. There are a lot of tools and a lot of different ways to implement the the flow. That said I think it's important to know the common practices in the industry and to build our CI/CD based on the common practice. This help avoid common pitfalls

I see this recommendation as part of the modern practices of DevOps and CI/CD. This is definitely an advice that I would keep in mind when designing a DevOps workflow.

Decouple Build and Deploy

The recommendation in this post is to decouple CI (to simplify: the build part) and CD (the deployment part).

There are multiple ways to see it:

• CI and CD are not done in 1 continuous action
• The output of CI is kept as artifact and this aftifact will be used during the CD

In this setup CI and CD are not 1 continuous action, they are building blocks. This doesn't mean that we can't deploy automatically the CD building block can be triggered automatically after the CI building blocks. It means that you have the control on how and when to sequence them.

A pattern that is not decoupling CI and CD can be as follows:

• The code is stored in Git
• There is a branch for each environment
• Every change in a branch the code is built and deployed in the related environment

On the opposite a pattern that is decoupling CI and CD can be as follows:

• The code is stored in Git
• When a specific branch is changed the code is built and the result is stored in an artifact repository
• Each environment has a separate process that is pulling from the artifact repository and deploying in the environment

The benefits

There are a few benefits that I would like to highlight.

Decouple CD from your source control.

By nature the CI is more strongly coupled to source control. Services are built from the code. On the other hand CD is more coupled to the way your handle release. The idea is to have 1 more level of abstraction between your code and your production.

Source control is a very important aspects of DevOps. There are topics like monorepos, library management, branching strategies. You want to be able to experiment with those practices and modernize your source control practice without directly impacting your production environment.

Some applications may also come from different kind of sources that you don't control directly. It may be another vendor or an open source component.

This is also a discussion that will be important in the context of Microservices. One of the key aspect of Microservices are their ability to be deployed independently. Repositories for monolitic application, containers and cloud functions can be different.

Handle multiple environments

You will be often handling complex environments. You may have multiple environments, some environments may be connected to the Internet, some environments may be offline, the application source control may even be unreachable from the environment.

You may even have a number of environments that is more dynamic that you initially thought. For example because of a new request, a hotfix that need to be tested in a temporary environment.

You don't want to redesign your CI/CD for each change in the environment.

Security

There is an added benefit of using an artifact repository as intermediary: security.

An artifact can be scanned for vulnerabilities. An extra mesure that you can take is to make artifacts immuables. Once an artifact is built, if they don't change you can guarantee that a given version is correctly validated and and exact version of an artifact is deployed in an environment. It you build a service twice you can end up with 2 different services. The same source code will produce different build artifacts depending on the library used, the Docker base image used or other components that can be variable during the build phase.

GitOps

This can appear like a paradox but decoupling CI and CD is one of the key of GitOps.

GitOps practices states that each environment should be declared in a manifest, ideally in Git. So it means that the environments are directly inferred from the source control.

That said one important distinction here is that the deployment manifest is different from the application source that is used to build the artifacts.

• The application code is the code hat will be used to build the binary of the application
• The deployment manifest is a manifest that describe the required state for the system (including naming the services and the versions for each environment)
  GitOps is using the description of the required state but not the build instructions.

Simplicity(?)

Decoupling introduce a bigger initial effort in the CI CD design and potentially more steps. That said, once done a less coupled architecture and based on CI and CD building blocks can also generate a system that is more simple to reason about. Less coupling tend also to simplify the workflow.

Advanced release

Release management is becoming more complex. CI and CD have been the norm for some time but there are discussion of Continuous Release. This can include simple release policies or more advanced patterns like blue/green deployment and A/B testing.

The drawback: the need of an artifact repository

The consequence of this decoupling is that a there is the need for an artifact repository. This introduce a necessary additional component in the system.

This will then depends on the technology used. If you are using containers artifact repository is pretty much a must (for image resgisty) and GitOps is becoming a standard. If you are using other technologies, there can still be a lot of benefit to this approach. The artifact repository can be a dedicated software or even a cloud storage bucket (AWS S3, Azure Storage Account, AlibabaCloud OSS,...)

Final notes....

The goal of this post is to share what are in my opinion some key aspects of modern CI CD:

• Both CI and CD are evolving and new pattern are emerging. You may want to decouple them to benefit from the best practices of each of them
• When designing a workflow, it's important to not only look at the requirements but also to check what are best (or common) pattern in the industry.
  Decoupling is definitely one of my go-to recommendations. This will help you scale your DevOps process. To do so you should approach the challenge as designing building blocks for the DevOps process.