GitHub Actions Shared Library

From Jenkins to GitHub Actions: the beginnings

For a while now, the Vox Media Development Experience team, formerly known as the Systems Engineering team, has relied on Jenkins to perform CI and CD tasks. It has proven to be a decent tool thanks to its customizable platform and low costs. However, we’ve run into bumps over the years:

• Regular breakage of agents requiring manual cleanup, including workspaces not getting cleaned up as expected in between jobs and resulting in, for example, duplicated docker networks that prevent jobs from running.
• While it is technically possible to have a Jenkins cluster with HA, it isn’t possible to ensure high availability on our Kubernetes cluster by increasing the number of replicas. A single master instance cannot be duplicated as there is no clear way to have multiple instances share states.
• a UI that can be difficult to parse with a logging view that is dense and confusing to read. The Blue Ocean view looks more modern but it is buggy (e.g. broken replay stage button, broken logs table), and it lacks some important features from the original UI including a way to add parameters to jobs.
• Jenkins uses its own version of a scripting language called Groovy which makes it difficult for teams to take ownership of their pipelines since they’re not necessarily familiar with it.

Since these issues were affecting engineers’ productivity, we’ve had to find some alternatives and explored three different tools: Circle CI, Semaphore, and GitHub Actions.

After some extensive research, our team chose GitHub Actions. It allows us to keep a level of customization similar to that provided by Jenkins, it is straightforward to set up and being included with GitHub means that it could empower engineers to work on their CI/CD pipelines. Finally, the docs and UX are great so far, and frankly, it is simply new and exciting!

Building the GitHub Actions Shared Library

Our Jenkins architecture uses a private shared library with scripts that let us deploy to Kubernetes, create database snapshots, and more. We first considered turning these scripts into individual Actions, but GitHub doesn’t support private shared actions yet. Instead, we opted to house all scripts under one private repo, appropriately named the GitHub Actions Shared Library (GASL).

We chose Bash as the initial official language because the whole team is familiar with it and it’s easier to get started as it doesn’t come with any boilerplate code. Ease of development mattered a lot since we were eager to get a functional release of GASL within a few months.

Given that the generic scripts provided by the Jenkins shared library, such as a helm linter, are available in the GitHub Actions marketplace, we only needed to port custom scripts from it. We also had to add some scripts to prepare GitHub runners’ environments for each workflow run; these included authenticating our GASL service account with ECR, setting up docker to work with our registry, and installing a docker container to allow the install of private dependencies inside containers by injecting the right SSH key.

Things looked simple enough but we lacked a testing infrastructure and a general directory structure. More importantly, in my opinion, writing complex custom scripts in Bash was going to be hellish. For example, our Docker images are environment specific and are tagged with a particular base64 encoding scheme that utilizes the git SHA. Writing this in Bash might’ve been possible, but at what cost? We were headed down a bad path and had to pivot.

Luckily, Github provides TypeScript Actions templates. It made more sense to get rid of the testing and structural gaps, and continue iterating on GASL with a framework specifically made for Actions. Getting to work with a new language was a welcomed bonus!

Using GASL

To use the shared library, repos must add the `actions/checkout` step to their workflow:

The `ref` parameter is particularly helpful because it lets us specify the desired GASL version. We use git tagging to create/update releases and pin versions to releases. It also makes updated releases automatically available to all the repos using it.

After checking out the Bash shared library, a repo owner would’ve had to manually add a series of secrets in order for the authenticating scripts to work. This is no longer the case since adding a `push-secrets` action in the TypeScript shared library. This action pushes the necessary secrets to a list of repos within our organization and makes using the GASL even easier. A repo owner only needs to add their application to the list in order to get the secrets, et voila! They’re ready to use the shared library.

The road ahead

GASL is still under development and it is looking better everyday. We’ve added several actions after the secrets pusher. They include:

• a linter that ensures repos have a proper container setup. Amongst other things, it checks that Dockerfiles are using images from the right registry or that we’re always pointing to specific image tags versions instead of the umbrella `latest`.
• a tool that creates GASL releases upon merges to master. It avoids having to create tags manually.
• the helm setup needed to access our private charts.
• and more!

Before we can invite other teams to our big GASL reveal party, we need to finish building a tool that will post real time job updates to Slack, and do more testing within our team. We’re getting close, though! Stay tuned...

---

---

Special thanks to Casey Kolderup, Ben Lee, Jason Ormand, and Peter Salvatore for thoughtful edits!