This article showcases a successful, automated approach to a common, complex CI/CD migration challenge. It provides valuable insights into leveraging existing tools and AI to reduce manual effort and accelerate infrastructure shifts, directly impacting developer productivity and system reliability.
The past few months have been exciting times for Slack’s CI infrastructure. After years of developer frustration with Jenkins (everything from security issues to downtime to generally poor UX) internal pressure led us to move a majority of Slack’s CI jobs from Jenkins to GitHub Actions.
My intern project at Slack this summer involved creating a conversion tool that could automatically migrate Jenkins pipelines to GitHub Actions, saving developers time and accelerating the migration. The project was successful, and is expected to cut the migration time by half and save over 1,300 hours. This blog is focused on that conversion tool, and the 7 week journey to design, implement, and improve it.
Before we start though, you should know some terminology:
The success criteria for the project are as follows:
After doing some research, we identified a few useful tools to use to convert from Jenkins to GHA:
Over the span of a week, we ran the Importer on several of our Jenkins instances. The tool was simple. You install it from the GitHub CLI, configure it, and run a command like this:
gh actions-importer audit jenkins --output-dir <output-dir>The Importer would then output GitHub Actions workflow files for each Jenkins pipeline on the Jenkins instance being audited. Upon running the Importer, we were curious as to how well it performed. Fortunately, the Importer creates an audit summary upon completing its audit of a Jenkins instance. The audit file contains information about how many Jenkins jobs were fully converted, partially converted, and failed to convert from Jenkins to GHA.
About 50% of the time the Importer moved the pipeline from Jenkins to GHA without error. 5% of the time the importer failed completely, and the rest of the jobs were partially-imported — it created a workflow YAML file, but wasn’t able to populate it fully.
In addition to the success and failure rates, we were curious as to exactly what Jenkins build steps and environments were unsupported by the Importer. Fortunately the audit summaries contain information about this as well. Glancing at the audit summary, it was initially concerning to see that there were hundreds of Jenkins pipelines that weren’t able to fully convert to GitHub Actions. A deeper inspection revealed a pleasant surprise – only eight unsupported Jenkins build steps and environments were responsible for over 90% of those failures.
This was good news. Instead of needing to spread ourselves thin looking at dozens of unsupported Jenkins steps, we instead only had to look at eight. For those key few unsupported items, we’d need to look at the following:
At this point in the project, it was known that the Importer’s workflows had a lot of room for improvement. Around half of the pipelines didn’t fully convert, so a big objective was to look into the unsupported Jenkins steps that caused them to not fully convert. Another task though was to investigate the pipelines that fully converted. Can we really trust the Importer when it claims that it could fully convert something? So really every pipeline that got (or didn’t get) converted to a workflow had to be looked at.
After looking through several dozen of the YAML files the Importer made, it was clear that there were 4 main types of corrections we’d need to make to the workflow YAML files:
Over the course of 2 weeks, we studied each of these:
RoI: Very High
By default, if you use an arbitrary action from the GitHub Actions marketplace, like say actions/github-script, any requests made by the action will hit the github.com API. There are rate limits for this. Actions that hit our self-hosted GitHub Enterprise instance have far higher rate limits however. So a task we had to do was replace rate-limited actions with internal mirrors of those actions that we have on our GitHub Enterprise instance.
Correcting rate-limited actions is simple to do on our end, and prevents end users from dealing with the headache that is having jobs fail from rate limits. So high RoI here.
RoI: High
There were a few instances where, to perform a certain task, the Importer would use one action whereas we’d prefer if it had used another. For instance, for sending Slack messages the Importer often elected to use the rtCamp/action-slack-notify action. Naturally, Slack has internal actions for sending Slack messages, and we’d rather use internal libraries for this purpose. So something to look into was replacing rtCamp/action-slack-notify with Slack’s internal messaging action.
This type of task seemed suitable for an LLM to do, since two actions often have disparate syntax which is hard to convert between by just using Python’s string methods. Preliminary testing also showed perfect LLM performance for this task which was reassuring.
Replacing one action with a different action would require end users to know both actions well. So automating this saves quite a bit of time and effort on their end, and it’s not that hard to write LLM prompts for these tasks. The RoI is thus high.
RoI: Medium
Some action items in the workflow files could only be done manually. For example, the syntax used in the workflow files to access secrets assumed that the secrets would be stored on GitHub. However, we don’t store our secrets on GitHub, and it would be time prohibitive and against our security policies to move them to GitHub. It would instead be better for end users to edit the workflow files such that secrets get read from where we store them instead of from GitHub. This couldn’t really be automated on the conversion tool side, since only the end users know where their secrets are and how to access them. All the conversion tool could do in such situations was leave a comment like: “Change these lines so that secrets are being read from where we store them instead of from GitHub. Go to a certain slack channel to find relevant docs about this.”
It’s definitely useful for end users to know where they should look to fix problems, but they still have to do the actual work of fixing the problem. So overall, medium RoI.
RoI: Low-medium
Recall that around 45% of pipelines only partially converted. What this means is that the Importer couldn’t find a GHA equivalent for a Jenkins environment or build step. When this happened, the Importer left a comment in the workflow that looked something like “X Jenkins item was unsupported.”
As it turns out, the reason why a lot of Jenkins pipelines were unsupported is because that piece of functionality in Jenkins was either already included or not necessary in GHA. So most of those “X was unsupported” comments weren’t too important and would probably be better off removed.
The benefits here to end users was small, as all it really does is declutter the workflows. So overall this had a low return. However, a script to delete strings is also very easy to write. So overall low-medium RoI.
The last part of the implementation is the corrections tool, which would correct the Importer’s workflows. We knew from the start that the tool would have a balance of using Python’s string methods and using LLMs. From the last section, note that out of the 4 categories of action items for the correction tool, the only one that needed AI was “replacing actions with other actions”. So the corrections tool was largely non-AI. It took around 3 weeks to make the corrections tool.
Here’s the bird’s-eye view. The input to the corrections tool was the path to a directory containing workflow yaml files. The following chain of events then happened:
The end result is that every yaml file in the directory inputted to the corrections tool was edited in place.
As mentioned previously, a large part of what needed to happen to the Importer’s yaml files boiled down to “add a comment after a line containing a certain string” or “replace a string with another string”. Python’s String.replace() method was used extensively here.
Prompt engineering is both science and art. It’s science, because there’s a formula that’s useful for doing prompt engineering:
Prompt engineering is also art, because experimenting with the prompt and seeing how you feel about the AI’s output is a key part too. There were a few rounds of this process:
An example of a prompt we’d end up at:
I’m going to provide you with a GitHub Actions workflow yaml file. In the file an action called rtCamp/action-slack-notify@v2.2.1 is being used. Your job is to replace every rtCamp/action-slack-notify@v2.2.1 action with the slack/message-action@v1.0 action, and output the resulting workflow yaml file. For reference, I will provide you with the syntax for the slack/message-action@v1.0 action. I’ll put it between <syntax></syntax> XML tags:
<syntax>
- uses: slack/message-action@v1.0
with:
channel: channel-name
text: text-message
</syntax>Now, perform the following steps:
Also, there are a few things you should absolutely never do:
I will now provide you with the workflow file you will be making edits to between <workflow></workflow> XML tags. When providing your output, only provide the contents of a YAML file. Do not describe what you did. Do not provide any justifications for your decisions. Only provide the corrected GitHub Actions yaml file, with nothing else before or after the yaml file in your output. Now, please perform the instructions I gave you:
We examined about 2 dozen of the LLM’s outputs and noticed something stunning — 100% accuracy in performing the requested tasks. A big concern with asking LLMs to do these corrections were hallucinations and other unexpected side effects, so seeing the LLM’s performance was reassuring. Overall, the LLM ended up being really useful for the migration. Even though prompt engineering isn’t trivial, it doesn’t take that long either, and after doing it the LLM could perform as well as any human for these “replace A with B” tasks. LLMs filled a crucial gap for fixing those errors that were too hard to fix with Python’s string methods.
First, consider the process of converting a pipeline from Jenkins to GHA manually:
Note how the steps are in descending order of difficulty and time cost.
The conversion tool lets end users skip step 1 entirely and have a much easier time with step 2. End users are given workflow files that have few, if any, flaws. They can skip straight to debugging, which will be accelerated since the workflows they get are highly accurate.
There are about 242 pipelines that this tool attempted to convert from Jenkins to GitHub Actions. The amount of time it would take a developer to manually convert one Jenkins pipeline to GHA varies. Here are some estimates for averages based on the type of developer:
Assuming 10% of end users are type 1, 40% are type 2 and 50% are type 3, it gives a total amount of time of about 1700 hours to convert all pipelines to GHA
Suppose that with this tool, half of pipelines run with no edits necessary (this is realistic, since half of pipelines are fully converted), and that the other half may take an average of 3 hours to debug. It would then take about 360 hours to convert all pipelines to GHA.
This tool is thus projected to save over 1,300 hours, 80% of the time it would take to move every Jenkins pipeline to GHA.
At time of writing, two weeks have passed since the outputs from the conversion tool have been published, and end-user feedback is now starting to arrive. It’s clear that developers at Slack have found the tool useful, but I’ve also heard that there are a few errors in the generated workflows. Even from the very start of my internship, I expected this to happen, but was worried that it might be hard to triage those incidents.
As it turns out, there wasn’t a problem. I made a Canvas doc for this conversion, and whenever end-users of the tool encountered an issue I updated the canvas with details about the problem and its solution. As more users used the tool, I got more feedback, and the canvas became more comprehensive.
If I were to pick the best thing about Slack, both the app and the company, it would be the ease of communication and collaboration. It’s easy to let people know things, to find out who to go to for help, to meet people, to take notes and keep track of info. All these things have made my internship experience more fun, pleasant, and educational than it would’ve been otherwise. And, if you join us, it’d make your time here great too 
.
Thank you everyone who contributed your knowledge and helped this project get to where it is:
Jerry Shen
Buddhadev Veeramallu
Ellen Wong
Shane Gearon
Sergii Gorbachov
Catherine Li
Nick Matute
Now more than ever, Slack needs GHA specialists. Does that sound like you? If so, we’re hiring 
!
Continue reading on the original blog to support the author
Read full articleThis article demonstrates how applying core software engineering principles like caching and parallelization to build systems can drastically improve developer experience and delivery speed, transforming slow pipelines into agile ones.
This article demonstrates a practical approach to enhancing configuration management safety and reliability in large-scale cloud environments. Engineers can learn how to reduce deployment risks and improve system resilience through environment segmentation and phased rollouts.
This article details Slack's successful Deploy Safety Program, which drastically cut customer impact from deployments. It provides a practical framework for improving reliability, incident response, and development velocity in complex, distributed systems.
This article demonstrates a practical approach to significantly improve CI/CD pipeline efficiency and developer experience. By intelligently caching and reusing build artifacts, engineering teams can drastically reduce build times and infrastructure costs.