Accelerating CI with AWS CodeBuild: Parallel take a look at execution now obtainable

I’m excited to announce that AWS CodeBuild now helps parallel take a look at execution, so you may run your take a look at suites concurrently and cut back construct instances considerably.

With the demo challenge I wrote for this put up, the entire take a look at time went down from 35 minutes to 6 minutes, together with the time to provision the environments. These two screenshots from the AWS Administration Console present the distinction.

Sequential execution of the take a look at suite

Parallel execution of the take a look at suite

Very lengthy take a look at instances pose a big problem when operating steady integration (CI) at scale. As tasks develop in complexity and crew measurement, the time required to execute complete take a look at suites can improve dramatically, resulting in prolonged pipeline execution instances. This not solely delays the supply of recent options and bug fixes, but additionally hampers developer productiveness by forcing them to attend for construct outcomes earlier than continuing with their duties. I’ve skilled pipelines that took as much as 60 minutes to run, solely to fail on the final step, requiring an entire rerun and additional delays. These prolonged cycles can erode developer belief within the CI course of, contribute to frustration, and finally decelerate your entire software program supply cycle. Furthermore, long-running assessments can result in useful resource rivalry, elevated prices due to wasted computing energy, and lowered general effectivity of the event course of.

With parallel take a look at execution in CodeBuild, now you can run your assessments concurrently throughout a number of construct compute environments. This function implements a sharding strategy the place every construct node independently executes a subset of your take a look at suite. CodeBuild supplies surroundings variables that establish the present node quantity and the entire variety of nodes, that are used to find out which assessments every node ought to run. There isn’t a management construct node or coordination between nodes at construct time—every node operates independently to execute its assigned portion of your assessments.

To allow take a look at splitting, configure the batch fanout part in your buildspec.xml, specifying the specified parallelism stage and different related parameters. Moreover, use the codebuild-tests-run utility in your construct step, together with the suitable take a look at instructions and the chosen splitting technique.

The assessments are break up based mostly on the sharding technique you specify. codebuild-tests-run provides two sharding methods:

• Equal-distribution. This technique kinds take a look at information alphabetically and distributes them in chunks equally throughout parallel take a look at environments. Adjustments within the names or amount of take a look at information would possibly reassign information throughout shards.
• Stability. This technique fixes the distribution of assessments throughout shards through the use of a constant hashing algorithm. It maintains present file-to-shard assignments when new information are added or eliminated.

CodeBuild helps computerized merging of take a look at stories when operating assessments in parallel. With computerized take a look at report merging, CodeBuild consolidates assessments stories right into a single take a look at abstract, simplifying outcome evaluation. The merged report contains aggregated go/fail statuses, take a look at durations, and failure particulars, decreasing the necessity for guide report processing. You’ll be able to view the merged ends in the CodeBuild console, retrieve them utilizing the AWS Command Line Interface (AWS CLI), or combine them with different reporting instruments to streamline take a look at evaluation.

Let’s have a look at the way it works
Let me reveal methods to implement parallel testing in a challenge. For this demo, I created a really fundamental Python challenge with tons of of assessments. To hurry issues up, I requested Amazon Q Developer on the command line to create a challenge and 1,800 take a look at instances. Every take a look at case is in a separate file and takes one second to finish. Operating all assessments in a sequence requires half-hour, excluding the time to provision the surroundings.

On this demo, I run the take a look at suite on ten compute environments in parallel and measure how lengthy it takes to run the suite.

To take action, I added a buildspec.yml file to my challenge.

model: 0.2 batch:   fast-fail: false   build-fanout:     parallelism: 10 # ten runtime environments      ignore-failure: false phases:   set up:     instructions:       - echo 'Putting in Python dependencies'       - dnf set up -y python3 python3-pip       - pip3 set up --upgrade pip       - pip3 set up pytest   construct:     instructions:       - echo 'Operating Python Checks'       - |          codebuild-tests-run            --test-command 'python -m pytest --junitxml=report/test_report.xml'            --files-search "codebuild-glob-search 'assessments/test_*.py'"            --sharding-strategy 'equal-distribution'   post_build:     instructions:       - echo "Check execution accomplished" stories:   pytest_reports:     information:       - "*.xml"     base-directory: "report"     file-format: JUNITXML 

There are three elements to focus on within the YAML file.

First, there’s a build-fanout part underneath batch. The parallelism command tells CodeBuild what number of take a look at environments to run in parallel. The ignore-failure command signifies if failure in any of the fanout construct duties may be ignored.

Second, I exploit the pre-installed codebuild-tests-run command to run my assessments.

This command receives the whole record of take a look at information and decides which of the assessments should be run on the present node.

• Use the sharding-strategy argument to decide on between equally distributed or steady distribution as I clarify above.
• Use the files-search argument to go all of the information which might be candidates for a run. We advocate to make use of the supplied codebuild-glob-search command for efficiency causes, however any file search device, similar to discover(1), will work.
• I go the precise take a look at command to run on the shard with the test-command argument.

Lastly, the stories part instructs CodeBuild to gather and merge the take a look at stories on every node.

Then, I open the CodeBuild console to create a challenge and a batch construct configuration for this challenge. There’s nothing new right here, so I’ll spare you the small print. The documentation has all the small print to get you began.  Parallel testing works on batch builds. Be sure to configure your challenge to run in batch.

Now, I’m able to set off an execution of the take a look at suite. I can commit new code on my GitHub repository or set off the construct within the console.

After a couple of minutes, I see a standing report of the totally different steps of the construct; with a standing for every take a look at surroundings or shard.

When the take a look at is full, I choose the Reviews tab to entry the merged take a look at stories.

The Reviews part aggregates all take a look at knowledge from all shards and retains the historical past for all builds. I choose my most up-to-date construct within the Report historical past part to entry the detailed report.

As anticipated, I can see the aggregated and the person standing for every of my 1,800 take a look at instances. On this demo, they’re all passing, and the report is inexperienced.

The 1,800 assessments of the demo challenge take one second every to finish. Once I run this take a look at suite sequentially, it took 35 minutes to finish. Once I run the take a look at suite in parallel on ten compute environments, it took six minutes to finish, together with the time to provision the environments. The parallel run took 17.1 % of the time of the sequential run. Precise numbers will fluctuate together with your tasks.

Further issues to know
This new functionality is suitable with all testing frameworks. The documentation contains examples for Django, Elixir, Go, Java (Maven), Javascript (Jest), Kotlin, PHPUnit, Pytest, Ruby (Cucumber), and Ruby (RSpec).

For take a look at frameworks that don’t settle for space-separated lists, the codebuild-tests-run CLI supplies a versatile various by the CODEBUILD_CURRENT_SHARD_FILES surroundings variable. This variable comprises a newline-separated record of take a look at file paths for the present construct shard. You should utilize it to adapt to totally different take a look at framework necessities and format take a look at file names.

You’ll be able to additional customise how assessments are break up throughout environments by writing your individual sharding script and utilizing the CODEBUILD_BATCH_BUILD_IDENTIFIER surroundings variable, which is robotically set in every construct. You should utilize this method to implement framework-specific parallelization or optimization.

Pricing and availability
With parallel take a look at execution, now you can full your take a look at suites in a fraction of the time beforehand required, accelerating your improvement cycle and bettering your crew’s productiveness. The demo challenge I created as an example this put up consumes 18.7 % of the time of a sequential construct.

Parallel take a look at execution is out there on all three compute modes supplied by CodeBuild: on-demand, reserved capability, and AWS Lambda compute.

This functionality is out there at the moment in all AWS Areas the place CodeBuild is obtainable, with no extra value past the usual CodeBuild pricing for the compute assets used.

I invite you to attempt parallel take a look at execution in CodeBuild at the moment. Go to the AWS CodeBuild documentation to study extra and get began with parallelizing your assessments.

— seb

PS: Right here’s the immediate I used to create the demo software and its take a look at suite: “I’m writing a weblog put up to announce codebuild parallel testing. Write a quite simple python app that has tons of of assessments, every take a look at in a separate take a look at file. Every take a look at takes one second to finish.”

---

How is the Information Weblog doing? Take this 1 minute survey!

(This survey is hosted by an exterior firm. AWS handles your data as described within the AWS Privateness Discover. AWS will personal the information gathered through this survey and won’t share the knowledge collected with survey respondents.)