Use Jenkins’ parallel job execution behavior to reduce the job execution time and save engineering hours.

1 Problem

We have a scheduled Jenkins pipeline that runs every night. This pipeline is taking approximately 15 hours to complete. We wanted to reduce the time it takes to complete the jobs and improve the efficiency of our engineering teams.

2 Background

The scheduled pipeline runs a heavy test suite on our application. This test suite consists of three test stages which are independent from one another. But the problem was that the stages were set to run sequentially. So the current Jenkins script looks like below (following script is an example),

pipeline {
  agent any
  stages {
    
    stage ('Stage 1') {
      steps {
        sh "echo 'Stage 1'"
        sh "sleep 5"
      }
    }
       
    stage ('Stage 2') {
      steps {
        sh "echo 'Stage 2'"
        sh "sleep 5"
      }
    }
      
    stage ('Stage 3') {
      steps {
        sh "echo 'Stage 3'"
        sh "sleep 5"
      }
    }
    
  }
}

Following image shows the Job stage execution when it runs in sequential order.

Assume that each stage takes T1, T2 and T3 approximately and T2 > T3 > T1. So the total time T is

T = T1 + T2 + T3

In above example Total execution time is equal to 17 seconds.

3 Solution

Solution is to run the stages in parallel and save the time. Since the stages are independent from one another, it was easier for us to run the stages in parallel. So now the total time it takes to complete T should be equal to T2 approximately (because T2 > T3 > T1).

3.1 Solution implementation

We can simply use the parallel stage keyword in pipeline scripts. So our new script looks like below,

pipeline {    
  agent any
  stages { 
    stage ("Parallel Jobs") {

      parallel {
        
        stage ("Stage 1") {
          steps {
            sh "echo 'Stage 1'"
            sh "sleep 5"
          }
        }
           	 
        stage ("Stage 2") {
          steps {
            sh "echo 'stage 2'"
            sh "sleep 5"
          }
        }
        
        stage ("Stage 3") {
          steps {
            sh "echo 'stage 3'"
            sh "sleep 5"
          }
        }
        
      }
      
    }
  }
}

Job stage execution is like below now, all the stages are run in parallel.

Now the total execution time has been reduced to 6.7 seconds (which is approximately equal to T2).