Fixing Jenkins Job Waiting In Queue
I was trying to solve a weird issue in Jenkins when jobs were stuck in queue for a long time. When jobs were in the queue, there were idle executors available. The reason for waiting was
Finished waiting. These jobs were triggered by a service account through Jenkins API. If the same job was triggered by a normal user, then the job would be executed immediately.
Finding the Cause
To find out the root cause, I clone Jenkins source code at GitHub. I started by searching the message
Finished waiting in the source code. This message was defined in the resource bundle with key
Queue.FinishedWaiting. This bundle key was referenced in the
hudson.model.Queue class. It simply means the job was blocked.
Looking at the surrounding
WaitingItem class, this class represents an item waiting in the queue. It has
leave() methods. It seems that the waiting item was added in the queue, but was only removed from the queue for a long time.
Searching the usage of
enter() method, the item was added in
scheduleInternal() method. Searching the usage of
leave() method, the item will be removed in the
maintain() method maintains the queue and moves projects between different states. Jenkins internally invokes this method by itself whenever there is a change that can affect the scheduling. In the
scheduleInternal() method, after adding an item to the queue, it calls
scheduleMaintenance() method to submit a
Runnable task to call the
So the cause of a job waiting in the queue may be the submitted task didn't see the waiting job. So the job has to wait until next time the
maintain() method was invoked.
I don't know exactly why this could happen. My solution is to manually call
scheduleMaintenance() method of
Queue class to force the check.
scheduleMaintenance() method is
public, so I can simply call this method on a
Jenkins has a Script Console to run Groovy scripts. In Groovy script, I can get the instance of the Jenkins queue and call the
To automate this, I used a HTTP client to send a
POST request to
POST request was send using
application/x-www-form-urlencoded content type. The request has one parameter
script with value
After trigging a new job in Jenkins, I use the HTTP client to call the
scheduleMaintenance() method through Script Console. No jobs in the waiting queue any more.