How to properly scale Jenkins?

27

5

In my project we have one AWS server running Jenkins Master + 1 Jenkins slave (2 executors)... and we need more
In order to augment our build power we have three options:

  1. Scale up: Make AWS instance bigger and add more executors.
  2. Scale up: Make AWS instance bigger and add another jenkins slave process.
  3. Scale out: Create another AWS instance with a jenkins slave and connect it to master

We want to do 2. as we are in a big organization and our current Jenkins Master has already access to every place he needs. Option 3. "New server" is complicated as it needs more bureaucratic approvals that will take weeks.

So my questions are:

  • Is there any technical issues in option 2?. Maybe the executors of each jenkins slave are not aware of the other slave executors?
  • In general, what is the best approach to scale Jenkins? Scaling up or scaling out?

Oscar Foley

Posted 2017-03-09T10:19:25.910

Reputation: 465

You'll have a pitfall, changing an instance type could be problematic if you move to a different hardware type, as your volume would have to be backed up and restored in the new instance.Tensibai 2017-03-09T10:40:20.550

2Why not number 3? The usual way to send jobs to Jenkins is to master. And based on certain criterias master will send it seamless to appropriate slaveRomeo Ninov 2017-03-09T13:32:02.627

FWIW, you need to also analyze your build structure to see how it uses the build machine's resources - scaling up might not help - I encountered cases in which the build time for 2 parallel builds on the same machine was longer than the combined build times of the same 2 builds executed sequentially, non-overlapping. In such case #3 would really be the only practical option available.Dan Cornilescu 2017-03-09T15:02:12.057

I agree that #3 is better but I don't have arguments for it or arguements against #1 and #2...Oscar Foley 2017-03-09T15:53:33.577

If you have the opportunity within your environment, I would look to an ephemeral solution. Seeing that you're already in AWS, you could easily bring machines up and down as needed while handling the workload. https://wiki.jenkins.io/display/JENKINS/Amazon+EC2+Plugin

casey vega 2019-05-25T15:54:18.237

Answers

11

There are no fundamental technical issues with running multiple jenkins slaves on the same machine. In fact Running Multiple Slaves on the Same Machine lists several good reasons for doing it:

While the correct use of executors largely obviates the need for multiple slave instances on the same machine, there are some unique use cases to consider:

  • You want more configurability between the configured nodes. Say you have one node set to be used as much as possible, and the other node to be used only when needed.
  • You may have multiple Jenkins master installations building different things, and so this configuration would allow you to have slaves for more than one master on the same box. That's right, with Jenkins you really can serve two masters.
  • You may wish to leverage the easiness of starting/stopping/replacing virtual machines, perhaps in conjunction with Jenkins plugins such as the Libvirt Slaves Plugin.
  • You wish to maximize your hardware investment and utilization, at the same time minimizing operating cost (e.g. utility expenses for running idling slaves).

In general scaling out is preferred, primarily because ability to scale up is typically limited by the types/sizes of the available physical resources.

In particular for augmenting build power I'd recommend an analysis of your actual build to determine how it uses the machine resources, which/where its bottlenecks are and what scalability limitations it raises to reveal if scaling up even helps.

For example I encountered cases in which the build time for 2 parallel builds on the same machine was longer than the combined build times of the same 2 builds executed sequentially (non-overlapping) on the same machine. In such case I wouldn't even consider scaling up as it would actually decrease the overall building capacity.

Dan Cornilescu

Posted 2017-03-09T10:19:25.910

Reputation: 5 812

5

Use Kubernetes and Helm.

I would recommend using the Jenkins Helm chart. It installs with helm install stable/jenkins and automatically scales.

https://github.com/kubernetes/charts/tree/master/stable/jenkins

David West

Posted 2017-03-09T10:19:25.910

Reputation: 671

3

I think you should do neither ;)

Well kinda. I think you need more executors, maybe your builds are really resource intensive? I would run at least 4 but we run 6 to 8 depending on jobs. I like to match # of cores to exectors. So you might want to scale up your nodes, I think we run a M4 large for our 4-8 executors.

I also think you should scale-out but you should do so smartly. Jenkins has a plugin to automatically scale-out on AWS depending on what is in the build queue. Basically you tell it how many jobs and how long the wait before it stands up a slave and sends the jobs to the new slave. You can also set the max amount of slaves, the min amount etc.

Levi

Posted 2017-03-09T10:19:25.910

Reputation: 450

2

I'd scale out instead of scale up, going for option 3. We have made a setup where we have all Jenkins agents running on an ECS (custom Docker based Jenkins) with an auto-scaling group. We have all our Jenkins masters communicating to the ECS, thus sharing the workload on the ECS, and no need to recreate the Jenkins master in a scale up exercise.

Jon Brohauge

Posted 2017-03-09T10:19:25.910

Reputation: 21