5/2/2019 Deploy a Jenkins Cluster on AWS – A Cloud Guru

Deploy a Jenkins Cluster on AWS

The automated platform has a Jenkins cluster with a
dedicated Jenkins master and workers inside an
autoscaling group

Mohamed Labouardy Following

Nov 4, 2018 · 6 min read

A few months ago, I gave a talk at Nexus User Conference 2018 on how
to build a fully automated CI/CD platform on AWS using Terraform,
Packer & Ansible.

The session illustrated how concepts like infrastructure as code,

immutable infrastructure, serverless, cluster discovery, etc can be used to
build a highly available and cost-effective pipeline.

The platform I built is represented in the following diagram:

https://read.acloud.guru/deploy-a-jenkins-cluster-on-aws-35dcf66a1eca 1/18
5/2/2019 Deploy a Jenkins Cluster on AWS – A Cloud Guru

The platform has a Jenkins cluster with a dedicated Jenkins master and
workers inside an autoscaling group. Each push event to the code
repository will trigger the Jenkins master which will schedule a new
build on one of the available slave nodes.

The slave nodes will be responsible of running the unit and pre-
integration tests, building the Docker image, storing the image to a
private registry and deploying a container based on that image to
Docker Swarm cluster.

https://read.acloud.guru/deploy-a-jenkins-cluster-on-aws-35dcf66a1eca 2/18
5/2/2019 Deploy a Jenkins Cluster on AWS – A Cloud Guru

CI/CD Platform in AWS with Terraform, Ans…

Ans…

If you missed my talk, you can watched it again on YouTube

On this post, I will walk through how to deploy the Jenkins cluster on
AWS using the latest automation tools.

The cluster will be deployed into a VPC with 2 public and 2 private
subnets across 2 availability zones. The stack will consists of an
autoscaling group of Jenkins workers in a private subnets and a private
instance for the Jenkins master sitting behind an elastic Load balancer.

To add or remove Jenkins workers on-demand, the CPU utilization of

the ASG will be used to trigger a scale out (CPU > 80%) or scale in
(CPU < 20%) event.

https://read.acloud.guru/deploy-a-jenkins-cluster-on-aws-35dcf66a1eca 3/18
5/2/2019 Deploy a Jenkins Cluster on AWS – A Cloud Guru

To get started, we will create 2 AMIs (Amazon Machine Image) for our
instances. To do so, we will use Packer, which allows you to bake your
own image.

The first AMI will be used to create the Jenkins master instance. The
AMI uses the Amazon Linux Image as a base image and for provisioning
part it uses a simple shell script:

1 {
2 "variables" : {
3 "region" : "eu-west-3",
4 "source_ami" : "ami-0ebc281c20e89ba4b"
5 },
6 "builders" : [
7 {
8 "type" : "amazon-ebs",
9 "profile" : "default",
10 "region" : "{{user `region`}}",
11 "instance_type" : "t2.micro",
12 "source_ami" : "{{user `source_ami`}}",
13 "ssh_username" : "ec2-user",
14 "ami_name" : "jenkins-master-2.107.2",
15 "ami_description" : "Amazon Linux Image wi
16 "run_tags" : {
17 "Name" : "packer-builder-docker"
18 },
19 "tags" : {
20 "Tool" : "Packer",
21 "Author" : "mlabouardy"
22 }
23 }
24 ],

The shell script will be used to install the necessary dependencies,

packages and security patches:

https://read.acloud.guru/deploy-a-jenkins-cluster-on-aws-35dcf66a1eca 4/18
5/2/2019 Deploy a Jenkins Cluster on AWS – A Cloud Guru

1 #!/bin/bash
2
3 echo "Install Jenkins stable release"
4 yum remove -y java
5 yum install -y java-1.8.0-openjdk
6 wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenki
7 rpm --import https://jenkins-ci.org/redhat/jenkins-ci.
8 yum install -y jenkins
9 chkconfig jenkins on
10
11 echo "Install Telegraf"
12 wget https://dl.influxdata.com/telegraf/releases/teleg
13 yum localinstall -y /tmp/telegraf.rpm
14 rm /tmp/telegraf.rpm
15 chkconfig telegraf on
16 mv /tmp/telegraf.conf /etc/telegraf/telegraf.conf
17 service telegraf start
18
19 echo "Install git"
20 yum install -y git
21
22 echo "Setup SSH key"
23 mkdir /var/lib/jenkins/.ssh
24 touch /var/lib/jenkins/.ssh/known_hosts
25 chown -R jenkins:jenkins /var/lib/jenkins/.ssh
26 chmod 700 /var/lib/jenkins/.ssh
27 mv /tmp/id_rsa /var/lib/jenkins/.ssh/id_rsa
28 chmod 600 /var/lib/jenkins/ ssh/id rsa

It will install the latest stable version of Jenkins and configure its
settings:

• Create a Jenkins admin user.

• Create a SSH, GitHub and Docker registry credentials.

• Install all needed plugins (Pipeline, Git plugin, Multi-branch

Project, etc).

• Disable remote CLI, JNLP and unnecessary protocols.

• Enable CSRF (Cross Site Request Forgery) protection.

• Install Telegraf agent for collecting resource and Docker metrics.

https://read.acloud.guru/deploy-a-jenkins-cluster-on-aws-35dcf66a1eca 5/18
5/2/2019 Deploy a Jenkins Cluster on AWS – A Cloud Guru

The second AMI will be used to create the Jenkins workers, similarly to
the first AMI, it will be using the Amazon Linux Image as a base image
and a script to provision the instance:

1 #!/bin/bash
2
3 echo "Install Java JDK 8"
4 yum remove -y java
5 yum install -y java-1.8.0-openjdk
6
7 echo "Install Docker engine"
8 yum update -y
9 yum install docker -y
10 usermod -aG docker ec2-user
11 service docker start
12
13 echo "Install git"
14 yum install -y git
15
16 echo "Install Telegraf"

A Jenkins worker requires the Java JDK environment and Git to be

installed. In addition, the Docker community edition (building Docker
images) and a data collector (monitoring) will be installed.

Now our Packer template files are defined, issue the following
commands to start baking the AMIs:

1 # validate packer template

2 packer validate ami.json 
3
4 # build ami

Packer will launch a temporary EC2 instance from the base image
specified in the template file and provision the instance with the given
shell script. Finally, it will create an image from the instance. The
following is an example of the output:

https://read.acloud.guru/deploy-a-jenkins-cluster-on-aws-35dcf66a1eca 6/18
5/2/2019 Deploy a Jenkins Cluster on AWS – A Cloud Guru

Sign in to AWS Management Console, navigate to “EC2 Dashboard”

and click on “AMI”, 2 new AMIs should be created as below:

Now our AMIs are ready to use, let’s deploy our Jenkins cluster to AWS.
To achieve that, we will use an infrastructure as code tool called
Terraform, it allows you to describe your entire infrastructure in
templates files.

I have divided each component of my infrastructure to a template file.

The following template file is responsible of creating an EC2 instance
from the Jenkins master’s AMI built earlier:

1 resource "aws_instance" "jenkins_master" {

2 ami = "${data.aws_ami.jenkins-mas
3 instance_type = "${var.jenkins_master_insta
4 key_name = "${var.key_name}"
5 vpc_security_group_ids = ["${aws_security_group.jenk
6 subnet_id = "${element(var.vpc_private_
7
8 root_block_device {
9 volume_type = "gp2"
10 volume_size = 30
11 delete_on_termination = false
12 }
13

Another template file used as a reference to each AMI built with Packer:

https://read.acloud.guru/deploy-a-jenkins-cluster-on-aws-35dcf66a1eca 7/18
5/2/2019 Deploy a Jenkins Cluster on AWS – A Cloud Guru

1 data "aws_ami" "jenkins-master" {

2 most_recent = true
3 owners = ["self"]
4
5 filter {
6 name = "name"
7 values = ["jenkins-master-2.107.2"]
8 }
9 }
10
11 data "aws_ami" "jenkins-slave" {
12 most_recent = true
13 owners = ["self"]

The Jenkins workers (aka slaves) will be inside an autoscaling group of

a minimum of 3 instances. The instances will be created from a launch
configuration based on the Jenkins slave’s AMI:

https://read.acloud.guru/deploy-a-jenkins-cluster-on-aws-35dcf66a1eca 8/18
5/2/2019 Deploy a Jenkins Cluster on AWS – A Cloud Guru

1 // Jenkins slaves launch configuration

2 resource "aws_launch_configuration" "jenkins_slave_lau
3 name = "jenkins_slaves_config"
4 image_id = "${data.aws_ami.jenkins-slave.id}"
5 instance_type = "${var.jenkins_slave_instance_type
6 key_name = "${var.key_name}"
7 security_groups = ["${aws_security_group.jenkins_sla
8 user_data = "${data.template_file.user_data_sl
9
10 root_block_device {
11 volume_type = "gp2"
12 volume_size = 30
13 delete_on_termination = false
14 }
15
16 lifecycle {
17 create_before_destroy = true
18 }
19 }
20
21 // ASG Jenkins slaves
22 resource "aws_autoscaling_group" "jenkins_slaves" {
23 name = "jenkins_slaves_asg"
24 launch_configuration = "${aws_launch_configuration.j
25 vpc_zone_identifier = "${var.vpc_private_subnets}"
26 min_size = "${var.min_jenkins_slaves}"
27 max_size = "${var.max_jenkins_slaves}"
28
29 depends_on = ["aws_instance.jenkins_master", "aws_el
30
31 lifecycle {
32 create_before_destroy = true
33 }
34

To leverage the power of automation, we will make the worker instance

join the cluster automatically (cluster discovery) using Jenkins RESTful
API:

https://read.acloud.guru/deploy-a-jenkins-cluster-on-aws-35dcf66a1eca 9/18
5/2/2019 Deploy a Jenkins Cluster on AWS – A Cloud Guru

1 #!/bin/bash
2
3 JENKINS_URL="${jenkins_url}"
4 JENKINS_USERNAME="${jenkins_username}"
5 JENKINS_PASSWORD="${jenkins_password}"
6 TOKEN=$(curl -u $JENKINS_USERNAME:$JENKINS_PASSWORD ''
7 INSTANCE_NAME=$(curl -s 169.254.169.254/latest/meta-da
8 INSTANCE_IP=$(curl -s 169.254.169.254/latest/meta-data
9 JENKINS_CREDENTIALS_ID="${jenkins_credentials_id}"
10
11 sleep 60
12
13 curl -v -u $JENKINS_USERNAME:$JENKINS_PASSWORD -H "$TO
14 import hudson.model.Node.Mode
15 import hudson.slaves.*
16 import jenkins.model.Jenkins
17 import hudson.plugins.sshslaves.SSHLauncher
18 DumbSlave dumb = new DumbSlave("'$INSTANCE_NAME'",

At boot time, the user-data script above will be invoked and the instance
private IP address will be retrieved from the instance meta-data and a
groovy script will be executed to make the node join the cluster:

1 data "template_file" "user_data_slave" {

2 template = "${file("scripts/join-cluster.tpl")}"
3
4 vars {
5 jenkins_url = "http://${aws_instance.je
6 jenkins_username = "${var.jenkins_username}"
7 jenkins_password = "${var.jenkins_password}"

Moreover, to be able to scale out and scale in instances on demand, I

have defined 2 CloudWatch metric alarms based on the CPU utilisation
of the autoscaling group:

https://read.acloud.guru/deploy-a-jenkins-cluster-on-aws-35dcf66a1eca 10/18
5/2/2019 Deploy a Jenkins Cluster on AWS – A Cloud Guru

1 // Scale out
2 resource "aws_cloudwatch_metric_alarm" "high-cpu-jenki
3 alarm_name = "high-cpu-jenkins-slaves-alarm
4 comparison_operator = "GreaterThanOrEqualToThreshold
5 evaluation_periods = "2"
6 metric_name = "CPUUtilization"
7 namespace = "AWS/EC2"
8 period = "120"
9 statistic = "Average"
10 threshold = "80"
11
12 dimensions {
13 AutoScalingGroupName = "${aws_autoscaling_group.je
14 }
15
16 alarm_description = "This metric monitors ec2 cpu ut
17 alarm_actions = ["${aws_autoscaling_policy.scale
18 }
19
20 resource "aws_autoscaling_policy" "scale-out" {
21 name = "scale-out-jenkins-slaves"
22 scaling_adjustment = 1
23 adjustment_type = "ChangeInCapacity"
24 cooldown = 300
25 autoscaling_group_name = "${aws_autoscaling_group.je
26 }
27
28 // Scale In
29 resource "aws_cloudwatch_metric_alarm" "low-cpu-jenkin
30 alarm_name = "low-cpu-jenkins-slaves-alarm"
31 comparison_operator = "LessThanOrEqualToThreshold"
32 evaluation_periods = "2"
33 metric_name = "CPUUtilization"
34 namespace = "AWS/EC2"

Finally, an Elastic Load Balancer will be created in front of the Jenkins

master’s instance and a new DNS record pointing to the ELB domain
will be added to Route 53:

https://read.acloud.guru/deploy-a-jenkins-cluster-on-aws-35dcf66a1eca 11/18
5/2/2019 Deploy a Jenkins Cluster on AWS – A Cloud Guru

1 resource "aws_route53_record" "jenkins_master" {

2 zone_id = "${var.hosted_zone_id}"
3 name = "jenkins.slowcoder.com"
4 type = "A"
5
6 alias {
7 name = "${aws_elb.jenkins_elb.dn
8 zone_id = "${aws_elb.jenkins_elb.zo

Once the stack is defined, provision the infrastructure with terraform

apply command:

1 # Install the AWS provider plugin

2 terraform int
3
4 # Dry-run check
5 terraform plan
6

The command takes an additional parameter, a variables file with the

AWS credentials and VPC settings (You can create a new VPC with
Terraform from here):

1 region = ""
2
3 aws_profile = ""
4
5 shared_credentials_file = ""
6
7 key_name = ""
8
9 hosted_zone_id = ""
10
11 bastion_sg_id = ""
12
13 jenkins_username = ""
14
15 jenkins_password = ""
16
17 jenkins credentials id = ""

https://read.acloud.guru/deploy-a-jenkins-cluster-on-aws-35dcf66a1eca 12/18
5/2/2019 Deploy a Jenkins Cluster on AWS – A Cloud Guru

Terraform will display an execution plan (list of resources that will be

created in advance), type yes to confirm and the stack will be created in
few seconds:

Jump back to EC2 dashboards, a list of EC2 instances will created:

In the terminal session, under the Outputs section, the Jenkins URL will
be displayed:

https://read.acloud.guru/deploy-a-jenkins-cluster-on-aws-35dcf66a1eca 13/18
5/2/2019 Deploy a Jenkins Cluster on AWS – A Cloud Guru

Point your favorite browser to the URL displayed, the Jenkins login
screen will be displayed. Sign in using the credentials provided while
baking the Jenkins master’s AMI:

If you click on “Credentials” from the navigation pane, a set of

credentials should be created out of the box:

The same goes for “Plugins”, a list of needed packages will be installed
also:

https://read.acloud.guru/deploy-a-jenkins-cluster-on-aws-35dcf66a1eca 14/18
5/2/2019 Deploy a Jenkins Cluster on AWS – A Cloud Guru

Once the Autoscaling group finished creating the EC2 instances, the
instances will join the cluster automatically as you can see in the
following screenshot:

You should now be ready to create your own CI/CD pipeline!

https://read.acloud.guru/deploy-a-jenkins-cluster-on-aws-35dcf66a1eca 15/18
5/2/2019 Deploy a Jenkins Cluster on AWS – A Cloud Guru

You can take this further and build a dynamic dashboard in your
favorite visualization tool like Grafana to monitor your cluster resource
usage based on the metrics collected by the agent installed on each EC2
instance:

https://github.com/mlabouardy/grafana-dashboards

Drop your comments, feedback, or suggestions below — or connect with

me directly on Twitter @mlabouardy.

mlabouardy/nexususerconference-
infrastructure

Nexus User Conference 2018. Contribute to

mlabouardy/nexususerconference-infrastructure…
github.com

https://read.acloud.guru/deploy-a-jenkins-cluster-on-aws-35dcf66a1eca 16/18
5/2/2019 Deploy a Jenkins Cluster on AWS – A Cloud Guru

https://read.acloud.guru/deploy-a-jenkins-cluster-on-aws-35dcf66a1eca 17/18
5/2/2019 Deploy a Jenkins Cluster on AWS – A Cloud Guru

https://read.acloud.guru/deploy-a-jenkins-cluster-on-aws-35dcf66a1eca 18/18