Create a Jenkins CICD Pipeline to build a Docker Image with Splunk Integration

Published in

AWS in Plain English

13 min readApr 3, 2021

In this article, I will share a step-by-step walkthrough on how to build a Continuous Integration and Continuous Delivery (CI/CD) pipeline using Jenkins. And how this pipeline is configured to build Docker images. Also, I have integrated Jenkins with Splunk to provide the capability to monitor and assess their operational performance.

Brushing up some important concepts

Continuous Integration focuses on blending the work products of individual developers together into a repository. This is often done various times every single day, and the prime purpose is to enable early detection of integration bugs, which should eventually result in tight cohesion & more development collaboration.

The aim of Continuous Delivery is to reduce the friction points that are inherent in the deployment or release processes. Generally, the implementation covers automating every single step for build deployments such that a safe code release can be done eventually — at any time or moment.

Continuous Deployment is a higher degree of automation, in which a build/deployment occurs automatically whenever an enormous modification is made to the code.

Jenkins is a self-sufficient, open-source automation tool written in Java. Jenkins uses plugins to build & test your project code continuously, making new changes a laid-back approach for developers. Jenkins facilitates Continuous Integration, It builds and tests your software continuously and monitors the execution and status of remote jobs, making it easier for team members and users to regularly obtain the latest stable code.

Jenkins has a handful of core concepts:

Job — A unit of work such as linting, running a test, or doing an owner's check.
Pipeline — A sequencing system where one action occurs typically if and only if the previous action successfully took place. Options for parallelizing your work are also available.
Queue — Jenkins automatically maintains a queue of jobs that should happen once sufficient capacity is available.
Plugin — Like WordPress, Jenkins has many plugins you can install to supplement the default features.
Master/Slave — A Jenkins “master” server powers the website and coordinates the work performed in your Jenkins installation. It communicates to “slave” machines responsible for actually performing your work. This means you can scale up/down the number of Jenkins workers you are running in your cluster to control capacity.

Docker is a tool that allows you to easily share, deploy, and run applications consistently in many environments through containers. First, I needed to get access to the Docker command line (CLI). The installation instructions for Docker are not very clear or straightforward, but you need to download Docker Desktop (for Mac or Windows).

Additionally, Docker Hub is similar to GitHub for git repositories or the npmjs registry for JavaScript packages — it’s an online repository of Docker images and the login that Docker Desktop will connect to.

A container is an executable package that contains everything needed to run an application. It will always run the same, regardless of infrastructure, in a sandboxed environment. It is a running instance of an image.

A Dockerfile is a text file that Docker reads in from top to bottom. It contains a bunch of instructions that inform Docker how the Docker image should get built.

To use a container, you’ll need to give instructions to Docker via a file called Dockerfile at the root of your project. Dockerfile seems a little intimidating at first, but it’s not that much different from how I set up my local environment. It just requires a few Dockerfile specific commands.

FROM — Start the Dockerfile and pull from a base image
COPY — Copy files from a local source to container target
WORKDIR — Set working directory for subsequent commands
RUN — Run commands
EXPOSE — Set a port
ENTRYPOINT — Set executable command

Splunk is a software platform to do data analytics in almost real-time by cleaning, segregating, extracting, classifying streams of machine-generated data. It helps in visualization & analysis of the data in the form of charts & statistics by creating dashboards. Splunk is a horizontal technology used for application management, security, and compliance, as well as business and web analytics. Splunk can ingest a variety of data formats like JSON, XML, and unstructured machine data like web and application logs.

It thereby helps pro-actively monitor systems, troubleshoots failure, store & retrieves data for later use, investigate particular outcomes, identify operational loopholes, monitor business metrics, derive business insights, analyze system performance, security threats, maintain operational hygiene by identifying scaling requirements, etc.

Splunk is proprietary software and today comes in 3 flavors -

Splunk Enterprise — The most popular, on-prem solution & Splunk stream processing system.
Splunk Lite — This is a subset of the functionality offered by Splunk enterprise, but this does not come with a commercial license.
Splunk Cloud — Splunk offered on the cloud as a Saas offering. But companies having data-governance concerns should evaluate this carefully as the data is stored over cloud

Implementation of the Project

Now learn about the implementation of the above concepts together by following this step-by-step tutorial. Here, we will first create and configure a Jenkins build pipeline to build, compile, and package your code into a runnable Docker image and further push the docker image to the Docker Hub. A Jenkins pipeline is basically a way to execute a Jenkins job sequentially in a defined way by codifying it and structuring it inside multiple blocks that can include multiple steps containing tasks

Let’s begin with Jenkins Setup for the project

Here, for this project, I have set up Jenkins in an Amazon Linux EC2 instance. We will first launch the instance and then you’ll learn about the various commands used to install and use Jenkins.

Step 1: Launching an Amazon Linux EC2 instance

i. Using the AWS console, From the Amazon EC2 dashboard, choose Launch Instance.

ii. The Choose an Amazon Machine Image (AMI) page displays a list of basic configurations called Amazon Machine Images that serve as templates for your instance. Select an Amazon Linux 2 AMI.

iii. On the Choose an Instance Type page, choose the default instance type t2.micro as the hardware configuration of your instance.

iv. In the Configure Instance Details, launch the instance in the public subnet along with Public IP enabled. Further, in the Advanced Details section, add the following user data script to install docker, create a docker group, and start the docker service on the instance:

#! /bin/sh
yum update -y
amazon-linux-extras install docker
service docker start
usermod -a -G docker ec2-user
chkconfig docker on

v. Continue with the default settings and move to Step 6: Configure Security Group, choose to Create a new security group, now give a name to it and add following rules.

vi. Now move to the Review and Launch section, launch the instance with a new key pair and download it.

vii. After successfully launching the instance, connect to your instance using the downloaded key pair.

Step 2: Installation of Jenkins on EC2 Instance

i. Run the command to update all the packages.

sudo yum update

ii. Check if java is installed or not using the command

java -version

If java is not installed, install using the following command

sudo yum install java-1.8.0-openjdk

iii. Now, to download the latest Jenkins package

sudo wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat/jenkins.repo

iv. To enable the installation of the package, import the key file from Jenkins-CI:

sudo rpm — import https://pkg.jenkins.io/redhat/jenkins.io.key

v. Install Jenkins on the EC2 instance

sudo yum install jenkins

vi. To start the Jenkins service

sudo service jenkins start

vii. Access the Jenkins server using the public DNS of the EC2 instance or public IP of the instance on port 8080.

http://{ec2-public-dns}:8080 or http://3.89.79.74:8080/

viii. Login using the username admin and to get the initial admin password execute the following command:

sudo su –
cd /var/lib/jenkins/secrets/
cat initialAdminPassword

ix. To stop Jenkins service

sudo service jenkins stop

Finally, we are done with the setup and now we will start to build our CICD pipeline to build the docker image.

Step 3: Create a CICD pipeline in Jenkins to build Docker Image and push it to Docker Hub

In this project, we will configure a Jenkins build pipeline to build, compile, and package a small maven project into a runnable Docker image. The Jenkins build process will create the Docker image using a Dockerfile. Basically, the Jenkins pipeline is designed in such a way that whenever a change is pushed from the local repository to the Git repository, Jenkins will detect the change and will automatically start to build the project. Thus it would create the docker image and push it to the Docker hub repository. In order to complete the project follow the steps below.

i. Firstly, as prerequisites setup git in your local machine and create a GitHub repository. Now, push all your code to the remote Git repository.

ii. Now, in your previously created EC2 instance run the following commands to install git and maven required for our project.

Steps to Install Git:

sudo yum install git -y
git version

Steps to Install Maven:

sudo wget http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo -O /etc/yum.repos.d/epel-apache-maven.reposudo sed -i s/\$releasever/6/g /etc/yum.repos.d/epel-apache-maven.reposudo yum install -y apache-mavenmvn --version

iii. Next we will install the required plugins for our project, log in to your Jenkins console, and go via the Jenkins GUI -> Manage Jenkins -> Manage Plugins. Select the tab “Available” and select all the below-listed plugins and then click on Install without Restart. This step is the most important one as it will allow us to access the code from our git repository and help us with the creation of a docker image.

Required plugins for Git:

Required plugins for docker:

CloudBees Docker Build and Publish Plugin
Docker Plugin
docker-build-step

iv. On to the next step, we will create our pipeline for this project. So after the installation of plugins is completed, return to the Dashboard. Via Jenkins -> New Item you’ll get to a page that will let you specify which kind of item you want to create. Select the Freestyle Project option, give a suitable name, and hit OK.

Step 4: Configure and execute the pipeline job

In order to configure the pipeline, select the GitHub Project option and add your GitHub repository URL. Similarly, in Source Code Management select Git and add your Git repository URL as shown in the image below.

This will allow Jenkins to access the source code from the git repository including the Dockerfile which is used for creating docker images.

In the Build Triggers, select Poll SCM and in Schedule, enter five asterisks (i.e ‘* * * * *’), this indicates that Jenkins will automatically trigger a build for all the incoming git commits.

Now, move to the Build section, and from the dropdown select the Invoke top-level Maven targets, and in Goals, mention install to perform the installs for maven.

Now in order to complete the pipeline, we have to create and push the created Docker image to Docker Hub. For that click on Add build step and choose Docker Build and Publish. Enter your Docker Hub’s repository name and add your Docker Hub credentials.

Finally, to see the output, click on Build Now and view the Console. Also, observe the image pushed in the Docker Hub registry.

Next, we need to integrate Jenkins with Splunk

Splunk plugin for Jenkins provides deep insights into your Jenkins master and node infrastructure, job and build details such as console logs, status, artifacts, and an incredibly efficient way to analyze test results. The plugin is used together with a Splunk App for Jenkins that provides out-of-the-box dashboards and search capabilities to enable organizations to run a high-performing Jenkins cluster and bring operational intelligence into the software development life cycle.

Note: Before we move further, create an account on Splunk.com, in order to access it.

Step 5: Installation of Splunk on AWS EC2 using Splunk Enterprise AMI

For this project, I have set up Splunk in an AWS EC2 instance using Splunk Enterprise AMI. Follow the below steps to launch the instance.

i. Once logged in, go to AWS Console, now go via Services -> EC2 -> Launch Instances. On the Choose an Amazon Machine Image (AMI) page, click on AWS Marketplace and search Splunk as shown below and Select Splunk Enterprise AMI.

ii. On the Choose an Instance Type page, choose the default instance type t2.micro as the hardware configuration of your instance.

iii. In the Configure Instance Details, launch the instance in the public subnet along with Public IP enabled.

iv. Continue with the default settings and move to Step 6: Configure Security Group, choose to Create a new security group, you will notice Splunk AMI has already enabled these following ports.

Add one more port i.e. 8088 to this list to configure Splunk HTTP Event Collector (HEC).

v. Now move to the Review and Launch section, launch the instance with a new key pair and download it.

Step 6: Installation of Splunk App for Jenkins

Once your instance is ready, log in to your Splunk Web UI by using the URL http://<PublicDNSOfSplunkInstance>:8000. Here, the default username is admin and the password is SPLUNK-<EC2InstanceID>.

Now once logged in, go via Manage Apps -> Browse More Apps, and search for Jenkins as shown in the snapshot below. Now, install the Splunk App for Jenkins. When you click on install, you are asked to enter your credentials of Splunk.com in order to download the app.

The Splunk Jenkins plugin uses the HTTP Event Collector to send the data to Splunk. To know more, click on ‘View on Splunkbase’.

Now, moving back to the dashboard, you can see the Splunk App for Jenkins is installed successfully.

Now to integrate Splunk with Jenkins, go via Settings -> Data inputs -> HTTP Event Collector. Create a new HTTP Event Collector(HEC) token in Splunk by providing a valid name. For now, go with the default settings and complete the process and copy the generated token for further use.

Step 7: Configure Jenkins for Splunk Integration

i. Now move to Jenkins, navigate to Manage Jenkins -> Manage Plugins, on this page search for ‘Splunk’ plugin, click on the check box for the Splunk plugin and then select your preference of restart for the installation.

ii. Once the plugin is installed, go to Manage Jenkins -> Configure system. Here, scroll to the Splunk for Jenkins Configuration section. Now, fill in the details as mentioned below:

Enable — Click on this checkbox to make the configuration for this plugin active.

HTTP Input Host — Mention the hostname of the Splunk Indexer where you have installed the Splunk App for Jenkins.

HTTP Input Port — Provide the port on which to communicate with Splunk, by default it’s 8088.

HTTP Input Token — Copy the HTTP Event Collector token you generated above.

SSL Enabled — If you want the communication to be encrypted between your Jenkins and Splunk instances.

Send all pipeline console logs — Check this as we want all the pipeline logs to be indexed in Splunk.

Jenkins Master Hostname — Provide the Hostname that should appear in your Splunk for the events received from this node.

iii. Now finally, click on Test Connection, you should get this message ‘Splunk connection verified’, if yes then you are done with the integration.

iv. Now to view the changes done, restart both Jenkins and Splunk.

Step 8: Splunk Dashboards for Jenkins

The Splunk App for Jenkins will use the data being sent by the Splunk plugin for Jenkins and show various dashboards and search capabilities. Navigate to the Splunk App for Jenkins on your Splunk instance, Here is an example screenshot of my project, this dashboard is located under the build analysis option.

Here are some of the key features provided by Splunk —

Overview — Visualize multiple masters and associated slaves on a single page. View build status trends and be able to drill down and get details information about any build.
Build Analysis — Easily find any Jenkins build using a variety of easy-to-use filters. View build summary or drill down to see build status trends, build time and queue time analysis, test pass/fail trends, test runtime distribution, and console logs.
Test Analysis — Test Results shows all the failing tests with stack traces, flags regression failures, groups test failures by errors, captures Jenkins environment variables, and provides nifty filters to find tests.
Jenkins Health — Splunk Jenkins Apps captures Jenkins internal JVM information as well as keys metrics like queue size, executors and nodes stats, Jenkins master logs, and Jenkins node stats.
Jenkins Nodes — This feature is extremely helpful in identifying problematic components in a Jenkins cluster and optimizing your team’s throughput.
Audit Trail — The audit trail feature allows you to see who has logged into your Jenkins system and done any activity like starting/aborting/changing jobs.

The below snapshot shows the logs and artifacts for the build pipeline. Thus, in this way you can go on and explore more about the Splunk tool.