1747913854550
1747913854550
DevOps Shack
GitHub Comprehensive Guide
Ultimate GitHub Actions CI/CD Pipeline Guide
1. What is CI/CD?
Continuous Deployment
2
This is a step beyond continuous delivery. In continuous deployment, every
change that passes all stages of the production pipeline is automatically
deployed to users, without manual intervention.
3
4. Real-World Use Cases for GitLab CI/CD
C. Microservices Architectures
D. Monorepo Management
• Directory-based triggers
E. Machine Learning/AI
F. Infrastructure as Code
4
5. Key Benefits of GitLab CI/CD
GitLab provides several benefits that make it a preferred CI/CD tool for many
organizations:
• Single Source of Truth: One platform for code, CI/CD, security, and
monitoring.
Understanding the basic flow helps visualize how GitLab CI/CD operates:
3. Runner Execution: A GitLab Runner picks up the job and starts executing
tasks.
5
4. Build & Test: Jobs defined in stages (build, test, deploy) run in order.
5. Artifacts & Reports: Output (e.g., test results, coverage reports) is stored
as artifacts.
6
How GitLab CI/CD Works
Basic Flow:
7
Example: A pipeline:run webhook might be triggered from a JIRA card
status change.
Stages:
• build
• test
• package
• deploy
• review
• cleanup
Stages run sequentially — all jobs in build must succeed before moving to
test.
Jobs:
Each stage can contain one or more jobs. These are tasks like:
• Running tests
Defined using the script: keyword, they’re the actual shell commands
executed.
8
1. Runner picks up the job from GitLab’s queue.
2. Pre-job phase:
3. Job execution:
4. Post-job phase:
o Uploading artifacts
o Storing logs
Runners are lightweight agents responsible for executing your pipeline jobs.
Types of Runners:
• Shared Runners: Available across all GitLab projects (good for general
use).
Executors:
Use Docker or K8s for consistent build environments and better isolation.
9
6. Authentication & Permissions
One of GitLab CI/CD’s key strengths is how deeply it's integrated with the
GitLab ecosystem.
• You can re-run failed jobs or entire pipelines from the UI or CLI.
10
• Pipelines can also include rollback steps in case deployments fail.
Let’s walk through a typical GitLab CI/CD pipeline for a web app:
Code Flow:
o Stage 5: deploy_staging
11
12. Connecting External Systems
• Slack (notifications)
• Jobs run as soon as their dependencies are met (more efficient than
linear execution).
12
Key Components and Terminologies in GitLab CI/CD
1. Pipeline
• Stages are the main steps in the process (e.g., build → test → deploy).
Example:
stages:
- build
- test
- deploy
build_job:
stage: build
script: echo \"Building...\"
test_job:
stage: test
script: echo \"Testing...\"
deploy_job:
stage: deploy
script: echo \"Deploying...\"
2. Jobs
13
A job is a single task that runs as part of a pipeline. Each job runs in its own
isolated environment, and jobs in the same stage can run in parallel (if
resources allow).
Job attributes:
Example job:
test_job:
stage: test
script:
- npm install
3. Stages
Stages define the execution order of jobs in a pipeline. All jobs in one stage
must complete before the next stage begins.
Common stages:
• build
• test
• review
• staging
• production
Stages are declared in the stages: keyword, and every job must specify which
stage it belongs to.
14
Example:
stages:
- build
- test
- deploy
4. GitLab Runner
A GitLab Runner is an agent that executes the jobs defined in the pipeline. It's
the executor that runs your scripts, builds containers, runs tests, etc.
5. Artifacts
Artifacts are files or directories generated by a job and saved to be used later
in the pipeline or for download. Common artifacts include:
• Test reports
• Build binaries
• Coverage results
• Logs
15
Example:
build:
stage: build
script:
- make build
artifacts:
paths:
- build/
Artifacts can also be set to expire automatically after a certain period.
6. Environments
Example:
deploy_prod:
stage: deploy
script:
- ./deploy.sh
environment:
name: production
url: https://app.example.com
7. Variables
16
• Masked and Protected for sensitive data
Example:
script:
- echo $MY_SECRET
• Project level
• Group level
• Runner level
• Job level
8. Cache
Example:
cache:
paths:
- node_modules/
You can also define key: for versioning cache uniquely per branch or job.
Example:
include:
- project: 'my-group/common-templates'
file: '/templates/deploy.yml'
17
You can also use built-in GitLab templates for things like Node.js, Docker,
Python, etc.
10. Triggers
• Upstream/downstream pipelines
• Scheduled pipelines
deploy_prod:
stage: deploy
script: ./deploy.sh
when: manual
Protected features in GitLab limit who can push or run jobs on sensitive
branches or environments like main or production.
18
Example:
job:
script: run-something
retry: 2
timeout: 10 minutes
13. Reports
• Code Coverage
• License Scanning
Summary Table
Component Description
19
Component Description
Includes modularization
This section equips you with a solid understanding of all the essential terms
and moving parts in GitLab CI/CD. You’ll now be able to read, write, and
debug .gitlab-ci.yml files with much more clarity.
20
.gitlab-ci.yml Structure
The .gitlab-ci.yml file is the heart of GitLab CI/CD. It defines the entire pipeline,
including what jobs to run, how to run them, and under what conditions.
Written in syntax, this file lives at the root of your repository and gets picked
up automatically by GitLab when changes are pushed.
Let’s break it down step by step and explore everything you need to master
this file.
1. Basic Structure
stages:
- build
- test
- deploy
build_job:
stage: build
script:
test_job:
stage: test
script:
deploy_job:
stage: deploy
21
script:
Each job is mapped to a stage, and each job contains at minimum a script:
section.
2. Defining Stages
Use the stages: keyword to define the order in which stages (and thus jobs)
run. All jobs within a stage will run in parallel, but stages themselves run
sequentially.
stages:
- lint
- build
- test
- deploy
All jobs in lint run → then build → then test → then deploy.
3. Writing Jobs
job_name:
stage: <stage_name>
script:
- <command 1>
- <command 2>
22
• artifacts: – files to preserve
Global image:
image: node:18
stages:
- test
test_job:
script:
- npm install
- npm test
Job-level image:
test_job:
image: python:3.9
script:
- pytest
23
5. Advanced Job Configuration
Parallel Jobs
Run multiple instances of the same job (e.g., for matrix builds):
test:
script: run_tests.sh
parallel: 5
Retry on Failure
job:
script: ./test.sh
retry: 2
Timeout
job:
script: heavy_task.sh
timeout: 20 minutes
job:
script: run.sh
only:
- main
job:
script: run.sh
rules:
- if: '$CI_COMMIT_BRANCH == "main"'
when: always
24
You can also use rules for dynamic pipelines (more on that later).
Artifacts:
build:
artifacts:
paths:
- build/
expire_in: 1 week
Cache:
cache:
paths:
- node_modules/
cache:
key: "$CI_COMMIT_REF_NAME"
paths:
- node_modules/
25
8. Using Variables
variables:
NODE_ENV: production
job:
script:
- echo $NODE_ENV
Also supports secrets via CI/CD variables UI, Vault, or group-level settings.
Local include:
include: 'templates/deploy.yml'
include:
- project: 'group/project'
file: '/templates/test.yml'
ref: 'main'
Multiple includes:
include:
- 'common.yml'
- 'docker-build.yml'
26
10. Best Practices
.default_job: &default_job
script:
job1:
<<: *default_job
job2:
<<: *default_job
27
12. Real-World Example: CI for Node.js App
image: node:18
stages:
- install
- test
- deploy
install_dependencies:
stage: install
script:
- npm install
cache:
paths:
- node_modules/
unit_tests:
stage: test
script:
- npm run test
deploy_prod:
stage: deploy
script:
- npm run build
- ./scripts/deploy.sh
environment:
name: production
url: https://yourapp.com
only:
- main
Directive Description
28
Directive Description
This section gives you a full grasp of how .gitlab-ci.yml is structured and how
to write clean, scalable, and powerful pipelines using GitLab’s DSL.
Artifacts
Artifacts are job outputs that you want to preserve after the job ends. They’re
useful for:
build:
script: npm run build
artifacts:
paths:
- dist/
expire_in: 1 week
29
🗃️ Caching
Caching helps speed up pipeline execution by reusing dependencies (e.g.,
node_modules/, .m2/, or .venv/).
cache:
paths:
- node_modules/
Use key: to control cache versioning and avoid invalidation:
cache:
key: "$CI_COMMIT_REF_SLUG"
Dependencies
The dependencies: keyword allows one job to fetch artifacts from a previous
job, even if they’re not in the same stage.
test:
stage: test
dependencies:
- build
deploy_staging:
environment:
name: staging
url: https://staging.myapp.com
30
Review Apps
deploy_review:
environment:
name: review/$CI_COMMIT_REF_NAME
url: https://$CI_ENVIRONMENT_SLUG.example.com
Rollbacks
rollback_prod:
stage: deploy
script: ./rollback.sh
when: manual
CI/CD Variables
script:
31
Security Scanning
• Dependency Scanning
• License Compliance
Includes
include:
- project: my-group/common
file: '/templates/deploy.yml'
Supports:
• Local paths
• Remote URLs
• Project files
Anchors
.default-job: &default_job
image: node:18
script:
- npm install
32
job1:
<<: *default_job
job2:
<<: *default_job
Monorepo Strategy
job:
script: ./run.sh
rules:
- changes:
- service-a/**
trigger:
trigger:
include: generated-pipeline.yml
Microservices Setup
33
• Keep Pipelines Fast
Use parallel jobs, proper caching, and limit unnecessary tasks.
• Fail Fast
Run lightweight checks (lint, format) early in the pipeline.
Parent-Child Pipelines
main:
trigger:
include: .child-pipeline.yml
Dynamic Pipelines
Matrix Builds
parallel:
matrix:
34
Custom Metrics & Observability
Use:
Cache is for speed. Artifacts are for passing files between jobs.
Security Gaps
Never expose secrets. Avoid using public runners for sensitive work.
GitLab CI/CD is one of the most powerful and flexible pipeline tools in the
DevOps ecosystem. Whether you're building simple web apps or managing
enterprise-scale microservices, GitLab provides everything you need:
35