Final DevSecOps Enterprise Container Hardening Guide 1.2
Final DevSecOps Enterprise Container Hardening Guide 1.2
Version 1, Release 2
24 August 2022
UNCLASSIFIED
UNCLASSIFIED
Container Hardening Process Guide, V1R2 DISA
24 August 2022 Developed by DISA for the DOD
Trademark Information
Names, products, and services referenced within this document may be the trade names,
trademarks, or service marks of their respective owners. References to commercial vendors and
their products or services are provided strictly as a convenience to our users and do not constitute
or imply endorsement by the Defense Information Systems Agency of any nonfederal entity,
event, product, service, or enterprise.
ii
UNCLASSIFIED
UNCLASSIFIED
Container Hardening Process Guide, V1R2 DISA
24 August 2022 Developed by DISA for the DOD
TABLE OF CONTENTS
Page
1. OVERVIEW .............................................................................................................................1
1.1 Introduction .......................................................................................................................1
1.2 Definitions .........................................................................................................................2
1.2.1 DOD Hardened Containers (DHC) ..........................................................................2
1.2.2 Container Hardening Team (DHT) ..........................................................................2
1.2.3 DOD Container Factory (DCF) ...............................................................................3
1.2.4 Repo One/DOD Centralized Container Source Code Repository (DCCSCR) ........3
1.2.5 Iron Bank/DOD Centralized Artifacts Repository (DCAR) ....................................3
1.2.6 DOD Approved CI/CD Orchestration Solution (DACCOS) ...................................3
1.2.7 DOD Base Container Image Approved Source (DBCIAS) .....................................3
2. CONTAINER HARDENING PROCESS ..............................................................................5
2.1 Container Hardening Prerequisite Steps ............................................................................5
2.2 Container Hardening Process ............................................................................................6
2.3 Container Scanning Process ..............................................................................................8
2.4 Findings Mitigation Reporting ..........................................................................................9
2.5 Approving Process ...........................................................................................................11
2.6 Automated Hardening......................................................................................................13
2.7 Third-Party Container Sharing ........................................................................................13
2.8 Keycloak Integration .......................................................................................................13
3. APPENDIX A: FOLDER STRUCTURE EXAMPLE .......................................................15
3.1 Inside Repo One DCCSCR .............................................................................................15
3.2 Inside Iron Bank DCAR ..................................................................................................16
4. APPENDIX B: DOD HARDENED CONTAINERS CYBERSECURITY
REQUIREMENTS.................................................................................................................17
5. APPENDIX C: SECURITY CONTROL INHERITANCE ...............................................18
6. APPENDIX D: CONTAINER SECURITY/REMEDIATION CONSIDERATIONS ....19
6.1 False Positives Commentary............................................................................................19
7. APPENDIX E: VISUAL PROCESS FLOW .......................................................................20
7.1 Iron Bank Steps ...............................................................................................................20
8. APPENDIX E: CONTAINER BRANCHING STRATEGY .............................................23
iii
UNCLASSIFIED
UNCLASSIFIED
Container Hardening Process Guide, V1R2 DISA
24 August 2022 Developed by DISA for the DOD
LIST OF FIGURES
Page
Figure 1-1: DevSecOps Example.................................................................................................... 1
Figure 1-2: Iron Bank Repository ................................................................................................... 2
Figure 3-1: Example 1: Repo One DCCSCR ............................................................................... 15
Figure 3-2: Example 2: Iron Bank DCAR .................................................................................... 16
Figure 7-1: Visual Iron Bank Flow Diagram ................................................................................ 20
Figure 8-1: Container Branching Process ..................................................................................... 23
Figure 8-2: Process Flow .............................................................................................................. 24
iv
UNCLASSIFIED
UNCLASSIFIED
Container Hardening Process Guide, V1R2 DISA
24 August 2022 Developed by DISA for the DOD
LIST OF TABLES
Page
Table 2-1: Hardener Process ........................................................................................................... 7
Table 2-2: Scanning Process Steps ................................................................................................. 8
Table 2-3: Mitigation Recommendations ....................................................................................... 9
Table 2-4: Approved Container Deployment Process .................................................................. 11
v
UNCLASSIFIED
UNCLASSIFIED
Container Hardening Process Guide, V1R2 DISA
24 August 2022 Developed by DISA for the DOD
1. OVERVIEW
1.1 Introduction
This document focuses on the Department of Defense (DOD) Enterprise DevSecOps Initiative
(DSOP) and was created to detail the Enterprise DevSecOps Container Hardening Process and
ensure it meets the DOD Hardened Containers Cybersecurity Requirements. It is important to
understand both DevSecOps and cybersecurity concepts and principals, as well as have
knowledge of containers and container platforms. Refer to the Master Approach Document for
more information on how the DSOP platform functions.
Note: This document focuses on container security. It is understood that any application code or
library must be scanned by static/dynamic code analysis tools and passed or have
mitigated/accepted the risk prior to being integrated into a container for DOD use. If the
application is already approved for use (and scanned) by the Intelligence Community
(IC)/National Security Agency (NSA)/DOD Chief Information Officer (CIO)/Defense
Information Systems Agency (DISA), reciprocity can take effect. This document does not
describe that process.
1
UNCLASSIFIED
UNCLASSIFIED
Container Hardening Process Guide, V1R2 DISA
24 August 2022 Developed by DISA for the DOD
1.2 Definitions
In describing the hardening process, it is necessary to define concepts and terminology. This
section will define and describe items discussed in the process as well as other sections of this
document.
2
UNCLASSIFIED
UNCLASSIFIED
Container Hardening Process Guide, V1R2 DISA
24 August 2022 Developed by DISA for the DOD
For onboarding containers into Repo One, view the prerequisite information shown at
https://repo1.dsop.io/dsop/dccscr/tree/master/contributor-onboarding or view the process detail
in Section 2.
The artifact repository supports both files (traditional artifacts) and containers, as well as
container registry capabilities. It provides a secure mechanism to store, track, sign, and distribute
approved containers. It is accessible at: https://ironbank.dsop.io.
There are two types of repositories: trusted and untrusted. Trusted repositories are used as an
approved source for downloading container base images. Untrusted resources cannot be used in
3
UNCLASSIFIED
UNCLASSIFIED
Container Hardening Process Guide, V1R2 DISA
24 August 2022 Developed by DISA for the DOD
this fashion unless a Memorandum of Understanding (MOU) is reached between the DoD and
the vendor or mission partner using the untrusted source. The MOU will dictate strict levels of
security and ensure access to base images and their updates.
The following sources are currently approved and should be used in order of priority:
• Iron Bank/DCAR (approved DOD-wide; trusted).
• Product vendor proprietary repository (untrusted).
• Docker Hub (untrusted).
• Red Hat Container Repository (untrusted).
A trusted repository indicates that the containers were hardened by the DHT and can be used as
is. Refer to Section 2.1 for the vendor/mission partner prerequisite review steps before selecting
a base image.
4
UNCLASSIFIED
UNCLASSIFIED
Container Hardening Process Guide, V1R2 DISA
24 August 2022 Developed by DISA for the DOD
5
UNCLASSIFIED
UNCLASSIFIED
Container Hardening Process Guide, V1R2 DISA
24 August 2022 Developed by DISA for the DOD
DevSecOps Reference Design. The platform and hardened containers must be compliant
with the minimum viable product requirements or it cannot be accredited.
7. It is critical for containers to be able to be built on Tech Note: Platform One spends a great
classified and air gapped environments. Dockerfiles deal of time to ensure dependencies are
should not download any artifacts. Instead, a not downloaded without their knowledge.
prephase step called Download will occur so the Initially, Platform One will list all
dependencies, download them while
platform can download the required dependencies
connected to internet, go Offline, and
online. Then, the rest DOD Container Hardening build the container validating the
process will operate in a disconnected and/or dependency downloads are not attempted.
classified environment that does not have internet This critical security process ensures
access. This will ensure the process will not be able decoupling the download from the build
process.
to pull files from the internet that are required to
build.
8. If an organization that has a capability, has two license options (for example, open source
and an enterprise managed version), both containers must be provided to Repo One. The
organization cannot only provide the paid version. The DOD Hardening Team will not
accept enterprise containers if the open-source container is not also provided.
It is important to understand that containers must run on a DevSecOps Platform compliant with
the DoD Enterprise DevSecOps Ref Design and its MVP requirements, including the Sidecar
Container Security Stack, which enforces behavior detection and zero trust in runtime.
Without a CNCF-compliant Kubernetes platform and the MVP Ref Design capabilities, a
standalone container is not accredited and its certificate to field does not apply.
The container hardening and scanning pipeline is located at https://repo1.dsop.io/dsop. Note that
third parties are not allowed master branch write access to the Repo One repository; only
members of the Hardening Team are authorized to make changes that will affect hardened
containers including, Dockerfiles, readme files, license files, and anything else required for the
scanning and hardening of the container. A Container Hardener will approve the container into
the Repo One and Iron Bank repository after validation and approval. This process should also
support free and open-source software (FOSS) as well as commercial off-the-shelf (COTS)
software.
Using the container software factory or pipeline, the Hardener will begin the hardening process
as depicted in the Figure 7-1 sub-process titled “Begin Hardener process”. The detailed steps and
examples are shown in Table 2-1 below.
6
UNCLASSIFIED
UNCLASSIFIED
Container Hardening Process Guide, V1R2 DISA
24 August 2022 Developed by DISA for the DOD
7
UNCLASSIFIED
UNCLASSIFIED
Container Hardening Process Guide, V1R2 DISA
24 August 2022 Developed by DISA for the DOD
8
UNCLASSIFIED
UNCLASSIFIED
Container Hardening Process Guide, V1R2 DISA
24 August 2022 Developed by DISA for the DOD
9
UNCLASSIFIED
UNCLASSIFIED
Container Hardening Process Guide, V1R2 DISA
24 August 2022 Developed by DISA for the DOD
Vendor/Partner
Step Process Description Resource
Recommended Action
1d Emit logs to stdout. Logs must be emitted to stdout or Hardener
accessible locations.
1e User privileges Do not run with elevated privileges. Hardener
Root ID (UID 0) must not be used
unless risk accepted and documented.
1f Decouple configurations to make them Use arguments, environment variables Hardener
more portable. Use secrets for and configmaps for configurations and
sensitive information such as tokens, key value pairs. Also, use encoding
keys, and passwords. such as base64 to obfuscate and
encrypt yaml secrets.
1g User access Integrate with the DoD hardened Hardener
Keycloak SSO stack for single sign on
capability or Require Common Access
Card (CAC) access through SAML
when possible. If not possible, MFA
may be used based on risk acceptance.
1h Ensure secure connections are Use TLS and secure certificates when Hardener
leveraged. connecting with other services and
containers. Refer to 1b.
1i Remove unnecessary packages, Remove anything not core to the Hardener
services, and applications. operation of the container.
1j Remove User and Group permissions. Remove the setuid and setgid Hardener
permissions so that file permissions
cannot be modified for malicious
purposes.
1k Comply with DoD Hardened Refer to Appendix B: DoD hardened Hardener
Containers Cybersecurity Containers Cybersecurity
Requirements. Requirements.
1l Avoid command injections. Be cautious with command injections Hardener
in RUN, particularly when using ENV
or ARGs.
1m Leverage Kubernetes secrets Ensure that Kubernetes Secrets and/or Hardener
secrets vault are used to store secrets.
2 Perform a container HEALTHCHECK Hardener
to ensure proper overall functionality
as well as the health and readiness of
the container.
10
UNCLASSIFIED
UNCLASSIFIED
Container Hardening Process Guide, V1R2 DISA
24 August 2022 Developed by DISA for the DOD
The Approver will reach a decision on each container via a process that depicts the number and
severity of findings from the reporting performed in Section 2.4. For example, if the number of
findings is high but most are high to low, the approver may reject the container altogether due to
the magnitude of findings. If the Approver determines a critical finding exists that is key to
keeping the DOD secure, the container may be rejected. Conversely, if the finding report dictates
that the container only has a few low to medium findings, the container may be approved in its
entirety.
The level of acceptable risk for cybersecurity findings that cannot be remediated at the time of
processing will be decided on a case-by-case basis. These will all be reviewed by the Iron Bank
or the PMO Authorizing Official (AO).
Regardless of the outcomes from the Approval process, all findings and recommendations are
shared with the mission partner or vendor. If a container is placed in the Approval Time Limit
process, the container is placed into a file with a trigger for rerun at X time and date. For any
containers approved, the following steps will be performed.
11
UNCLASSIFIED
UNCLASSIFIED
Container Hardening Process Guide, V1R2 DISA
24 August 2022 Developed by DISA for the DOD
12
UNCLASSIFIED
UNCLASSIFIED
Container Hardening Process Guide, V1R2 DISA
24 August 2022 Developed by DISA for the DOD
Note: Ensure any changes to the Repo One folder are changed in Iron Bank as well. If a
Dockerfile changes, the new artifact must go through the hardening process again.
If a new CVE is detected in an upgrade, the build will stop and a manual assessment of the new
CVE will be conducted.
It is allowed to leverage binaries (as long as they are downloaded by the Download.json or
Download.yaml script) in Dockerfile for the application layer but the source of the Dockerfile
must be provided so the container can be rebuilt. The goal is to integrate existing Dockerfiles
into the DOD hardening process and Repo One so that new containers benefit from the same
automation.
As Keycloak is fully integrated with the DOD Active Directory Federation Services
(ADFS)/PKI, it is strongly recommended that any system operating in Impact Level 4 or higher
13
UNCLASSIFIED
UNCLASSIFIED
Container Hardening Process Guide, V1R2 DISA
24 August 2022 Developed by DISA for the DOD
use Keycloak for any user authentication. For nonperson entity-to-entity communication, signed
x509 certificates and PKI should be used.
14
UNCLASSIFIED
UNCLASSIFIED
Container Hardening Process Guide, V1R2 DISA
24 August 2022 Developed by DISA for the DOD
15
UNCLASSIFIED
UNCLASSIFIED
Container Hardening Process Guide, V1R2 DISA
24 August 2022 Developed by DISA for the DOD
16
UNCLASSIFIED
UNCLASSIFIED
Container Hardening Process Guide, V1R2 DISA
24 August 2022 Developed by DISA for the DOD
17
UNCLASSIFIED
UNCLASSIFIED
Container Hardening Process Guide, V1R2 DISA
24 August 2022 Developed by DISA for the DOD
PaaS/Platform Host providers rely on technical concepts such as process isolation, service
routing, redundancy, firewall controls, and many other security layers. These typically align with
DOD security concepts “out of the box”. Security overlays should always be leveraged (STIG,
NIST 800-53, etc.). Ensure the selected host operating system has significant DISA STIG
involvement, including security guidance/standards where available.
Leverage hardened Infrastructure as Code and Configuration as Code from Repo One whenever
available to install various Kubernetes distributions supported by the Container Hardening team.
With a properly locked down hosting environment, containers inherit most of the security
controls and benefits from infrastructure to host OS-level remediation requirements. The focus
then shifts to container security requirements and application security requirements (e.g., static
code analysis, unit testing, library Common Vulnerabilities and Exposure [CVE] testing, etc.).
18
UNCLASSIFIED
UNCLASSIFIED
Container Hardening Process Guide, V1R2 DISA
24 August 2022 Developed by DISA for the DOD
For false positives or nonapplicable items, there are two options: either remediate to force the
scanner to detect it as corrected or modify scanning profiles to mark specific components as
nonapplicable. For example, there are 363 configuration settings applicable to RHEL 7 base OS,
and 93 of these controls are not inherited and/or applicable. If any of the 93 controls need to be
remediated, 85 of them can be automated. Always automate all container-hardening processes
leveraging CI/CD pipelines whenever possible. Manual processes should be avoided if possible.
As stated previously, it is very important to select a proper base image. Refer to Section 2.1
Container Hardening Pre-Process Steps.
For example:
• If a container requires a base OS (such as RHEL/Alpine), the DoD hardened-base image
of these OSs must be used.
• If a container needs OpenJDK such as a Java application or a COTS product using Java,
the DOD OpenJDK base image must be reused whenever possible. This ensures that
hardening of JDK is centralized and consolidated across all containers by leveraging
inheritance with those base images.
19
UNCLASSIFIED
UNCLASSIFIED
Container Hardening Process Guide, V1R2 DISA
24 August 2022 Developed by DISA for the DOD
20
UNCLASSIFIED
UNCLASSIFIED
Container Hardening Process Guide, V1R2 DISA
24 August 2022 Developed by DISA for the DOD
d. The action of merging into the development branch will inform the CI server to start
orchestrating the pipeline.
e. The first CI Runner will have egress to the Internet. It will look at the Download.json
file to identify the components necessary to pull the contributors information into the
environment securely. Once in the environment signatures and checksums will be
validated to ensure providence.
f. Each item downloaded will be sent through a Clam AV scan. If there are threats
identified, the download will be quarantined.
g. If there are no threats detected, dependencies will be pushed into a private Container
Registry.
2. Build Container
a. After dependencies have been validated, the CI server will start another CI Runner
without any egress to perform the build operations.
b. The runner will connect to the Container Registry and pull down the scanned
dependencies.
c. The runner will build the contributor's container without Internet access.
d. After a successful build, the container will be pushed into the Container Registry
3. Evaluate Container
a. After a successful build, the runner will execute an OpenSCAP/InSpec,
Prisma/StackRox, and Anchore scans.
b. Results of the scans will be uploaded to the Vulnerability Assessment Tracker (VAT)
c. Contributor will connect to the VAT and justify any findings from the scans.
d. Iron Bank CVE Approvers will review all the justifications submitted and validate the
information as accurate and appropriate to satisfy the finding.
4. Approve Container
a. With the findings of the scans validated, the AO or the AO Designated Representative
(DR) will review the entire body of evidence and make the decision to approve the
container.
5. Publish Container
a. Once approved, the Vulnerability Assessment Tracker (VAT) will merge the
Development Branch into the Master Branch.
b. This will trigger the CI server to start a publish pipeline. Another CI Runner without
egress will be started to perform the actions.
c. The runner will pull the container in, sign the container, generate a checksum, and
pull the body of evidence.
d. The package will be pushed to multiple locations segregating the check sum and
public key from the container and body of evidence.
6. Deliver Container
a. The Iron Bank application will retrieve container information from the storage
location utilized by the VAT.
b. Users will be able to access the Iron Bank application and obtain the body of
evidence and containers.
21
UNCLASSIFIED
UNCLASSIFIED
Container Hardening Process Guide, V1R2 DISA
24 August 2022 Developed by DISA for the DOD
c. Kubernetes clusters will have a container that connects to the Iron Bank or Registry
One (new container registry) authenticating with a machine-to-machine certificate
and synchronize containers.
22
UNCLASSIFIED
UNCLASSIFIED
Container Hardening Process Guide, V1R2 DISA
24 August 2022 Developed by DISA for the DOD
23
UNCLASSIFIED
UNCLASSIFIED
Container Hardening Process Guide, V1R2 DISA
24 August 2022 Developed by DISA for the DOD
24
UNCLASSIFIED