Container Security Scanning: A Complete Pipeline

July 11, 2022

Every container image includes an operating system, libraries, and your code. Each component can have vulnerabilities. Without scanning, you’re shipping unknown risks to production. A good scanning pipeline catches issues early and often.

Here’s how to build container security scanning that works.

What to Scan For

Vulnerability Types

container_vulnerabilities:
  os_packages:
    examples: OpenSSL, glibc, curl
    source: Base image packages
    frequency: Very common

  language_dependencies:
    examples: npm packages, Python libraries
    source: Application dependencies
    frequency: Common

  misconfigurations:
    examples: Running as root, exposed secrets
    source: Dockerfile/runtime config
    frequency: Very common

  malware:
    examples: Cryptominers, backdoors
    source: Compromised images/packages
    frequency: Rare but severe

  secrets:
    examples: API keys, passwords in layers
    source: Build process mistakes
    frequency: Common

Scanning Tools

Trivy

# Trivy - comprehensive scanner
trivy_capabilities:
  scans:
    - OS vulnerabilities
    - Language dependencies
    - Misconfigurations
    - Secrets

  formats:
    - Images (local and remote)
    - Filesystem
    - Git repositories
    - Kubernetes manifests
# Scan local image
trivy image myapp:latest

# Scan with severity filter
trivy image --severity HIGH,CRITICAL myapp:latest

# Output as JSON for CI
trivy image --format json --output results.json myapp:latest

# Scan filesystem during build
trivy fs --security-checks vuln,config .

# Scan IaC files
trivy config ./terraform/

Grype

# Grype - vulnerability scanner
grype myapp:latest

# With SBOM (faster subsequent scans)
syft packages myapp:latest -o json > sbom.json
grype sbom:sbom.json

# CI integration
grype myapp:latest --fail-on high

Dockerfile Linting

# Hadolint - Dockerfile best practices
hadolint_rules:
  DL3000: Use absolute WORKDIR
  DL3001: No SSH keys in image
  DL3002: Don't set USER root
  DL3003: Use WORKDIR for cd
  DL3006: Pin image versions
  DL3007: Don't use latest tag
  DL3008: Pin package versions
# Lint Dockerfile
hadolint Dockerfile

# Ignore specific rules
hadolint --ignore DL3008 Dockerfile

# CI integration
hadolint Dockerfile || exit 1

CI/CD Integration

GitHub Actions

name: Container Security

on:
  push:
    branches: [main]
  pull_request:

jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Lint Dockerfile
        uses: hadolint/hadolint-action@v3.1.0
        with:
          dockerfile: Dockerfile

      - name: Build image
        run: docker build -t myapp:${{ github.sha }} .

      - name: Scan for vulnerabilities
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: myapp:${{ github.sha }}
          format: 'sarif'
          output: 'trivy-results.sarif'
          severity: 'HIGH,CRITICAL'

      - name: Upload scan results
        uses: github/codeql-action/upload-sarif@v2
        with:
          sarif_file: 'trivy-results.sarif'

      - name: Fail on critical vulnerabilities
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: myapp:${{ github.sha }}
          exit-code: '1'
          severity: 'CRITICAL'

GitLab CI

stages:
  - build
  - scan
  - deploy

build:
  stage: build
  script:
    - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA

container_scan:
  stage: scan
  image:
    name: aquasec/trivy:latest
    entrypoint: [""]
  script:
    - trivy image --exit-code 0 --severity HIGH $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    - trivy image --exit-code 1 --severity CRITICAL $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
  artifacts:
    reports:
      container_scanning: gl-container-scanning-report.json

deploy:
  stage: deploy
  needs: [container_scan]
  script:
    - kubectl set image deployment/app app=$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA

Registry Scanning

Continuous Registry Scanning

registry_scanning:
  why:
    - New CVEs discovered daily
    - Images in registry age
    - Catch issues in deployed images

  implementation:
    - Scan on push
    - Scheduled rescanning
    - Alert on new findings
# Harbor registry scanning policy
apiVersion: harbor.io/v1alpha1
kind: ScanPolicy
metadata:
  name: daily-scan
spec:
  schedule: "0 2 * * *"  # 2 AM daily
  includeUnscanned: true
  projectSelectors:
    - repository: "*"
  vulnerabilitySeverity: High
  notificationTargets:
    - slack: "#security-alerts"

Image Signing

image_signing:
  purpose: Verify image authenticity and integrity
  tools:
    - cosign (Sigstore)
    - Notary
    - Docker Content Trust

  workflow:
    - Build image
    - Scan for vulnerabilities
    - If pass, sign image
    - Registry/runtime verifies signature
# Sign with cosign
cosign sign --key cosign.key myregistry.io/myapp:v1.0.0

# Verify signature
cosign verify --key cosign.pub myregistry.io/myapp:v1.0.0

# Keyless signing (Sigstore)
cosign sign myregistry.io/myapp:v1.0.0
# Uses OIDC identity, logs to Rekor transparency log

Admission Control

Preventing Unscanned Images

# Kubernetes admission controller
# OPA Gatekeeper policy
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sImageScanned
metadata:
  name: require-scanned-images
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
    namespaces: ["production"]
  parameters:
    allowedRegistries:
      - "myregistry.io"
    maxCriticalVulns: 0
    maxHighVulns: 5
# Kyverno policy
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-image-signature
spec:
  validationFailureAction: enforce
  rules:
    - name: verify-signature
      match:
        resources:
          kinds:
            - Pod
      verifyImages:
        - image: "myregistry.io/*"
          key: |-
            -----BEGIN PUBLIC KEY-----
            ...
            -----END PUBLIC KEY-----

Vulnerability Management

Prioritization

vulnerability_prioritization:
  critical:
    criteria:
      - CVSS >= 9.0
      - Actively exploited
      - Network exploitable
    action: Immediate patch required

  high:
    criteria:
      - CVSS >= 7.0
      - Exploit available
    action: Patch within 7 days

  medium:
    criteria:
      - CVSS >= 4.0
    action: Patch within 30 days

  low:
    criteria:
      - CVSS < 4.0
    action: Address in regular updates

  false_positives:
    - Document accepted risk
    - Use .trivyignore for known acceptable

Exception Management

# .trivyignore - Documented exceptions
# CVE-2021-12345: False positive, function not used
CVE-2021-12345

# CVE-2021-67890: Accepted risk until vendor fix available
# Expires: 2022-09-01
CVE-2021-67890

Secure Base Images

Minimal Images

# BAD: Full OS
FROM ubuntu:22.04
RUN apt-get update && apt-get install -y python3

# BETTER: Slim image
FROM python:3.11-slim

# BEST: Distroless
FROM gcr.io/distroless/python3-debian11

Hardened Images

hardened_images:
  sources:
    - Google Distroless
    - Chainguard Images
    - Red Hat UBI
    - AWS ECR Public

  benefits:
    - Minimal attack surface
    - Regular updates
    - Security-focused

Key Takeaways

Shift security left. The earlier you find vulnerabilities, the cheaper they are to fix.