Container Orchestration: Docker Swarm vs Kubernetes vs Mesos

Running a few containers on a single host is straightforward. Running hundreds of containers across dozens of hosts—handling scheduling, networking, health checks, scaling, and updates—requires orchestration.

Three platforms dominate container orchestration in 2016: Docker Swarm, Kubernetes, and Mesos with Marathon. Each takes a different approach to the same fundamental problems. Understanding their tradeoffs helps you choose the right platform for your needs.

What Orchestration Provides

Before comparing platforms, let’s establish what container orchestration actually does:

Scheduling: Placing containers on hosts based on resource requirements, constraints, and availability.

Service discovery: Enabling containers to find each other despite dynamic scheduling.

Load balancing: Distributing traffic across container instances.

Health monitoring: Detecting unhealthy containers and replacing them.

Scaling: Adjusting container count based on demand or configuration.

Rolling updates: Deploying new versions without downtime.

Networking: Connecting containers across hosts with consistent addressing.

Storage: Attaching persistent storage to containers.

All three platforms provide these capabilities. They differ in architecture, complexity, and operational characteristics.

Docker Swarm

Docker Swarm provides native clustering for Docker. It extends the Docker API to operate on clusters rather than single hosts.

Architecture

Swarm uses a simple architecture:

• Manager nodes: Handle cluster management, scheduling, and API requests
• Worker nodes: Run containers

Managers use Raft consensus for leadership election and state replication. The cluster tolerates manager failures with enough managers (3 or 5 for production).

Strengths

Simplicity: Swarm is the easiest orchestrator to learn. The API is Docker’s familiar API. Existing Docker commands work with minimal changes.

Docker integration: Swarm is built into Docker Engine (since 1.12). No separate installation. Native integration with Docker Compose for service definitions.

Low overhead: Managers and workers are lightweight. Swarm can run on smaller instances than Kubernetes.

Good enough for many use cases: For straightforward deployments—a handful of services, basic scaling, rolling updates—Swarm provides what you need without complexity overhead.

Weaknesses

Limited features: Swarm lacks advanced features found in Kubernetes: pods, custom resource types, sophisticated scheduling constraints.

Smaller ecosystem: Fewer integrations, tools, and community resources compared to Kubernetes.

Unclear future: Docker’s focus has shifted. Swarm development continues but at slower pace than Kubernetes.

Swarm Configuration

Deploying a service in Swarm:

# Create service with 3 replicas
docker service create --name web --replicas 3 -p 80:80 nginx

# Scale service
docker service scale web=5

# Rolling update
docker service update --image nginx:1.11 web

Or using Docker Compose v3:

version: '3'
services:
  web:
    image: nginx
    deploy:
      replicas: 3
      update_config:
        parallelism: 1
        delay: 10s

Kubernetes

Kubernetes (K8s) is Google’s container orchestrator, open-sourced in 2014 and now governed by the Cloud Native Computing Foundation (CNCF).

Architecture

Kubernetes has a more complex architecture:

Control plane:

• API Server: Entry point for all operations
• etcd: Distributed key-value store for cluster state
• Scheduler: Assigns pods to nodes
• Controller Manager: Runs controllers that manage cluster state

Worker nodes:

• Kubelet: Manages containers on the node
• Kube-proxy: Handles networking rules
• Container runtime: Actually runs containers (Docker, containerd)

Core Concepts

Kubernetes introduces abstractions beyond containers:

Pod: Smallest deployable unit. Contains one or more containers that share network and storage.

Service: Stable network endpoint for a set of pods. Provides load balancing and service discovery.

Deployment: Declares desired state for pods. Handles rolling updates and rollbacks.

ConfigMap/Secret: External configuration for pods.

Namespace: Logical isolation for resources within a cluster.

Strengths

Feature-rich: Kubernetes handles sophisticated scenarios: stateful workloads, batch processing, complex networking, custom scheduling.

Extensibility: Custom Resource Definitions (CRDs) extend Kubernetes with new resource types. Operators encode operational knowledge.

Ecosystem: Massive community, extensive integrations, abundant tooling. If you need something, someone’s probably built it.

Cloud support: All major cloud providers offer managed Kubernetes (GKE, EKS, AKS). This reduces operational burden significantly.

Industry momentum: Kubernetes is becoming the standard. Skills transfer between organizations. Job market values Kubernetes experience.

Weaknesses

Complexity: Kubernetes has steep learning curve. Many concepts, many configuration options, many things to understand.

Operational overhead: Running Kubernetes yourself requires significant expertise. Upgrades, networking, storage, security all need attention.

Resource overhead: Control plane components consume resources. Small deployments may find the overhead disproportionate.

Kubernetes Configuration

Deployment in Kubernetes:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: nginx
        image: nginx:1.11
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector:
    app: web
  ports:
  - port: 80
  type: LoadBalancer

Mesos with Marathon

Apache Mesos is a distributed systems kernel that abstracts CPU, memory, storage, and other resources. Marathon is a container orchestration platform that runs on Mesos.

Architecture

Mesos:

• Master: Coordinates resource offers to frameworks
• Agent: Runs on each node, reports resources
• Framework: Applications that use Mesos resources (Marathon is one framework)

Marathon:

• Schedules long-running services on Mesos
• Handles health checks, scaling, and rolling updates
• Provides REST API for management

Strengths

Proven at scale: Mesos runs at massive scale at companies like Twitter, Airbnb, and Apple. It’s proven for tens of thousands of nodes.

Mixed workloads: Mesos supports different frameworks for different workloads: Marathon for services, Chronos for scheduled jobs, Spark for analytics. One cluster, multiple workload types.

Resource efficiency: Two-level scheduling (Mesos offers resources, frameworks decide what to run) enables sophisticated resource optimization.

Weaknesses

Complexity: Mesos + Marathon has more moving pieces than Swarm. Not as complex as Kubernetes, but not simple.

Declining mindshare: The industry is consolidating around Kubernetes. Mesos community is smaller and less active than Kubernetes.

Two systems: Mesos provides resources; Marathon provides orchestration. Understanding both is necessary.

Marathon Configuration

Application in Marathon:

{
  "id": "/web",
  "instances": 3,
  "container": {
    "type": "DOCKER",
    "docker": {
      "image": "nginx:1.11",
      "network": "BRIDGE",
      "portMappings": [
        {"containerPort": 80, "hostPort": 0}
      ]
    }
  },
  "healthChecks": [
    {"protocol": "HTTP", "path": "/", "portIndex": 0}
  ]
}

Comparison Matrix

Making the Choice

Choose Docker Swarm When:

• You want the fastest path to running containers in production
• Your workloads are straightforward (web services, APIs)
• You have limited operational capacity
• You’re already familiar with Docker

Choose Kubernetes When:

• You need sophisticated orchestration features
• You’re building for significant scale
• You want to use managed cloud services
• You’re willing to invest in learning
• You want skills that transfer across organizations

Choose Mesos/Marathon When:

• You’re running at massive scale
• You need mixed workloads (services, batch, analytics) on one cluster
• You have existing Mesos infrastructure
• You have operational expertise for complex systems

The Realistic View

For most organizations starting with containers in 2016, the choice is between Swarm and Kubernetes:

Swarm for simplicity. Get containers running quickly without significant learning investment. Accept that you might outgrow it.

Kubernetes for capability. Invest in learning. Either use managed services or dedicate operational resources. Benefit from ecosystem and industry momentum.

Mesos/Marathon makes sense for organizations with existing Mesos infrastructure or requirements at scale that few organizations actually have.

The market is consolidating around Kubernetes. If you’re choosing for the long term, Kubernetes is the safer bet. If you need to ship containers next month with minimal learning, Swarm gets you there faster.

Key Takeaways

• All three platforms solve the same fundamental problems: scheduling, discovery, scaling, updates
• Docker Swarm offers simplicity and Docker integration at the cost of features
• Kubernetes offers comprehensive features and ecosystem at the cost of complexity
• Mesos/Marathon offers proven scale and mixed workloads for teams with operational expertise
• For most teams, the practical choice is between Swarm (simplicity) and Kubernetes (capability)
• Kubernetes is gaining dominant market share; long-term bets favor Kubernetes