13
Feb

Monitoring Docker containers with cAdvisor from Marathon

Problem:

Monitor resource utilisation of Docker containers in a Mesos cluster. This is useful when deciding how much CPU and Memory to give to each container or for understanding when to scale up / down.

Solution:

cAdvisor is a simple to use monitoring tool for Docker containers. It provides a Docker container ready to run on each of the Mesos slaves.

With Marathon and Mesos is very easy to deploy a cAdvisor agent on each of the slaves. Marathon allows you to define constraints to make sure you can distribute the cAdvisor container evenly across all the Mesos slaves.

Bellow is the body of the HTTP POST request to be made to Marathon and deploy cAdvisor.

{
  "container": {
    "type": "DOCKER",
    "docker": {
      "image": "google/cadvisor:latest"
    },
    "volumes": [
      {
        "containerPath": "/rootfs",
        "hostPath": "/",
        "mode": "RO"
      },
      {
        "containerPath": "/var/run",
        "hostPath": "/var/run",
        "mode": "RW"
      },
      {
        "containerPath": "/sys",
        "hostPath": "/sys",
        "mode": "RO"
      },
      {
        "containerPath": "/var/lib/docker",
        "hostPath": "/var/lib/docker",
        "mode": "RO"
      },
      {
        "containerPath": "/cgroup",
        "hostPath": "/cgroup",
        "mode": "RO"
      }
    ],
    "network": "BRIDGE",
    "portMappings": [
      { "containerPort": "0.0.0.0:8080", "hostPort": "0.0.0.0:8080", "protocol": "tcp" }
    ]
  },
  "id": "cadvisor",
  "instances": 1,
  "cpus": 0.5,
  "mem": 512,
  "constraints": [
    [
      "hostname",
      "UNIQUE"
    ]
  ],
  "ports": [
    8080
  ]
}

If the Mesos slaves run on CentOS 7 as in my case, then you also need to open port 8080 in the firewall. The following commands allow you to do so:

 

$ firewall-cmd --zone=public --add-port=8080/tcp --permanent
$ firewall-cmd --reload

 

UPDATE: Also, make sure IP forward is ON when you run Docker in BRIDGE mode:

$ sysctl -w net.ipv4.ip_forward=1

Once the deployment is complete, you can access the cAdvisor UI from each of the slaves on port 8080.

cAdvisor-screenshot

 

2 Comments

  • samek Said:

    How did you solve the naming ?
    For example you have nginx dockers started all over the place, they get marathon name like ‘mesos-802b92c7-af76-4127-9a68-90e77eda6426’

    How would you then group them by service when streaming data to influxdb ?

  • Dragos Said:

    I didn’t get to influxdb yet, but if you did and found a solution to your problem I’d be interested to know more.

Sorry, the comment form is closed at this time.

Comments RSS Feed