13
Feb

Monitoring Docker containers with cAdvisor from Marathon

Problem:

Monitor resource utilisation of Docker containers in a Mesos cluster. This is useful when deciding how much CPU and Memory to give to each container or for understanding when to scale up / down.

Solution:

cAdvisor is a simple to use monitoring tool for Docker containers. It provides a Docker container ready to run on each of the Mesos slaves.

With Marathon and Mesos is very easy to deploy a cAdvisor agent on each of the slaves. Marathon allows you to define constraints to make sure you can distribute the cAdvisor container evenly across all the Mesos slaves.

Bellow is the body of the HTTP POST request to be made to Marathon and deploy cAdvisor.

{
  "container": {
    "type": "DOCKER",
    "docker": {
      "image": "google/cadvisor:latest"
    },
    "volumes": [
      {
        "containerPath": "/rootfs",
        "hostPath": "/",
        "mode": "RO"
      },
      {
        "containerPath": "/var/run",
        "hostPath": "/var/run",
        "mode": "RW"
      },
      {
        "containerPath": "/sys",
        "hostPath": "/sys",
        "mode": "RO"
      },
      {
        "containerPath": "/var/lib/docker",
        "hostPath": "/var/lib/docker",
        "mode": "RO"
      },
      {
        "containerPath": "/cgroup",
        "hostPath": "/cgroup",
        "mode": "RO"
      }
    ],
    "network": "BRIDGE",
    "portMappings": [
      { "containerPort": "0.0.0.0:8080", "hostPort": "0.0.0.0:8080", "protocol": "tcp" }
    ]
  },
  "id": "cadvisor",
  "instances": 1,
  "cpus": 0.5,
  "mem": 512,
  "constraints": [
    [
      "hostname",
      "UNIQUE"
    ]
  ],
  "ports": [
    8080
  ]
}

If the Mesos slaves run on CentOS 7 as in my case, then you also need to open port 8080 in the firewall. The following commands allow you to do so:

 

$ firewall-cmd --zone=public --add-port=8080/tcp --permanent
$ firewall-cmd --reload

 

UPDATE: Also, make sure IP forward is ON when you run Docker in BRIDGE mode:

$ sysctl -w net.ipv4.ip_forward=1

Once the deployment is complete, you can access the cAdvisor UI from each of the slaves on port 8080.

cAdvisor-screenshot

 

02
Feb

An easy way to mount a local directory in a Docker container via Boot2Docker

Problem:

On an OSX the docker process runs in the VirtualBox VM causing all mounts to be created inside that VM and not on a local host.

So starting a docker container with a mounting option

docker run -v /tmp/folder:/usr/local/my-data ...

means that the /tmp/folder within the VM is mounted to the docker container and not the folder on the local computer.


Solution:

VirtualBox Guest Additions¬†- The ${HOME} on the local machine is automatically mounted on the VM running docker, keeping the same path: /Users/<user>/…

This means we can create a folder inside the home dir and mount it to the docker container, via the Boot2Docker VM. My script looks like:

mkdir -p ~/tmp/my-data
docker run --volume=${HOME}/tmp/my-data:/usr/local/my-data ...

18
Apr

Verifying SSL certs after Heartbleed

After the recent heartbleed storm that hit our beloved wide web, a lot of people and companies have revocated and then reissued new SSL certificates.

In order to verify that the certs have been correctly setup, here’s a simple command that I used to check the date of the certs:

$ openssl s_client -connect google.com:443 < /dev/null 2>/dev/null \
     | openssl x509 -startdate -noout

This commands outputs today:

notBefore=Apr 9 12:05:08 2014 GMT