Setup mesos

Abstract resources with Mesos

Think in datacenter scale, we have multiple servers with different specifications and capacities. One way to run a task say web application is to run a web application in a server. Seriously, do you think that its okay to use the whole server with 8GB memory to run an application which uses 1GB memory at max. So, we need a way for applications to specify the required amount of resources in order to successfully run. On the other hand we need a way for the servers to specify how much they are capable of in terms on cpu, memory, disk etc. If we have both these information flowing through a brain center which which again has to allocate and orchestrate the whole system to place the tasks at the right server so as to make full utilization of servers.

The brain center is Mesos, the fellow which sits on all servers / VM instances and gives capacity reports to brain is Mesos Agent. The tasks are any applications like web app or database.

In order to make mesos so generic they have delegated the brain process itself to frameworks. The frameworks has two components one on master (the scheduler) and other on agents (the executor). Now the Mesos master is actually a facilitator or middle man which provides

1. amount of resources that are available across datacenter and redirects the framework requests to required executors sitting along with mesos agent in a cluster node.
2. boilerplate functionalities like health checks, authentication etc to all frameworks

The mesos agents will provide the capacity information in the standard format having key : values just like json but there are types like scalar, float, range, text etc.

Example :

[
  {
    "name": "cpus",
    "type": "SCALAR",
    "scalar": {
      "value": 24
    }
  },
  {
    "name": "gpus",
    "type": "SCALAR",
    "scalar": {
      "value": 2
    }
  },
  {
    "name": "mem",
    "type": "SCALAR",
    "scalar": {
      "value": 24576
    }
  },
  {
    "name": "disk",
    "type": "SCALAR",
    "scalar": {
      "value": 409600
    }
  },
  {
    "name": "ports",
    "type": "RANGES",
    "ranges": {
      "range": [
        {
          "begin": 21000,
          "end": 24000
        },
        {
          "begin": 30000,
          "end": 34000
        }
      ]
    }
  },
  {
    "name": "bugs",
    "type": "SET",
    "set": {
      "item": [
        "a",
        "b",
        "c"
      ]
    },
    "role": "debug_role"
  }
]
# reference : http://mesos.apache.org/documentation/latest/attributes-resources/

Reference : Topics in mesos documentation

In the code repository cloned you will have a mesos folder which has the following scripts too.

TODO change the installation procedure from docker image to install in instance directly so as to maintain consistency with mesos agent installation procedure.

Installing mesos in master instance :

We are going to use the mesos docker image but we need some variables and configurations in place to start mesos.

CLUSTER_NAME=dc1
IP_ADDRESS=$(curl -s http://169.254.169.254/latest/meta-data/local-ipv4)

sudo tee /etc/docker/mesos-master <<-EOF
CONSUL_IP=localhost
DC=$CLUSTER_NAME
MESOS_HOSTNAME_LOOKUP=false
MESOS_IP=$IP_ADDRESS
MESOS_ZK=zk://$CLUSTER_NAME.zookeeper.service.consul:2181/mesos
MESOS_PORT=5050
MESOS_LOG_DIR=/var/log/mesos
MESOS_QUORUM=1
MESOS_WORK_DIR=/var/lib/mesos
MESOS_CLUSTER=$CLUSTER_NAME
EOF

Run the mesos master docker container

# Run the consul container with --env-file as environment variable to master with above configuration file
docker run --restart=always -d --name=mesos-master --net=host --env-file=/etc/docker/mesos-master flyinprogrammer/mesos-master

Check installation

docker logs mesos-master

Goto consul ui
http://<public-ip-of-mesos-master-instance>:8500/ui/#/dc1/services
You should see the mesos-master as one of the service in green tag

Troubleshooting

java.lang.UnsatisfiedLinkError: /usr/lib/libmesos-1.2.0.so: libcurl-nss.so.4: cannot open shared object file: No such file or directory

This error should not occur as we are installing mesos from apt package manager where mesos specifies libcurl4-nss as dependency so it will be downloaded and will be ready as shared library

docker stop mesos-master && docker rm mesos-master
mkdir mesos && cd mesos
wget https://raw.githubusercontent.com/flyinprogrammer/mesos-master/master/app.json .

tee Dockerfile <<-EOF
FROM flyinprogrammer/mesos-base
RUN apt-get -y update && \
    apt-get -y install libcurl4-nss-dev
COPY app.json /app.json
ENV CONTAINERPILOT=file:///app.json
EXPOSE 5050
CMD ["/bin/containerpilot", "mesos-master"]
EOF

docker build -t "my-mesos-master" .
docker run --restart=always -d --name=mesos-master --net=host --env-file=/etc/docker/mesos-master my-mesos-master

Installing mesos in agent instance :

We are going to use apt package manager to install marathon and container pilot as docker

Why we are installing mesos agent directly on instance rather than mesos docker?

Because, mesos has some code tied up with systemd (a init system for Unix) and it needs full access to systemd which does not happen every time via docker.

CLUSTER_NAME=dc1

# install mesos which installs zookeeper too
sudo tee /etc/apt/sources.list.d/mesosphere.list <<-EOF
deb http://repos.mesosphere.com/ubuntu xenial main
EOF

sudo apt-get update
sudo apt-get install --allow-unauthenticated -y mesos

# Stopping zookeeper and mesos master which is by default started after installation.
sudo systemctl stop zookeeper
sudo systemctl disable zookeeper
sudo systemctl stop mesos-master
sudo systemctl disable mesos-master

# Create configurations file for mesos slave
sudo tee /etc/default/mesos-slave <<-EOF
MESOS_CGROUPS_LIMIT_SWAP=false
MESOS_CGROUPS_ENABLE_CFS=true
MESOS_CONTAINERIZERS=docker,mesos
MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins
MESOS_ISOLATION=cgroups/cpu,cgroups/mem
MESOS_LOG_DIR=/var/log/mesos/agent
MESOS_MASTER=zk://$CLUSTER_NAME.zookeeper.service.consul:2181/mesos
MESOS_PORT=5051
MESOS_WORK_DIR=/var/lib/mesos/agent
MESOS_CLUSTER=$CLUSTER_NAME
EOF

# starting mesos slave
sudo systemctl start mesos-slave

TODO install container pilot within host instead of docker and create a systemd file to start mesos slave

# Container pilot configurations
sudo tee /etc/docker/cp-mesos-slave.json <<-EOF
{
    "consul": "localhost:8500",
    "logging": {
        "level": "INFO",
        "format": "default",
        "output": "stdout"
    },
    "services": [{
        "name": "mesos-agent",
        "port": 5051,
        "health": [
            "/usr/bin/curl",
            "--fail",
            "-s",
            "http://localhost:5051/health"
        ],
        "tags": ["$CLUSTER_NAME"],
        "poll": 10,
        "ttl": 30
    }]
}
EOF

docker run --restart=always -d --net=host -v /etc/docker/cp-mesos-slave.json:/app.json flyinprogrammer/cp

Check installation

Goto consul ui
http://<public-ip-of-mesos-master-instance>:8500/ui/#/dc1/services
You should see the mesos-agent as one of the service in green tag

Goto mesos ui
http://<public-ip-of-mesos-master-instance>:5050/#/agents
You should see mesos agents

Play around !