Container pilot

Helping agent placed inside a container

We use container pilot an implementation for Autopilot pattern, which sits on every container and solves

• the service registration to consul
• docker PID 1 problem

The service registration to consul

The containers that we spawn must be registered with local consul agent waiting in host instance which will propagate the service information across the clusters. But, how can we register a service in consul?

One possible approach can be to hit the consul rest api for registering the service, but there are three problems !

1. How can we re-register whenever the container dies and spawn again, because we do it on startup (one time)
2. Another problem with this approach is its does not give the exact status of all the services
3. What about the side cars (services running in the same container). If container pilot finds that one of the side cars are failing it sends events to consul that particular service is unhealthy and restart the side car.

Docker PID 1 problem

For explaining this we need to go into some os internals :

Code

Text which are converted to instructions which intern is understood by the operating system for execution

Process

A cpu sets all its context from process to execute an instruction. Context switch needs some memory swapping because they do not share the same memory space.

Thread

A lightweight process within a process which does memory sharing with other thread within same process so switching execution between threads are faster.

Concurrency

When the time is shared by processes for execution in the same cpu

Parallelism

When you have multiple cpus itself and each takes care of a process

Process Groups

Multiple processes can be spawned a process which falls into same process group. First process to enter a process group is group leader who can take control of terminal

Sessions

Multiple process groups are grouped under same session. First process to enter a session becomes a session leader and has more priority than group leaders in controlling the terminal

Foreground process

The process which runs interacts with the controlling terminal.

Background process

The process is demonized and runs under the hood, cannot interact with the controlling terminal. Cannot stop by CTRL+C !

Parent process

The process can fork itself the give birth to child processes which can run independently but has to give a waitpid() system call for the child process so that it can read the exit status and other process parameters later.

Child process

The child process is born takes some space in process table, does some work example grep from bash and dies after executing. When the parent gives a waitpid() the child is reaped after parent reading the required status.

Process table

It keeps record of all process id and parent and other related fields in a table format. There is a restriction on number of process that can be spawned by a user. In unix ulimit -u it gives around 800 beyond which you cannot create anymore processes and which means zombie processes can fill this limit and normal well-behaved processes cannot start too.

Zombie process

When parent is not well behaving by reaping their children after execution with waitpid(). Child process stays in the process table and possible there might be a resource leak and starvation of resources too.

Orphan process

When the parent process dies (happens because of bugs, errors) and so the child process after coming back from execution will become an orphan.

Init process

The first process PID 1 which spawns other processes like the sshd used for sshing the machine. The init process is responsible for the reaping the orphan child processes by adopting them and then reaping. Many daemons and applications rely on init process to handle their orphan children. Reference : expectations of init processes

Docker relation

All unix systems will have an init process (like init process) but can be bulky when compared to container level restriction on size. In docker the containers are created which in turn spawns the application process but it does not reap the orphan children. So after sometime you will not know why the container is not working as it was working initially because it cannot create new processes. Reference : Docker PID 1 problem

Solutions

• You can execute bash as entrypoint which then calls the application and bash has the capability to reap the child processes too. The problem is when you give SIGTERM or any other signals its not propagated to the child processes instead the parent dies and there is a non graceful shutdown for the application as you stop the container. The will cause corruption problems like when the child process is writing to some file.
• You can also propagate signals in bash to child but then you will still be not sure of the application is stopped as it gives some signal and parent dies. Hold on the next solution is great !
• You can use some existing tools like base_image and tini which gives a lightweight init system which reaps the child processes if necessary. Now docker does not have these init processes natively but they support tini with --init parameter in docker run. Hold on the next solution is awesome !
• You can use container pilot which sits inside your container itself as a lightweight init system which can kick start your applications also side cars and registers the services with consul catalog. Performs health checks, sends these information to consul. If something goes wrong it informs consul and restarts the application, re-registers the service.