Release 0.9.4

Fixed SFTP not working

Changelog.md

@@ -1,3 +1,21 @@
+## 0.9.4 (release date: 2014-02-03)
+
+ * Fixed syslog-ng startup problem.
+
+## 0.9.3 (release date: 2014-01-31)
+
+ * It looks like Docker changed their Ubuntu 12.04 base image, thereby breaking our Dockerfile. This has been fixed.
+ * The init system (`/sbin/my_init`) now supports running scripts during startup. You can put startup scripts `/etc/my_init.d`. `/etc/rc.local` is also run during startup.
+ * To improve security, the base image no longer contains pregenerated SSH host keys. Instead, users of the base image are encouraged to regenerate one in their Dockerfile. If the user does not do that, then random SSH host keys are generated during container boot.
+
+## 0.9.2 (release date: 2013-12-11)
+
+ * Fixed SFTP support. Thanks Joris van de Donk!
+
+## 0.9.1 (release date: 2013-11-12)
+
+ * Improved init process script (`/sbin/my_init`): it now handles shutdown correctly. Previously, `docker stop` would not have any effect on `my_init`, causing the whole container to be killed with SIGKILL. The new init process script gracefully shuts down all runit services, then exits.
+
 ## 0.9.0 (release date: 2013-11-12)
 
  * Initial release

LICENSE.txt

@@ -0,0 +1,19 @@
+Copyright (c) 2013-2014 Phusion
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

Makefile

@@ -1,16 +1,20 @@
 NAME = phusion/baseimage
-VERSION = 0.9.0
+VERSION = 0.9.4
 
-.PHONY: all build tag_latest release
+.PHONY: all build test tag_latest release
 
 all: build
 
 build:
 	docker build -t $(NAME):$(VERSION) -rm image
 
+test:
+	env NAME=$(NAME) VERSION=$(VERSION) ./test/runner.sh
+
 tag_latest:
 	docker tag $(NAME):$(VERSION) $(NAME):latest
 
-release: tag_latest
+release: test tag_latest
+	@if ! docker images phusion/baseimage | awk '{ print $$2 }' | grep -q -F $(VERSION); then echo "$(NAME) version $(VERSION) is not yet built. Please run 'make build'"; false; fi
 	docker push $(NAME)
 	@echo "*** Don't forget to create a tag. git tag rel-$(VERSION) && git push origin rel-$(VERSION)"

README.md

@@ -1,43 +1,80 @@
-# A minimal Docker base image with a correct and usable system
+# A minimal Ubuntu base image modified for Docker-friendliness
 
-Baseimage-docker is a [Docker](http://www.docker.io) image meant to serve as a good base for any other Docker container. It contains a minimal base system with the most important things already installed and set up correctly.
+Baseimage-docker is a special [Docker](http://www.docker.io) image that is configured for correct use within Docker containers. It is Ubuntu, plus modifications for Docker-friendliness. You can use it as a base for your own Docker images.
 
- * **Github**: https://github.com/phusion/baseimage-docker
- * **Discussion forum**: https://groups.google.com/d/forum/passenger-docker
- * **Twitter**: https://twitter.com/phusion_nl
- * **Blog**: http://blog.phusion.nl/
+Baseimage-docker is available for pulling from [the Docker registry](https://index.docker.io/u/phusion/baseimage/)!
 
+### What are the problems with the stock Ubuntu base image?
+
+Ubuntu is not designed to be run inside docker. Its init system, Upstart, assumes that it's running on either real hardware or virtualized hardware, but not inside a Docker container. But inside a container you don't want a full system anyway, you want a minimal system. But configuring that minimal system for use within a container has many strange corner cases that are hard to get right if you are not intimately familiar with the Unix system model. This can cause a lot of strange problems.
+
+Baseimage-docker gets everything right. The "Contents" section describes all the things that it modifies.
+
+<a name="why_use"></a>
 ### Why use baseimage-docker?
 
-Why use baseimage-docker instead of doing everything yourself in Dockerfile?
+You can configure the stock `ubuntu` image yourself from your Dockerfile, so why bother using baseimage-docker?
 
+ * Configuring the base system for Docker-friendliness is no easy task. As stated before, there are many corner cases. By the time that you've gotten all that right, you've reinvented baseimage-docker. Using baseimage-docker will save you from this effort.
  * It reduces the time needed to write a correct Dockerfile. You won't have to worry about the base system and can focus on your stack and your app.
- * It sets up the base system **correctly**. It's very easy to get the base system wrong, but this image does everything correctly.
  * It reduces the time needed to run `docker build`, allowing you to iterate your Dockerfile more quickly.
+ * It reduces download time during redeploys. Docker only needs to download the base image once: during the first deploy. On every subsequent deploys, only the changes you make on top of the base image are downloaded.
 
-## Contents
+-----------------------------------------
+
+**Related resources**:
+  [Website](http://phusion.github.io/baseimage-docker/) |
+  [Github](https://github.com/phusion/baseimage-docker) |
+  [Docker registry](https://index.docker.io/u/phusion/baseimage/) |
+  [Discussion forum](https://groups.google.com/d/forum/passenger-docker) |
+  [Twitter](https://twitter.com/phusion_nl) |
+  [Blog](http://blog.phusion.nl/)
+
+**Table of contents**
+
+ * [What's inside the image?](#whats_inside)
+   * [Overview](#whats_inside_overview)
+   * [Wait, I thought Docker is about running a single process in a container?](#docker_single_process)
+ * [Inspecting baseimage-docker](#inspecting)
+ * [Using baseimage-docker as base image](#using)
+   * [Getting started](#getting_started)
+   * [Adding additional daemons](#adding_additional_daemons)
+   * [Running scripts during container startup](#running_startup_scripts)
+   * [Login to the container](#login)
+ * [Building the image yourself](#building)
+ * [Conclusion](#conclusion)
+
+-----------------------------------------
+
+<a name="whats_inside"></a>
+## What's inside the image?
+
+<a name="whats_inside_overview"></a>
+### Overview
 
 *Looking for a more complete base image, one that is ideal for Ruby, Python, Node.js and Meteor web apps? Take a look at [passenger-docker](https://github.com/phusion/passenger-docker).*
 
 | Component        | Why is it included? / Remarks |
 | ---------------- | ------------------- |
 | Ubuntu 12.04 LTS | The base system. |
-| A **correct** init process | According to the Unix process model, [the init process](https://en.wikipedia.org/wiki/Init) -- PID 1 -- inherit all [orphaned child processes](https://en.wikipedia.org/wiki/Orphan_process) and must [reap them](https://en.wikipedia.org/wiki/Wait_(system_call)). Most Docker containers do not have an init process that does this correctly, and as a result their containers become filled with [zombie processes](https://en.wikipedia.org/wiki/Zombie_process) over time. Baseimage-docker comes with an init process `/sbin/my_init` that performs reaping correctly. |
+| A **correct** init process | According to the Unix process model, [the init process](https://en.wikipedia.org/wiki/Init) -- PID 1 -- inherits all [orphaned child processes](https://en.wikipedia.org/wiki/Orphan_process) and must [reap them](https://en.wikipedia.org/wiki/Wait_(system_call). Most Docker containers do not have an init process that does this correctly, and as a result their containers become filled with [zombie processes](https://en.wikipedia.org/wiki/Zombie_process) over time. <br><br>Furthermore, `docker stop` sends SIGTERM to the init process, which is then supposed to stop all services. Unfortunately most init systems don't do this correctly within Docker since they're built for hardware shutdowns instead. This causes processes to be hard killed with SIGKILL, which doesn't give them a chance to correctly deinitialize things. This can cause file corruption. <br><br>Baseimage-docker comes with an init process `/sbin/my_init` that performs both of these tasks correctly. |
 | Fixes APT incompatibilities with Docker | See https://github.com/dotcloud/docker/issues/1024. |
 | syslog-ng | A syslog daemon is necessary so that many services - including the kernel itself - can correctly log to /var/log/syslog. If no syslog daemon is running, a lot of important messages are silently swallowed. <br><br>Only listens locally. |
-| ssh server | Allows you to easily login to your container to inspect or administer things. <br><br>Password and challenge-response authentication are disabled by default. Only key authentication is allowed.<br>It allows an predefined key by default to make debugging easy. You should replace this ASAP. See instructions. |
+| ssh server | Allows you to easily login to your container to inspect or administer things. <br><br>Password and challenge-response authentication are disabled by default. Only key authentication is allowed.<br>By default, it allows a predefined key, in order to make debugging easy. You should replace this ASAP. See instructions. |
 | cron | The cron daemon must be running for cron jobs to work. |
-| [runit](http://smarden.org/runit/) | For service supervision and management. Much easier to use than SysV init and supports restarting daemons when they crash. Much easier to use and more lightweight than Upstart. |
+| [runit](http://smarden.org/runit/) | Replaces Ubuntu's Upstart. Used for service supervision and management. Much easier to use than SysV init and supports restarting daemons when they crash. Much easier to use and more lightweight than Upstart. |
 | `setuser` | A tool for running a command as another user. Easier to use than `su`, has a smaller attack vector than `sudo`, and unlike `chpst` this tool sets `$HOME` correctly. Available as `/sbin/setuser`. |
 
-Baseimage-docker is very lightweight: it only consumes 4 MB of memory.
+Baseimage-docker is very lightweight: it only consumes 6 MB of memory.
 
+<a name="docker_single_process"></a>
 ### Wait, I thought Docker is about running a single process in a container?
 
 Absolutely not true. Docker runs fine with multiple processes in a container. In fact, there is no technical reason why you should limit yourself to one process - it only makes things harder for you and breaks all kinds of essential system functionality, e.g. syslog.
 
 Baseimage-docker *encourages* multiple processes through the use of runit.
 
+<a name="inspecting"></a>
 ## Inspecting baseimage-docker
 
 To look around in the image, run:
@@ -46,8 +83,12 @@ To look around in the image, run:
 
 You don't have to download anything manually. The above command will automatically pull the baseimage-docker image from the Docker registry.
 
+<a name="using"></a>
 ## Using baseimage-docker as base image
 
+<a name="getting_started"></a>
+### Getting started
+
 The image is called `phusion/baseimage`, and is available on the Docker registry.
 
 By default, it allows SSH access for the key in `image/insecure_key`. This makes it easy for you to login to the container, but you should replace this key as soon as possible.
@@ -64,19 +105,25 @@ By default, it allows SSH access for the key in `image/insecure_key`. This makes
     # Remove authentication rights for insecure_key.
     RUN rm -f /root/.ssh/authorized_keys /home/*/.ssh/authorized_keys
     
-    # Use baseimage-docker's init process.
+    # Regenerate SSH host keys. baseimage-docker does not contain any, so you
+    # have to do that yourself. You may also comment out this instruction; the
+    # init system will auto-generate one during boot.
+    RUN /etc/my_init.d/00_regen_ssh_host_keys.sh
+    
+    # Use baseimage-docker's init system.
     CMD ["/sbin/my_init"]
     
-    # ...put other build instructions here...
+    # ...put your own build instructions here...
     
     # Clean up APT when done.
     RUN apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
 
+<a name="adding_additional_daemons"></a>
 ### Adding additional daemons
 
 You can add additional daemons to the image by creating runit entries. You only have to write a small shell script which runs your daemon, and runit will keep it up and running for you, restarting it when it crashes, etc.
 
-The shell script must be called `run`, must be executable, and is to be placed in the directory `/etc/services/<NAME>`.
+The shell script must be called `run`, must be executable, and is to be placed in the directory `/etc/service/<NAME>`.
 
 Here's an example showing you how to a memached server runit entry can be made.
 
@@ -87,11 +134,53 @@ Here's an example showing you how to a memached server runit entry can be made.
     exec chpst -u memcache /usr/bin/memcached >>/var/log/memcached.log 2>&1
 
     ### In Dockerfile:
-    RUN mkdir /etc/services/memcached
-    ADD redis.sh /etc/services/memcached/run
+    RUN mkdir /etc/service/memcached
+    ADD memcached.sh /etc/service/memcached/run
 
 Note that the shell script must run the daemon **without letting it daemonize/fork it**. Usually, daemons provide a command line flag or a config file option for that.
 
+<a name="running_startup_scripts"></a>
+### Running scripts during container startup
+
+The baseimage-docker init system, `/sbin/my_init`, runs the following scripts during startup, in the following order:
+
+ * All executable scripts in `/etc/my_init.d`, if this directory exists. The scripts are run during in lexicographic order.
+ * The script `/etc/rc.local`, if this file exists.
+
+All scripts must exit correctly, e.g. with exit code 0. If any script exits with a non-zero exit code, the booting will fail.
+
+The following example shows how you can add a startup script. This script simply logs the time of boot to the file /tmp/boottime.txt.
+
+    ### In logtime.sh (make sure this file is chmod +x):
+    #!/bin/sh
+    date > /tmp/boottime.txt
+
+    ### In Dockerfile:
+    RUN mkdir -p /etc/my_init.d
+    ADD logtime.sh /etc/my_init.d/logtime.sh
+
+<a name="login"></a>
+### Login to the container
+
+You can use SSH to login to any container that is based on baseimage-docker.
+
+Start a container based on baseimage-docker (or a container based on an image based on baseimage-docker):
+
+    docker run phusion/baseimage
+
+Find out the ID of the container that you just ran:
+
+    docker ps
+
+Once you have the ID, look for its IP address with:
+
+    docker inspect <ID> | grep IPAddress
+
+Now SSH into the container. In this example we're using [the default insecure key](https://github.com/phusion/baseimage-docker/blob/master/image/insecure_key), but if you're followed the instructions well then you've already replaced that with your own key. You did replace the key, didn't you?
+
+    ssh -i insecure_key root@<IP address>
+
+<a name="building"></a>
 ## Building the image yourself
 
 If for whatever reason you want to build the image yourself instead of downloading it from the Docker registry, follow these instructions.
@@ -115,10 +204,11 @@ If you want to call the resulting image something else, pass the NAME variable,
 
     make build NAME=joe/baseimage
 
+<a name="conclusion"></a>
 ## Conclusion
 
  * Using baseimage-docker? [Tweet about us](https://twitter.com/share) or [follow us on Twitter](https://twitter.com/phusion_nl).
- * Having problems? Please post a message at [the discussion forum](https://groups.google.com/d/forum/passenger-docker).
+ * Having problems? Want to participate in development? Please post a message at [the discussion forum](https://groups.google.com/d/forum/passenger-docker).
  * Looking for a more complete base image, one that is ideal for Ruby, Python, Node.js and Meteor web apps? Take a look at [passenger-docker](https://github.com/phusion/passenger-docker).
 
 [<img src="http://www.phusion.nl/assets/logo.png">](http://www.phusion.nl/)

Vagrantfile

@@ -9,7 +9,7 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
   config.vm.box = "phusion-open-ubuntu-12.04-amd64"
   config.vm.box_url = "https://oss-binaries.phusionpassenger.com/vagrant/boxes/ubuntu-12.04.3-amd64-vbox.box"
   config.ssh.forward_agent = true
-  if File.directory?("#{ROOT}/passenger-docker")
+  if File.directory?("#{ROOT}/../passenger-docker")
     config.vm.synced_folder File.expand_path("#{ROOT}/../passenger-docker"),
       "/vagrant/passenger-docker"
   end

image/00_regen_ssh_host_keys.sh

@@ -0,0 +1,6 @@
+#!/bin/bash
+set -e
+if [[ ! -e /etc/ssh/ssh_host_rsa_key ]]; then
+	echo "No SSH host key available. Generating one..."
+	dpkg-reconfigure openssh-server
+fi

image/cleanup.sh

@@ -6,3 +6,5 @@ set -x
 apt-get clean
 rm -rf /build
 rm -rf /tmp/* /var/tmp/*
+
+rm -f /etc/ssh/ssh_host_*

image/config/sshd_config

@@ -123,7 +123,7 @@ ChallengeResponseAuthentication no
 #Banner none
 
 # override default of no subsystems
-Subsystem	sftp	/usr/libexec/sftp-server
+Subsystem	sftp	/usr/lib/sftp-server
 
 # Example of overriding settings on a per-user basis
 #Match User anoncvs

image/my_init

@@ -1,5 +1,116 @@
-#!/bin/bash
-set -e
-# No exec. We want bash to be the init process so that it can kill
-# zombie processes.
-/usr/sbin/runsvdir-start
+#!/usr/bin/python2
+import os, sys, stat, signal, errno
+
+pid = None
+status = None
+
+def listdir(path):
+	try:
+		result = os.stat(path)
+	except OSError:
+		return []
+	if stat.S_ISDIR(result.st_mode):
+		return sorted(os.listdir(path))
+	else:
+		return []
+
+def is_exe(path):
+	try:
+		return os.path.isfile(path) and os.access(path, os.X_OK)
+	except OSError:
+		return False
+
+def reap_child(signum, frame):
+	global pid, status, waiting_for_runit
+	try:
+		result = os.wait3(os.WNOHANG)
+		if result is not None and pid == result[0]:
+			status = result[1]
+	except OSError:
+		pass
+
+def stop_child_process(name):
+	global pid
+	print("*** Shutting down %s (PID %d)..." % (name, pid))
+	try:
+		os.kill(pid, signal.SIGHUP)
+	except OSError:
+		pass
+
+def run_command_killable(*argv):
+	global pid
+	filename = argv[0]
+	pid = os.spawnvp(os.P_NOWAIT, filename, argv)
+	signal.signal(signal.SIGINT, lambda signum, frame: stop_child_process(filename))
+	signal.signal(signal.SIGTERM, lambda signum, frame: stop_child_process(filename))
+	try:
+		this_pid, status = os.waitpid(pid, 0)
+	except OSError as e:
+		if e.errno == errno.EINTR:
+			sys.exit(2)
+		else:
+			raise
+	finally:
+		signal.signal(signal.SIGINT, signal.SIG_DFL)
+		signal.signal(signal.SIGTERM, signal.SIG_DFL)
+	if status != 0:
+		sys.stderr.write("*** %s failed with exit code %d\n" % (filename, status))
+		sys.exit(1)
+
+# Run /etc/my_init.d/*
+for name in listdir("/etc/my_init.d"):
+	filename = "/etc/my_init.d/" + name
+	if is_exe(filename):
+		print("*** Running %s..." % filename)
+		run_command_killable(filename)
+
+# Run /etc/rc.local.
+if is_exe("/etc/rc.local"):
+	print("*** Running /etc/rc.local...")
+	run_command_killable("/etc/rc.local")
+
+# Start runit.
+signal.signal(signal.SIGCHLD, reap_child)
+print("*** Booting runit...")
+pid = os.spawnl(os.P_NOWAIT, "/usr/sbin/runsvdir-start", "/usr/sbin/runsvdir-start")
+print("*** Runit started as PID %d" % pid)
+signal.signal(signal.SIGTERM, lambda signum, frame: stop_child_process("runit"))
+
+# Wait for runit, and while waiting, reap any adopted orphans.
+done = False
+while not done:
+	try:
+		this_pid, status = os.waitpid(pid, 0)
+		done = True
+	except OSError as e:
+		if e.errno == errno.EINTR:
+			# Try again
+			pass
+		else:
+			# The SIGCHLD handler probably caught it.
+			done = True
+
+# Runit has exited. Reset signal handlers.
+print("*** Runit exited with code %s. Waiting for all services to shut down..." % status)
+signal.signal(signal.SIGCHLD, signal.SIG_DFL)
+signal.signal(signal.SIGTERM, signal.SIG_DFL)
+signal.siginterrupt(signal.SIGCHLD, False)
+signal.siginterrupt(signal.SIGTERM, False)
+
+# Wait at most 5 seconds for services to shut down.
+import time
+
+def shutdown(signum = None, frame = None):
+	global status
+	if status is not None:
+		sys.exit(status)
+
+signal.signal(signal.SIGALRM, shutdown)
+signal.alarm(5)
+done = False
+while not done:
+	done = os.system("/usr/bin/sv status /etc/service/* | grep -q '^run:'") != 0
+	if not done:
+		time.sleep(0.5)
+shutdown()
+

image/prepare.sh

@@ -14,7 +14,7 @@ $minimal_apt_get_install apt-transport-https
 ## Fix some issues with APT packages.
 ## See https://github.com/dotcloud/docker/issues/1024
 dpkg-divert --local --rename --add /sbin/initctl
-ln -s /bin/true /sbin/initctl
+ln -sf /bin/true /sbin/initctl
 
 ## Upgrade all packages.
 echo "initscripts hold" | dpkg --set-selections

image/system_services.sh

@@ -5,6 +5,7 @@ set -x
 
 ## Install init process.
 cp /build/my_init /sbin/
+mkdir -p /etc/my_init.d
 
 ## Install runit.
 $minimal_apt_get_install runit
@@ -13,6 +14,7 @@ $minimal_apt_get_install runit
 $minimal_apt_get_install syslog-ng-core
 mkdir /etc/service/syslog-ng
 cp /build/runit/syslog-ng /etc/service/syslog-ng/run
+mkdir -p /var/lib/syslog-ng
 
 ## Install the SSH server.
 $minimal_apt_get_install openssh-server
@@ -20,6 +22,7 @@ mkdir /var/run/sshd
 mkdir /etc/service/sshd
 cp /build/runit/sshd /etc/service/sshd/run
 cp /build/config/sshd_config /etc/ssh/sshd_config
+cp /build/00_regen_ssh_host_keys.sh /etc/my_init.d/
 
 ## Install default SSH key for root and app.
 mkdir -p /root/.ssh

test/runner.sh

@@ -0,0 +1,34 @@
+#!/bin/bash
+set -e
+
+function abort()
+{
+	echo "$@"
+	exit 1
+}
+
+function cleanup()
+{
+	echo " --> Stopping container"
+	docker stop $ID >/dev/null
+	docker rm $ID >/dev/null
+}
+
+echo " --> Starting container"
+PWD=`pwd`
+ID=`docker run -d -v $PWD/test:/test $NAME:$VERSION`
+sleep 1
+
+echo " --> Obtaining IP"
+IP=`docker inspect $ID | grep IPAddress | sed -e 's/.*: "//; s/".*//'`
+if [[ "$IP" = "" ]]; then
+	abort "Unable to obtain container IP"
+fi
+
+trap cleanup EXIT
+
+echo " --> Logging into container and running tests"
+chmod 600 image/insecure_key
+sleep 1 # Give container some more time to start up.
+ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i image/insecure_key root@$IP \
+	/bin/bash /test/test.sh

test/test.sh

@@ -0,0 +1,22 @@
+#!/bin/bash
+set -o pipefail
+
+function ok()
+{
+	echo "  OK"
+}
+
+function fail()
+{
+	echo "  FAIL"
+	exit 1
+}
+
+echo "Checking whether all services are running..."
+services=`sv status /etc/service/*`
+status=$?
+if [[ "$status" != 0 || "$services" = "" || "$services" =~ down ]]; then
+	fail
+else
+	ok
+fi