FreeBSD Jails using ZFS and bsdinstall

Posted on 13 January 2013

The new bsdinstall installer and ZFS make it particularly easy to create and deploy FreeBSD Jails.

Jails indeed are a "very powerful tool for system administrators" as described in the handbook. However, the handbook describes a time consuming and tedious process to create jails involving compiling userland. This article shows how to quicky get started with jails using ZFS datasets and the bsdinstall script on FreeBSD 9.0 and higher.

Before you begin, identify the ZFS pool where you intend to deploy jails. Lets say it's called tank.

Create your First Jail.

It is useful to have one 'base jail' that is a minimal FreeBSD instance with no additional packages. This instance can later be cloned or mounted to create other jails.

Prepare a ZFS dataset for this jail. In this example the jail will be installed in /jails/fbsd.

# zfs create -o mountpoint=/jails tank/jails
# zfs create tank/jails/fbsd

Install the FreeBSD userland into /jails/fbsd

# bsdinstall jail /jails/fbsd

The bsdinstall script will display a series of dialogs and finally download and install FreeBSD into the specified directory. I recommend the following options:

Distribution: Deselect 'games' and 'ports' leaving just 'lib32' selected.
User Accounts: Do not add any users.

That's it - the jail folder is ready! Before the jail can be started, networking needs to be set up. In this example a loopback address is created on lo0 so that it can be used in the jail. Any IP address can be used instead as long as it is available on the host.

# ifconfig lo0 127.0.0.2/32 alias

Start a jailed shell.

# jail -c path=/jails/fbsd mount.devfs host.hostname=fbsd 
  ip4.addr=127.0.0.2 command=/bin/sh

This runs a shell in the jailed environment. Verify that it is jailed by running ifconfig - you should only see the jailed IP. Also, there will be no network access from the jail as it was not configured. Exit the shell to drop back to the host shell.

Snapshot this jail containing FreeBSD in a pristine state.

# zfs snapshot tank/jails/fbsd@9.1

This critical step will enable cloning this snapshot to create more jails in the future.

Create New Jails

There are two ways in which new jails can be built from the base jail:

Using ZFS clones: This uses a clone of the base jail's ZFS snapshot to create a new jail. ZFS clones do not use additional disk space. Updates to the base jail are not reflected in the cloned jail. The userland in the cloned jail is writeable and can diverge from the source.
Using nullfs mounts: This uses mount_nullfs to mount many directories from the base jail into the new jail. The mounted directories can be made read-only. Updating the base jail to a new userland version automatically updates the new jail. This technique is not described in this article.

The following example creates a new jail that will be used to build ports using technique #1. Since downloading ports requires internet access, first we set up a firewall NAT for the IP address used by this jail.

Enabling Network Access from a Jail

There are many ways to configure networking for a jail. The following example demonstrates running a jail using a loopback address (e.g. 127.0.0.3) that can access the internet via a NAT rule configured using Packet Filter. This example assumes Packet Filter is not configured on this host.

# ifconfig lo0 127.0.0.3/32 alias
# echo 'nat on em0 from 127.0.0.3 to any -> (em0)' >> /etc/pf.conf
# kldload pf
# pf -e -f /etc/pf.conf

If pf was already enabled on the host, run pf -d to disable it first. Also verify other rules do not interfere with the NAT rule.

Creating the Ports Jail using ZFS Snapshot Clones

Use ZFS to clone the base jail. This takes less than a second.

# zfs clone tank/jails/fbsd@9.1 tank/jails/ports

Assuming you have an IP address that can access the internet, use that to start a shell in the jail.

# jail -c path=/jails/ports mount.devfs host.hostname=ports 
  ip4.addr=127.0.0.3 command=/bin/sh

Now you can proceed to install ports in the jail.

# hostname
ports
# portsnap fetch extract

That's it - this jail can now be used to build ports.

Creating More Jails

The 'zfs clone' command can be used to create any number of jails. Also, 'zfs snapshot' can be used to snapshot a jail in any state and the jail can later be reverted to any snapshot using 'zfs rollback'. See ZFS in the FreeBSD Handbook for details about these ZFS commands.

Starting Jails Automatically

The examples above show how to start a shell in a jailed environment. To start a jail automatically when the host system starts up, use the rc.conf system on the host:

### Contents of /etc/rc.conf
jail_enable="YES"
jail_list="ports"
jail_ports_rootdir="/jails/ports"
jail_ports_hostname="ports.example.org"
jail_ports_ip="127.0.0.3"
jail_ports_devfs_enable="YES"

Starting a jail implies starting the /etc/rc system within the jailed environment. For more details about automatic startup, see rc.conf.

For more information about jails, see Jails in the FreeBSD Handbook. Also see jail.conf - a cleaner way to configure jails that has appeared in FreeBSD 9.1, but doesn't seem to be described in the handbook.

« Previous post