VocalBit« assorted thoughts on technology, computers and programming »

FreeBSD Jails using ZFS and bsdinstall

The new bsdinstall installer and ZFS make it particularly easy to create and deploy FreeBSD Jails. This hands-on tutorial will get you familiar with manually configuring and starting a jail.

Jails indeed are a "very powerful tool for system administrators" as described in the handbook. However, the handbook describes a time consuming and tedious process to create jails involving compiling userland. This article shows how to quicky get started with jails using ZFS datasets and the bsdinstall script on FreeBSD 9.0 and higher.

Before you begin, identify the ZFS pool where you intend to deploy jails. Lets say it's called tank.

Create your First Jail

It is useful to have one 'base jail' that is a minimal FreeBSD instance with no additional packages. This instance can later be cloned or mounted to create other jails.

Prepare a ZFS dataset for this jail. In this example the jail will be installed in /jails/fbsd

# zfs create -o mountpoint=/jails tank/jails
# zfs create tank/jails/fbsd

Install the FreeBSD userland into /jails/fbsd

# bsdinstall jail /jails/fbsd

The bsdinstall script will display a series of dialogs and finally download and install FreeBSD into the specified directory. I recommend the following options:

  • Distribution: Deselect games and ports leaving just lib32 selected.
  • User Accounts: Do not add any users.

That's it - the jail folder is ready! Before the jail can be started, networking needs to be set up. In this example a new loopback address is created on lo0 so that it can be used in the jail. Any IP address can be used as long as it is available on the host.

# ifconfig lo0 127.0.0.2/32 alias

Start a jailed shell

# jail -c path=/jails/fbsd mount.devfs host.hostname=fbsd ip4.addr=127.0.0.2 command=/bin/sh

This runs a shell in the jailed environment. Verify that it is jailed by running ifconfig - you should only see the jailed IP. Also, there will be no network access from the jail as it was not configured. Exit the jailed shell to drop back to the host shell.

Snapshot this jail containing FreeBSD in a pristine state.

# zfs snapshot tank/jails/fbsd@9.1

This critical step will enable cloning this snapshot to create more jails in the future.

Create New Jails

There are two ways in which new jails can be built from the base jail:

  1. Using ZFS clones : This uses a clone of the base jail's ZFS snapshot to create a new jail. ZFS clones do not use any additional disk space until you start writing to them. Updates to the base jail are not reflected in the cloned jail. The userland in the cloned jail is writeable and can diverge from the base jail.
  2. Using nullfs mounts : This uses mount_nullfs to mount many directories from the base jail into the new jail. The mounted directories can be made read-only. Updating the base jail to a new userland version automatically updates the new jail. This technique is not described in this article. The following example creates a new jail that will be used to build ports using technique #1. Since downloading ports requires internet access, first we set up a firewall NAT for the IP address used by this jail.

Enabling Network Access from a Jail

There are many ways to configure networking for a jail. The following example demonstrates running a jail using a loopback address (e.g. 127.0.0.3) that can access the internet via a NAT rule configured using Packet Filter. This example assumes Packet Filter is not configured on this host.

Configure the ip address of the jail on the interface:

# ifconfig lo0 127.0.0.3/32 alias

To make the above change stick after reboots, you need to add an ipv4_addrs_<iface> line in /etc/rc.conf. See the rc.conf man page for details. Now you can enable internet access from within the jail by setting up a NAT using Packet Filter:

# echo 'nat on em0 from 127.0.0.3 to any -> (em0)' >> /etc/pf.conf
# kldload pf
# pf -e -f /etc/pf.conf

If Packet Filter was already enabled on the host, run pf -d to disable it first. Also verify other rules do not interfere with this NAT rule.

Creating the Ports Jail using ZFS Snapshot Clones

Use ZFS to clone the base jail. This takes less than a second. This example creates a new jail called 'ports':

# zfs clone tank/jails/fbsd@9.1 tank/jails/ports

Assuming you have an IP address that can access the internet, use that to start a shell in the jail:

# jail -c path=/jails/ports mount.devfs host.hostname=ports ip4.addr=127.0.0.3 command=/bin/sh

Now you can proceed to install ports in the jail:

# hostname
ports
# portsnap fetch extract

That's it - this jail can now be used to build ports.

Creating More Jails

The zfs clone command can be used to create any number of jails. Also, zfs snapshot can be used to snapshot a jail in any state and the jail can later be reverted to any snapshot using zfs rollback. See ZFS in the FreeBSD Handbook for details about these ZFS commands.

Starting Jails Automatically

The examples above show how to start a shell in a jailed environment. To start a jail automatically when the host system starts up, use the rc.conf system on the host:

### Contents of /etc/rc.conf
jail_enable="YES"
jail_list="ports"
jail_ports_rootdir="/jails/ports"
jail_ports_hostname="ports.example.org"
jail_ports_ip="127.0.0.3"
jail_ports_devfs_enable="YES"

Starting a jail implies starting the /etc/rc system within the jailed environment. For more details about automatic startup, see rc.conf.

For more information about jails, see Jails in the FreeBSD Handbook.

Tips

See jail.conf - a cleaner way to configure jails that has appeared in FreeBSD 9.1, but doesn't seem to be described in the handbook.

Instead of adding the jail's ip address permanently to your interface (using ifconfig and rc.conf as described above), you can set the ip address on the jail command line itself using the format <iface>|<ip-address>, e.g.:

# jail -c path=/jails/ports mount.devfs host.hostname=ports "ip4.addr=lo0|127.0.0.3" command=/bin/sh

This will add the ip 127.0.0.3 to the lo0 interface at jail startup time, and remove it when the jail is killed. (This feature is in FreeBSD 9.0 and newer).

Take a look at ezjail which does a lot of things for you and makes managing jails easy. It includes ZFS integration as well. It uses the 'Using nullfs mounts' technique mentioned above.