VNET + Thin Jail(Snapshot)

备注

本文是 OpenZFS快照Thin Jail 的完整(补充)实践，以及结合 FreeBSD VNET Jail 完成的方案。

同时，也是我实践 FreeBSD Linux Jail 的基础(前置部分)

主机激活 jail

执行以下命令配置在系统启动时启动 Jail :

激活jail

# 配置在系统启动时激活Jail功能
sysrc jail_enable="YES"

# 配置所有jails在后台启动
sysrc jail_parallel_start="YES"

Jail目录树

设置环境变量:

设置 jail目录和release版本环境变量

export jail_dir="zdata/jails"
export bsd_ver="14.3"
# 在FreeBSD中root用户的shell默认是sh，所以调整 ~/.shrc
echo 'export jail_dir="zdata/jails"' >> ~/.shrc
echo 'export bsd_ver="14.3"' >> ~/.shrc

jail目录结构

zfs create $jail_dir
zfs create $jail_dir/media
zfs create $jail_dir/templates
zfs create $jail_dir/containers

完成后检查 df -h 可以看到磁盘如下:

jail目录的zfs形式

Filesystem                   Size    Used   Avail Capacity  Mounted on
...
zdata                        6.1T     96K    6.1T     0%    /zdata
zdata/jails                  6.1T    104K    6.1T     0%    /zdata/jails
zdata/jails/media            6.1T     96K    6.1T     0%    /zdata/jails/media
zdata/jails/templates        6.1T     96K    6.1T     0%    /zdata/jails/templates
zdata/jails/containers       6.1T     96K    6.1T     0%    /zdata/jails/containers

快照(snapshot)

备注

快照(snapshot) 型 vs. 模板和NullFS 型 Thin Jail 说明两种不同的Thin Jail，本文是 OpenZFS Snapshots 类型的Thin Jail构建记录:

ZFS snapshot Thin Jail: 将FreeBSD Release base存放在只读的 14.3-RELEASE@base 快照上
NullFS Thin Jail: 将FreeBSD Release base存放在读写的 14.3-RELEASE-base 数据集上

从这步开始可以看出两者不同

为OpenZFS Snapshot Thin Jail 准备模版dataset:

创建 14.3-RELEASE 模版

zfs create -p $jail_dir/templates/$bsd_ver-RELEASE

下载用户空间:

下载用户空间

fetch https://download.freebsd.org/ftp/releases/amd64/amd64/$bsd_ver-RELEASE/base.txz -o /$jail_dir/media/$bsd_ver-RELEASE-base.txz

将下载内容解压缩到模版目录: 内容解压缩到模板目录(后续要在14.3-RELEASE上创建快照，这就是和NullFS的区别)

解压缩

tar -xf /$jail_dir/media/$bsd_ver-RELEASE-base.txz -C /$jail_dir/templates/$bsd_ver-RELEASE --unlink

将时区和DNS配置复制到模板目录

cp /etc/resolv.conf /$jail_dir/templates/$bsd_ver-RELEASE/etc/resolv.conf
cp /etc/localtime   /$jail_dir/templates/$bsd_ver-RELEASE/etc/localtime

更新模板补丁:

更新模板补丁

freebsd-update -b /$jail_dir/templates/$bsd_ver-RELEASE/ fetch install

备注

从这里开始 Snapshot 类型Thin Jail 和 NullFS 类型Thin Jail开始出现步骤差异: 这里需要为base模版建立快照，然后clone(NullFS则创立快照，但只clone skeleton 部分保持读写)

为模版创建快照(完整快照):

为RELEASE模版创建base快照

zfs snapshot $jail_dir/templates/$bsd_ver-RELEASE@base

基于快照(snapshot)的Thin Jail 比 NullFS 类型简单很多，只要在模块快照基础上创建clone就可以生成一个新的Thin Jail:

为了能够灵活创建jail，这里定义一个 jail_name 环境变量，方便后续调整jail命名

# 假设这里创建名为ldev的jail
jail_name=ldev

clone出一个名为 ldev 的Thin Jail(后续将进一步改造为 FreeBSD Linux Jail )

zfs clone $jail_dir/templates/$bsd_ver-RELEASE@base $jail_dir/containers/$jail_name

现在可以看到相关ZFS数据集如下:

ZFS数据集显示jail存储

Filesystem                                     Size    Used   Avail Capacity  Mounted on
...
zdata/jails                                    1.3T    116K    1.3T     0%    /zdata/jails
zdata/jails/media                              1.3T    201M    1.3T     0%    /zdata/jails/media
zdata/jails/containers                         1.3T    104K    1.3T     0%    /zdata/jails/containers
zdata/jails/templates                          1.3T    104K    1.3T     0%    /zdata/jails/templates
...
zdata/jails/templates/14.3-RELEASE             1.3T    460M    1.3T     0%    /zdata/jails/templates/14.3-RELEASE
zdata/jails/containers/ldev                    1.3T    460M    1.3T     0%    /zdata/jails/containers/ldev

配置Jail

备注

在 VNET + Thin Jail(NullFS) 配置基础上完成

适合不同Jail的公共配置 /etc/jail.conf :

混合多种jail的公共 /etc/jail.conf

# STARTUP/LOGGING
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.consolelog = "/var/log/jail_console_${name}.log";

# PERMISSIONS
allow.raw_sockets;
exec.clean;
mount.devfs;

allow.mount;
allow.mount.devfs;

enforce_statfs = 1;

# HOSTNAME
host.hostname = "${name}";

# NETWORK - VNET/VIMAGE
#ip4 = inherit;
interface = igc0bridge;
vnet;
vnet.interface = "${epair}b";
# common NETWORK config
$gateway = "192.168.7.221";
$bridge = "igc0bridge";
$epair = "epair${id}";

# ADD TO bridge INTERFACE
exec.prestart += "ifconfig ${epair} create up";
exec.prestart += "ifconfig ${epair}a up descr jail:${name}";
exec.prestart += "ifconfig ${bridge} addm ${epair}a up";
exec.start    += "ifconfig ${epair}b ${ip} up";
exec.start    += "route add default ${gateway}";
exec.poststop = "ifconfig ${bridge} deletem ${epair}a";
exec.poststop += "ifconfig ${epair}a destroy";

.include "/etc/jail.conf.d/*.conf";

用于Snapshot类型的 ldev 独立配置 /etc/jail.conf.d/ldev.conf :

用于Snapshot类型 /etc/jail.conf.d/ldev.conf

ldev {
  # thin jail devfs_ruleset 5 和Linux Jail的4不同
  devfs_ruleset=5;

  # HOSTNAME/PATH - Snapshot
  path = "/zdata/jails/containers/${name}";

  # NETWORKS/INTERFACES
  $id = "253";
  $ip = "192.168.7.${id}/24";

}

备注

挂载路径和NullFS类型的Thin Jail不同

启动jail

最后启动 ldev :

启动 ldev

service jail start ldev

通过 jexec ldev 进入jail

设置Jail ldev 在操作系统启动时启动，修改 /etc/rc.conf :

/etc/rc.conf

# Jails
jail_enable="YES"
jail_parallel_start="YES"
jail_list="jdev ldev ludev"
jail_reverse_stop="YES"