How to compile Incus – Mi blog lah!
Incus is a manager for virtual machines and system containers.
A virtual machine is an instance of an operating system that runs on a computer, along with the main operating system. A virtual machine uses hardware virtualization features for the separation from the main operating system.
A system container is an instance of an operating system that also runs on a computer, along with the main operating system. A system container, instead, uses security primitives of the Linux kernel for the separation from the main operating system. You can think of system containers as software virtual machines.
In this post we are going to see how to compile and install Incus using the instructions on how to install from source.
Prerequisites
This post assumes that
- You are familiar with compiling software from source.
- You are familiar with using Incus.
- If you are not installing on your workstation, you already have Incus installed so that you can create a VM.
Setting up the environment
We are going to create a virtual machine running Ubuntu 22.04 from the image images:debian/12
. By default Incus allocates 10GB of space for the virtual machine and this image is quite lean, leaving around 8.5GB free space for our compilation. We are using a virtual machine instead of a system container because we want to run Incus in there. It’s easier to do that instead of introducing nesting
in this tutorial.
$ incus launch --vm images:debian/12 incus-compile Creating incus-compile Starting incus-compile
Then, get a shell into the virtual machine.
$ incus exec incus-compile -- sudo --login --user debian debian@incus-compile:~$
Installing development packages
We follow the instructions from https://linuxcontainers.org/incus/docs/main/installing/ on how to install from source. The commands have the -y
flag so that you can easily copy and paste directly into the terminal window of your VM.
First, install the prerequisite development packages. Note that liblxc-dev
on Ubuntu is lxc-dev
on Debian. Also note that Incus requires a recent version of Go. We are not installing Go from the distro repositories, we are getting a Go binary from the official site. The go version
command below should show a version 1.21.6
or newer.
sudo apt update sudo apt install -y acl attr autoconf automake dnsmasq-base git libacl1-dev libcap-dev liblxc1 lxc-dev libsqlite3-dev libtool libudev-dev liblz4-dev libuv1-dev make pkg-config rsync squashfs-tools tar tcl xz-utils ebtables sudo apt install -y wget wget https://go.dev/dl/go1.21.6.linux-amd64.tar.gz rm -rf /usr/local/go && sudo tar -C /usr/local -xzf go1.21.6.linux-amd64.tar.gz echo 'export PATH=$PATH:/usr/local/go/bin' >> ~/.bashrc export PATH=$PATH:/usr/local/go/bin go version
Second, install the prerequisite development packages for the different storage drivers. We are not installing zfsutils-linux
because it is not a standard Debian package; it requires to enable the contrib
repository, and we do not need that at the moment.
sudo apt install -y btrfs-progs sudo apt install -y ceph-common sudo apt install -y lvm2 thin-provisioning-tools
Thirdly, install the packages for the test suite.
sudo apt install -y busybox-static curl gettext jq sqlite3 socat bind9-dnsutils
Getting the source
We can clone the Incus repository and either use the latest version of the source code or a tagged version. We will use the latest version here.
git clone https://github.com/lxc/incus cd incus
Compiling the source: dependencies
Then, we run make deps
so that we sort out the dependencies for the compilation. This step takes about 30 seconds.
make deps
At the end of the output of that command, we get the following instructions.
Please set the following in your environment (possibly ~/.bashrc)
export CGO_CFLAGS="-I/home/ubuntu/go/deps/raft/include/ -I/home/ubuntu/go/deps/cowsql/include/"
export CGO_LDFLAGS="-L/home/ubuntu/go/deps/raft/.libs -L/home/ubuntu/go/deps/cowsql/.libs/"
export LD_LIBRARY_PATH="/home/ubuntu/go/deps/raft/.libs/:/home/ubuntu/go/deps/cowsql/.libs/"
export CGO_LDFLAGS_ALLOW="(-Wl,-wrap,pthread_create)|(-Wl,-z,now)"
To do that, we run the following to append the lines to ~/.bashrc. We run it and the shell is waiting for us to paste the four lines.
cat >> ~/.bashrc
We paste these lines. Please use the corresponding lines from your output.
export CGO_CFLAGS="-I/home/ubuntu/go/deps/raft/include/ -I/home/ubuntu/go/deps/cowsql/include/" export CGO_LDFLAGS="-L/home/ubuntu/go/deps/raft/.libs -L/home/ubuntu/go/deps/cowsql/.libs/" export LD_LIBRARY_PATH="/home/ubuntu/go/deps/raft/.libs/:/home/ubuntu/go/deps/cowsql/.libs/" export CGO_LDFLAGS_ALLOW="(-Wl,-wrap,pthread_create)|(-Wl,-z,now)"
Finally, press Ctrl+D to get out of the cat
command and return to your shell. Then exit
to exit the VM. We need that so that we get a fresh shell that would get updated with the new lines from ~/.bashrc
.
Ctrl+D exit
Get back into the virtual machine and we are ready to actually compile.
incus exec incus-compile -- sudo --login --user ubuntu ubuntu@incus-compile:~$ cd incus/ ubuntu@incus-compile:~/incus$
Compiling the sources: compilation
We compile Incus by running make
. This process downloads the source code of several Go packages and then performs the compilation. Therefore, the speed is related to how fast your Internet connection is, and how fast your computer is. A compilation from scratch takes about 3 minutes.
debian@incus-compile:~/incus$ make ... Incus built successfully debian@incus-compile:~/incus$
Installation of compiled binaries
There are two binaries, incus
the client and incusd
the server. They are found in the Go bin/
directory. Let’s add the directory to the $PATH
and add this configuration to our shell configuration.
export PATH="${PATH}:$(go env GOPATH)/bin" echo 'export PATH="${PATH}:$(go env GOPATH)/bin"' >> ~/.bashrc
System setup
Before Incus is ready to run, we need to prepare our system. Incus uses a Linux kernel feature called namespaces. With namespaces, we specify a range for User IDs and Group IDs to be used for the unprivileged containers in Incus. Containers are process trees that live in children namespaces. From within the container you would think you are root
with UID 0, but thanks to namespaces, that actual ID on the host would be something like 1000000. Totally unprivileged. Incus requires about 10M of such ID space. Those IDs are sub
ordinate IDs. Here is how to add them manually to our system.
echo "root:1000000:1000000000" | sudo tee -a /etc/subuid /etc/subgid
Running incusd
We run incusd
with the following command.
sudo -E PATH=${PATH}:/sbin LD_LIBRARY_PATH=${LD_LIBRARY_PATH} $(go env GOPATH)/bin/incusd --group sudo
Running incusd
with --verbose
We run incusd
with --verbose
so that we can view and understand the related messages that are printing.
core scheduling
is no
. I do not know yet what is this, and what are the implications. Other than that, all look fine.
The incusd
server keeps running until we hit Ctrl+C to stop it. We keep it running and open a new terminal window to run other commads.
debian@incus-compile:~$ sudo -E PATH=${PATH}:/sbin LD_LIBRARY_PATH=${LD_LIBRARY_PATH} $(go env GOPATH)/bin/incusd --group sudo --verbose INFO [2024-01-29T01:34:20Z] Starting up mode=normal path=/var/lib/incus version=0.5.1 INFO [2024-01-29T01:34:20Z] System idmap (root user): INFO [2024-01-29T01:34:20Z] - u 0 1000000 1000000000 INFO [2024-01-29T01:34:20Z] - g 0 1000000 1000000000 INFO [2024-01-29T01:34:20Z] Selected idmap: INFO [2024-01-29T01:34:20Z] - u 0 1000000 1000000000 INFO [2024-01-29T01:34:20Z] - g 0 1000000 1000000000 INFO [2024-01-29T01:34:20Z] Kernel features: INFO [2024-01-29T01:34:20Z] - closing multiple file descriptors efficiently: yes INFO [2024-01-29T01:34:20Z] - netnsid-based network retrieval: yes INFO [2024-01-29T01:34:20Z] - pidfds: yes INFO [2024-01-29T01:34:20Z] - core scheduling: no INFO [2024-01-29T01:34:20Z] - uevent injection: yes INFO [2024-01-29T01:34:20Z] - seccomp listener: yes INFO [2024-01-29T01:34:20Z] - seccomp listener continue syscalls: yes INFO [2024-01-29T01:34:20Z] - seccomp listener add file descriptors: yes INFO [2024-01-29T01:34:20Z] - attach to namespaces via pidfds: yes INFO [2024-01-29T01:34:20Z] - safe native terminal allocation : yes INFO [2024-01-29T01:34:20Z] - unprivileged file capabilities: yes INFO [2024-01-29T01:34:20Z] - cgroup layout: cgroup2 INFO [2024-01-29T01:34:20Z] - idmapped mounts kernel support: yes INFO [2024-01-29T01:34:20Z] Instance type operational driver=lxc features="map[]" type=container INFO [2024-01-29T01:34:20Z] Instance type operational driver=qemu features="map[cpu_hotplug:{} io_uring:{} vhost_net:{}]" type=virtual-machine INFO [2024-01-29T01:34:20Z] Initializing local database INFO [2024-01-29T01:34:20Z] Set client certificate to server certificate fingerprint=04e709e2c64c90b007a643bfe9c9252ca10b706a695cfb0292468268684fb41a INFO [2024-01-29T01:34:20Z] Starting database node id=1 local=1 role=voter INFO [2024-01-29T01:34:20Z] Loading daemon configuration INFO [2024-01-29T01:34:20Z] Binding socket socket=/var/lib/incus/unix.socket type="REST API Unix socket" INFO [2024-01-29T01:34:20Z] Binding socket socket=/var/lib/incus/guestapi/sock type="devIncus socket" INFO [2024-01-29T01:34:20Z] Binding socket socket="host(2):13517" type="VM socket" INFO [2024-01-29T01:34:20Z] Initializing global database INFO [2024-01-29T01:34:20Z] Connecting to global database INFO [2024-01-29T01:34:20Z] Connected to global database INFO [2024-01-29T01:34:20Z] Initialized global database INFO [2024-01-29T01:34:20Z] Firewall loaded driver driver=nftables INFO [2024-01-29T01:34:20Z] Initializing storage pools INFO [2024-01-29T01:34:20Z] Initializing daemon storage mounts INFO [2024-01-29T01:34:20Z] Initializing networks INFO [2024-01-29T01:34:20Z] All networks initialized INFO [2024-01-29T01:34:20Z] Cleaning up leftover image files INFO [2024-01-29T01:34:20Z] Done cleaning up leftover image files INFO [2024-01-29T01:34:20Z] Starting device monitor INFO [2024-01-29T01:34:20Z] Initialized filesystem monitor driver=fanotify path=/dev INFO [2024-01-29T01:34:20Z] Started seccomp handler path=/run/incus/seccomp.socket INFO [2024-01-29T01:34:20Z] Pruning expired images INFO [2024-01-29T01:34:20Z] Done pruning expired images INFO [2024-01-29T01:34:20Z] Pruning expired backups INFO [2024-01-29T01:34:20Z] Done pruning expired backups INFO [2024-01-29T01:34:20Z] Expiring log files INFO [2024-01-29T01:34:20Z] Daemon started INFO [2024-01-29T01:34:20Z] Updating images INFO [2024-01-29T01:34:20Z] Pruning resolved warnings INFO [2024-01-29T01:34:20Z] Done expiring log files INFO [2024-01-29T01:34:20Z] Done pruning resolved warnings INFO [2024-01-29T01:34:20Z] Updating instance types INFO [2024-01-29T01:34:20Z] Done updating images INFO [2024-01-29T01:34:22Z] Done updating instance types
Initializing Incus
We initialize the Incus server incusd
through the Incus client incus
.
debian@incus-compile:~$ incus admin init Would you like to use clustering? (yes/no) [default=no]: Do you want to configure a new storage pool? (yes/no) [default=yes]: Name of the new storage pool [default=default]: Name of the storage backend to use (ceph, dir, lvm, btrfs) [default=btrfs]: Create a new BTRFS pool? (yes/no) [default=yes]: Would you like to use an existing empty block device (e.g. a disk or partition)? (yes/no) [default=no]: Size in GiB of the new loop device (1GiB minimum) [default=5GiB]: 3 Would you like to create a new local network bridge? (yes/no) [default=yes]: What should the new bridge be called? [default=incusbr0]: What IPv4 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]: What IPv6 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]: Would you like the server to be available over the network? (yes/no) [default=no]: Would you like stale cached images to be updated automatically? (yes/no) [default=yes]: Would you like a YAML "init" preseed to be printed? (yes/no) [default=no]: yes config: {} networks: - config: ipv4.address: auto ipv6.address: auto description: "" name: incusbr0 type: "" project: default storage_pools: - config: size: 3GiB description: "" name: default driver: btrfs profiles: - config: {} description: "" devices: eth0: name: eth0 network: incusbr0 type: nic root: path: / pool: default type: disk name: default projects: [] cluster: null debian@incus-compile:~$
Launching a container in a VM
We can now launch system containers through the freshly-compiled Incus installation.
debian@incus-compile:~$ incus launch images:debian/12/cloud mycontainer Launching mycontainer debian@incus-compile:~$ incus list -c ns4t +-------------+---------+-------------------+-----------+ | NAME | STATE | IPV4 | TYPE | +-------------+---------+-------------------+-----------+ | mycontainer | RUNNING | 10.10.50.7 (eth0) | CONTAINER | +-------------+---------+-------------------+-----------+ debian@incus-compile:~$
Getting a shell in a container in a VM
We get a shell in a container in a virtual machine.
debian@incus-compile:~$ incus exec mycontainer -- sudo --login --user debian debian@mycontainer:~$
Closing up
You can shutdown the server by terminating the server process. For example, hit Ctrl+C on the terminal window with Incus running.
Conclusion
It’s quite easy to build Incus and test it out. The process is effortless once you setup a VM. As Incus is developed, you can run git pull
and receive the source code updates. Then, you can compile again and get the updated version running.
Troubleshooting
Error: VM agent isn't currently running
You have launched a virtual machine with Incus and you try to get a shell in it. But instead you got this message.
The virtual machine is still starting up and you are very fast with the keyboard. Try to connect again in a few moments.
Error: sudo: unknown user debian
Error: sudo: error initializing audit plugin sudoers_audit
You started a virtual machine with Incus and you try to get a shell in it as a non-root account using sudo
. You get these two error messages.
It is possible that you are launching an image that does not create a non-root account (here, debian
for the Debian image). Or, you are trying to get access too fast that the VM has not yet completed the booting process. The /cloud
images have instructions that create a non-root account. You can get a shell with incus shell mycontainer
and investigate.
Error: make: *** [Makefile:37: build] Error 2
You are compiling Incus by running make
and the compilation fails with this output. Sudden failure without much information.
... github.com/lxc/incus/internal/server/db/generate/file github.com/lxc/incus/internal/server/db/generate/db github.com/lxc/incus/internal/server/db/generate github.com/lxc/incus/test/dev_incus-client github.com/lxc/incus/test/syscall/sysinfo make: *** [Makefile:37: build] Error 2 debian@incus-compile:~/incus$
You have not enabled a recent version of Go and you are using your distro’s Go package. You may have to edit the $PATH
so that the fresh Go takes precedence. You can verify that this is the issue by running make
again. In that case, the error message would be different, sort of the following. Need a newer Go compiler.
debian@incus-compile:~/incus$ make CC="cc" CGO_LDFLAGS_ALLOW="(-Wl,-wrap,pthread_create)|(-Wl,-z,now)" go install -v -tags "libsqlite3" ./... github.com/lxc/incus/shared/idmap # github.com/lxc/incus/shared/idmap shared/idmap/set_linux.go:142:23: undefined: filepath.SkipAll note: module requires Go 1.20 github.com/go-acme/lego/v4/certificate # github.com/go-acme/lego/v4/certificate ../go/pkg/mod/github.com/go-acme/lego/v4@v4.14.2/certificate/errors.go:31:16: undefined: errors.Join note: module requires Go 1.20 make: *** [Makefile:37: build] Error 2 debian@incus-compile:~/incus$
Error: AppArmor support has been disabled because 'apparmor_parser' couldn't be found
You have compiled Incus and you try to run the Incus server (incusd
). You get an error related to AppArmor while you do have AppArmor support in your Linux distribution. The binaries are in /sbin/
.
You need to add /sbin
to the $PATH
when you run incusd
.
Error: Firewall failed to detect any compatible driver, falling back to "xtables"
You have compiled Incus and you try to run the Incus server (incusd
). You get an error related to the Linux kernel firewall while you do have firewall support in your Linux distribution. The binaries are in /sbin/
.
You need to add /sbin
to the $PATH
when you run incusd
.
Error: QEMU command not available for CPU architecture
You have compiled Incus and you try to run the Incus server (incusd
). You get an error related to Qemu support for running virtual machines with Incus. The packages are missing.
You need to install the package qemu-system
to get support for multiple architectures. Or, just qemu-system-x86
if you want to save space. Or, ignore the message if you only plan to launch only system containers. Or, ignore if your system does not support VM nesting.