Hi, author of the article, pleased to see it here! I'd really like to hear more from folks about how they're using LXC and/or LXD, and what they think their greatest strengths are compared to Docker or Kubernetes.

I ran LXD for about a year in my home lab. Unfortunately, the easiest way to get it running is by installing Snap; which I didn't do. Instead I ran it on Alpine Edge (one of the few distros that actually has it in their package manager). LXD kept breaking after system updates and I got tired of troubleshooting. I suppose that's just part of the perils of running bleeding edge.

When LXD was running, I found that it feels like something between a VM and OCI container. In the whole pets vs cattle analogy, LXD containers definitely feel more like pets.

On the host, concepts are similar to OCI containers (networking, macvlan, port mapping, etc). I like its approach to building up containers with several commands instead of one long command. You can add and remove ports, volumes, and even GPUs to running containers. It doesn't have compose files, I just used bash scripts to create reproducible containers.

Inside the LXD container is where things get different. You're containerizing a whole OS, not an application. So you end up administering your container like a VM. You jump inside it, update its packages, install whatever application and dependencies you need. It doesn't work like a glorified package manager (which is how most homelabbers use Docker). As a long time VM user I actually prefer installing things myself, but I can see this turning away many people in the selfhosting community.

I liked LXD and would've kept using it if it weren't so intimately linked to Snap. Last month it failed to start my containers after a system update...again. So I moved that box over to Rocky with Podman containers running as SystemD units. I do kinda miss having fast, throwaway OSes on that server for experiments (sometimes its nice to have pets).

I fucking love LXC! It allowed me to “virtualize” several physical servers on a single newer server, keeping all of the original setup (except some networking changes). Combined with ipvlan, I could even work around the MAC address restriction of my server provider, with both IPv4 and IPv6.

I also use it to host a Jitsi instance. Jitsi is rather picky about the underlying distribution.

All this without the overhead of an actual virtual machine of course.

The only pain I have with LXC is its behavior when errors occur. It’s not easy to tell what’s wrong, the error messages are worthless. Sometimes the debug log can help, sometimes not so much.

I have been running a handful of Debian systems in containers since OpenVZ was state-of-the art. As the hardware expired, I moved to linux-vserver, then LXC, then LXD ... at which point it all shifted from straightforward text configuration files to SQLite mediated by CLI programs. When I could not figure out the incantation required for some gnarly configuration rotations ("computer says no"), I was forced to shut everything down, alter the SQLite files directly, and spin it all back up again. On the next hardware refresh, I switched back to LXC.

This sort of mediation makes sense and works great for ephemeral containers in Kubernetes. For full virtual servers, LXD is the worst of both worlds.

I run everything in LXD containers on my home server(s), with Debian as the host (manually compiled, not the snap). Most of the setup is automated with Ansible that I hacked to read Python scripts instead of YAML.

LXD is a treat to work with, and I feel its container model is perfect for home/small business servers that need to run lots of third-party software, much of which would work poorly in a "proper" stateless container.

I use LXD + ansible for my personal projects. IMO the main benefit I see is it requires very low cognitive overhead. It's insanely easy to set up a container, I only need to remember like 3 commands ever. These programs will never be clustered either so theres no point in having a scheduler.

Lxd is fantastic but it became unusable for me once it started being distributed only as a snap.

I still use lxd on an Ubuntu 18.04 host that still has the deb instal but looking forward I’m trying to get things migrated to systemd-nspawn.

My last company had hundreds possibly thousands of LXC containers, and we orchestrated everything via saltstack (which is similar to ansible or puppet if you aren't familiar).

The justification was that we needed our SaaS to also work on-prem for financial companies and government entities, and thus we could not count on kubernetes or any specific cloud vendor to be available, so we rolled our own orchestration built on ubuntu hosts and (for some stupid reasons) centos LXC containers running on top of those.

LXC is a good system and all, but what we were doing with it was a total nightmare, we were basically trying to reinvent kubernetes using traditional configuration management and LXC with two flavors of linux and a million dependencies.

These seem like cool tech and I know things like proxmox can run them pretty easily.

Still Dockerfiles and all the magic that goes with them was just so much easier to pick up and convince others on the team to try.

Do either of them support the concept of “layers” like Docker does?

I think that feature combined with overlayfs2 is quite useful, despite my many criticisms of Docker.

It is sort of a middle ground between Nix like granularity which requires rewriting upstream, and big LXC blobs created with shell scripts.

Although I also think we need some kind of middle ground between docker and nix :)

I tried to invent docker and kubernetes with LXC back when docker was super alpha and kubernetes was barely a fetus. I failed, obviously, but learned a lot. It's kinda the only reason I managed to pick up kubernetes later on, honestly. Sometimes, not knowing what you're doing is an asset.

systemd-nspawn is worth a look - I found it more intuitive, well designed and security conscious (within reason for containers) than the other container options. It must get some serious use behind closed doors somewhere, because the quality outstrips the amount of publicity it gets

I use both lxc and docker and have uses cases for both, I think it really comes down to how stateful something is or how lazy I feel about writing a Dockerfile. I had a really hard time with learning lxd and really only got into using lxc without the daemon.

One trend with docker that I personally don't like is that a lot of projects prefer docker-compose over regular Dockerfiles (though some of them support both), and this leads to a lot of bloat with how many containers it takes to run one application and duplication where each app will need its own database, redis, webserver, etc. containers.

That's not a problem when you are at the scale to need that kind of orchestration and have the resources to run that many containers, but personally I would rather have one database host that all my containers can talk to and one reverse proxy/webserver to make them accessible to make better use of resources and get better density on one host.

One downside with lxc is that there's been some fragmentation with the image creation side of it, I had been used to using lxc-templates for years, which are just shell scripts for bootstrapping and configuring your container and pretty common between distros. I found a bug with creating containers for the latest version of Alpine that I fixed, and only then did I find out that lxc-templates are deprecated and essentially unmaintained, and that now distrobuilder is the preferred way to build lxc containers, at least per Canonical. That seems to be another Canonical-ism, since the distrobuilder docs say that snap is the only way to install it unless you build from source, so that's a huge no from me, and I also didn't feel like learning the yaml config for it.

I was actually considering moving my docker daemon into an unprivileged lxc container like the article mentions, but haven't gotten around to it.

lxc/lxd was the reason I went with ubuntu, but snap is why I migrated my containers to docker (and will soon move to debian)

My shop went from VMware to OpenShift around the time I transitioned from systems to whatever it is I do now. In my private life I've used kvm since it was first rolled out, but over the last few years have redeployed all my home services to docker. I did try lxc for a while, and think it could replace kvm for lab work. On the other hand, lxd never seemed worth the effort: especially with its snapd dependency. Podman seems like a good alternative to docker, but I'm not sure migrating to it would be worth the effort as long as docker sticks around.

How do you all manage the security side for LXC containers? As I understand it, any local user allowed to run lxc containers can effectively spin up a root level access container on the host, example: run a privileged container and mount / inside the container? Is there anyway to mitigate this?

I love using LXC, Finding documentation and such online regarding things however is an absolute pain in the ass in trying to figure out if the LXC commands they are talking about older depreciated syntax's, LXD which uses 'lxc' for their command line, Or the modern version of LXC which you are trying to make use of....

I use an LXC to run Docker and a bunch of containers on my Proxmox VE machine :). LXC is interesting, but you can't beat the ecosystem of Docker images available (and maintained, updated, etc.)

Have a few apps running in plain LXC containers (like AdguardHome) but maintenance is non-free (unlike a docker-compose stack with Watchtower keeping everything nice and fresh).

Back in 2016, I thought it'd be nice if LXC/D had something like a Dockerfile. It would just be a small layer or tool to do it, proved with a little python script: https://github.com/jonatron/lxf

Does LXC and LXD support the functional equivalent of Dockerfile or docker-compose.yml? I appreciate how Docker has become widely accepted for defining reproducible environments but have always found it to be rather heavy on resources.

I used LXC to virtualize Mastodon instances for a while. It works as intended. The illusion was that there were separate kernels all on the same machine.

If you are interested in this kind of lightweight containerization, the best tool to consider would be FreeBSD jails. They are available in FreeBSD since version 4.0 released on March 14, 2000 and at this point are a very mature solution to this problem. You can even attach dedicated network cards to jails and prevent them from being visible on the host. Moreover their integration with the base os and system utilities is seamless and very mature.

Linux folks love saying: use the right tool for the right job when it comes to telling people why Kubernetes is a strong enough reason for any company to switch to linux, however when it comes to things such as lightweight containers (or ZFS-on-root for the matter) for some reason the same reasoning isn't applied.

LXC and LXD look like a patchwork compared to FreeBSD jails. If you are on the desktop and want to run graphical apps, maybe it can make some kind of sense if you are already using Linux for everything else. But if you want to have a lightweight container setup on a headless server, FreeBSD is by far the best option.

In FreeBSD land, no one boasts of running 5 virtualized environments on a single machine because this is just a common thing for us and part of every workfow. You want to work on some github repository? You create a new jail in 1 minute using a tool such as cbsd.

A server with 32gb of RAM can easily run 1000s of jails, even a small VPS with 1-2gb of RAM can run a dozen jails or more without problems. It is intriguing that so many people dismiss the power of that.

And it is intriguing that people easily accept: "there is no equivalent to dockerfiles but it's fine because things can be scripted easily with basic shell commands" when it comes to LXC/LXD but people dismiss this very argument when FreeBSD folks try to explain why there is no need for Kubernetes on FreeBSD.

Anyway, if you run a startup not operating a resource intensive products (like ML requiring spinning up massive amounts of cores and RAM in an irregular basis) FreeBSD and jails can really save massive amounts of time and allow you to focus on the right thing. just run everything from a single Root VPS (usually hosting companies already handle things such as RAID replication and hardware redundancy transparently) and before you reach scalability problems, you'll already be making millions of dollars in revenue.

Even after, running thousands of jails on a single hosy essentially means doing what AWS and other cloud providers are doing. This doesn't prevent you from builing a small high-availability layer on top to ensure that critical apps run from at least two different physical hosts. This isn't very difficult to build with things like Consul, HA proxy, built-in system utilities and a small custom web interface.

I use LXD for most of my dev and testing environments, as well as the container for most network services.

Even my mail server runs in LXC, and it is a direct descendant of a Red Hat 5 server from ~20 years ago!

I've often seen LXD vs Docker described as LXD being better suited to running distros and distro-like environments, with Docker being better suited to single-application environments.

So that's what I do. For many of dev and testing environments, there's an LXD container, which I use over SSH/rsync daily. Compiles, long-running tests, bandwidth heavy network actions and so on are done in those containers, on a few fast servers in data centres. It's much faster than my laptop.

One motivation for running Linux in LXD containers instead of on bare metal was to allows me to decouple changes to host OS version and networking from the containers in which I do most my work. Previously I used bare metal, i.e. ran Linux on servers, but found it annoying that I couldn't update the host OS and especially the kernel or disto major version, without shutting down everything in Screen sessions and long-lived networking tests that are not so easy to shut down and restart (without a rewrite anyway).

Being containers, they can use host filesystems and devices almost directly. So they are great for things like I/O performance tests, with confidence that it's basically testing host performance and behaviour.

Unfortunately, after getting deep into both those things, I found LXD wasn't quite as run-a-distro friendly as it first appeared, and it wasn't as reliable at replicating host I/O performance either.

Doing file I/O with "shift=true" host-filesystem mounts in LXD (the most sensible mode) turns out to be have very much slower O_DIRECT performance than it should, so that screwed up my database storage tests unexpctedly until I realised. Now I run low-level storage performance tests outside LXD, because I don't trust it.

As for distro-like environments, I eventually found stateful snapshot+restore or migation of dev and test environments is permanently broken. No LXD container I've used over many years has ever successfully been able to live-snapshot/migrate, without hitting an error which prevents it. This isn't some obscure bug, either. It's never worked, and my browsing of forums and issue trackers leads to the view that it's not actually expected to work for almost any distro-like environment, despite being one of the headline features.

As a result, one of the main factors motivating using LXD instead of bare metal for me turned out not to work. It's always possible to shut down a container, but that loses so much state, long-lived Screen sessions, running processes and so on that it's about as disruptive as updating bare metal. I.e. no real advantage.

I could switch back to VMs, which are excellent for snapshotting and migration, as I used to use (with libvirt+kvm) but they have a different problem for much of my work: Host filesystem/blockdev sharing is relatively slow. Not only could I not use them to measure performance against various kernels on real hardware (it would be measuring the VM as much as anything), I also run many data-intensive jobs, and the closer I can get to those host storage devices, the better.

The last reason it's felt buggy and rough is when removing a host-container filesystem mount from a container, it has often deleted the mount point on the host as well, disrupting other processes using it. Operations on the container are not supposed to change the host itself. You get used to working around this, but it's annoying.

Ah well, nothing's perfect. It's still a very useful tool, with some rough edges. I'd look into fixing the I/O performance and host-unmounting bug if I thought the snapshot/migration feature would be made to work someday, but because that looks unlikely, plus issues that came with Snap, I'm not as motivated and will live with hackish workarounds.

For things like my mail, web, SpamAssassin and other services, LXD has been great, and the issues I encountered aren't really a problem. This is the sort of thing which Docker is pretty good for as well. However, due to history I've tended to keep servers going for a long time (with my mail server winning the crown as it's an image that's been gradually modified and upgraded for about 20 years, across many different ways of running Linux in container-like environments, starting with chroot). LXC/LXD is better than Docker for this use case, though either can be made to work.

No layer caching mechanisms

LXC is particular useful on shared web hosting.

Docker's easier and does all the same stuff. It's integrated with everything else, has an interface everyone is familiar with, uses OCIs, is extensible, is supported on basically every platform, has a simpler config file, tons of community support, and now runs rootless.

Does anyone have a use case where they couldn't use Docker? I'm sure they exist but the list must be tiny. Listed in the article is:

  - run systemd in a container
    - why?? does your system not have systemd or equivalent?
    - bad design; how will you monitor what's running and what's not, or upgrade an individual service in this container? restart everything?
    - apparently systemd can run in docker?? https://medium.com/swlh/docker-and-systemd-381dfd7e4628
  
  - run lxc/docker in a container
    - docker-in-docker