multi lxc with haproxy

Near the beginning of last year I hit a few issues with some of my Docker containers and part of my CI/CD pipeline. Around the same time I seemed to be reading more about LXC, and a few people on IRC mentioned that it was worth learning. I decided to take a step back from Docker and give LXC a go.

what the chroot

LXC or Linux Containers, is a virtualisation method allowing the kernel to be used between multiple environments or containers. While traditionally with Docker you would run single applications inside a container then network them together (web server, database, etc.) LXC gives you a “full” Linux system but unlike a virtual machine it shares the same kernel as the host.

There are pros and cons to LXC but I don’t want to get into that in this post. If you would like to know more about LXC check out the official website. I should also point out that I have stuck with LXC and not LXD, which is a next generation container manager.

Setting up LXC is straightforward by following the official guide.

Creating a container is as easy as

lxc-create -t download -n <name>

selecting an image from the list shown.

Or if you know the image you want to use you can specify it

lxc-create -n <name> -t download -- --dist <distro> --release <release_number> --arch <architecture>

After I created my container I started it and set it up as I would any other system. This then became my “base image”. Any new container I wanted could be cloned from this so it is already set up. I renew the base image periodically with updates etc.

Cloning a container can be done by incanting

lxc-copy -n ${BASE} -N ${NEW}

This command is suppose to change the hostname of the cloned container but I found it didn’t. To remedy that incant

sudo sed -i "s/${BASE}/${NEW}/" ${HOME}/.local/share/lxc/${NEW}/rootfs/etc/hostname

virtualise all the things

I was using Docker to run a number of things on a single VPS, using an Nginx container as a proxy.

For no particular reason, with LXC I opted for HAProxy. My HAProxy is running in a container. On the host server I set the following firewall rules to send traffic to the HAProxy container

iptables -t nat -I PREROUTING \
        -i ${INTERFACE} \
        -p TCP \
        -d ${PUBLIC_IP_ADDRESS}/${CIDR} \
        --dport 80 \
        -j DNAT \
        --to-destination ${HAPROXY_CONTAINER_IP}:80

iptables -t nat -I PREROUTING \
        -i ${INTERFACE} \
        -p TCP \
        -d ${PUBLIC_IP_ADDRESS}/${CIDR} \
        --dport 443 \
        -j DNAT \
        --to-destination ${HAPROXY_CONTAINER_IP}:443

Then I could login to HAProxy container to configure it. The config file may be either /etc/haproxy.cfg or /etc/haproxy/haproxy.cfg, on my container it is the latter.

Of course I want to use SSL and it is advised to set the Diffie-Hellman parameter to 2048 bits instead of the default 1024. I included the following to the global section of haproxy.cfg

tune.ssl.default-dh-param 2048

I am using LetsEncrypt for my SSL certificates, so I installed certbot. This will be used later to generate our SSL certificates. One of the best solutions I found for LetsEncrypt with HAProxy is from janeczku on Github. I put a copy of the acme-http01-webroot.lua script into /etc/haproxy/ and added the following to the global section of haproxy.cfg

lua-load /etc/haproxy/acme-http01-webroot.lua

To tell HAProxy to use SSL I had to configure a couple of frontends after the default section

frontend http_frontend
    bind *:80

    acl url_acme_http01 path_beg /.well-known/acme-challenge/
    http-request use-service lua.acme-http01 if METH_GET url_acme_http01

    redirect scheme https

frontend https_frontend
    bind *:443

This config will redirect HTTP traffic on port 80 to HTTPS on 443.

Now I can declare a backend and acl to route traffic. For the sake of example my LXC container is called “pyratelog” and the domain I am pointing to is “log.pyratebeard.net”.

The acl is declared in the https_frontend section

frontend https_frontend
    bind *:443

    acl pyratelog hdr(host) -i log.pyratebeard.net
    use_backend pyratelog if pyratelog

Then beneath the frontend the backend section is configured

backend pyratelog
    balance leastconn
    http-request set-header X-Client-IP %[src]
    server pyratelog pyratelog:80 check

LXC has built in container name resolution, so you can use the name of the container instead of its IP address.

A reload of HAProxy picks up the changes.

I used `certbot to request a new SSL cert

certbot certonly --text \
    --webroot --webroot-path /var/lib/haproxy \
    -d log.pyratebeard.net \
    --renew-by-default \
    --agree-tos \
    --email me@email.com

This created two PEM files, a private key and a chain file. I combined these into one file to be read by HAProxy

cat /etc/letsencrypt/live/log.pyratebeard.net/privkey.pem \
    /etc/letsencrypt/live/log.pyratebeard.net/fullchain.pem \
    | tee /etc/letsencrypt/live/pem/pyratelog.pem

Now I had to alter the https_frontend section to point to the SSL cert directory

frontend https_frontend
    bind *:443 ssl crt /etc/letsencrypt/live/pem/

and reloaded HAProxy.

When I added another LXC container behind HAProxy I simply add a new backend and include an acl in the https_frontend, so it would looks something like this

frontend https_frontend
    bind *:443

    acl pyratelog hdr(host) -i log.pyratebeard.net
    use_backend pyratelog if pyratelog

    acl pyrateweb hdr(host) -i pyratebeard.net
    use_backend pyrateweb if pyrateweb

backend pyratelog
    balance leastconn
    http-request set-header X-Client-IP %[src]
    server pyratelog pyratelog:80 check
    
backend pyrateweb
    balance leastconn
    http-request set-header X-Client-IP %[src]
    server pyrateweb pyrateweb:80 check

Then I ran the certbot command again, and combine the PEM files

certbot certonly --text \
    --webroot --webroot-path /var/lib/haproxy \
    -d pyratebeard.net \
    --renew-by-default \
    --agree-tos \
    --email me@email.com

cat /etc/letsencrypt/live/pyratebeard.net/privkey.pem \
    /etc/letsencrypt/live/pyratebeard.net/fullchain.pem \
    | tee /etc/letsencrypt/live/pem/pyrateweb.pem

A reload of HAProxy picks up the changes.

From now on renewing an SSL cert is done by incanting

sudo certbot certonly --text --webroot --webroot-path /var/lib/haproxy -d log.pyratebeard.net

then combine the PEM files again, overwriting the previous file, and reloading HAProxy.

I was happy with how easy it was to get LXC running with HAProxy, and now comfortably run a number of containers on a single host.

Docker hasn’t completely been removed from my systems, depending on the use case I do lean towards LXC a bit more these days. I have been running my LXC setup for over a year and have had no issues. The “CI/CD” has had to change though, and I will cover how I publish these blog posts onto my LXC container in a later post.