I’ve seen a couple of instances around asking if it’s possible to self host Lemmy fronted by Traefik, figured I can share my test setup with the community. This configuration is still in testing phase but seems to work in a federated configuration.
The Design
To run in an elastic and highly available way (where possible with my current hardware) I’ve opted to use Docker Swarm as it’s one of the easiest ways to get started with multi-node deployment. There are a few limitations with Swarm and I’m planning on moving to a K3s cluster in the future.
For added protection I’ve chosen to front my services with Cloudflare for the time being, there’s a definite tradeoff of privacy vs security here and since I’m not able to afford traffic scrubbing center and deal with any ISP issues for now it’ll be used.
The Setup
A few prerequisites are needed for this type of deployment although I think it should be fairly easy to adapt it to a single node setup.
- Docker Swarm - 2x Manger nodes 2x Worker nodes
- NFS - TrueNAS, used to store LE certs and pictrs blobs
- Cloudflare’d domain - optional but makes life much easier
- PostgreSQL Database - A dedicated postgresql VM for persistent database
The Config
I’ll skip some of the basic setup configurations since there are plenty of guides out there and focus on the docker compose files for the cloudflare, traefik, and lemmy stacks.
Starting with the simplest, you can follow any guide to create a new tunnel to use for this setup, the compose file should look something like this:
cloudflare/docker-compose.yml
version: "3.8"
services:
cloudflare:
image: cloudflare/cloudflared:2023.5.1-amd64 # always hardcode your versions to avoid unexpected changes
deploy:
mode: global # in swarm we will deploy one instance per manager node (more nodes more redudnacy/LB)
placement:
constraints:
- node.role == manager
networks:
- bridge # a default created network allowing connectivity to the rest of the network (and internet to connect upstream)
- traefik_dmz # an overlay internal private network for LB'd services
command:
- tunnel
- --no-autoupdate
- run
- --token=*** # replace with the token issued by cloudflare
networks:
bridge:
external: true
traefik_dmz:
external: true
important: Once you’ve created your tunnal add a public host pointing
your-lemmy.domain
tohttps://traefik
and under TLS enable theNo TLS Verify
option. Since both cloudflare and traefik services are on a common network cloudflare can access the internal service VIP by calling traefik (matching the service name)traefik/docker-compose.yml
version: '3.8'
services:
traefik:
image: traefik:v2.10
environment:
- "CF_DNS_API_TOKEN=***" # your cloudflare API token with zone read and DNS edit permissions
ports:
- target: 80
published: 80
protocol: tcp
mode: host
- target: 443
published: 443
protocol: tcp
mode: host
- target: 8082
published: 8082
protocol: tcp
mode: host
deploy:
mode: global
placement:
constraints:
- node.role == manager
labels:
- traefik.enable=true
- traefik.docker.network=traefik_dmz
- traefik.http.routers.traefik-dashboard.rule=HostRegexp(`{subhost:your-hostname-pattern[\d]+}.homelab`) # To access the dashboard you can use your hosts' FQDN pattern and access https://your-hostname-pattern1.homelab
- traefik.http.routers.traefik-dashboard.entrypoints=https
- traefik.http.routers.traefik-dashboard.tls=true
- traefik.http.routers.traefik-dashboard.service=api@internal
- traefik.http.services.traefik.loadbalancer.server.port=8080
command:
- "--log=true" # System logs good for debugging
- "--log.level=WARN"
- "--accesslog=false" # Access logs good for debugging but so noisy
- "--accesslog.format=json"
- "--accesslog.fields.defaultmode=keep"
- "--accesslog.fields.headers.defaultmode=keep"
- "--accesslog.fields.headers.names.Authorization=drop"
- "--api=true" # enable for dashboard access
- "--api.dashboard=true"
- "--ping=true" # enable for external LB health checks
- "--ping.entrypoint=ping"
- "--serverstransport.insecureskipverify=true"
- "--global.checknewversion=false"
- "--global.sendanonymoususage=false"
- "--entrypoints.http=true"
- "--entrypoints.http.address=:80"
- "--entrypoints.http.http.redirections.entrypoint.to=https"
- "--entrypoints.http.http.redirections.entrypoint.scheme=https"
- "--entrypoints.https=true"
- "--entrypoints.https.address=:443"
- "--entrypoints.https.forwardedheaders.insecure=true"
- "--entrypoints.https.forwardedheaders.trustedips=10.0.0.0/8" # limit the x-forwarded-for header trust to your external LB
- "--entrypoints.ping=true"
- "--entrypoints.ping.address=:8082"
- "--providers.docker=true" # This will allow us to auto detect configured services and forward traffic
- "--providers.docker.exposedbydefault=false"
- "--providers.docker.swarmmode=true"
- "--certificatesresolvers.cloudflare.acme.dnschallenge=true" # LetsEncrypt using cloudflare DNS challenge
- "--certificatesresolvers.cloudflare.acme.dnschallenge.provider=cloudflare"
- "--certificatesresolvers.cloudflare.acme.caserver=https://acme-v02.api.letsencrypt.org/directory"
- "--certificatesresolvers.cloudflare.acme.email=your@email.address"
- "--certificatesresolvers.cloudflare.acme.storage=/config/secrets/acme.json" # you'll need to create this file ahead of time with chmod 600 perms
- "--certificatesresolvers.cloudflare.acme.dnschallenge.delayBeforeCheck=42"
- "--certificatesresolvers.cloudflare.acme.dnschallenge.resolvers=1.1.1.1,1.0.0.1"
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro # Needed in R/O mode to allow the docker provider to work
- traefik_config:/config # Local store for our LE certs
networks:
- bridge
- traefik_dmz
volumes:
traefik_config:
driver_opts:
type: nfs
o: addr=your-nfs-ip-or-name,nolock,soft,rw
device: :/path/to/your/nfs/export
networks:
bridge:
external: true
traefik_dmz:
external: true
lemmy/docker-compose.yml
version: "3.8"
services:
lemmy-server:
image: dessalines/lemmy:0.17.3
configs:
- source: lemmy.hjson # A pre created static config file supported by docker swarm
target: /config/config.hjson # See https://join-lemmy.org/docs/en/administration/configuration.html
deploy:
mode: replicated
replicas: 1
placement:
constraints:
- node.role == worker
labels:
- traefik.enable=true
- traefik.docker.network=traefik_dmz
# the following roughly translates to the Nginx config from the offical
# doc at https://github.com/LemmyNet/lemmy-ansible/blob/main/templates/nginx.conf#L63
- traefik.http.routers.leddit-api.rule=Host(`your-lemmy.domain`) && (PathPrefix(`/api`, `/pictrs`, `/feeds`, `/nodeinfo`, `/.well-known`) || Method(`POST`) || Headers(`accept`, `application/activity+json`) || Headers(`accept`, `application/ld+json; profile="https://www.w3.org/ns/activitystreams"`))
- traefik.http.routers.leddit-api.entrypoints=https
- traefik.http.routers.leddit-api.tls=true
- traefik.http.routers.leddit-api.tls.certresolver=cloudflare
- traefik.http.routers.leddit-api.tls.domains[0].main=your-lemmy.domain
- traefik.http.services.leddit-api.loadbalancer.server.port=8536
networks:
- traefik_dmz
- bridge
- app
lemmy-ui:
image: dessalines/lemmy-ui:0.17.3
environment:
# this needs to match the hostname defined in the lemmy service
- LEMMY_UI_LEMMY_INTERNAL_HOST=lemmy-server:8536
# set the outside hostname here
- LEMMY_UI_LEMMY_EXTERNAL_HOST=your-lemmy.domain
- LEMMY_HTTPS=true
- NODE_ENV=production
deploy:
mode: replicated
replicas: 1
placement:
constraints:
- node.role == worker
labels:
- traefik.enable=true
- traefik.docker.network=traefik_dmz
- traefik.http.routers.leddit-web.rule=Host(`your-lemmy.domain`)
- traefik.http.routers.leddit-web.entrypoints=https
- traefik.http.routers.leddit-web.tls=true
- traefik.http.routers.leddit-web.tls.certresolver=cloudflare
- traefik.http.routers.leddit-web.tls.domains[0].main=your-lemmy.domain
- traefik.http.services.leddit-web.loadbalancer.server.port=1234
networks:
- traefik_dmz
- app
pictrs:
image: asonix/pictrs
environment:
- PICTRS__API_KEY=***
user: 991:991
volumes:
- pictrs_data:/mnt
deploy:
mode: replicated
replicas: 1
placement:
constraints:
- node.role == worker
networks:
- app
configs:
lemmy.hjson:
external: true
networks:
app: # a stack specific network for limited communication scope (only services on this network can connect to pictrs)
internal: true
traefik_dmz:
external: true
bridge:
external: true
volumes:
pictrs_data:
driver_opts:
type: nfs
o: addr=your-nfs-ip-or-name,nolock,soft,rw
device: :/path/to/your/nfs/export
Since we have multiple manager nodes you can deploy HAProxy and load balance across the manager nodes to their traefik instances using /ping for health checks and a split DNS on your local network to reduce Cloudflare traffic. Using another LE certificate on HAProxy and exposing it externally will allow you to bypass Cloudflare all together if/when is needed.
I hope this little(?) guide will be helpful.
I am currently also using traefik on top of the lemmy stack but I was too lazy so I just proxied the nginx there and prayed to the god of routing lol
Might steal your traefik paths for lemmy though, thanks
Funny enough I already made a few changes to the traefik configs, I saw someone else’s post and if it’s safe to assume that any request with Accept header starting with application/ should be routed to the Lemmy server, the following would work as well:
- traefik.http.routers.leddit-api.rule=Host(`leddit.social`) && (PathPrefix(`/api`, `/pictrs`, `/feeds`, `/nodeinfo`, `/.well-known`) || Method(`POST`) || HeadersRegexp(`Accept`, `^[Aa]pplication/.+`))
I’ve also added caching policies to make sure none of the API responses are cached and having the UI be cached explicitly since it’s not done today.
services: lemmy-server: deploy: labels: - traefik.http.routers.leddit-api.middlewares=no-cache - traefik.http.middlewares.no-cache.headers.customresponseheaders.Cache-Control=no-store ... lemmy-ui: deploy: labels: - traefik.http.routers.leddit-web.middlewares=cache-control - traefik.http.middlewares.cache-control.headers.customresponseheaders.Cache-Control=public, max-age=86400