☕ Coffee anyone ?
Sign in Subscribe
8 min read Dev Log

Migrating Funkwhale 1.4 to 2.0 with Docker Compose

Funkwhale logo on the seam between 1.4.0 and 2.0.12, with a broken flute

Dev Log — Funkwhale. Upgrade across a major version by building a second host next to the first, rehearsing, then switching the reverse proxy.

Note : this MO has been generated by Claude Code with my own directives == the ones I use in my professionnal life :
- you backup anything you can : VM, Database
- you can always rollback
- you dry-run test when you can
- you don't delete anything as long as daily use has been confirmed functional

Principle

  • OLD (the 1.4 host) is never modified: it is the rollback.
  • NEW is built from scratch with the official 2.0.12 Compose files. Go straight to 2.0.12, never stop at 2.0.0.
  • Data moves as a database dump plus a copy of media/. The music library stays where it is.
  • Cutover = one line in the reverse proxy.

Requirements: NEW with Docker Engine and the Compose plugin (2 vCPU, 6 GB RAM and 40 GB disk is a reasonable start), same hostname and same music paths as OLD, and a reverse proxy that terminates TLS and forwards X-Forwarded-Proto. On a VM, use CPU type host.

Setup (on the workstation)

export OLD=old.example.internal NEW=new.example.internal NEW_IP=192.168.0.20
export FQDN=funkwhale.example.org V=2.0.12
fw()  { ssh $NEW "cd /srv/funkwhale && $(printf '%q ' "$@")"; }   # run a command on NEW in /srv/funkwhale
ofw() { ssh $OLD "$(printf '%q ' "$@")"; }                         # run a command on OLD
Q="select 'users',count(*) from users_user union all select 'tracks',count(*) from music_track union all select 'uploads',count(*) from music_upload"

Run everything from a Bash terminal (Linux, macOS, or WSL on Windows) on a workstation that can SSH to both machines. Both VMs run on Proxmox here, but nothing below depends on it. fw and ofw are not installed commands, just shell functions defined above: fw <command> runs the command on NEW over SSH from /srv/funkwhale, and ofw <command> runs it on OLD. printf '%q ' escapes each argument so quotes and $ survive the trip. They only exist in the shell where you pasted this block, so define them again in a new terminal.

Use sudo in front of docker on a host where your user is not in the docker group.

Procedure

1. Inventory OLD (read-only)

ofw docker exec funkwhale-postgres-1 psql -U postgres -At -c "$Q" > fw-ref-old.txt
ofw grep -e ^FUNKWHALE_HOSTNAME= -e ^MUSIC_DIRECTORY_PATH= -e ^MUSIC_DIRECTORY_SERVE_PATH= -e ^NGINX_MAX_BODY_SIZE= -e ^EMAIL_CONFIG= -e ^DEFAULT_FROM_EMAIL= /srv/funkwhale/.env
ofw docker exec funkwhale-postgres-1 psql -U postgres -At -c "select version()"

Watch points. fw-ref-old.txt holds the reference counters: you compare them at step 16. The two MUSIC_DIRECTORY_* values must be identical on NEW (import-in-place paths are stored as-is in the database). PostgreSQL must be 12 or newer.

2. Install Docker on NEW

Follow the official guide for your distribution: https://docs.docker.com/engine/install/

ssh $NEW 'sudo systemctl enable --now docker && docker compose version'

Watch points. Use the official Docker repository, not the distribution's own docker package. Expected: a Compose v2 version string.

3. Rotate container logs

ssh $NEW 'echo {\"log-driver\":\"json-file\",\"log-opts\":{\"max-size\":\"10m\",\"max-file\":\"3\"}} | sudo tee /etc/docker/daemon.json && sudo systemctl restart docker'

Watch points. Do it from day one: unbounded JSON logs fill the disk. Check with docker info --format '{{.LoggingDriver}}' (expected json-file).

4. Create directories, mount the music share read-only

ssh $NEW 'sudo install -d -o $USER -g $USER /srv/funkwhale/data/media /srv/funkwhale/data/static /srv/funkwhale/data/music'
ssh $NEW 'ln -sfn /srv/music /srv/funkwhale/data/music/library'

Watch points. Adapt the paths to your layout. If the library lives on a network share, mount it read-only (ro) until the cutover: nothing on NEW can write to it by mistake. Mount it before Docker starts (RequiresMountsFor= in a Docker systemd drop-in), otherwise a container sees an empty folder.

5. Restrict port 5002 to the proxy

Save script A (see appendix) as fw-docker-user, replace PROXY_IP in it, then:

scp fw-docker-user $NEW:/tmp/ && ssh $NEW 'sudo install -m 755 /tmp/fw-docker-user /usr/local/sbin/fw-docker-user'
ssh $NEW "sudo mkdir -p /etc/systemd/system/docker.service.d && printf '[Service]\nExecStartPost=/usr/local/sbin/fw-docker-user\n' | sudo tee /etc/systemd/system/docker.service.d/20-fw.conf"
ssh $NEW 'sudo systemctl daemon-reload && sudo systemctl restart docker && sudo iptables -S DOCKER-USER'

Watch points. Docker publishes ports through NAT before host firewall rules (firewalld, ufw) see them, so those rules do not protect this port: the restriction goes in the DOCKER-USER chain, re-applied at every Docker start. Expected: an ACCEPT for the proxy then a DROP on port 5002. Reloading the host firewall can flush Docker chains: restart Docker afterwards.

6. Fetch the official 2.0.12 files

fw curl -fsSL https://dev.funkwhale.audio/funkwhale/funkwhale/-/raw/$V/deploy/docker-compose.yml -o docker-compose.yml
fw curl -fsSL https://dev.funkwhale.audio/funkwhale/funkwhale/-/raw/$V/deploy/env.prod.sample -o .env
fw chmod 600 .env

Watch points. Always take both files from the release tag, never from develop.

7. Reuse the Django secret of OLD

fw sed -i '/^DJANGO_SECRET_KEY=/d' .env
ofw grep ^DJANGO_SECRET_KEY= /srv/funkwhale/.env | ssh $NEW 'cat >> /srv/funkwhale/.env'
fw grep -c '^DJANGO_SECRET_KEY=.' .env

Watch points. Same secret = existing sessions and client tokens stay valid. Never put it on a command line. Expected count: 1.

8. Set the variables

fw sed -i -e "s#^FUNKWHALE_VERSION=.*#FUNKWHALE_VERSION=$V#" -e "s#^FUNKWHALE_HOSTNAME=.*#FUNKWHALE_HOSTNAME=$FQDN#" -e "s#^FUNKWHALE_PROTOCOL=.*#FUNKWHALE_PROTOCOL=https#" -e "s#^FUNKWHALE_API_IP=.*#FUNKWHALE_API_IP=$NEW_IP#" -e "s#^FUNKWHALE_API_PORT=.*#FUNKWHALE_API_PORT=5002#" -e "s#^REVERSE_PROXY_TYPE=.*#REVERSE_PROXY_TYPE=nginx#" -e "s#^MUSIC_DIRECTORY_PATH=.*#MUSIC_DIRECTORY_PATH=/srv/funkwhale/data/music/library#" -e "s#^MUSIC_DIRECTORY_SERVE_PATH=.*#MUSIC_DIRECTORY_SERVE_PATH=/srv/funkwhale/data/music/library#" .env
ssh $NEW "printf 'DATABASE_HOST=postgres\nDATABASE_USER=postgres\nDATABASE_NAME=postgres\nCACHE_URL=redis://redis:6379/0\n' >> /srv/funkwhale/.env"

Watch points. Set the two music paths to the exact values noted at step 1. CACHE_URL and the three DATABASE_* lines are required for Docker installs: without CACHE_URL the default points to 127.0.0.1, unreachable from a container. Copy the other values noted at step 1 (NGINX_MAX_BODY_SIZE, EMAIL_CONFIG, DEFAULT_FROM_EMAIL) the same way. The front container is an nginx, hence REVERSE_PROXY_TYPE=nginx.

9. Pull images, test the CPU

fw docker compose pull
fw docker compose run --rm --no-deps api python -c "import numpy, autobahn; print('cpu-ok')"

Watch points. Expected: cpu-ok. On Illegal instruction, set FUNKWHALE_VERSION=2.0.12-legacycpu in .env and pull again. On a VM, prefer CPU type host.

10. Start the database and cache only

fw docker compose up -d postgres redis

Watch points. No migrate yet: the database must stay empty for the restore. Take a VM snapshot here.

11. Dump OLD and verify the transfer

ofw docker exec funkwhale-postgres-1 pg_dump -U postgres -Fc postgres > fw-old.dump
sha256sum fw-old.dump
ssh $NEW 'cat > /srv/funkwhale/fw-old.dump' < fw-old.dump
fw sha256sum fw-old.dump

Watch points. OLD keeps serving during the dump. The two checksums must be identical.

12. Copy media/

ssh $OLD 'sudo tar -C /srv/funkwhale/data -cf - media' | ssh $NEW 'sudo tar -C /srv/funkwhale/data -xpf -'

Watch points. Covers, avatars and transcodes only, not the music library. -p keeps owners: compare ls -ln /srv/funkwhale/data with OLD and chown -R if they differ.

13. Restore the dump

fw docker compose exec -T postgres pg_restore -U postgres -d postgres --no-owner --no-privileges < fw-old.dump

Watch points. --no-owner --no-privileges makes it independent of OLD's role names. "already exists" warnings on schema public are expected.

14. Optional: remove duplicate upload rows

Dry run:

fw docker compose exec -T postgres psql -U postgres -At -c "select count(*), count(*) filter (where import_status='finished'), count(distinct (library_id, source)) from music_upload"

Save script B (see appendix) as purge.sql, then:

fw docker compose exec -T postgres psql -U postgres -At < purge.sql
fw docker compose exec -T postgres psql -U postgres -c "VACUUM (ANALYZE) music_upload"

Watch points. On NEW only, before migrate. Every full rescan adds one skipped row per file, so music_upload can be several times larger than the library, and 2.0 rewrites all of it: purging first shortens migrate noticeably. The script runs in one transaction and aborts if the number of finished rows, of distinct files or of playable tracks changes. finished and pending rows are never touched.

15. Migrate

fw time docker compose run --rm api funkwhale-manage migrate
fw docker compose run --rm api funkwhale-manage check

Watch points. Do not interrupt migrate. Expected: no traceback, then System check identified no issues. Time it: this duration is your downtime at the real cutover.

16. Compare with the reference

fw docker compose exec -T postgres psql -U postgres -At -c "$Q" | diff - fw-ref-old.txt && echo identical

Watch points. users and tracks must be identical. uploads differs only if you ran step 14. In 2.0, former "libraries" become playlists: check them in the UI.

17. Start the full stack

fw docker compose up -d
fw docker compose ps

Watch points. Every container Up; api, celeryworker, celerybeat, front, postgres, redis.

18. Generate thumbnails

fw docker compose exec -T api funkwhale-manage fw media generate-thumbnails

Watch points. The 2.0 front requests 50×50 thumbnails that 1.4 never produced and 2.0 no longer creates on demand: without this, covers and avatars return 404. Never add -d (it deletes existing thumbnails). Files that are not images log a KeyError; the command carries on.

19. Check logs and endpoints

fw docker compose logs --tail=100 api celeryworker | grep -i -E "error|traceback|critical" || echo no-error
curl -s -o /dev/null -w '%{http_code}\n' -H "Host: $FQDN" http://$NEW_IP:5002/api/v1/instance/nodeinfo/2.0/
curl -m 5 http://$NEW_IP:5002/

Watch points. Expected: no-error and 200. The last command, run from any machine other than the proxy, must time out (port restricted at step 5). Use /api/v1/instance/nodeinfo/2.0/: the /api/v2/ route only serves nodeinfo 2.1. A Cannot connect to redis://127.0.0.1 in the Celery logs means CACHE_URL is missing.

20. Test without touching users

Point only your workstation at NEW (an /etc/hosts entry or a dedicated proxy rule), log in, play a track, check covers.

Watch points. Same FQDN and same certificate: other clients stay on OLD. Check that generated links use https (X-Forwarded-Proto set by the proxy).

21. Final sync on the day

ofw docker compose -f /srv/funkwhale/docker-compose.yml stop

Repeat steps 11 to 16, then make the music library read-write on NEW (remount the share without ro).

Watch points. Freeze OLD first so nothing changes after the dump. Compare the counters one last time. Make the music library writable only now.

22. Cutover

Point the proxy upstream to NEW_IP:5002, validate the configuration, reload.

HAProxy:

server funkwhale NEW_IP:5002 check

haproxy -c -f /etc/haproxy/haproxy.cfg && systemctl reload haproxy

nginx: change the proxy_pass / upstream address, then:

nginx -t && systemctl reload nginx

Watch points. One line to change, the upstream address. Always validate the syntax before the reload. Downtime = the duration measured at step 15 plus the reload.

23. Rollback (if needed)

Set the upstream back to OLD_IP:5002, validate and reload the proxy, then start OLD:

ofw docker compose -f /srv/funkwhale/docker-compose.yml start

Watch points. OLD was never modified. Keep it stopped, not deleted, for a couple of weeks after the cutover.

Appendix

A. Docker-user script

fw-docker-user (replace PROXY_IP):

#!/bin/sh
add() { iptables -C DOCKER-USER "$@" 2>/dev/null || iptables -I DOCKER-USER 1 "$@"; }
add -p tcp -m conntrack --ctorigdstport 5002 --ctdir ORIGINAL -j DROP
add -p tcp -s PROXY_IP -m conntrack --ctorigdstport 5002 --ctdir ORIGINAL -j ACCEPT

B. Purge script

purge.sql:

\set ON_ERROR_STOP on
begin;
create temp table before as
  select count(*) n, count(*) filter (where import_status='finished') fin,
         count(distinct (library_id, source)) src,
         count(distinct track_id) filter (where import_status='finished') tracks
  from music_upload;
create temp table todel as
  select id from (
    select id, import_status,
      bool_or(import_status='finished') over (partition by library_id, source) has_fin,
      row_number() over (partition by library_id, source
        order by (import_status='finished') desc, creation_date desc, id desc) rn
    from music_upload where source is not null and source<>''
  ) t
  where import_status in ('skipped','errored') and (has_fin or rn>1);
delete from music_upload u using todel d where u.id = d.id;
do $$ declare b record; n int; begin
  select * into b from before;
  select count(*) into n from music_upload where import_status='finished';
  if n <> b.fin then raise exception 'finished rows changed'; end if;
  select count(distinct (library_id, source)) into n from music_upload;
  if n <> b.src then raise exception 'distinct files changed'; end if;
  select count(distinct track_id) filter (where import_status='finished') into n from music_upload;
  if n <> b.tracks then raise exception 'playable tracks changed'; end if;
end $$;
commit;

End of log.

Published by:

billisdead

Member discussion