Remote checks fail fast when the worker is down and no longer pile up

While the worker was stopped (maintenance), checks were queued with nobody to
run them; agents timed out ("fetch failed", 502 after a restart) and every
retry queued another copy. Now: the worker writes a heartbeat and a check is
refused with 503 at once while it is down; a retry of the same files joins
the job already queued or running; a job nobody waits for any more is
dropped. TD_KEEP_WORKER=1 deploys without touching the worker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RNwtXPYqFu3oKaaXwYwNWW
This commit is contained in:
Saty 2026-10-18 05:00:59 +02:00
commit 9cfa55c9b2
6 changed files with 73 additions and 20 deletions

View file

@ -1,6 +1,7 @@
#!/bin/sh
# Updates the coordinator's code without taking the site down. Run as root on the server:
# sh deploy/update.sh <unpacked-source-dir>
# TD_KEEP_WORKER=1 leaves the worker alone (during maintenance with the worker stopped).
#
# The new release is copied and syntax-checked next to the running one; nothing changes for the
# running site until then. /opt/togetherdecomp is a symlink that switches to the new release in
@ -67,8 +68,13 @@ else
systemctl is-active --quiet togetherdecomp.service || { journalctl -u togetherdecomp -n 20 --no-pager; exit 1; }
# The worker finishes its running verification, merge or check before it stops (no work is
# redone), then starts on the new release. That can take minutes, so don't wait for it here.
echo "== the worker restarts once its running work is done"
systemctl restart --no-block togetherdecomp-worker.service
if [ -n "${TD_KEEP_WORKER:-}" ]; then
# Maintenance (e.g. a rebuild with the worker stopped): leave it; it runs the new code when started.
echo "== worker left as it is (TD_KEEP_WORKER)"
else
echo "== the worker restarts once its running work is done"
systemctl restart --no-block togetherdecomp-worker.service
fi
fi
# Keep the last three releases for a quick rollback (ln -sfn <release> /opt/togetherdecomp).