Remote checks fail fast when the worker is down and no longer pile up
While the worker was stopped (maintenance), checks were queued with nobody to
run them; agents timed out ("fetch failed", 502 after a restart) and every
retry queued another copy. Now: the worker writes a heartbeat and a check is
refused with 503 at once while it is down; a retry of the same files joins
the job already queued or running; a job nobody waits for any more is
dropped. TD_KEEP_WORKER=1 deploys without touching the worker.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RNwtXPYqFu3oKaaXwYwNWW
This commit is contained in:
parent
0710882703
commit
9cfa55c9b2
6 changed files with 73 additions and 20 deletions
|
|
@ -1,6 +1,7 @@
|
|||
#!/bin/sh
|
||||
# Updates the coordinator's code without taking the site down. Run as root on the server:
|
||||
# sh deploy/update.sh <unpacked-source-dir>
|
||||
# TD_KEEP_WORKER=1 leaves the worker alone (during maintenance with the worker stopped).
|
||||
#
|
||||
# The new release is copied and syntax-checked next to the running one; nothing changes for the
|
||||
# running site until then. /opt/togetherdecomp is a symlink that switches to the new release in
|
||||
|
|
@ -67,8 +68,13 @@ else
|
|||
systemctl is-active --quiet togetherdecomp.service || { journalctl -u togetherdecomp -n 20 --no-pager; exit 1; }
|
||||
# The worker finishes its running verification, merge or check before it stops (no work is
|
||||
# redone), then starts on the new release. That can take minutes, so don't wait for it here.
|
||||
echo "== the worker restarts once its running work is done"
|
||||
systemctl restart --no-block togetherdecomp-worker.service
|
||||
if [ -n "${TD_KEEP_WORKER:-}" ]; then
|
||||
# Maintenance (e.g. a rebuild with the worker stopped): leave it; it runs the new code when started.
|
||||
echo "== worker left as it is (TD_KEEP_WORKER)"
|
||||
else
|
||||
echo "== the worker restarts once its running work is done"
|
||||
systemctl restart --no-block togetherdecomp-worker.service
|
||||
fi
|
||||
fi
|
||||
|
||||
# Keep the last three releases for a quick rollback (ln -sfn <release> /opt/togetherdecomp).
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue