clank/clank-together
1
0
Fork 0
Code Issues Wiki Activity

Restarts no longer hang: bounded shutdown, wait only for batches

Browse source 
The web process waited on server.close() for open requests, among them agents
waiting minutes for a remote check, until systemd killed it after 90 s; the
site was down meanwhile. It now closes idle connections and exits within 3 s
(TimeoutStopSec=15 as a backstop). update.sh restarts the worker once no batch
verification or merge runs; remote checks are re-queued and still answered.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RNwtXPYqFu3oKaaXwYwNWW
This commit is contained in:
Saty 2026-10-18 04:37:31 +02:00
commit 5b8c821401
4 changed files with 12 additions and 5 deletions
Download patch file Download diff file Expand all files Collapse all files

1
deploy/togetherdecomp-worker.service
View file

@ -13,6 +13,7 @@ ExecStart=/opt/node/bin/node --disable-warning=ExperimentalWarning /opt/together
Restart=on-failure
# Restarting re-queues whatever was being verified.
RestartSec=5
TimeoutStopSec=15
Environment=NODE_ENV=production
# Hardening. The process needs: its data dir, the docker socket, git.

1
deploy/togetherdecomp.service
View file

@ -11,6 +11,7 @@ WorkingDirectory=/var/lib/togetherdecomp
ExecStart=/opt/node/bin/node --disable-warning=ExperimentalWarning /opt/togetherdecomp/src/server/cli.js start web --data /var/lib/togetherdecomp
Restart=on-failure
RestartSec=5
TimeoutStopSec=15
Environment=NODE_ENV=production
Environment=TD_WEB_DIR=/opt/togetherdecomp/src/server/web

7
deploy/update.sh
View file

@ -66,10 +66,11 @@ else
systemctl restart togetherdecomp.service
sleep 2
systemctl is-active --quiet togetherdecomp.service || { journalctl -u togetherdecomp -n 20 --no-pager; exit 1; }
# A restart re-queues the worker's running verification or merge; wait (up to 15 min) for it to finish.
echo "== restarting the worker when it is idle"
# A restart re-queues what the worker is doing. Remote checks are cheap to redo (the waiting agent
# still gets its result); a batch verification or merge is waited for, up to 15 minutes.
echo "== restarting the worker once no batch is being verified or merged"
waited=0
while [ -n "$(ls -A "$JOBS" 2>/dev/null)" ] && [ "$waited" -lt 900 ]; do
while ls "$JOBS" 2>/dev/null | grep -qE '^(batch|merge)-' && [ "$waited" -lt 900 ]; do
sleep 5
waited=$((waited + 5))
done