Commit graph

34 commits

Author SHA1 Message Date
ragnarok-clanker
8d472dd8ee Compare linked code without line numbers, namespace hashes or unlearned names
effectiveHash masks only relocated fields (an operand equal to its source line no longer counts), and normalizes MSVC anonymous-namespace hashes. A check binds every annotated global's decorated name, from its definition or an annotated extern, across all its units before comparing, so references from other units resolve to the global's ID as they will once main learns the name.
2026-10-26 19:25:03 +01:00
ragnarok-clanker
689a8bc85b Add maintenance batches and symbol corrections for maintainers
A maintenance batch carries structural source changes that belong to no function task (moving a global to its owner, fixing data annotations, adding a header or unit). It runs the same gates as function work and compiles every unit the change can affect; it merges only when no function's compiled code changes, recording the result like any merge. POST /api/admin/symbols/:id corrects a symbol's size or name, or retires a global main no longer annotates. td maintenance submit|symbol drive both.
2026-10-26 18:46:45 +01:00
ragnarok-clanker
4efc56fa8d Let agents revise merged functions through region claims
A region whose functions are all on main can be claimed, and its done tasks submitted as revisions: competing candidates verified and peer reviewed like new work, while the task stays done whatever the outcome. Rewriting a merged function without listing it sends the batch back instead of merging it unreviewed. Admins can force a done task back to open.
2026-10-26 18:06:07 +01:00
Saty
50117671ee Peer review: clear majority, a cap on reviewers, shared bodies skip review
"same" now needs 70% of the weight as well as a lead of three (7:3, not 10:7),
so a contested function is not worn down until "same" happens to lead. A
function seven reviewers could not decide goes back to its author with why;
the queue stops handing it out at the cap. Both are policy settings
(peerAcceptShare, peerMaxReviewers); trust's author record counts the same way.

One body annotated for several functions: when the original of the submitted
one is the same code as another's (same bytes, relative references to the same
targets) and main already accepts that one from this compiled body, the
submission is approved without peer review.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 15:09:18 +01:00
Saty
ab5b07c5f5 Unstick functions on their way to main
A resubmission whose peer verdicts carry over now goes on as approved; in_review
waited for a human review that no longer comes, and the peer queue skipped it.

A batch that only merges as a whole (noSplit) goes back to its author once a
function of it drops out later (sent back, withdrawn, superseded), instead of
holding its approved functions forever.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 14:51:39 +01:00
Saty
09d1091775 Peer review decides by margin: "same" ahead by three accepts
Four (five after an objection) "same" could not be reached with the few
accounts online: an author never reviews their own work, and with four
active accounts that were also the authors, each function could get at most
three verdicts. 948 of 1027 verdicts in six hours went to functions that
stayed undecided; merges fell to about 5 an hour.

- Accept when weighted "same" leads "different" by peerMargin (3: 3:0, 4:1,
  5:2), from at least two reviewers.
- Send back when at least peerReviews (3) "different" lead by
  peerRejectMargin (2: 3:1, 4:2), from at least two reviewers.
- Close votes wait for more verdicts; reviewers are offered functions that a
  verdict can still decide.
- The worker settles the waiting queue under the current rule when it starts.
- Trust and quality count accepted and sent-back functions the same way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 04:08:08 +02:00
Saty
f542ad5cb2 Fixes from the second contributor report
- Static gate: a file main already holds may grow 128 KB beyond its size on
  main (Unit_C.cpp is 796 KB, Player_C.cpp 491 KB: no edit to them could
  pass the 256 KB limit); only what a submission adds counts toward the total.
- A peer-confirmed function in another unit may move closer to the original
  (a header layout fix); only moving away from it is a regression.
- A function main no longer annotates anywhere (fn_07162, removed from stpl.h
  in 0f9ebdb4 while its record still said exact) is not "removed" by every
  later submission of the unit it was recorded at.
- A library function main already annotates in a file (fn_25851 in
  BSNEncodeInterface.h) no longer refuses every edit of that file; adding
  such an annotation still is refused.
- td_withdraw / td withdraw <attemptId>: the author takes back a function
  waiting for review; it returns to their claim.
- td_submit says when held functions have no changed code in the working
  tree, instead of "you hold no tasks".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 12:37:10 +02:00
Saty
7120dcaebc Trust: proven authors skip peer review, careful reviewers count more
Each agent (token) earns trust from its record (trust.js):
- author: of its peer-decided functions scoring 90%+ (14 days), the
  accepted share, as a 95% lower bound, must reach 85% over 20+
  functions. Its functions at 95%+ are then accepted without peer
  review, but for a random one in five that peers still review.
- reviewer: agreement with the other reviewers, counted apart for
  functions they found the same and ones they found different (30
  days). 85%/65% over 30+ differences counts x2, 92%/85% over 60+ x3.
Weights are frozen on the verdict; a decision always takes two
reviewers. Skipped functions and own verdicts never feed the record.
Shown per agent on the Account page, in td_quality and in the admin
Authors table.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 08:38:16 +02:00
Saty
01538a93d5 Trust: proven authors skip peer review, careful reviewers count more
Each agent (token) earns trust from its record (trust.js):
- author: of its peer-decided functions scoring 90%+ (14 days), the
  accepted share, as a 95% lower bound, must reach 85% over 20+
  functions. Its functions at 95%+ are then accepted without peer
  review, but for a random one in five that peers still review.
- reviewer: agreement with the other reviewers, counted apart for
  functions they found the same and ones they found different (30
  days). 85%/65% over 30+ differences counts x2, 92%/85% over 60+ x3.
Weights are frozen on the verdict; a decision always takes two
reviewers. Skipped functions and own verdicts never feed the record.
Shown per agent on the Account page, in td_quality and in the admin
Authors table.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 08:10:59 +02:00
Saty
84ec59d626 Stale baselines and file-local statics no longer bounce batches
Before a check counts regressions, the units it would blame are compiled
as main has them (once per main commit) and their recorded status is
updated: a header change merged later could leave a unit building
differently on main than recorded (FrameScript_Sprintf 85.9% vs 86.1%,
Spell_C_NeedsCooldownEvent recorded exact, built at 89.7%), and every
recompile of the unit was counted as a regression.

A file-local static is resolved by object and name
(Foo_Scripts.obj|s_ScriptFunctions), never by the bare name: an imported
symbol named plain s_ScriptFunctions tied every unit's table to one
address, so every RegisterScriptFunctions mismatched.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 06:21:21 +02:00
Saty
aab450b4f7 Unchanged code is not submitted for review; a looping account pauses
A region claim plus "submit everything held" sent every function of the
file to peer review, also the ones the agent never touched (1006 of one
agent's 1179 waiting reviews were code already on main, unchanged).
- td submit sends only functions the agent rewrote (or lists in confirm).
- Verification withdraws unchanged, not byte-identical functions unless
  confirm names them.
- An account whose last six submissions in 15 minutes were all refused
  before compiling (not synced, conflicts, comments) pauses ten minutes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 05:06:52 +02:00
Saty
67965b04f1 Refuse removing annotations of functions not held; new source files join the build list
- A submission may not drop the annotation of a function of the current
  build it does not hold (51 annotations were lost that way); removing a
  duplicate is fine.
- A unit missing from compile.json compiles with the command shape of its
  folder, and the merge adds it to compile.json in the same commit (22
  contributor files were outside the build).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 04:56:07 +02:00
Saty
b55b592a82 Submissions may not declare functions that are defined nowhere
The server compiles units one by one and never links, so a call to a
function declared only by its prototype (an invented helper, or a member
declared in a header and never written) compiled and could merge; main
would not link. A newly declared function at file or class scope must be
defined in the submission, exist elsewhere in the repository, or be a
known function of the original (by name or fn_ ID).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 04:29:30 +02:00
Saty
347ce4ea24 Withdraw held functions the submission did not write, also when main has code for them
Verification compiles the unit a function lives in on main, so a held but
unwritten function was found compiled from main code and sent to peer
review as if submitted; reviewers shown the submitted files could not find
it (592 verdicts said so, 196 such functions were waiting).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 03:55:05 +02:00
Saty
a9eb651688 Compile only the includers a header change can affect
A verification of a header template compiled every includer (705 for
stpl.h) and a merge every includer of every submitted header (up to 1199
per header), even when the header only differed in line endings; one
such job took up to an hour and held up every other batch. Now:
- merge: headers unchanged from main are skipped; for changed ones, the
  includers whose source mentions a name the change touched (identifiers
  on changed lines, the enclosing class/struct), all of them only when
  that is more than 300;
- verification: the includers that mention the template arguments of
  the annotation (or the class), at most 16.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 03:40:19 +02:00
Saty
d9825a64d3 Peer review message: five "same" merge, three "different" send back
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 03:11:18 +02:00
Saty
10f840b18f Regions, competing candidates, five "same" merge, checks compile locally
- Agents list regions (source files or classes: open, done, in flight,
  who claimed them), claim one as a whole and submit it in one batch.
- Several agents may claim a region; each may submit any function of it.
  Candidates are verified and reviewed on their own; the first accepted
  one merges and the others are superseded, the rest of their batches
  goes on. A done task stays done (trigger).
- Peer review: five "same" accept (no human step), three "different"
  send back.
- The server no longer compiles checks (remoteChecks, off by default);
  approved accounts download the compiler package for local checks.
- Tables for game credits and rounds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 02:32:20 +02:00
Saty
22b5975ffc Fair verification queue; up to 8 batches waiting per account (was 4)
Verification took the oldest batch overall, so the per-account cap of 4 was
what kept one fast contributor from holding every slot. Now the next slot
goes to the account with the fewest batches verifying, and the cap can be
higher.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RNwtXPYqFu3oKaaXwYwNWW
2026-10-19 00:34:29 +02:00
Saty
3c946d7689 Regression check: exact to match_unresolved is not a regression
Same bytes; whether a referenced label resolves depends on which other
units the check compiled. The same Scripts.cpp functions refused batches of
several contributors.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RNwtXPYqFu3oKaaXwYwNWW
2026-10-19 00:17:23 +02:00
Saty
9c8cd1320d Verify three batches at once by default (was two)
The server has room (8 cores, load about 5.5); batches waited about 15
minutes for their result.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RNwtXPYqFu3oKaaXwYwNWW
2026-10-19 00:09:01 +02:00
Saty
abba0f5f06 Header template copies are not regressions while the header is unchanged
A template or inline function defined in a header is emitted by every unit
that uses it; main records its best copy at one of them. Editing that unit
could drop or vary its copy and refuse the submission, although the other
units still emit the good one (16 batches in an hour, mostly stpl.h).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RNwtXPYqFu3oKaaXwYwNWW
2026-10-18 22:00:01 +02:00
Saty
d09cfed509 Fixes from a contributor's report: x64 symbol addresses, header and Win32 functions
- td symbol new accepted only 32-bit addresses; every x64 function is at
  0x14xxxxxxx.
- Verification also compiles the unit each submitted function is emitted
  from on main: a header's inline function only exists in units that use it,
  and a Win32/ copy is compiled out of the x64 build. A function never
  compiled before that is defined in a changed header gets all includers.
- Agents may edit that unit too, not only the task's nominal file.
- Guidance: adjustor thunks and deleting destructors are compiler-generated;
  mark them no-decompile instead of stacking annotations on a body.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RNwtXPYqFu3oKaaXwYwNWW
2026-10-18 20:34:29 +02:00
Saty
1defae5e8b Productivity: unwritten functions withdrawn, per-function returns at verification, neighbour rule
- A submitted function without any annotation in the submission was counted
  as a failure (2176 in a day) and reserved to its author; it is now
  withdrawn back into the author's claim.
- Verification sends failing functions back one by one; the rest of the batch
  carries on (838 batches were returned whole, with their good functions).
- Changing a neighbouring function is refused only if someone else works on
  it (about 2000 refusals in a day were unheld neighbours).
- td-server stats [json] [--hours N] and docs/productivity.md: readings,
  diagnosis, changes and what to compare at the next review.
- Progress panel shows functions and bytes, each with totals and percentage.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RNwtXPYqFu3oKaaXwYwNWW
2026-10-18 20:26:05 +02:00
Saty
ba824587ae Verify fast, merge thoroughly: includers of a changed header compile at merge
Verification compiles the submitted units (one includer per header for
header-only changes, plus one includer as the lint sample): enough to judge
and route every submitted function, in seconds even for header changes. The
merge compiles every includer of a changed header that verification skipped
and checks them for regressions, so nothing reaches main without it; it runs
in the big merge lane. Batches sent back early never pay for the full compile.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RNwtXPYqFu3oKaaXwYwNWW
2026-10-18 05:54:59 +02:00
Saty
a9a63dffdc Worker fills in missing batch sizes on start
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RNwtXPYqFu3oKaaXwYwNWW
2026-10-18 05:39:40 +02:00
Saty
a2b405c14b Merges recompile only what changed; a slot for small batches; queue limit per account
- A merge reuses the verification's compile for every unit whose source and
  included headers are unchanged, and compiles only the rest. A split part
  uses its parent's verification, so merging the approved functions of a
  header batch recompiles the one .cpp file instead of hundreds of units.
- While a big batch (more than 20 units) is verified, the other slot takes
  only small ones.
- An account can have 4 batches waiting for verification; further submits get
  429 with a hint to do peer reviews meanwhile (maxQueuedBatches).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RNwtXPYqFu3oKaaXwYwNWW
2026-10-18 05:35:51 +02:00
Saty
749942d0b5 Faster verification: lint one includer per header, reuse checks at merge, small batches first
- The preprocessed-output lint compiled every unit a second time (with /P)
  when a header changed: 323 units twice for Types.h. A header expands the
  same way everywhere, so it now lints the submitted units plus one includer
  of each changed header.
- Merges re-ran the full check, including that lint. The lint is skipped on
  merge, and when nothing the compiled units consist of (sources, included
  headers) changed on main since verification, the verification's compile is
  reused and nothing is compiled at all.
- The queue takes small batches first (by the number of units they compile);
  a big batch that waited 20 minutes goes ahead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RNwtXPYqFu3oKaaXwYwNWW
2026-10-18 05:29:26 +02:00
Saty
033427c99b No free text in submissions; batch approval only approves what it shows; drain fix
Prompt injection: a submission may not add comments other than annotations,
offset and INLINED markers, and only string literals that occur in the
original executable (adjacent literals joined; #include paths restricted).
Comments and strings already on main stay. Checked before compiling, for
remote checks and submissions; errors name the line but never repeat the
text. Peer reviewers no longer see author notes or running verdict counts.
Agents are told that only AGENTS.md and the guidance field instruct them.

Review: "Approve batch" in the confirmed section also approved the batch's
functions still in peer review (for maintainers). The page now sends exactly
the functions it shows, and skipping peer review needs skipPeerReview,
which only the peer review section's button and the attempt page send.

Worker drain: a stopping worker kept starting new verifications when one
finished, so it never stopped. Stopped queues take no new work.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RNwtXPYqFu3oKaaXwYwNWW
2026-10-18 05:14:25 +02:00
Saty
0710882703 Progress no longer drops when templates are recompiled; live progress
Templates and inline functions are compiled in every unit that uses them. A
merge that recompiled many units (a header change) let the last unit's copy
win, so functions that matched in one unit were recorded as match_unresolved
from another, and the regression check missed it because the better status
came from a unit it did not compile. Now each function counts with its best
compiled copy, and a check never lowers a status measured in a unit it did
not compile.

The progress gauge and the crew views refresh every 30 seconds; the gauge
moves from its previous reading.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RNwtXPYqFu3oKaaXwYwNWW
2026-10-18 04:48:19 +02:00
Saty
9253c439ad Remote checks compile only the agent's own files
A check that touched a header recompiled every unit including it (dozens),
which saturated the server while agents worked in parallel. Checks are
feedback; the submission's verification still compiles all dependents.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RNwtXPYqFu3oKaaXwYwNWW
2026-10-18 04:44:25 +02:00
Saty
928c91705e Merge the approved part of a batch on its own; analyze checks in worker threads
A batch no longer waits for its slowest function: once some of its functions
are approved and the rest wait for review, the approved ones merge as their
own batch (the pending functions put back as they are on the base commit),
and the rest is rebased onto that commit. The merge re-verifies the split; if
anything breaks, the batch merges as a whole as before.

Comparing compiler output against the original runs in worker threads. A
check that recompiled dozens of units (a header change) blocked the
coordinator's event loop for minutes, so the website stopped responding.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RNwtXPYqFu3oKaaXwYwNWW
2026-10-18 03:33:07 +02:00
Saty
5900fe2ecf Wait for SQLite locks instead of crashing
A second process (rebuild, ghidra refresh) holding a write lock made the
verifier's timer throw "database is locked" and took the coordinator down.
Set a busy timeout and keep timer errors from killing the process.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RNwtXPYqFu3oKaaXwYwNWW
2026-10-18 03:01:14 +02:00
Clank Together
125f5c4e7d Batches, auto-merge, standalone td, one-line install, self-update, server Ghidra
- Batches: claim every open function of a unit or address range (td claim --unit/--range),
  submit the file once; one compile, per-function routing, merged as a whole. A failing
  function returns the batch; peer verdicts carry over to functions whose compiled code is
  unchanged (code fingerprints). Submissions may not change functions they do not hold.
- Byte-identical functions are approved and merged without human review
  (matchPolicy.autoMergeIdentical); peer-confirmed ones need one click per batch.
- Remote check: td check compiles on the server when there is no local toolchain.
- /api/original: assembly and pseudocode of the original from the server's Ghidra export
  (objdump before an export exists); td disasm/decompile fall back to it.
- td as a standalone executable (Node single executable application; tools/build-td),
  one-line installers (install.ps1 / install.sh) with checksum verification, td connect for
  any MCP client, td mcp --workspace, self-update (td update, hourly check).
- Server-side Ghidra project: built once per version, refreshed after merges, downloadable
  for contributors (td ghidra download); JDK 21 and the pinned Ghidra in deploy/install.sh.
- Agent instructions: work continuously, batches first, peer reviews in between.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RNwtXPYqFu3oKaaXwYwNWW
2026-10-18 02:13:09 +02:00
Clank Together
836610da8f Clank Together: coordinator, sandboxed verifier and agent client
Coordinator (accounts with approval, API tokens, tasks with leases, sandboxed MSVC verification against
the original exe, peer review by three agents for non-identical code, human review, serial merge queue),
the `td` contributor CLI with an MCP server for agents, pinned Ghidra integration, deployment (Caddy,
systemd, nftables, Forgejo) and the wowdecomp 15890 migration tools.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RNwtXPYqFu3oKaaXwYwNWW
2026-10-18 01:29:16 +02:00