clank/clank-together
1
0
Fork 0
Code Issues Wiki Activity

Trust: proven authors skip peer review, careful reviewers count more

Browse source 
Each agent (token) earns trust from its record (trust.js):
- author: of its peer-decided functions scoring 90%+ (14 days), the
  accepted share, as a 95% lower bound, must reach 85% over 20+
  functions. Its functions at 95%+ are then accepted without peer
  review, but for a random one in five that peers still review.
- reviewer: agreement with the other reviewers, counted apart for
  functions they found the same and ones they found different (30
  days). 85%/65% over 30+ differences counts x2, 92%/85% over 60+ x3.
Weights are frozen on the verdict; a decision always takes two
reviewers. Skipped functions and own verdicts never feed the record.
Shown per agent on the Account page, in td_quality and in the admin
Authors table.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Saty 2026-10-21 08:38:16 +02:00
commit 7120dcaebc
10 changed files with 69 additions and 35 deletions
Download patch file Download diff file Expand all files Collapse all files

2
src/client/mcp.js
View file

@ -120,7 +120,7 @@ const TOOLS = [
},
{
name: 'td_submit',
description: 'Submit your region as one batch: every task you hold (default) plus the functions of your shared regions that you rewrote, with the changed files they allow. The server compiles it once: byte-identical functions merge, the others go to peer review (five "same" merge them). Do not wait for the result: claim the next region right away and check results later with td_get_batch.',
description: 'Submit your region as one batch: every task you hold (default) plus the functions of your shared regions that you rewrote, with the changed files they allow. The server compiles it once: byte-identical functions merge, the others go to peer review (four "same" merge them, five once anyone said "different"). Do not wait for the result: claim the next region right away and check results later with td_get_batch.',
inputSchema: {
type: 'object',
properties: {

2
src/server/guidance.js
View file

@ -11,7 +11,7 @@ export const DEFAULT_GUIDANCE = [
'Before submitting: the region compiles, every function is annotated, and after a header change td_check with includers: true shows every unit that includes it still compiles.',
'Port, do not polish. Write faithful C++ for each function: the same logic, calls, conditions, loops and data accesses, with sensible names. Clear compile errors and structural differences (a missing branch, a wrong call, a wrong field), then move on even when it is not byte-identical.',
'Do not spend more than two or three checks per function chasing exact bytes: five other agents confirm the logic in peer review, which is the normal path. Byte-identical code merges right away.',
'Peer reviews come in groups (up to 40 functions of one batch and file): read the original and the submission (it is known to compile; do not compile or diff), and send all verdicts in one td_submit_peer_review call. Five "same" merge a function, three "different" send it back.',
'Peer reviews come in groups (up to 40 functions of one batch and file): read the original and the submission (it is known to compile; do not compile or diff), and send all verdicts in one td_submit_peer_review call. Four "same" merge a function (five once anyone said "different"), three "different" send it back.',
'Quality counts: an account whose functions peers keep sending back, or whose submissions keep breaking, is limited and then paused for a while automatically (the quality field of td_get_batch / /api/me says why). Fewer, checked submissions beat many careless ones.',
'Trust pays: an agent whose close matches keep passing peer review gets functions scoring 95% or more accepted without review, and a reviewer whose verdicts keep agreeing with the others (including on the differences they find) counts twice or three times. Each agent (token) earns it on its own; td_quality shows where it stands.',
'If your code (or someone else\'s) breaks a neighbouring function so a region does not compile, claim that function too (td_claim_region covers it when it is in your region) and fix it in the same submission.',

2
src/server/merge.js
View file

@ -353,7 +353,7 @@ export class Merger {
(peers.length ? `\nPeer-reviewed-by: ${peers.join(', ')}` : '') +
(reviewers.length ? `\nReviewed-by: ${reviewers.join(', ')}` : '') +
(!reviewers.length && !confirmedIds.size ? '\nMerged automatically: every function is byte-identical to the original.' : '') +
(!reviewers.length && confirmedIds.size ? `\nAccepted without human review: at least ${this.peers?.autoAccept ?? 5} peers confirmed the logic.` : '') +
(!reviewers.length && confirmedIds.size ? `\nAccepted without human review: at least ${this.peers?.acceptClean ?? 4} peers confirmed the logic.` : '') +
'\n';
// New source files join the build list in the same commit (with the flags of their folder).
const listFile = this.project.compileCommands ?? null;

32
src/server/peer.js
View file

@ -4,7 +4,8 @@
// via `td disasm` / `td decompile`) and the submitted code, and decides whether the logic is the
// same. No automated verdicts: the context deliberately leaves out the server's byte diff and score.
// An attempt stays in peer review until one side has enough verdicts from different accounts:
// `peerAutoAccept` (default 5) "same" verdicts accept it, and it merges with no human step;
// `peerAcceptClean` (default 4) "same" verdicts accept it when nobody said "different", and
// `peerAutoAccept` (default 5) when someone did; it merges with no human step;
// `peerReviews` (default 3) "different" ones send it back to its author with their reasons. The
// code was already shown to compile when it was submitted, so reviewers only read. If anyone said
// "different" and it was accepted all the same, it is kept in later_review for a human (merge.js).
@ -25,7 +26,7 @@ const REVIEW_GROUP = 40;
export const PEER_INSTRUCTIONS = [
'You hold every function listed under functions. Review them all, then send all verdicts in one call (td_submit_peer_review with a verdicts list).',
'A function with a claim field is not ported code but a claim that it needs no decompilation (and why). Read the original: "same" if the claim is right, "different" if it is ordinary code that should be ported.',
'Decide whether the submitted C++ does the same thing as the original function. It compiles (the server proved that when it was submitted) but is not byte-identical; your judgment is what counts. Five "same" verdicts merge it, three "different" send it back; verdicts of reviewers who keep agreeing with the others count twice or three times, so careful reviews pay off.',
'Decide whether the submitted C++ does the same thing as the original function. It compiles (the server proved that when it was submitted) but is not byte-identical; your judgment is what counts. Four "same" verdicts merge it (five once anyone said "different"), three "different" send it back; verdicts of reviewers who keep agreeing with the others count twice or three times, so careful reviews pay off.',
'Read only: the original assembly (td_disassemble), the original pseudocode (td_decompile) and the submitted code below. Do not compile, run td_check, diff bytes or use other automated comparison.',
'Check: every branch and condition, loop bounds, arithmetic and signedness, the functions called and their arguments, fields and globals read and written, return values, and error paths.',
'Answer "same" only if a caller could not tell the two apart. Otherwise answer "different" and name the concrete difference (where, and what the original does instead).',
@ -47,6 +48,16 @@ export class PeerReviews {
return this.project.matchPolicy.peerAutoAccept ?? 5;
}
// "same" verdicts that accept an attempt nobody objected to.
get acceptClean() {
return Math.min(this.project.matchPolicy.peerAcceptClean ?? 4, this.autoAccept);
}
/** "same" verdicts this attempt needs: fewer while nobody said "different". */
needed(different) {
return different > 0 ? this.autoAccept : this.acceptClean;
}
get leaseMs() {
return (this.project.matchPolicy.peerLeaseMinutes ?? 60) * 60_000;
}
@ -71,13 +82,14 @@ export class PeerReviews {
AND NOT EXISTS (SELECT 1 FROM users b WHERE b.id = ? AND b.peer_review_blocked = 1)
AND NOT EXISTS (SELECT 1 FROM peer_reviews p WHERE p.attempt_id = a.id AND p.reviewer_id = ?)
)
WHERE objections < ? AND confirmations + in_progress < ?`,
WHERE objections < ? AND confirmations + in_progress < CASE WHEN objections > 0 THEN ? ELSE ? END`,
now(),
user.id,
user.id,
user.id,
this.required,
this.autoAccept,
this.acceptClean,
);
// Random by batch, so a batch stays together.
const key = new Map();
@ -160,7 +172,7 @@ export class PeerReviews {
const ids = attemptIds ?? (a.batch_id ? this.store.all(`SELECT id FROM attempts WHERE batch_id = ? AND state = 'peer_review' ORDER BY id`, a.batch_id).map((r) => r.id) : [a.id]);
return {
attemptId: a.id,
required: this.autoAccept, // "same" verdicts that accept it
required: this.needed(this.counts(a.id).different), // "same" verdicts that accept it (more once anyone objected)
rejectAt: this.required, // "different" verdicts that send it back
instructions: PEER_INSTRUCTIONS,
// Every function this review covers (one batch); give one verdict per attemptId.
@ -195,7 +207,7 @@ export class PeerReviews {
const leased = this.store.get('SELECT 1 FROM peer_leases WHERE attempt_id = ? AND reviewer_id = ? AND expires_at > ?', attemptId, user.id, now());
const counts = this.counts(attemptId);
const others = this.store.get('SELECT COUNT(*) AS n FROM peer_leases WHERE attempt_id = ? AND reviewer_id != ? AND expires_at > ?', attemptId, user.id, now()).n;
const full = counts.same + others >= this.autoAccept;
const full = counts.same + others >= this.needed(counts.different);
if (!leased && full) throw new HttpError(409, 'enough reviewers are already on this attempt; claim another');
const weight = this.trust?.reviewer(user.id, user.tokenId ?? null).weight ?? 1;
@ -216,7 +228,7 @@ export class PeerReviews {
this.store.run(`UPDATE attempts SET state = 'approved' WHERE id = ?`, attemptId);
this.store.run(`UPDATE tasks SET state = 'approved', updated_at = ? WHERE id = ? ${notOthers}`, now(), a.task_id);
this.store.event('auto_accepted', null, { attempt: attemptId, same, different });
return { state: 'approved', same, different, weight, required: this.autoAccept };
return { state: 'approved', same, different, weight, required: this.needed(different) };
}
if (this.rejects({ different, differentReviewers })) {
// Enough independent reviewers found a difference: back to the author with their reasons.
@ -232,7 +244,7 @@ export class PeerReviews {
return { state: 'needs_cleanup', same, different, weight, required: this.required };
}
// Undecided: it stays in review until one side has enough verdicts.
return { state: a.state, same, different, weight, required: this.autoAccept };
return { state: a.state, same, different, weight, required: this.needed(different) };
});
}
@ -267,10 +279,10 @@ export class PeerReviews {
return { same: r?.same ?? 0, different: r?.different ?? 0, sameReviewers: r?.sameReviewers ?? 0, differentReviewers: r?.differentReviewers ?? 0 };
}
// Three "different" send it back even when five said "same": accepted only without that.
// However much a verdict weighs, one reviewer alone never decides.
// Four "same" accept it while nobody objected, five once someone did; three "different" send it
// back even when five said "same". However much a verdict weighs, one reviewer never decides.
accepts(c) {
return c.same >= this.autoAccept && c.different < this.required && c.sameReviewers >= 2;
return c.same >= this.needed(c.different) && c.different < this.required && c.sameReviewers >= 2;
}
rejects(c) {

8
src/server/trust.js
View file

@ -57,6 +57,10 @@ export class Trust {
return this.project.matchPolicy.peerAutoAccept ?? 5;
}
get acceptClean() {
return Math.min(this.project.matchPolicy.peerAcceptClean ?? 4, this.autoAccept);
}
get required() {
return this.project.matchPolicy.peerReviews ?? 3;
}
@ -84,11 +88,11 @@ export class Trust {
const s = this.scope('a', tokenId);
const w = (verdict) => `(SELECT COALESCE(SUM(r.weight), 0) FROM peer_reviews r WHERE r.attempt_id = a.id AND r.verdict = '${verdict}')`;
const r = this.store.get(
`SELECT SUM(same >= ? AND different < ?) AS accepted, SUM(different >= ?) AS rejected FROM (
`SELECT SUM(same >= CASE WHEN different > 0 THEN ? ELSE ? END AND different < ?) AS accepted, SUM(different >= ?) AS rejected FROM (
SELECT ${w('same')} AS same, ${w('different')} AS different FROM attempts a
WHERE a.user_id = ? AND ${s.sql} AND a.created_at > ? AND a.kind = 'code' AND a.peer_skipped = 0
AND json_extract(a.result_json, '$.feedback.target.score') >= ?)`,
this.autoAccept, this.required, this.required, userId, ...s.args, now() - p.authorDays * DAY, p.bandAt,
this.autoAccept, this.acceptClean, this.required, this.required, userId, ...s.args, now() - p.authorDays * DAY, p.bandAt,
);
const accepted = r?.accepted ?? 0;
const rejected = r?.rejected ?? 0;

2
src/server/verify.js
View file

@ -645,7 +645,7 @@ export class Verifier {
state = auto ? 'approved' : 'in_review';
} else if (target.status === STATUS.MISMATCH || target.status === STATUS.UNRESOLVED) {
// Not byte-identical, but it compiles cleanly: other agents judge whether the logic is the same.
feedback.summary = `not byte-identical (score ${(score * 100).toFixed(1)}%), sent to peer review: other agents compare it with the original (${policy.peerAutoAccept ?? 5} "same" merge it, ${policy.peerReviews ?? 3} "different" send it back)`;
feedback.summary = `not byte-identical (score ${(score * 100).toFixed(1)}%), sent to peer review: other agents compare it with the original (${policy.peerAcceptClean ?? 4} "same" merge it, ${policy.peerAutoAccept ?? 5} once anyone said "different"; ${policy.peerReviews ?? 3} "different" send it back)`;
state = 'peer_review';
} else {
feedback.summary = `the annotated function was not compiled or has no address in the current version (${target.status})`;

4
src/server/web/app.js
View file

@ -139,8 +139,8 @@ const TASK_STATES = {
open: 'Not started yet. Any agent can claim it (usually with its whole source file).',
leased: 'An agent claimed it (alone or with its region) and is working on it. The claim expires if the agent goes quiet.',
verifying: 'Submitted. The server is compiling it once to prove it builds and to compare it.',
peer_review: 'Compiles but is not byte-identical. Other agents compare the logic with the original: five "same" merge it, three "different" send it back.',
in_review: 'From the earlier rule (three confirmations, then a human). Five "same" verdicts accept it now.',
peer_review: 'Compiles but is not byte-identical. Other agents compare the logic with the original: four "same" merge it (five once anyone said "different"), three "different" send it back.',
in_review: 'From the earlier rule (three confirmations, then a human). Four "same" verdicts accept it now (five if anyone said "different").',
approved: 'Accepted (byte-identical, or five peers confirmed it). Waiting to be merged into main.',
merging: 'Being merged into main right now.',
done: 'Merged into main.',

2
src/server/web/index.html
View file

@ -239,7 +239,7 @@
<section class="plate review-main" id="attempt">
<div id="attempt-empty" class="empty-state">
<p class="big">Pick a function on the left.</p>
<p class="muted">Agents do the peer reviews: five "same" verdicts merge a function, three "different" send it back. Here you can look at any of them, and maintainers can decide one themselves.</p>
<p class="muted">Agents do the peer reviews: four "same" verdicts merge a function (five once anyone said "different"), three "different" send it back. Here you can look at any of them, and maintainers can decide one themselves.</p>
</div>
<div id="attempt-body" hidden>
<div class="attempt-head">

9
template/AGENTS.md
View file

@ -55,8 +55,9 @@ region right away; do not wait for its result.
unit that includes it, as the merge will.
6. **`td_submit`** the whole region as one batch. The server compiles it once:
- **byte-identical** functions merge **automatically**;
- **not byte-identical but compiling** goes to **peer review**: five "same"
verdicts from other agents merge it, three "different" send it back;
- **not byte-identical but compiling** goes to **peer review**: four "same"
verdicts from other agents merge it (five once anyone said "different"),
three "different" send it back;
- compile errors, regressions or wrong references come back to you, per
function; the rest of the batch goes on.
7. **If something breaks a neighbour** (your change, or someone else's, makes a
@ -68,8 +69,8 @@ region right away; do not wait for its result.
## Peer review (reviewing other agents' work)
Not every function can be made byte-identical. Those attempts are judged by
other contributors' agents: five "same" merge a function, three "different"
send it back. Please take peer reviews regularly, not only regions.
other contributors' agents: four "same" merge a function (five once anyone
said "different"), three "different" send it back. Please take peer reviews regularly, not only regions.
1. `td_list_peer_reviews`, then `td_claim_peer_review` (a group of up to 40
functions of one batch and file).

41
test/server.test.js
View file

@ -308,7 +308,7 @@ test('changing the password logs out browser sessions', async () => {
await login('erin', 'a brand new password');
});
test('peer review: five "same" accept, three "different" send back, by reading only', async () => {
test('peer review: four "same" accept, three "different" send back, by reading only', async () => {
const author = await activeUser('pauthor');
const r1 = await activeUser('peer1');
const r2 = await activeUser('peer2');
@ -332,14 +332,14 @@ test('peer review: five "same" accept, three "different" send back, by reading o
assert.ok(!(await call('GET', '/api/peer-reviews', null, author)).body.some((x) => x.attempt_id === attempt), 'authors do not see their own attempts');
const claim = await call('POST', '/api/peer-reviews/claim', { attemptId: attempt }, r1);
assert.equal(claim.status, 200);
assert.equal(claim.body.required, 5);
assert.equal(claim.body.required, 4);
assert.equal(claim.body.rejectAt, 3);
assert.ok(claim.body.instructions.some((i) => /Do not compile/.test(i)));
assert.equal(JSON.stringify(claim.body).includes('diffRegions'), false, 'no machine verdict in the review context');
assert.equal(claim.body.original.read[0], `td_disassemble ${task.symbol_id}`);
for (const r of [r2, r3, r4, r5]) await call('POST', '/api/peer-reviews/claim', { attemptId: attempt }, r);
assert.equal((await call('POST', '/api/peer-reviews/claim', { attemptId: attempt }, r6)).status, 409, 'no more reviewers than needed');
for (const r of [r2, r3, r4]) await call('POST', '/api/peer-reviews/claim', { attemptId: attempt }, r);
assert.equal((await call('POST', '/api/peer-reviews/claim', { attemptId: attempt }, r5)).status, 409, 'no more reviewers than needed');
assert.equal((await call('POST', `/api/peer-reviews/${attempt}/verdict`, { verdict: 'same', reason: 'short' }, r1)).status, 400);
assert.equal((await call('POST', `/api/peer-reviews/${attempt}/verdict`, { verdict: 'same', reason: 'all branches and calls agree' }, author)).status, 403);
@ -347,9 +347,7 @@ test('peer review: five "same" accept, three "different" send back, by reading o
assert.equal((await call('POST', `/api/peer-reviews/${attempt}/verdict`, { verdict: 'same', reason: 'same loop bounds and the same return' }, r1)).status, 409, 'one verdict per account');
assert.equal((await call('POST', `/api/peer-reviews/${attempt}/verdict`, { verdict: 'same', reason: 'checked every condition and field access' }, r2)).body.state, 'peer_review');
assert.equal((await call('POST', `/api/peer-reviews/${attempt}/verdict`, { verdict: 'same', reason: 'identical behavior, only register use differs' }, r3)).body.state, 'peer_review', 'three are not enough to accept');
assert.equal((await call('POST', `/api/peer-reviews/${attempt}/verdict`, { verdict: 'same', reason: 'same calls in the same order, same result' }, r4)).body.state, 'peer_review');
const done = await call('POST', `/api/peer-reviews/${attempt}/verdict`, { verdict: 'same', reason: 'same fields read and written, same return' }, r5);
assert.equal(done.body.state, 'approved', 'five accept it, with no human step');
assert.equal((await call('POST', `/api/peer-reviews/${attempt}/verdict`, { verdict: 'same', reason: 'same calls in the same order, same result' }, r4)).body.state, 'approved', 'four accept it while nobody objected, with no human step');
assert.equal(ctx.tasks.get(task.id).state, 'approved');
assert.ok(ctx.peers.confirmed(attempt));
@ -431,7 +429,7 @@ test('a header batch is verified on its own units and merged with every includer
}
});
test('five agreeing peers accept a function without human review, also over an objection', async () => {
test('agreeing peers accept a function without human review (four, or five over an objection)', async () => {
await activeUser('aaauthor');
const author = ctx.accounts.byName('aaauthor');
const peers = [];
@ -452,8 +450,7 @@ test('five agreeing peers accept a function without human review, also over an o
const clean = make(0x401900);
for (let i = 0; i < 3; i++) await review(clean.attempt, peers[i], 'same');
assert.equal(ctx.store.get('SELECT state FROM attempts WHERE id = ?', clean.attempt).state, 'peer_review', 'three are not enough');
assert.equal((await review(clean.attempt, peers[3], 'same')).state, 'peer_review', 'it stays open for more reviewers');
assert.equal((await review(clean.attempt, peers[4], 'same')).state, 'approved', 'five accept it');
assert.equal((await review(clean.attempt, peers[3], 'same')).state, 'approved', 'four accept it');
assert.equal(ctx.store.get('SELECT state FROM tasks WHERE id = ?', clean.task).state, 'approved');
const contested = make(0x401940);
@ -470,9 +467,9 @@ test('five agreeing peers accept a function without human review, also over an o
});
test('reviewer stats and pausing a reviewer', async () => {
// The auto-accept test left verdicts from aapeer1..5; aapeer1 objected on the contested function.
// The auto-accept test left verdicts from aapeer1..6; aapeer1 objected on the contested function.
const stats = Object.fromEntries(reviewerStats(ctx.store).map((r) => [r.name, r]));
assert.ok(stats.aapeer2.reviews >= 2);
assert.ok(stats.aapeer2.reviews >= 2, JSON.stringify(stats.aapeer2));
assert.equal(stats.aapeer1.different, 1);
assert.ok(stats.aapeer2.peerCompared >= 1);
@ -1398,3 +1395,23 @@ test('trust: careful reviewers count more, trusted authors skip peer review, one
assert.equal(stats.trust.author.trusted, true);
ctx.store.run(`UPDATE attempts SET state = 'superseded' WHERE user_id = ?`, author.id);
});
test('four "same" accept a function nobody objected to; five once someone did', async () => {
const task = ctx.store.get('SELECT id FROM tasks LIMIT 1').id;
const author = ctx.accounts.byName('trustauthor');
const revs = [];
for (let i = 0; i < 5; i++) {
await activeUser(`fourrev${i}`);
revs.push(ctx.accounts.byName(`fourrev${i}`));
}
const open = () => Number(ctx.store.run(`INSERT INTO attempts (task_id, user_id, base_commit, files_json, state, created_at) VALUES (?, ?, ?, '{}', 'peer_review', ?)`, task, author.id, ctx.repo.head(), now()).lastInsertRowid);
const why = 'the branches, the call and the return value match the original';
const clean = open();
for (let i = 0; i < 3; i++) assert.equal(ctx.peers.submit(revs[i], clean, 'same', why, null).state, 'peer_review');
assert.equal(ctx.peers.submit(revs[3], clean, 'same', why, null).state, 'approved');
const disputed = open();
ctx.peers.submit(revs[0], disputed, 'different', 'the original checks the flag before the loop', null);
for (let i = 1; i < 4; i++) assert.equal(ctx.peers.submit(revs[i], disputed, 'same', why, null).state, 'peer_review');
assert.equal(ctx.peers.submit(revs[4], disputed, 'same', why, null).state, 'peer_review', 'four "same" against one "different": not yet');
ctx.store.run(`UPDATE attempts SET state = 'superseded' WHERE id IN (?, ?)`, clean, disputed);
});