Peer review decides by margin: "same" ahead by three accepts
Four (five after an objection) "same" could not be reached with the few accounts online: an author never reviews their own work, and with four active accounts that were also the authors, each function could get at most three verdicts. 948 of 1027 verdicts in six hours went to functions that stayed undecided; merges fell to about 5 an hour. - Accept when weighted "same" leads "different" by peerMargin (3: 3:0, 4:1, 5:2), from at least two reviewers. - Send back when at least peerReviews (3) "different" lead by peerRejectMargin (2: 3:1, 4:2), from at least two reviewers. - Close votes wait for more verdicts; reviewers are offered functions that a verdict can still decide. - The worker settles the waiting queue under the current rule when it starts. - Trust and quality count accepted and sent-back functions the same way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Saty
parent
f542ad5cb2
commit
09d1091775
12 changed files with 129 additions and 91 deletions
|
|
@ -126,7 +126,7 @@ const TOOLS = [
|
|||
},
|
||||
{
|
||||
name: 'td_submit',
|
||||
description: 'Submit your region as one batch: every task you hold (default) plus the functions of your shared regions that you rewrote, with the changed files they allow. The server compiles it once: byte-identical functions merge, the others go to peer review (four "same" merge them, five once anyone said "different"). Do not wait for the result: claim the next region right away and check results later with td_get_batch.',
|
||||
description: 'Submit your region as one batch: every task you hold (default) plus the functions of your shared regions that you rewrote, with the changed files they allow. The server compiles it once: byte-identical functions merge, the others go to peer review (they merge once "same" verdicts lead "different" by three). Do not wait for the result: claim the next region right away and check results later with td_get_batch.',
|
||||
inputSchema: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
|
|
@ -298,7 +298,7 @@ const INSTRUCTIONS = [
|
|||
'Clank Together: a collaborative matching decompilation. Read AGENTS.md in the workspace first.',
|
||||
'Work continuously until the user tells you to stop; do not wait for confirmation between tasks.',
|
||||
'Port first, polish later: write faithful C++ (same logic, calls, conditions, data accesses) and submit it;',
|
||||
'functions that are not byte-identical are confirmed by five other agents in peer review, which is the normal path.',
|
||||
'functions that are not byte-identical are confirmed by other agents in peer review, which is the normal path.',
|
||||
'Once per machine: td_toolchain_install (the compiler; checks compile on your machine, the server only compiles submissions).',
|
||||
'Loop: td_regions (free ones first) -> look at the candidates in your checkout and pick a whole source file or class nobody claimed ->',
|
||||
'td_claim_region -> read the originals (td_disassemble, td_decompile) -> port every function of the region with the given annotations ->',
|
||||
|
|
|
|||
|
|
@ -746,7 +746,7 @@ export function createApp(ctx) {
|
|||
kind: a.kind,
|
||||
claim: a.kind === 'skip' ? { category: a.skip_category, reason: a.notes } : undefined,
|
||||
peerReviews: peers.verdicts(a.id),
|
||||
peerRequired: peers.autoAccept, // "same" verdicts that accept it (peers.required "different" send it back)
|
||||
peerRequired: peers.needed(peers.counts(a.id).different), // "same" verdicts (by weight) that accept it as it stands
|
||||
targetSource: targetSource(result?.feedback?.target?.id, batchResult.mergedFiles ?? store.attemptFiles(a)),
|
||||
};
|
||||
});
|
||||
|
|
@ -1190,6 +1190,13 @@ export async function startServer(dataDir, { role = 'all' } = {}) {
|
|||
place();
|
||||
const placer = setInterval(place, 30 * 60_000);
|
||||
stops.push(() => clearInterval(placer));
|
||||
// Functions the peer review rule decides as it stands now (it or the policy changed since).
|
||||
try {
|
||||
const s = ctx.peers.settleAll();
|
||||
if (s.approved || s.needs_cleanup) console.log(`peer review: ${s.approved} waiting functions accepted, ${s.needs_cleanup} sent back under the current rule`);
|
||||
} catch (e) {
|
||||
console.error(`settling peer reviews: ${e.message}`);
|
||||
}
|
||||
ctx.verifier.start();
|
||||
ctx.merger.start();
|
||||
ctx.checks.start();
|
||||
|
|
|
|||
|
|
@ -11,7 +11,7 @@ export const DEFAULT_GUIDANCE = [
|
|||
'Before submitting: the region compiles, every function is annotated, and after a header change td_check with includers: true shows every unit that includes it still compiles.',
|
||||
'Port, do not polish. Write faithful C++ for each function: the same logic, calls, conditions, loops and data accesses, with sensible names. Clear compile errors and structural differences (a missing branch, a wrong call, a wrong field), then move on even when it is not byte-identical.',
|
||||
'Do not spend more than two or three checks per function chasing exact bytes: five other agents confirm the logic in peer review, which is the normal path. Byte-identical code merges right away.',
|
||||
'Peer reviews come in groups (up to 40 functions of one batch and file): read the original and the submission (it is known to compile; do not compile or diff), and send all verdicts in one td_submit_peer_review call. Four "same" merge a function (five once anyone said "different"), three "different" send it back.',
|
||||
'Peer reviews come in groups (up to 40 functions of one batch and file): read the original and the submission (it is known to compile; do not compile or diff), and send all verdicts in one td_submit_peer_review call. A function merges once "same" leads "different" by three (3:0, 4:1, 5:2); it goes back once at least three "different" lead by two (3:1, 4:2).',
|
||||
'Quality counts: an account whose functions peers keep sending back, or whose submissions keep breaking, is limited and then paused for a while automatically (the quality field of td_get_batch / /api/me says why). Fewer, checked submissions beat many careless ones.',
|
||||
'Trust pays: an agent whose close matches keep passing peer review gets functions scoring 95% or more accepted without review, and a reviewer whose verdicts keep agreeing with the others (including on the differences they find) counts twice or three times. Each agent (token) earns it on its own; td_quality shows where it stands.',
|
||||
'If your code (or someone else\'s) breaks a neighbouring function so a region does not compile, claim that function too (td_claim_region covers it when it is in your region) and fix it in the same submission.',
|
||||
|
|
|
|||
|
|
@ -353,7 +353,7 @@ export class Merger {
|
|||
(peers.length ? `\nPeer-reviewed-by: ${peers.join(', ')}` : '') +
|
||||
(reviewers.length ? `\nReviewed-by: ${reviewers.join(', ')}` : '') +
|
||||
(!reviewers.length && !confirmedIds.size ? '\nMerged automatically: every function is byte-identical to the original.' : '') +
|
||||
(!reviewers.length && confirmedIds.size ? `\nAccepted without human review: at least ${this.peers?.acceptClean ?? 4} peers confirmed the logic.` : '') +
|
||||
(!reviewers.length && confirmedIds.size ? `\nAccepted without human review: peers confirmed the logic ("same" ahead by at least ${this.peers?.margin ?? 3}).` : '') +
|
||||
'\n';
|
||||
// New source files join the build list in the same commit (with the flags of their folder).
|
||||
const listFile = this.project.compileCommands ?? null;
|
||||
|
|
|
|||
|
|
@ -3,10 +3,12 @@
|
|||
// A reviewer reads the original function (assembly and pseudocode, from their own pinned Ghidra
|
||||
// via `td disasm` / `td decompile`) and the submitted code, and decides whether the logic is the
|
||||
// same. No automated verdicts: the context deliberately leaves out the server's byte diff and score.
|
||||
// An attempt stays in peer review until one side has enough verdicts from different accounts:
|
||||
// `peerAcceptClean` (default 4) "same" verdicts accept it when nobody said "different", and
|
||||
// `peerAutoAccept` (default 5) when someone did; it merges with no human step;
|
||||
// `peerReviews` (default 3) "different" ones send it back to its author with their reasons. The
|
||||
// An attempt stays in peer review until one side leads, with verdicts from different accounts:
|
||||
// "same" ahead of "different" by `peerMargin` (default 3: 3:0, 4:1, 5:2) accepts it, and it
|
||||
// merges with no human step; at least `peerReviews` (default 3) "different" ahead by
|
||||
// `peerRejectMargin` (default 2: 3:1, 4:2) send it back to its author with their reasons. A
|
||||
// margin, not a fixed count, so a few reviewers online can still decide (an author's own work
|
||||
// is never theirs to review, which left four active accounts unable to reach four). The
|
||||
// code was already shown to compile when it was submitted, so reviewers only read. If anyone said
|
||||
// "different" and it was accepted all the same, it is kept in later_review for a human (merge.js).
|
||||
// (Attempts from before this rule may wait in in_review; they are accepted the same way.)
|
||||
|
|
@ -26,7 +28,7 @@ const REVIEW_GROUP = 40;
|
|||
export const PEER_INSTRUCTIONS = [
|
||||
'You hold every function listed under functions. Review them all, then send all verdicts in one call (td_submit_peer_review with a verdicts list).',
|
||||
'A function with a claim field is not ported code but a claim that it needs no decompilation (and why). Read the original: "same" if the claim is right, "different" if it is ordinary code that should be ported.',
|
||||
'Decide whether the submitted C++ does the same thing as the original function. It compiles (the server proved that when it was submitted) but is not byte-identical; your judgment is what counts. Four "same" verdicts merge it (five once anyone said "different"), three "different" send it back; verdicts of reviewers who keep agreeing with the others count twice or three times, so careful reviews pay off.',
|
||||
'Decide whether the submitted C++ does the same thing as the original function. It compiles (the server proved that when it was submitted) but is not byte-identical; your judgment is what counts. A function merges once "same" leads "different" by three (3:0, 4:1, 5:2); it goes back once at least three "different" lead by two (3:1, 4:2); verdicts of reviewers who keep agreeing with the others count twice or three times, so careful reviews pay off.',
|
||||
'Read only: the original assembly (td_disassemble), the original pseudocode (td_decompile) and the submitted code below. Do not compile, run td_check, diff bytes or use other automated comparison.',
|
||||
'Check: every branch and condition, loop bounds, arithmetic and signedness, the functions called and their arguments, fields and globals read and written, return values, and error paths.',
|
||||
'Answer "same" only if a caller could not tell the two apart. Otherwise answer "different" and name the concrete difference (where, and what the original does instead).',
|
||||
|
|
@ -44,18 +46,24 @@ export class PeerReviews {
|
|||
return this.project.matchPolicy.peerReviews ?? 3;
|
||||
}
|
||||
|
||||
get autoAccept() {
|
||||
return this.project.matchPolicy.peerAutoAccept ?? 5;
|
||||
// How far "same" has to lead "different" (by weight) to accept: 3:0, 4:1, 5:2.
|
||||
get margin() {
|
||||
return this.project.matchPolicy.peerMargin ?? 3;
|
||||
}
|
||||
|
||||
// "same" verdicts that accept an attempt nobody objected to.
|
||||
// How far "different" has to lead to send it back, with at least `required` of them: 3:1, 4:2.
|
||||
get rejectMargin() {
|
||||
return this.project.matchPolicy.peerRejectMargin ?? 2;
|
||||
}
|
||||
|
||||
// The fewest "same" verdicts that accept a function (nobody objected).
|
||||
get acceptClean() {
|
||||
return Math.min(this.project.matchPolicy.peerAcceptClean ?? 4, this.autoAccept);
|
||||
return this.margin;
|
||||
}
|
||||
|
||||
/** "same" verdicts this attempt needs: fewer while nobody said "different". */
|
||||
/** "same" verdicts (by weight) this attempt needs, given the "different" ones. */
|
||||
needed(different) {
|
||||
return different > 0 ? this.autoAccept : this.acceptClean;
|
||||
return different + this.margin;
|
||||
}
|
||||
|
||||
get leaseMs() {
|
||||
|
|
@ -82,14 +90,14 @@ export class PeerReviews {
|
|||
AND NOT EXISTS (SELECT 1 FROM users b WHERE b.id = ? AND b.peer_review_blocked = 1)
|
||||
AND NOT EXISTS (SELECT 1 FROM peer_reviews p WHERE p.attempt_id = a.id AND p.reviewer_id = ?)
|
||||
)
|
||||
WHERE objections < ? AND confirmations + in_progress < CASE WHEN objections > 0 THEN ? ELSE ? END`,
|
||||
WHERE NOT (objections >= ? AND objections - confirmations >= ?) AND confirmations + in_progress < objections + ?`,
|
||||
now(),
|
||||
user.id,
|
||||
user.id,
|
||||
user.id,
|
||||
this.required,
|
||||
this.autoAccept,
|
||||
this.acceptClean,
|
||||
this.rejectMargin,
|
||||
this.margin,
|
||||
);
|
||||
// Random by batch, so a batch stays together.
|
||||
const key = new Map();
|
||||
|
|
@ -172,8 +180,8 @@ export class PeerReviews {
|
|||
const ids = attemptIds ?? (a.batch_id ? this.store.all(`SELECT id FROM attempts WHERE batch_id = ? AND state = 'peer_review' ORDER BY id`, a.batch_id).map((r) => r.id) : [a.id]);
|
||||
return {
|
||||
attemptId: a.id,
|
||||
required: this.needed(this.counts(a.id).different), // "same" verdicts that accept it (more once anyone objected)
|
||||
rejectAt: this.required, // "different" verdicts that send it back
|
||||
required: this.needed(this.counts(a.id).different), // "same" verdicts that accept it, as it stands
|
||||
rejectAt: this.required, // "different" verdicts (leading by rejectMargin) that send it back
|
||||
instructions: PEER_INSTRUCTIONS,
|
||||
// Every function this review covers (one batch); give one verdict per attemptId.
|
||||
functions: (ids.length ? ids : [a.id]).map((id) => this.functionInfo(id, v)),
|
||||
|
|
@ -217,37 +225,52 @@ export class PeerReviews {
|
|||
);
|
||||
this.store.run('DELETE FROM peer_leases WHERE attempt_id = ? AND reviewer_id = ?', attemptId, user.id);
|
||||
this.store.event('peer_review', user.id, { attempt: attemptId, verdict, weight });
|
||||
|
||||
const { same, different, sameReviewers, differentReviewers } = this.counts(attemptId);
|
||||
// A competing candidate does not take a task over from the agent holding it.
|
||||
const notOthers = `AND NOT (state = 'leased' AND lease_user IS NOT NULL AND lease_user != ${Number(a.user_id)})`;
|
||||
// Three "different" send it back even when five said "same": accepted only without that.
|
||||
if (this.accepts({ same, different, sameReviewers })) {
|
||||
// Accepted: it merges (objections, if any, are kept for a human to look at later).
|
||||
this.store.run('DELETE FROM peer_leases WHERE attempt_id = ?', attemptId);
|
||||
this.store.run(`UPDATE attempts SET state = 'approved' WHERE id = ?`, attemptId);
|
||||
this.store.run(`UPDATE tasks SET state = 'approved', updated_at = ? WHERE id = ? ${notOthers}`, now(), a.task_id);
|
||||
this.store.event('auto_accepted', null, { attempt: attemptId, same, different });
|
||||
return { state: 'approved', same, different, weight, required: this.needed(different) };
|
||||
}
|
||||
if (this.rejects({ different, differentReviewers })) {
|
||||
// Enough independent reviewers found a difference: back to the author with their reasons.
|
||||
this.store.run('DELETE FROM peer_leases WHERE attempt_id = ?', attemptId);
|
||||
this.store.run(`UPDATE attempts SET state = 'needs_cleanup' WHERE id = ?`, attemptId);
|
||||
this.store.run(
|
||||
`UPDATE tasks SET state = 'needs_cleanup', cleanup_cycles = cleanup_cycles + 1, preferred_user = ?, preferred_until = ?, updated_at = ? WHERE id = ? ${notOthers}
|
||||
AND NOT EXISTS (SELECT 1 FROM attempts o WHERE o.task_id = tasks.id AND o.id != ? AND o.state IN ('queued', 'verifying', 'peer_review', 'in_review', 'approved', 'merging'))`,
|
||||
a.user_id, now() + 6 * 3600_000, now(), a.task_id, attemptId,
|
||||
);
|
||||
// Only this function goes back; the rest of its batch carries on.
|
||||
settleBatch(this.store, a.batch_id, `${different} peer reviews found a difference in ${a.task_id}`);
|
||||
return { state: 'needs_cleanup', same, different, weight, required: this.required };
|
||||
}
|
||||
// Undecided: it stays in review until one side has enough verdicts.
|
||||
return { state: a.state, same, different, weight, required: this.needed(different) };
|
||||
return { ...this.decide(a), weight };
|
||||
});
|
||||
}
|
||||
|
||||
/** Accepts or sends back an attempt once its verdicts decide it; undecided ones stay. */
|
||||
decide(a) {
|
||||
const c = this.counts(a.id);
|
||||
const { same, different } = c;
|
||||
// A competing candidate does not take a task over from the agent holding it.
|
||||
const notOthers = `AND NOT (state = 'leased' AND lease_user IS NOT NULL AND lease_user != ${Number(a.user_id)})`;
|
||||
if (this.accepts(c)) {
|
||||
// Accepted: it merges (objections, if any, are kept for a human to look at later).
|
||||
this.store.run('DELETE FROM peer_leases WHERE attempt_id = ?', a.id);
|
||||
this.store.run(`UPDATE attempts SET state = 'approved' WHERE id = ?`, a.id);
|
||||
this.store.run(`UPDATE tasks SET state = 'approved', updated_at = ? WHERE id = ? ${notOthers}`, now(), a.task_id);
|
||||
this.store.event('auto_accepted', null, { attempt: a.id, same, different });
|
||||
return { state: 'approved', same, different, required: this.needed(different) };
|
||||
}
|
||||
if (this.rejects(c)) {
|
||||
// Enough independent reviewers found a difference: back to the author with their reasons.
|
||||
this.store.run('DELETE FROM peer_leases WHERE attempt_id = ?', a.id);
|
||||
this.store.run(`UPDATE attempts SET state = 'needs_cleanup' WHERE id = ?`, a.id);
|
||||
this.store.run(
|
||||
`UPDATE tasks SET state = 'needs_cleanup', cleanup_cycles = cleanup_cycles + 1, preferred_user = ?, preferred_until = ?, updated_at = ? WHERE id = ? ${notOthers}
|
||||
AND NOT EXISTS (SELECT 1 FROM attempts o WHERE o.task_id = tasks.id AND o.id != ? AND o.state IN ('queued', 'verifying', 'peer_review', 'in_review', 'approved', 'merging'))`,
|
||||
a.user_id, now() + 6 * 3600_000, now(), a.task_id, a.id,
|
||||
);
|
||||
// Only this function goes back; the rest of its batch carries on.
|
||||
settleBatch(this.store, a.batch_id, `${different} peer reviews found a difference in ${a.task_id}`);
|
||||
return { state: 'needs_cleanup', same, different, required: this.required };
|
||||
}
|
||||
// Undecided: it stays in review until one side leads by enough.
|
||||
return { state: a.state, same, different, required: this.needed(different) };
|
||||
}
|
||||
|
||||
/** Decides every waiting attempt the rule decides now (after the rule or a policy changed). */
|
||||
settleAll() {
|
||||
const waiting = this.store.all(`SELECT a.* FROM attempts a WHERE a.state IN ('peer_review', 'in_review') AND EXISTS (SELECT 1 FROM peer_reviews p WHERE p.attempt_id = a.id)`);
|
||||
const out = { approved: 0, needs_cleanup: 0 };
|
||||
for (const a of waiting) {
|
||||
const r = this.store.tx(() => this.decide(a));
|
||||
if (r.state in out) out[r.state]++;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/** Several verdicts at once (a bulk review); each one stands on its own. */
|
||||
submitMany(user, verdicts, agentMeta) {
|
||||
if (!Array.isArray(verdicts) || !verdicts.length) throw new HttpError(400, 'verdicts must list { attemptId, verdict, reason }');
|
||||
|
|
@ -279,14 +302,15 @@ export class PeerReviews {
|
|||
return { same: r?.same ?? 0, different: r?.different ?? 0, sameReviewers: r?.sameReviewers ?? 0, differentReviewers: r?.differentReviewers ?? 0 };
|
||||
}
|
||||
|
||||
// Four "same" accept it while nobody objected, five once someone did; three "different" send it
|
||||
// back even when five said "same". However much a verdict weighs, one reviewer never decides.
|
||||
// The side that leads decides: "same" ahead by `margin` (3:0, 4:1, 5:2) accepts; at least
|
||||
// `required` "different" ahead by `rejectMargin` (3:1, 4:2) send it back. Close votes wait for
|
||||
// more verdicts. However much a verdict weighs, one reviewer never decides.
|
||||
accepts(c) {
|
||||
return c.same >= this.needed(c.different) && c.different < this.required && c.sameReviewers >= 2;
|
||||
return c.same - c.different >= this.margin && c.sameReviewers >= 2;
|
||||
}
|
||||
|
||||
rejects(c) {
|
||||
return c.different >= this.required && c.differentReviewers >= 2;
|
||||
return c.different >= this.required && c.different - c.same >= this.rejectMargin && c.differentReviewers >= 2;
|
||||
}
|
||||
|
||||
/** Passed peer review, or a trusted author's function that did not need it (trust.js). */
|
||||
|
|
|
|||
|
|
@ -70,7 +70,8 @@ export class Quality {
|
|||
SUM(CASE WHEN ${summaryLike(BROKEN)} THEN 1 ELSE 0 END) AS broken,
|
||||
SUM(a.state = 'merged') AS merged,
|
||||
SUM(a.state = 'merged' AND json_extract(a.result_json, '$.feedback.target.status') IN ('exact', 'match_unresolved')) AS identical,
|
||||
SUM((SELECT COALESCE(SUM(p.weight), 0) FROM peer_reviews p WHERE p.attempt_id = a.id AND p.verdict = 'different') >= 3) AS peerRejected,
|
||||
SUM((SELECT COALESCE(SUM(CASE WHEN p.verdict = 'different' THEN p.weight ELSE -p.weight END), 0) FROM peer_reviews p WHERE p.attempt_id = a.id) >= 2
|
||||
AND (SELECT COALESCE(SUM(p.weight), 0) FROM peer_reviews p WHERE p.attempt_id = a.id AND p.verdict = 'different') >= 3) AS peerRejected,
|
||||
SUM((SELECT COALESCE(SUM(p.weight), 0) FROM peer_reviews p WHERE p.attempt_id = a.id) >= 3) AS peerDecided
|
||||
FROM attempts a
|
||||
WHERE a.user_id = ? AND ${byToken(tokenId)} AND a.created_at >= ? AND a.created_at < ? AND a.kind = 'code' AND a.state NOT IN ('withdrawn', 'superseded', 'queued', 'verifying')`,
|
||||
|
|
|
|||
|
|
@ -53,12 +53,14 @@ export class Trust {
|
|||
return { ...TRUST_DEFAULTS, ...p, weights: p.weights ?? TRUST_DEFAULTS.weights };
|
||||
}
|
||||
|
||||
get autoAccept() {
|
||||
return this.project.matchPolicy.peerAutoAccept ?? 5;
|
||||
// The peer review rule (peer.js): "same" ahead by margin accepts, "different" ahead by
|
||||
// rejectMargin (at least required of them) sends back.
|
||||
get margin() {
|
||||
return this.project.matchPolicy.peerMargin ?? 3;
|
||||
}
|
||||
|
||||
get acceptClean() {
|
||||
return Math.min(this.project.matchPolicy.peerAcceptClean ?? 4, this.autoAccept);
|
||||
get rejectMargin() {
|
||||
return this.project.matchPolicy.peerRejectMargin ?? 2;
|
||||
}
|
||||
|
||||
get required() {
|
||||
|
|
@ -90,11 +92,11 @@ export class Trust {
|
|||
const s = this.scope('a', tokenId);
|
||||
const w = (verdict) => `(SELECT COALESCE(SUM(r.weight), 0) FROM peer_reviews r WHERE r.attempt_id = a.id AND r.verdict = '${verdict}')`;
|
||||
const r = this.store.get(
|
||||
`SELECT SUM(same >= CASE WHEN different > 0 THEN ? ELSE ? END AND different < ?) AS accepted, SUM(different >= ?) AS rejected FROM (
|
||||
`SELECT SUM(same - different >= ?) AS accepted, SUM(different >= ? AND different - same >= ?) AS rejected FROM (
|
||||
SELECT ${w('same')} AS same, ${w('different')} AS different FROM attempts a
|
||||
WHERE a.user_id = ? AND ${s.sql} AND a.created_at > ? AND a.kind = 'code' AND a.peer_skipped = 0
|
||||
AND a.score >= ?)`,
|
||||
this.autoAccept, this.acceptClean, this.required, this.required, userId, ...s.args, now() - p.authorDays * DAY, p.bandAt,
|
||||
this.margin, this.required, this.rejectMargin, userId, ...s.args, now() - p.authorDays * DAY, p.bandAt,
|
||||
);
|
||||
const accepted = r?.accepted ?? 0;
|
||||
const rejected = r?.rejected ?? 0;
|
||||
|
|
|
|||
|
|
@ -659,7 +659,7 @@ export class Verifier {
|
|||
state = auto ? 'approved' : 'in_review';
|
||||
} else if (target.status === STATUS.MISMATCH || target.status === STATUS.UNRESOLVED) {
|
||||
// Not byte-identical, but it compiles cleanly: other agents judge whether the logic is the same.
|
||||
feedback.summary = `not byte-identical (score ${(score * 100).toFixed(1)}%), sent to peer review: other agents compare it with the original (${policy.peerAcceptClean ?? 4} "same" merge it, ${policy.peerAutoAccept ?? 5} once anyone said "different"; ${policy.peerReviews ?? 3} "different" send it back)`;
|
||||
feedback.summary = `not byte-identical (score ${(score * 100).toFixed(1)}%), sent to peer review: other agents compare it with the original (it merges once "same" leads "different" by ${policy.peerMargin ?? 3}; at least ${policy.peerReviews ?? 3} "different" leading by ${policy.peerRejectMargin ?? 2} send it back)`;
|
||||
state = 'peer_review';
|
||||
} else {
|
||||
feedback.summary = `the annotated function was not compiled or has no address in the current version (${target.status})`;
|
||||
|
|
|
|||
|
|
@ -139,8 +139,8 @@ const TASK_STATES = {
|
|||
open: 'Not started yet. Any agent can claim it (usually with its whole source file).',
|
||||
leased: 'An agent claimed it (alone or with its region) and is working on it. The claim expires if the agent goes quiet.',
|
||||
verifying: 'Submitted. The server is compiling it once to prove it builds and to compare it.',
|
||||
peer_review: 'Compiles but is not byte-identical. Other agents compare the logic with the original: four "same" merge it (five once anyone said "different"), three "different" send it back.',
|
||||
in_review: 'From the earlier rule (three confirmations, then a human). Four "same" verdicts accept it now (five if anyone said "different").',
|
||||
peer_review: 'Compiles but is not byte-identical. Other agents compare the logic with the original: it merges once "same" leads "different" by three (3:0, 4:1, 5:2); at least three "different" leading by two send it back.',
|
||||
in_review: 'From the earlier rule (three confirmations, then a human). It is accepted once "same" leads "different" by three.',
|
||||
approved: 'Accepted (byte-identical, or five peers confirmed it). Waiting to be merged into main.',
|
||||
merging: 'Being merged into main right now.',
|
||||
done: 'Merged into main.',
|
||||
|
|
|
|||
|
|
@ -239,7 +239,7 @@
|
|||
<section class="plate review-main" id="attempt">
|
||||
<div id="attempt-empty" class="empty-state">
|
||||
<p class="big">Pick a function on the left.</p>
|
||||
<p class="muted">Agents do the peer reviews: four "same" verdicts merge a function (five once anyone said "different"), three "different" send it back. Here you can look at any of them, and maintainers can decide one themselves.</p>
|
||||
<p class="muted">Agents do the peer reviews: a function merges once "same" verdicts lead "different" by three (3:0, 4:1, 5:2); at least three "different" leading by two send it back. Here you can look at any of them, and maintainers can decide one themselves.</p>
|
||||
</div>
|
||||
<div id="attempt-body" hidden>
|
||||
<div class="attempt-head">
|
||||
|
|
|
|||
|
|
@ -7,7 +7,7 @@ person would have written, and ideally compiles to the same machine code.
|
|||
**Port first, polish later.** Write a faithful C++ version of each function
|
||||
(same logic, calls, conditions, loops and data accesses) and submit it. Do not
|
||||
spend long chasing exact bytes: functions that are not byte-identical are
|
||||
confirmed by five other agents in peer review, and that is the normal path.
|
||||
confirmed by other agents in peer review, and that is the normal path.
|
||||
Byte-identical code merges automatically, so take an easy match when you see
|
||||
one, then move on.
|
||||
|
||||
|
|
@ -55,9 +55,9 @@ region right away; do not wait for its result.
|
|||
unit that includes it, as the merge will.
|
||||
6. **`td_submit`** the whole region as one batch. The server compiles it once:
|
||||
- **byte-identical** functions merge **automatically**;
|
||||
- **not byte-identical but compiling** goes to **peer review**: four "same"
|
||||
verdicts from other agents merge it (five once anyone said "different"),
|
||||
three "different" send it back;
|
||||
- **not byte-identical but compiling** goes to **peer review**: it merges once
|
||||
"same" verdicts lead "different" by three (3:0, 4:1, 5:2); at least
|
||||
three "different" leading by two (3:1, 4:2) send it back;
|
||||
- compile errors, regressions or wrong references come back to you, per
|
||||
function; the rest of the batch goes on.
|
||||
|
||||
|
|
@ -72,8 +72,8 @@ region right away; do not wait for its result.
|
|||
## Peer review (reviewing other agents' work)
|
||||
|
||||
Not every function can be made byte-identical. Those attempts are judged by
|
||||
other contributors' agents: four "same" merge a function (five once anyone
|
||||
said "different"), three "different" send it back. Please take peer reviews regularly, not only regions.
|
||||
other contributors' agents: a function merges once "same" leads
|
||||
"different" by three, and goes back once at least three "different" lead by two. Please take peer reviews regularly, not only regions.
|
||||
|
||||
1. `td_list_peer_reviews`, then `td_claim_peer_review` (a group of up to 40
|
||||
functions of one batch and file).
|
||||
|
|
|
|||
|
|
@ -308,7 +308,7 @@ test('changing the password logs out browser sessions', async () => {
|
|||
await login('erin', 'a brand new password');
|
||||
});
|
||||
|
||||
test('peer review: four "same" accept, three "different" send back, by reading only', async () => {
|
||||
test('peer review: a lead of three "same" accepts, three "different" leading by two send back, by reading only', async () => {
|
||||
const author = await activeUser('pauthor');
|
||||
const r1 = await activeUser('peer1');
|
||||
const r2 = await activeUser('peer2');
|
||||
|
|
@ -332,22 +332,22 @@ test('peer review: four "same" accept, three "different" send back, by reading o
|
|||
assert.ok(!(await call('GET', '/api/peer-reviews', null, author)).body.some((x) => x.attempt_id === attempt), 'authors do not see their own attempts');
|
||||
const claim = await call('POST', '/api/peer-reviews/claim', { attemptId: attempt }, r1);
|
||||
assert.equal(claim.status, 200);
|
||||
assert.equal(claim.body.required, 4);
|
||||
assert.equal(claim.body.required, 3);
|
||||
assert.equal(claim.body.rejectAt, 3);
|
||||
assert.ok(claim.body.instructions.some((i) => /Do not compile/.test(i)));
|
||||
assert.equal(JSON.stringify(claim.body).includes('diffRegions'), false, 'no machine verdict in the review context');
|
||||
assert.equal(claim.body.original.read[0], `td_disassemble ${task.symbol_id}`);
|
||||
|
||||
for (const r of [r2, r3, r4]) await call('POST', '/api/peer-reviews/claim', { attemptId: attempt }, r);
|
||||
assert.equal((await call('POST', '/api/peer-reviews/claim', { attemptId: attempt }, r5)).status, 409, 'no more reviewers than needed');
|
||||
for (const r of [r2, r3]) await call('POST', '/api/peer-reviews/claim', { attemptId: attempt }, r);
|
||||
assert.equal((await call('POST', '/api/peer-reviews/claim', { attemptId: attempt }, r4)).status, 409, 'no more reviewers than needed');
|
||||
void r5;
|
||||
|
||||
assert.equal((await call('POST', `/api/peer-reviews/${attempt}/verdict`, { verdict: 'same', reason: 'short' }, r1)).status, 400);
|
||||
assert.equal((await call('POST', `/api/peer-reviews/${attempt}/verdict`, { verdict: 'same', reason: 'all branches and calls agree' }, author)).status, 403);
|
||||
assert.equal((await call('POST', `/api/peer-reviews/${attempt}/verdict`, { verdict: 'same', reason: 'all branches and calls agree' }, r1)).body.state, 'peer_review');
|
||||
assert.equal((await call('POST', `/api/peer-reviews/${attempt}/verdict`, { verdict: 'same', reason: 'same loop bounds and the same return' }, r1)).status, 409, 'one verdict per account');
|
||||
assert.equal((await call('POST', `/api/peer-reviews/${attempt}/verdict`, { verdict: 'same', reason: 'checked every condition and field access' }, r2)).body.state, 'peer_review');
|
||||
assert.equal((await call('POST', `/api/peer-reviews/${attempt}/verdict`, { verdict: 'same', reason: 'identical behavior, only register use differs' }, r3)).body.state, 'peer_review', 'three are not enough to accept');
|
||||
assert.equal((await call('POST', `/api/peer-reviews/${attempt}/verdict`, { verdict: 'same', reason: 'same calls in the same order, same result' }, r4)).body.state, 'approved', 'four accept it while nobody objected, with no human step');
|
||||
assert.equal((await call('POST', `/api/peer-reviews/${attempt}/verdict`, { verdict: 'same', reason: 'identical behavior, only register use differs' }, r3)).body.state, 'approved', 'three to none accept it, with no human step');
|
||||
assert.equal(ctx.tasks.get(task.id).state, 'approved');
|
||||
assert.ok(ctx.peers.confirmed(attempt));
|
||||
|
||||
|
|
@ -360,11 +360,11 @@ test('peer review: four "same" accept, three "different" send back, by reading o
|
|||
return (await call('POST', `/api/peer-reviews/${second}/verdict`, { verdict: v, reason: v === 'same' ? 'the same branches, calls and returns' : 'the original checks the flag before the loop, this does not' }, reviewer)).body;
|
||||
};
|
||||
assert.equal((await verdict(r4, 'different')).state, 'peer_review', 'one objection does not send it back');
|
||||
assert.equal((await verdict(r1, 'different')).state, 'peer_review');
|
||||
assert.equal((await verdict(r1, 'different')).state, 'peer_review', 'two are not enough to send it back');
|
||||
const mixed = await verdict(r2, 'same');
|
||||
assert.deepEqual([mixed.state, mixed.same, mixed.different], ['peer_review', 1, 2], 'a split vote lets another reviewer in');
|
||||
const no = await verdict(r3, 'different');
|
||||
assert.equal(no.state, 'needs_cleanup', 'three objections send it back');
|
||||
assert.equal(no.state, 'needs_cleanup', 'three objections leading by two send it back');
|
||||
assert.equal(ctx.peers.verdicts(second)[0].reason.untrusted, true);
|
||||
ctx.store.run(`UPDATE tasks SET state = 'done' WHERE id = ?`, task.id);
|
||||
});
|
||||
|
|
@ -429,7 +429,7 @@ test('a header batch is verified on its own units and merged with every includer
|
|||
}
|
||||
});
|
||||
|
||||
test('agreeing peers accept a function without human review (four, or five over an objection)', async () => {
|
||||
test('agreeing peers accept a function without human review (three ahead)', async () => {
|
||||
await activeUser('aaauthor');
|
||||
const author = ctx.accounts.byName('aaauthor');
|
||||
const peers = [];
|
||||
|
|
@ -448,17 +448,17 @@ test('agreeing peers accept a function without human review (four, or five over
|
|||
};
|
||||
|
||||
const clean = make(0x401900);
|
||||
for (let i = 0; i < 3; i++) await review(clean.attempt, peers[i], 'same');
|
||||
assert.equal(ctx.store.get('SELECT state FROM attempts WHERE id = ?', clean.attempt).state, 'peer_review', 'three are not enough');
|
||||
assert.equal((await review(clean.attempt, peers[3], 'same')).state, 'approved', 'four accept it');
|
||||
for (let i = 0; i < 2; i++) await review(clean.attempt, peers[i], 'same');
|
||||
assert.equal(ctx.store.get('SELECT state FROM attempts WHERE id = ?', clean.attempt).state, 'peer_review', 'two are not enough');
|
||||
assert.equal((await review(clean.attempt, peers[2], 'same')).state, 'approved', 'three to none accept it');
|
||||
assert.equal(ctx.store.get('SELECT state FROM tasks WHERE id = ?', clean.task).state, 'approved');
|
||||
|
||||
const contested = make(0x401940);
|
||||
await review(contested.attempt, peers[0], 'different');
|
||||
for (let i = 1; i <= 3; i++) await review(contested.attempt, peers[i], 'same');
|
||||
assert.equal(ctx.store.get('SELECT state FROM attempts WHERE id = ?', contested.attempt).state, 'peer_review');
|
||||
assert.equal((await review(contested.attempt, peers[4], 'same')).state, 'peer_review');
|
||||
assert.equal((await review(contested.attempt, peers[5], 'same')).state, 'approved', 'five "same" accept it over one objection (flagged for later at merge)');
|
||||
assert.equal((await review(contested.attempt, peers[4], 'same')).state, 'approved', 'four to one accept it (flagged for later at merge)');
|
||||
await review(contested.attempt, peers[5], 'same');
|
||||
|
||||
for (const x of [clean, contested]) {
|
||||
ctx.store.run(`UPDATE attempts SET state = 'rejected' WHERE id = ?`, x.attempt);
|
||||
|
|
@ -1226,7 +1226,7 @@ test('the live socket pushes new events, refuses other sites and ignores what cl
|
|||
}
|
||||
});
|
||||
|
||||
test('three "different" send a function back even when five said "same"', async () => {
|
||||
test('a close vote waits for more verdicts; the side that leads decides', async () => {
|
||||
await activeUser('odauthor');
|
||||
const author = ctx.accounts.byName('odauthor');
|
||||
const peers = [];
|
||||
|
|
@ -1245,7 +1245,10 @@ test('three "different" send a function back even when five said "same"', async
|
|||
void reviewers;
|
||||
await call('POST', '/api/peer-reviews/claim', { attemptId: attempt }, peers[7]);
|
||||
const r = await call('POST', `/api/peer-reviews/${attempt}/verdict`, { verdict: 'same', reason: 'the same calls and the same returns' }, peers[7]);
|
||||
assert.equal(r.body.state, 'needs_cleanup', 'five "same" do not outvote three "different"');
|
||||
assert.equal(r.body.state, 'in_review', 'five "same" against three "different": a lead of two, not yet');
|
||||
ctx.store.run(`INSERT INTO peer_reviews (attempt_id, reviewer_id, verdict, reason, created_at) VALUES (?, ?, 'same', 'checked every branch and call', ?)`, attempt, ctx.accounts.userByApiToken(await activeUser('odpeer9')).id, now());
|
||||
assert.deepEqual(ctx.peers.settleAll(), { approved: 1, needs_cleanup: 0 }, 'six to three: settled by the worker');
|
||||
assert.equal(ctx.store.get('SELECT state FROM attempts WHERE id = ?', attempt).state, 'approved');
|
||||
ctx.store.run(`UPDATE tasks SET state = 'done' WHERE id = ?`, task);
|
||||
});
|
||||
|
||||
|
|
@ -1396,7 +1399,7 @@ test('trust: careful reviewers count more, trusted authors skip peer review, one
|
|||
ctx.store.run(`UPDATE attempts SET state = 'superseded' WHERE user_id = ?`, author.id);
|
||||
});
|
||||
|
||||
test('four "same" accept a function nobody objected to; five once someone did', async () => {
|
||||
test('three "same" accept a function nobody objected to; one objection takes a fourth', async () => {
|
||||
const task = ctx.store.get('SELECT id FROM tasks LIMIT 1').id;
|
||||
const author = ctx.accounts.byName('trustauthor');
|
||||
const revs = [];
|
||||
|
|
@ -1407,12 +1410,13 @@ test('four "same" accept a function nobody objected to; five once someone did',
|
|||
const open = () => Number(ctx.store.run(`INSERT INTO attempts (task_id, user_id, base_commit, files_json, state, created_at) VALUES (?, ?, ?, '{}', 'peer_review', ?)`, task, author.id, ctx.repo.head(), now()).lastInsertRowid);
|
||||
const why = 'the branches, the call and the return value match the original';
|
||||
const clean = open();
|
||||
for (let i = 0; i < 3; i++) assert.equal(ctx.peers.submit(revs[i], clean, 'same', why, null).state, 'peer_review');
|
||||
assert.equal(ctx.peers.submit(revs[3], clean, 'same', why, null).state, 'approved');
|
||||
for (let i = 0; i < 2; i++) assert.equal(ctx.peers.submit(revs[i], clean, 'same', why, null).state, 'peer_review');
|
||||
assert.equal(ctx.peers.submit(revs[2], clean, 'same', why, null).state, 'approved');
|
||||
const disputed = open();
|
||||
ctx.peers.submit(revs[0], disputed, 'different', 'the original checks the flag before the loop', null);
|
||||
for (let i = 1; i < 4; i++) assert.equal(ctx.peers.submit(revs[i], disputed, 'same', why, null).state, 'peer_review');
|
||||
assert.equal(ctx.peers.submit(revs[4], disputed, 'same', why, null).state, 'peer_review', 'four "same" against one "different": not yet');
|
||||
for (let i = 1; i < 3; i++) assert.equal(ctx.peers.submit(revs[i], disputed, 'same', why, null).state, 'peer_review');
|
||||
assert.equal(ctx.peers.submit(revs[3], disputed, 'same', why, null).state, 'peer_review', 'three "same" against one "different": not yet');
|
||||
assert.equal(ctx.peers.submit(revs[4], disputed, 'same', why, null).state, 'approved', 'four to one');
|
||||
ctx.store.run(`UPDATE attempts SET state = 'superseded' WHERE id IN (?, ?)`, clean, disputed);
|
||||
});
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue