Methodology

1. What Fydell reads

Fydell reads the code in repositories an engineer connects or submits. It does not run that code. A finding says what a snapshot of the code contains, with the file and lines it came from, and never on its own what the person can do.

A finding about a repository becomes a claim about a person only when contribution evidence, such as authored commits or reviewed changes, links them to those lines.

Missing evidence is not evidence of inability. Fydell only describes what it was shown.

2. Kinds of evidence

Every piece of evidence records where it came from: inspected code, provider event, submitted by the engineer, run by fydell, engineer's statement, reviewer's observation.

Each finding is also marked as observed in the code or inferred from the evidence. An observation points at code that is there; an inference is a reading of that code, and is worded as one.

Code an engineer submitted directly is still read by Fydell, but its history did not come from the code host, so it is recorded as submitted rather than inspected.

3. How findings are checked

Fydell keeps a gold set: short pieces of code, each with the practices someone might claim it shows, including near misses written to look like a practice without being one.

Two engineers label each item independently. Neither sees Fydell's output or the other's label until both have submitted.

When they disagree, a third engineer who labelled neither side settles it. Every week a rotating sample of items both engineers agreed on goes to a third engineer, because two people can agree and still be wrong.

Fydell's findings are scored only against decisions the engineers agreed on or that were settled.

4. Current measurement

The gold set has 26 items. 0 of them have been labelled by two engineers so far, and the release gate needs 200.

Until that threshold is reached Fydell publishes no accuracy figures. Findings are worded as observations, with a note that accuracy for that kind of finding has not been measured yet.

The items written to seed the set were labelled by the tool that wrote them. Those labels show how the harness works; they are marked unreviewed and never count toward any figure on this page.

5. Release gate

A change to the analyzer ships only when at least 200 gold items have two engineer labels, overall precision is at least 90%, and no kind of finding with 10 or more labelled findings is below 80%.

Until the label threshold is met the gate reports that there are too few labels. It does not report a pass.

6. How wording follows accuracy

A kind of finding that is correct at least 90% of the time on the gold set is stated plainly. Between 75% and 90% it is stated as likely. Below that it is labelled as low accuracy.

A kind of finding without enough labelled examples is described as an observation, with a note that its accuracy has not been measured yet.

7. Corrections

Engineers can add context to a finding or mark it as inaccurate. Staff review corrections every week, grouped by the kind of finding, and fix the analyzer or add gold items where a pattern shows up. A correction never removes evidence on its own; it is shown next to the finding.

8. What Fydell does not claim

Tests in a repository are read, not run. A test suite is evidence that tests exist, not that they pass.

Production behaviour, performance under load and how code was written are not observable from source.