Is AI-Scored Code Review Fair to Candidates?

When you put the words "AI" and "hiring" in the same sentence, people have strong reactions, and they should. Automated decision-making in hiring carries real risks: encoded bias, lack of transparency, and the potential to scale unfairness faster than any human process could.

So let us address the question directly: is AI-scored code review fair to candidates?

The Fairness Question

The concern is straightforward. If an AI model evaluates a candidate's code, it might penalise patterns associated with certain educational backgrounds, coding bootcamps, or regional coding conventions. It might favour the idioms it saw most during training. It might encode the biases present in its training data in ways that are invisible and unauditable.

These are not hypothetical risks. They have occurred in other domains. Resume screening tools have been shown to discriminate on the basis of name, gender, and educational institution. The question is whether code evaluation carries the same risks, and if so, what can be done about them.

The question is not whether AI can be biased. Of course it can. The question is whether it is more or less biased than the process it replaces.

Where Bias Actually Lives

To evaluate fairness, we need to compare AI scoring not against a theoretical ideal, but against the actual process it replaces: manual code review by one or two engineers.

Here is what we know about manual code review in hiring:

Reviewers disagree with each other. When multiple engineers independently evaluate the same take-home test, they reach different pass/fail conclusions roughly a third of the time. This is not a controversial finding. It has been documented across multiple studies of code review in both hiring and production contexts.

Criteria are implicit. Most manual reviews operate against unstated criteria. The reviewer looks at the code, forms an impression, and makes a judgment. What weight did they give to test coverage versus code organisation versus naming conventions? They often cannot tell you themselves.

Anchoring effects are real. The order in which submissions are reviewed affects scores. The first strong submission raises the bar. A mediocre submission following a poor one looks better than it is. Reviewers are not aware this is happening.

When the criteria are written down, they can be examined. When they live in a reviewer's head, they cannot.

None of this means AI scoring is automatically fair. It means the baseline we are comparing against is not fair either, and it is worth understanding why.

How Brief-Driven Scoring Works

The key architectural decision that shapes fairness in automated code review is what the model is scoring against.

A naive approach would be to ask a language model: "Is this code good?" That approach inherits every bias in the model's training data about what "good code" looks like. It would likely favour patterns common in open-source projects by well-known developers, penalise unfamiliar conventions, and produce opaque judgments.

Brief-driven scoring takes a different approach:

1. The hiring team writes a brief that specifies exactly what they are evaluating: which technologies should be used, what architectural patterns matter, what the quality bar looks like for this specific role.
2. The model evaluates the submission against the brief, not against a general sense of code quality. If the brief says "we value pragmatic solutions over abstraction," then pragmatic code scores well.
3. Every evaluation produces a reasoning trace that explains which criteria were met, which were not, and why.

This matters for fairness because it makes the evaluation criteria explicit and auditable. If the brief contains biased criteria, you can see that and fix it. If the model misapplies a criterion, you can see the reasoning and correct it.

What Candidates Think

Fairness is not only about the process. It is also about how candidates experience it.

We have heard two consistent themes from candidates who have been through both manual and automated review processes:

Speed matters. Candidates who submit a take-home test and hear nothing for two weeks have a terrible experience, regardless of the outcome. Automated review delivers results within hours. For candidates who are evaluating multiple offers, this speed is not a nice-to-have. It is often the difference between accepting your offer and accepting someone else's.

Structured feedback matters more. Most manual processes end with a form email: "Thank you for your time, but we have decided to move forward with other candidates." No feedback, no explanation, no learning opportunity. Automated review can provide specific, structured feedback about what the submission did well and where it fell short.

This does not settle the fairness question, but it reframes it. A process that candidates experience as more transparent and more respectful of their time has a meaningful fairness advantage, even before we look at scoring consistency.

The Auditability Advantage

Perhaps the strongest argument for well-designed automated review is auditability.

Every automated evaluation produces a complete record: the brief it scored against, the code it analysed, the scores it assigned, and the reasoning behind each score. This record can be:

• Reviewed by the hiring team to verify the evaluation makes sense
• Compared across candidates to check for consistency
• Analysed in aggregate to detect patterns that might indicate bias
• Shared with candidates as structured feedback

Manual review produces almost none of this by default. A senior engineer reads code, forms a judgment, and communicates a thumbs up or down to the hiring manager. If a candidate asks why they were rejected, there is rarely a detailed answer available.

Fairness in hiring is not a feature you ship. It is a property you continuously measure and improve.

Auditability makes continuous improvement possible. If you discover that your brief inadvertently penalises a certain approach, you can update the brief. If you notice the model consistently undervalues a valid architectural pattern, you can adjust the prompt. The feedback loop exists because the data exists.

Limitations and Open Questions

We would undermine our own argument if we pretended AI scoring has no limitations. Here are the ones we think about most:

We are actively studying these limitations. Our approach is to treat fairness as a continuous measurement problem rather than a binary feature. We track scoring distributions, compare automated evaluations against expert panels, and publish what we find.

This article's hypothesis, stated in the frontmatter, is that brief-driven AI scoring produces more consistent and less biased evaluations than manual review. We believe the structural arguments support this, but we are committed to testing it rigorously and updating our position based on evidence.