When my LLM scorer told me what I wanted to hear:
recalibrating a job-fit pipeline from 93% false confidence to deduction-based honesty

PROBLEM

I built an autonomous job-search agent: 9 boards scraped daily, LLM-based fit scoring, ATS-ready CV generation. The pipeline ran overnight and surfaced the top matches in a Google Sheet each morning.

One morning it surfaced a Senior Inference Engineer role at a frontier AI lab with a 93% fit score. I read the posting carefully. The role required 5+ years of systems programming experience, GPU kernel optimization, and prior work on production ML infrastructure at scale. My background is applied AI and full-stack. Honest fit: roughly 55%.

The scorer wasn't lying to me. It was doing exactly what I'd asked: summing up signals of alignment between my profile and the job. The problem was structural. An additive rubric — “find all the ways this matches” — is biased upward by design, because LLMs over-match on surface keywords. “AI experience” matched “AI company.” “Systems thinking” matched “systems programming.” Each partial match added points. No signal subtracted any.

This is an eval integrity failure: an evaluator optimizing for encouragement, not accuracy.

ARCHITECTURE

The original pipeline had a standard additive prompt: surface all matching signals, weight them, sum to a score. I redesigned it around a deduction model.

scraper (9 boards, nightly)
  ↓
heuristic pre-filter
  ↓ ← [absolute title filter added here — NEW]
     blocks BEFORE the LLM sees the posting:
     finance, sales, non-AI roles matched by company name
  ↓
LLM scorer
  BEFORE: additive prompt
  — "find all ways this profile matches the job"
  — each match adds +points
  — no penalties, no ceiling

  AFTER: deduction-based prompt
  — start at 100
  — −15 per missing hard requirement
  — −8 per partial match / adjacent skill
  — seniority gap formula: if level_delta > 1, −20 per level
  — target-role recalibration: bonus only for AI-specific titles
  ↓
Google Sheets log
  ↓
CV generator (only if score ≥ FIT_THRESHOLD)

The key insight: deduction forces the LLM to argue against the match, not for it. Starting from 100 and losing points changes the frame. The model has to find the gaps, not the overlaps.

The absolute title filter runs before the LLM sees the posting at all. This prevents the domain bonus (“works at an AI company”) from rescuing roles that are fundamentally wrong — financial analyst at OpenAI is still a financial analyst.

REGLAS DE SCORING (escala 0.0–10.0):

Hard blocks — asigna score <= 3.0 si aplica CUALQUIERA:
- Requiere security clearance o ciudadanía US
- Rol es ML research puro (RL, pretraining, interpretability)
- Requiere ML training infra distribuida (CUDA/TPU)
[...]
Score 8.5–9.5 si el rol es:
- Forward Deployed Engineer: embed con cliente + build LLM/MCP/agents
- Applied AI Engineer: foco builder + Claude API context + Python en producción
[...]
Ajustes:
+0.5  Dublin o Irlanda mencionado explícitamente
+0.5  Visa sponsorship ofrecida explícitamente
+0.3  MCP o "Model Context Protocol" mencionado
-2.5  Requiere 7+ años de experiencia
-1.5  Job en US sin sponsorship mencionado

REGLAS DE SCORING HARD (APLICAR ANTES QUE CUALQUIER OTRA COSA):

Asigna score <= 4 si el JD cumple CUALQUIERA de estas condiciones:
- Screening question "7+ years pre-sales" como filtro binario
- Rol es ML research (RL, pretraining, interpretability, alignment science)
- Requiere ML training infra distribuida (checkpoints, CUDA/TPU)
- Compensation >= $500k USD (indica seniority muy fuera de rango)
- Requiere security clearance o ciudadanía US
[...]
Asigna score 8.5–9.5 si:
- Rol es Forward Deployed Engineer (embed + build + MCP/agents)
- Rol es Applied AI Engineer con foco en builder credibility + Claude API

Asigna score 7.5–8.5 si:
- Rol es Applied AI Engineer con foco advisory (más advisor que builder)
- Rol es technical post-sales implementation con LLMs

Code, prompts, and the full comparison harness: github.com/AzahidGarcia/llm-judge-calibration ↗

EVALUATION

I built a manual ground-truth set: job postings hand-labeled with what I considered an honest fit percentage, based on a careful read of requirements vs. my actual skills. The labels were written before running either scorer on them.

I ran both scorers on the same set and compared the score distributions to my ground truth. Two-track validation to catch geo-specific false positives:

• Track A: Remote-friendly, Mexico-based or global
• Track B: Relocation-viable, Europe-primary

The two tracks surfaced a different failure mode: geo filtering was partially handled by the LLM (which sometimes assigned domain bonuses based on company reputation rather than actual fit), and partially by heuristics. Without the two-track split, Europe-specific senior roles at well-known labs would inflate the remote-track scores.

I also manually audited the 3 confirmed false positives from the previous run — roles that scored above threshold but were clearly wrong on close read — and verified that the new pipeline would have caught them at the title filter stage before LLM scoring.

MEASURED DELTA

The threshold values (57 / 45) were calibrated empirically against the ground-truth set. They are not magic numbers — they're the point where the precision/recall tradeoff felt right for my specific profile and search criteria.

AUTOPSY

The deeper lesson isn't about this pipeline. It's about what happens when you give an LLM an evaluation task without constraining the direction of error.

The pipeline's structure selects for high scores: they're the only output that triggers action, so every ambiguity in the rubric resolves upward. Nothing “learns” — the incentive gradient is baked into the system design.

Deduction fixes the direction of error but doesn't eliminate it. What's still broken:

• Seniority gap estimation is noisy on hybrid roles. A role labeled “Senior ML Engineer” might require 3 years or 8 years depending on the company. The deduction formula applies a fixed penalty per level gap, but the level definition is itself an LLM estimate. Noise compounds.
• The threshold doesn't generalize. FIT_THRESHOLD=57 was calibrated to my profile, my job criteria, and the specific time window. If my skills or target roles shift, the threshold needs recalibration. There's no automatic recalibration loop.
• Ground truth is subjective and thin. My labeled set — hand-labeled postings rated before running either scorer — was labeled by me: the same person whose judgment the pipeline is trying to replicate. A real eval would need independent labelers and a larger sample.

The honest summary: I fixed the worst failure mode and made the scorer directionally more accurate. I did not build a well-calibrated evaluator. I built a less-optimistic one.