Do AI Cover Letter Generators Invent Facts? We Tested Ours
Last updated September 6, 2026 · By Andrew Johnson · OneTwo Resume editorial team
The short answer
We ran 5 fictional candidate profiles — each with a deliberately closed set of true facts — through our own production cover-letter generator, twice each, and checked every claim in all 10 letters against the inputs. Result: 0 of 10 letters invented a number, credential, or job title. But 4 of 10 fabricated a candidate activity ('I am actively exploring TypeScript resources to bridge this gap' — the candidate never said that), and 10 of 10 asserted qualities about the employer that appear nowhere in the job posting. All raw inputs and outputs are published below. We shipped a prompt fix in response, and this page describes the pre-fix behavior.
Everyone suspects AI writing tools embellish. Almost nobody measures their own. This is a small, reproducible test of our own production generator — the same two-stage pipeline, prompts, and model that the live product used on September 6, 2026 — designed so that any fabrication is detectable by construction.
Method
- 5 fictional candidate profiles (retail associate, junior developer, registered nurse, marketing coordinator, warehouse worker), each with a deliberately closed fact inventory: every number, skill, and activity the candidate legitimately has is in the resume text. Anything else in an output is unsupported by construction.
- 1 realistic job posting each, including deliberate stretch (the warehouse associate applies to a supervisor role; the coordinator applies to a manager role) — stretch is where embellishment pressure is highest.
- 2 generations per profile (the letter stage runs at temperature 0.7, so outputs vary) = 10 letters.
- Rubric: every factual assertion classified as supported (stated in the resume), reframed (honest paraphrase), invented candidate fact (a number, credential, title, or activity not in the input), invented employer attribute (a claim about the company absent from the posting), or scope inflation (execution upgraded to ownership). Annotation was done by the author; the raw data is published so anyone can re-annotate.
Results
| Failure mode | Letters affected | Example (verbatim) |
|---|---|---|
| Invented numbers, credentials, or job titles | 0 / 10 | — (every number in every letter traces to the resume) |
| Invented candidate activities | 4 / 10 | "I am actively exploring TypeScript resources and tutorials to bridge this gap"; "I have proactively sought opportunities to lead small-scale projects" |
| Invented employer attributes | 10 / 10 | "St. Anne's reputation for excellence in patient care"; "Delta Logistics' commitment to excellence and innovation" — neither appears in the postings |
| Scope inflation | 2 / 10 | "developed and implemented social media strategies" for a resume that says "write and schedule 3 social posts per week" |
| Understated experience (conservative errors) | 3 / 10 | "over two years of retail experience" for a candidate with six |
What the pattern means
The generator's failure mode is not the one people fear. It did not inflate metrics or invent credentials — the guardrail language in the production prompt ("be honest about gaps") appears to hold for hard facts. Instead, fabrication concentrated in exactly two places:
- The gap-bridging move. When the candidate lacks a required skill, the model wants to resolve the tension — so it invents a redemption arc: currently learning, actively seeking, already exploring. It happened in 4 of 10 letters, always attached to a real gap. This is the most dangerous class, because it reads as concrete and checkable in an interview.
- Company flattery. Every letter attributed values, reputation, or mission to the employer that the posting never stated. Recruiters read hundreds of letters; unearned specificity about "your commitment to innovation" is exactly what makes AI letters smell like AI letters.
What we changed because of this
Same day as this test, we added two rules to the production prompt: the generator must not claim the candidate is currently learning, pursuing, or exploring anything unless the resume says so (gaps are acknowledged with willingness, not invented activity), and it must not assert employer qualities that aren't in the provided posting. This page documents the pre-fix behavior; we'll rerun the same protocol on the fixed prompt and publish the comparison.
Limitations
Ten letters from one vendor is a small sample annotated by one person — the vendor. We mitigated that the only honest way available: the complete raw data (all five profiles, postings, intermediate analyses, and all ten letters, plus per-run files like this one) is published verbatim, and the runner script lives in our repo, so the test is reproducible and re-annotatable. We have not yet tested competitors' generators under the same protocol — doing that honestly requires accounts and their terms' blessing, and we'd rather publish a narrow true result than a broad padded one. If you run this protocol on any tool, we'll link your results here.
The practical takeaway for job seekers
Whatever tool writes your letter — ours, a competitor's, or ChatGPT — proofread for these three specific patterns before sending: delete any "currently learning / actively seeking" sentence that isn't literally true, verify or cut every claim about the company, and downgrade any verb that upgrades your role. The numbers, in our test at least, took care of themselves.
Frequently Asked Questions
Do AI cover letter generators make things up?
In our test of our own production generator (10 letters, 5 fictional candidates with closed fact inventories): it never invented a number or a credential, but it fabricated candidate activities — 'I am actively exploring TypeScript resources', 'I have proactively sought leadership opportunities' — in 4 of 10 letters, and asserted unverifiable employer qualities ('your commitment to innovation') in 10 of 10. Every AI-generated letter needs a factual proofread before sending.
What should I check before sending an AI-written cover letter?
Three things, based on the failure modes we actually measured: (1) any sentence claiming you are 'currently learning', 'actively seeking', or 'exploring' something — delete it unless it's true; (2) any claim about the company's values, reputation, or mission — verify it against their actual site, or cut it; (3) any scope upgrade, like 'developed strategies' when you executed tasks. Numbers were reliably preserved in our test, but check them anyway.
Why publish results showing your own product's flaws?
Because unverifiable marketing is the disease this industry has, and the only credible alternative is measurement you can check. All ten inputs and outputs are published verbatim, the rubric is stated, and the finding led to a concrete product fix. We'd rather be the tool that documents its failure modes than the tool that claims not to have any.