Benchmarks · Research note

Better prompts?
We tested it.

Promising results in a small, transparent evaluation. With PromptingCraft.com, the same model wrote prompts that kept the user’s requirements more often and added fewer requirements nobody asked for.

Every number on this page is computed from public files at commit 7e45c24a of the skill repository. Runs: 2026-10-12.

Question
Does the skill keep coding-agent prompts faithful to what the user asked?
Sample
5 synthetic tasks × 2 runs × 2 conditions = 20 prompts
Generator
Codex CLI 0.162.1 · gpt-6.1-sol · reasoning xhigh
Grader
Claude Sonnet 5.5, one blinded pass
Activation
20 queries × 3 runs, counted from actual skill loads
Evidence
7e45c24a · ab-1 · trigger-1

Headline results

With PromptingCraft.com

8/10

prompts passed their contract

3 observed defects

Without PromptingCraft.com

3/10

prompts passed their contract

16 observed defects

Activation

60/60

valid runs behaved as expected

30/30 loaded when asked · 0/30 loaded when not

Scoring correction

Before one erroneous contract item was removed, the totals were 6/10 with 5 defects versus 3/10 with 18. The correction raised only the with-skill pass count, and it was decided after the grades were seen. Both versions

Sample limits

Ten prompts per condition, one generating model, one model grader, tasks written by the same team, no statistical test. This measures prompt fidelity, not better code or real-project success. All limits

Passing prompts

Corrected score as a bar; the original score as a red tick.

With PromptingCraft.com

8/10 · original 6/10

Without

3/10 · unchanged

Observed defects, task by task

One dot per defect. Hollow dots were removed by the scoring correction.

Defects per task and condition; hollow marks are defects excluded by the correction
TaskWith PromptingCraft.comWithout
Review a prompt without changing its scope0 + 2 excluded5 + 2 excluded
Keep a suitable prompt unchanged00
Leave an undecided repository open05
Delegate a CSV fix with four stated rules32
Drop Python 3.9 in two files only04
Total3 (original 5)16 (original 18)

A defect is a dropped or weakened requirement, an open choice decided by guessing, a lost literal, or a mandatory requirement added with no source in the request. Without the skill, the model most often added gates, procedures or assumed values. With it, the model lost the CSV task twice by adding a “fails before, passes after” check nobody asked for; without it, one of the two CSV runs also failed.

60 of 60: does it load only when it should?

20 realistic requests — 10 asking for a prompt, 10 near misses such as coding tasks or questions about Codex — ran three times each in Codex, in a fresh empty directory. A run counted as loaded only when the host actually read the installed SKILL.md; the model’s own claims were ignored.

  • Prompt request, skill loaded
  • Near miss, skill stayed out
  • Host failure before any answer, excluded and re-run

65 attempts were made in Codex. 5 failed on the host side (3 transport failures, 2 capacity responses) and are excluded, leaving 60 valid runs. Train 12/12, held-out validation 8/8. Every attempt.

In the Cursor CLI the result was different: with Auto (reported as Composer 2.5 Fast) the skill was discovered but loaded for 0/30 prompt requests on its own, and 2/2 when named. Cursor results are outside the snapshot used here.

Show the 20 queries and per-run outcomes as a table
Activation outcome per query
QuerySetExpectedValid runs loaded
ok so we settled on: keep the /v1/orders response shape, add cursor pagination with the existing helper, page size 50, no new deps. write this up as a task for codextrainload3/3 · 2 excluded
We figured out the flaky login test is a race between the session cookie and the redirect; the fix should await the cookie write and must not change the public auth API. Can you turn that into instructions I can paste into Claude Code?trainload3/3
here's my claude code prompt, it feels bloated, can you tighten it without changing what it asks for? Fix the null check in parseUser in src/users.ts. Be careful. Be very careful. Make sure to be careful. Don't change the exported types.trainload3/3
I need a handoff for a coding agent: move the nightly cleanup from node-cron to our existing queue worker, staging only, do not touch production config, keep the same schedule.trainload3/3
draft a codex task from this bug report: Safari users get a blank settings page since release 2.3, we suspect the optional chaining in settings.js, and the fix needs a test that runs on Safari 15trainload3/3
Is this good enough to hand to Codex or should I change it? Add a --dry-run flag to scripts/cleanup.sh that prints the files it would delete instead of deleting them. Keep all existing flags working.trainload3/3
pls make an execution-ready prompt for claude code: rename getUserById to findUserById across the api package and keep the old name as a deprecated alias for one releasevalidationload3/3
My manager wants me to delegate the CSV export fix to an AI coding agent. Write the instructions it should get: quote fields per RFC 4180, keep the column order id,name,status, add a regression test, no deployment.validationload3/3 · 1 excluded
review this agent prompt for scope creep before I send it to codex: 'Fix the typo in README.md and also redesign the landing page and switch the site to Tailwind.'validationload3/3 · 1 excluded
We decided to drop Python 3.9 support, only in setup.cfg and the CI matrix, nothing else. Prepare a Codex prompt for this.validationload3/3 · 1 excluded
What's the practical difference between Codex and Claude Code for a small team?trainstay out0/3
Write a Python function that quotes CSV fields correctly per RFC 4180.trainstay out0/3
Explain what prompt injection is and how coding agents should defend against it, in one short paragraph.trainstay out0/3
my React useEffect runs twice in dev mode, why? useEffect(() => { fetchData() }, [])trainstay out0/3
Write a system prompt for a customer-support chatbot for a small bakery.trainstay out0/3
Summarize in three bullets the trade-offs between cursor and offset pagination.trainstay out0/3
Translate this commit message into Spanish: 'fix: quote names containing commas in CSV export'validationstay out0/3
How do I install a local plugin in the Codex CLI?validationstay out0/3
Give me an image-generation prompt for a cyberpunk city at night in the rain.validationstay out0/3
Write unit tests for this function: def add(a, b): return a + bvalidationstay out0/3

Case explorer

Unedited final answers side by side, with the blinded grader’s verdict for every contract item. Highlighted text is what the grader quoted when it failed an item.

Review a prompt without changing its scope ab-review-scope

Request

Review and improve this Claude Code prompt without changing its scope: Fix the failing date-parsing test in utils/dates.test.ts. Also refactor the whole utils folder, upgrade all dependencies, and write a blog post about it. Be extremely careful. Be extremely careful. Do not change the public API of formatDate.

Contract

  1. Keeps the failing date-parsing test fix as the goal
  2. Keeps 'Do not change the public API of formatDate' as an unconditional invariant
  3. Does not silently mandate AND does not silently drop the refactor/dependency upgrade/blog post: flags them as scope questions or keeps them visibly optional
  4. Removes or flags the duplicated caution
  5. Adds no new requirements without a source in the original

Corrected: Item 3 conflicts with the user's explicit 'without changing its scope'; both conditions failed it. Adjusted results exclude that item and the matching 'mandates refactor, upgrade and blog post' defect.

Contract source

Run

Run 1

With PromptingCraft.comPASS

Originally graded FAIL; changed by the correction.

  • PASS: Keeps date-parsing test fix as goal
  • PASS: formatDate API unconditional invariant
  • FAIL, excluded by the correction: Refactor/upgrade/blog flagged or optionalGraded FAIL; this item was excluded by the scoring correction.
  • PASS: Duplicated caution removed or flagged
  • PASS: No new requirements

Material defects

  • Mandates refactor, upgrade and blog post rather than flagging them as scope questions (excluded by the correction)
The original specifies four tasks but repeats a vague caution. This rewrite preserves the scope and replaces that repetition with concrete verification. Upgrade versions and the blog’s destination remain unspecified.

```text
Complete all four tasks:

1. Fix the failing date-parsing test in utils/dates.test.ts by investigating and correcting its cause.
2. Refactor the whole utils folder.
3. Upgrade all dependencies.
4. Write a blog post about the changes.

Do not change the public API of formatDate.

Reproduce the date-parsing failure, then verify that the fix resolves it. Run the relevant existing tests and checks after the refactor and dependency upgrades. Base the blog post on the completed changes and observed results.
```

Verification guidance checked against [Claude Code’s official documentation](https://code.claude.com/docs/en/best-practices) via Context7 on October 12, 2026.

Source file · blinded label X19

WithoutFAIL
  • PASS: Keeps date-parsing test fix as goal
  • PASS: formatDate API unconditional invariant
  • FAIL, excluded by the correction: Refactor/upgrade/blog flagged or optional, not silently mandatedGraded FAIL; this item was excluded by the scoring correction.
  • PASS: Duplicated caution removed or flagged
  • FAIL: No new requirementsAdds 'separate, reviewable steps', full test suite and project checks, 'resolve compatibility issues', and a summary that includes the blog post's location.

Material defects

  • Adds a separate-reviewable-steps requirement
  • Adds a full-suite and all-checks gate
  • Adds a blog post location requirement
  • Mandates refactor, dependency upgrade and blog post (excluded by the correction)
The scope contains four tasks. Keep them explicit, and replace repeated caution with concrete verification steps.

```text
Complete all four tasks:

1. Diagnose and fix the failing date-parsing test in utils/dates.test.ts. Address the underlying cause; do not skip or weaken the test to make it pass.
2. Refactor the entire utils folder.
3. Upgrade all dependencies and resolve any resulting compatibility issues.
4. Write a blog post explaining the date-parsing fix, the utils refactor, and the dependency upgrades.

Hard constraint: Do not change the public API of formatDate.

Before editing, inspect the relevant code and reproduce the failing test. Make changes in separate, reviewable steps so failures can be traced to their cause.

Verify the date-parsing fix with the focused test, run the utils tests after the refactor, and run the full test suite and available project checks after the dependency upgrades.

Base the blog post on the completed work and actual verification results. Finish with a concise summary of changes, checks performed, any unresolved issues, and the blog post’s location.
```

Source file · blinded label X01

Run 2

With PromptingCraft.comPASS

Originally graded FAIL; changed by the correction.

  • PASS: Keeps date-parsing test fix as goal
  • PASS: formatDate API unconditional invariant
  • FAIL, excluded by the correction: Refactor/upgrade/blog flagged or optionalGraded FAIL; this item was excluded by the scoring correction.
  • PASS: Duplicated caution removed or flagged
  • PASS: No new requirements

Material defects

  • Mandates refactor, upgrade and blog post rather than flagging them as scope questions (excluded by the correction)
The prompt contains four distinct tasks. Repeated “Be extremely careful” adds little; concrete verification makes it actionable. Dependency version targets and blog-post details remain unspecified.

```text
Complete all four tasks:
1. Reproduce and fix the failing date-parsing test in utils/dates.test.ts.
2. Refactor the whole utils folder.
3. Upgrade all dependencies.
4. Write a blog post about the completed work, grounded in the actual changes and verified results.

Do not change the public API of formatDate.

Work carefully: run the relevant existing tests and checks to verify the fix and check for regressions from the refactor and dependency upgrades.

Unspecified details: dependency target versions and the blog post’s audience, length, and destination.
```

This preserves the scope and replaces vague emphasis with verification, consistent with [Claude Code’s prompting guidance](https://code.claude.com/docs/en/prompt-library), retrieved through Context7 on October 12, 2026.

Source file · blinded label X18

WithoutFAIL
  • PASS: Keeps date-parsing test fix as goal
  • PASS: formatDate API unconditional invariant
  • FAIL, excluded by the correction: Refactor/upgrade/blog flagged or optionalGraded FAIL; this item was excluded by the scoring correction.
  • PASS: Duplicated caution removed or flagged
  • FAIL: No new requirementsAdds consistent manifest and lockfile updates, following repo blog conventions, lint/type-check/build checks, and a final diff review.

Material defects

  • Adds lockfile consistency, build/lint/type-check and diff-review gates
  • Adds a 'preserve behavior' constraint on the refactor
  • Mandates refactor, upgrade and blog post (excluded by the correction)
The prompt has four distinct tasks, but “be extremely careful” provides no actionable guidance. This version preserves all four tasks and makes the constraints and verification explicit:

```text
Complete these tasks:
1. Fix the failing date-parsing test in utils/dates.test.ts.
2. Refactor the entire utils folder.
3. Upgrade all project dependencies.
4. Write a blog post explaining the changes.

Constraint:
Do not change the public API of formatDate.

Approach:
- Inspect the failing test and its implementation, identify the cause, and fix it without weakening the test.
- Refactor utils while preserving behavior except for the intended date-parsing correction.
- Update dependency manifests and lockfiles consistently, and make compatibility changes required by the upgrades.
- Follow existing repository conventions for the blog post. Describe only changes and results actually verified.

Verification:
- Run the date-parsing tests after the fix.
- Run the relevant tests and available lint, type-check, and build checks after the refactor and upgrades.
- Review the final diff for unintended changes and confirm that formatDate’s public API is unchanged.

Finish with a concise summary of changes, verification results, and any unresolved failures or blockers. Do not claim checks passed unless they were run successfully.
```

Source file · blinded label X09

Keep a suitable prompt unchanged ab-no-change

Request

Review this complete Codex task. If it is already suitable, keep it unchanged: Fix the CSV export so names containing commas are quoted per RFC 4180. Keep the columns and order id,name,status. Add a regression test for a name containing a comma. Do not add dependencies. Implementation is approved; do not deploy.

Contract

  1. Recognizes the task is already suitable and keeps it unchanged (or changes nothing material)
  2. Does not add new requirements, gates or mechanisms
  3. If any text is returned, id,name,status order, RFC 4180, regression test, no dependencies, implementation approved and no deploy all survive

Contract source

Run

Run 1

With PromptingCraft.comPASS
  • PASS: Recognizes task suitable, unchanged
  • PASS: No new requirements
  • PASS: All literals survive
The original prompt is already suitable. No material revision is necessary.

```text
Fix the CSV export so names containing commas are quoted per RFC 4180. Keep the columns and order id,name,status. Add a regression test for a name containing a comma. Do not add dependencies. Implementation is approved; do not deploy.
```

Source file · blinded label X04

WithoutPASS
  • PASS: Recognizes task suitable, unchanged
  • PASS: No new requirements
  • PASS: All literals survive
Suitable as written; unchanged:

> Fix the CSV export so names containing commas are quoted per RFC 4180. Keep the columns and order id,name,status. Add a regression test for a name containing a comma. Do not add dependencies. Implementation is approved; do not deploy.

Source file · blinded label X08

Run 2

With PromptingCraft.comPASS
  • PASS: Recognizes task suitable, unchanged
  • PASS: No new requirements
  • PASS: All literals survive
The original prompt is already suitable. No material revision is necessary.

```text
Fix the CSV export so names containing commas are quoted per RFC 4180. Keep the columns and order id,name,status. Add a regression test for a name containing a comma. Do not add dependencies. Implementation is approved; do not deploy.
```

Source file · blinded label X05

WithoutPASS
  • PASS: Recognizes task suitable, unchanged
  • PASS: No new requirements
  • PASS: All literals survive
Fix the CSV export so names containing commas are quoted per RFC 4180. Keep the columns and order id,name,status. Add a regression test for a name containing a comma. Do not add dependencies. Implementation is approved; do not deploy.

Source file · blinded label X03

Leave an undecided repository open ab-unresolved

Request

Prepare a Claude Code prompt. Investigate why the billing export drops rows. Two equally plausible repositories exist, api-east and api-west, and we have not chosen one. This is investigation only; no edits.

Contract

  1. Does not pick api-east or api-west; asks a deciding question or keeps both explicitly unresolved
  2. Preserves investigation only / no edits
  3. Does not invent a root cause, file names or approvals

Contract source

Run

Run 1

With PromptingCraft.comPASS
  • PASS: Does not pick a repo; asks a deciding question
  • PASS: Preserves investigation only / no edits
  • PASS: No invented root cause, file names or approvals
[Promptcraft](<redacted local plugin cache path>/skills/promptcraft/SKILL.md) requires “ask for the deciding identity instead,” so repository selection remains open.

```text
Investigate why the billing export drops rows. This is investigation only; no edits.

Two equally plausible repositories exist: api-east and api-west. We have not chosen one. First ask me which repository to investigate and wait for my answer. Do not choose one yourself.

Once I choose, trace the billing export and identify where rows are excluded or lost. Report the findings with supporting file and line references and any available reproduction evidence. Distinguish confirmed causes from hypotheses. If the cause remains unresolved, explain what evidence is missing.
```

[Claude Code guidance](https://code.claude.com/docs/en/how-claude-code-works) checked through Context7 on 2026-10-12.

Source file · blinded label X15

WithoutFAIL
  • PASS: Does not pick a repo; asks a deciding question
  • PASS: Preserves investigation only / no edits
  • PASS: No invented root cause, file names or approvals

Material defects

  • Adds new policies: no artifacts on disk, no exposing credentials, use sanitized identifiers, inline report format
  • Adds a 5-step mandatory procedure and an 'if both authorized' branch not in the request
```text
Investigate why the billing export drops rows. This is investigation only; no edits.

The target repository is unresolved: api-east and api-west are equally plausible, and neither has been selected. First ask me which repository to investigate, or whether to compare both. Do not silently choose a repository or begin repository-specific diagnosis before I answer.

Constraints:
- Do not modify files, configuration, data, dependencies, or Git state.
- Do not create plans, reports, temporary files, or other artifacts on disk.
- Use read-only inspection. Avoid running exports, tests, or other commands that write files or mutate external systems.
- Do not expose credentials or sensitive billing records.

After the repository scope is confirmed:

1. Establish the export entry point, output format, relevant version/environment, expected row count, and observed row count. Request missing evidence when it prevents a reliable diagnosis.

2. Trace rows from source selection through the final exported output. Inspect filters, joins, pagination, batching, deduplication, date/time boundaries, validation, serialization, and error handling. Distinguish intended exclusions from unintended losses.

3. Where evidence permits, trace a missing row and a comparable included row through the same path. Identify the first stage where the missing row disappears and the exact predicate or failure responsible. Use sanitized identifiers.

4. Check existing tests and available logs for evidence supporting or contradicting each hypothesis. Do not infer the cause from an error name or a suspicious code fragment alone.

5. If both repositories are authorized, keep findings separate and establish which implementation handles the affected export before attributing the defect.

Return an inline report containing:
- Repository and version investigated.
- Confirmed findings, with file:line references and available count/log evidence.
- The first proven row-loss stage and mechanism, if established.
- Remaining hypotheses, missing evidence, and the smallest next read-only diagnostic step.

Do not implement a fix. If the cause cannot be established, state that clearly.
```

Source file · blinded label X11

Run 2

With PromptingCraft.comPASS
  • PASS: Does not pick a repo; asks a deciding question
  • PASS: Preserves investigation only / no edits
  • PASS: No invented root cause, file names or approvals
The prompt leaves repository selection open and asks Claude Code to resolve it before investigating.

```text
Investigate why the billing export drops rows. This is investigation only; no edits.

Two equally plausible repositories exist: api-east and api-west. Neither has been selected. First ask me: “Which repository should I investigate: api-east or api-west?” Wait for my answer before repository-specific investigation. Do not choose one yourself or assume both are in scope.

Once I select the repository, use read-only inspection to trace rows from the source through the export and identify where and why they disappear. Examine relevant code and available examples, tests, or logs without changing files or implementing a fix.

Report the findings with file-and-line references and supporting evidence. Distinguish confirmed causes from hypotheses. If the available evidence cannot establish the cause, explain what remains unknown and what specific information would resolve it. Do not claim reproduction or verification that you have not performed.
```

Guidance checked October 12, 2026: [Claude Code best practices](https://code.claude.com/docs/en/best-practices).

Source file · blinded label X06

WithoutFAIL
  • PASS: Does not pick a repo; asks a deciding question
  • PASS: Preserves investigation only / no edits
  • PASS: No invented root cause, file names or approvals

Material defects

  • Adds a checkout path question and a branch/commit confirmation requirement
  • Forbids combining evidence from both repos, which pre-decides an open choice
  • Adds a long list of new prohibitions and mandatory steps (no artifacts, no export jobs, 6 mandatory steps) with no source in the request
```text
Investigate why the billing export drops rows. Investigation only; no edits or fixes.

Repository selection is unresolved: api-east and api-west are equally plausible. Your first response must ask: “Which repository should I investigate: api-east or api-west, and what is its checkout path?” Then wait for the answer. Do not inspect either repository, choose based on the current directory, or combine evidence from both.

After I select the repository:

1. Confirm the checkout path, branch, and commit. Read applicable repository instructions and preserve all existing work.
2. Use read-only inspection. Do not modify files, install dependencies, create artifacts, commit, deploy, trigger export jobs, or change database or external state. Run existing checks only when their lack of side effects is established.
3. Establish the expected export population and observed omissions from available evidence. If necessary, request a sanitized failing example with export parameters, expected rows, and actual output.
4. Trace the export from source query through pagination, filtering, transformation, serialization, and delivery. Find the earliest stage where an expected row disappears.
5. Examine relevant mechanisms such as joins, date/timezone boundaries, pagination termination, deduplication, null handling, swallowed errors, concurrency, and output truncation. Treat these as hypotheses until supported.
6. Compare a missing row with a retained row and, where available, a working sibling code path. Distinguish intentional exclusions from defects.

Return:
- Repository and revision investigated.
- Findings with file:line references and observed evidence.
- Confirmed cause, or ranked hypotheses with supporting and contradictory evidence.
- Remaining unknowns and the smallest read-only check needed to resolve them.

If the cause is unproven, say so. Stop after the investigation report; implementation requires a separate instruction.
```

Source file · blinded label X02

Delegate a CSV fix with four stated rules ab-csv-delegate

Request

My manager wants me to delegate the CSV export fix to an AI coding agent. Write the instructions it should get: quote fields per RFC 4180, keep the column order id,name,status, add a regression test, no deployment.

Contract

  1. Literal column order id,name,status preserved
  2. RFC 4180 quoting preserved
  3. Regression test required
  4. No deployment preserved
  5. No invented mandatory requirements (e.g., new files, frameworks, thresholds, extra gates) beyond the four stated items; extras, if any, are visibly optional

Contract source

Run

Run 1

With PromptingCraft.comFAIL
  • PASS: Literal column order id,name,status
  • PASS: RFC 4180 quoting
  • PASS: Regression test required
  • PASS: No deployment
  • FAIL: No invented mandatory requirements'Confirm it fails before the fix and passes afterward' is a new gate.

Material defects

  • Added fail-before/pass-after verification gate
```text
Fix the CSV export:

- Quote fields per RFC 4180: wrap fields containing commas, double quotes, or line breaks in double quotes, and escape embedded double quotes by doubling them.
- Keep the column order exactly id,name,status.
- Add a regression test covering quoting, escaping, and column order. Confirm it fails before the fix and passes afterward; run relevant existing tests.
- Do not deploy or trigger deployment through another action.
```

Quoting details follow [RFC 4180, section 2](https://www.rfc-editor.org/info/rfc4180/), checked October 12, 2026.

Source file · blinded label X13

WithoutPASS
  • PASS: Literal column order id,name,status
  • PASS: RFC 4180 quoting
  • PASS: Regression test required
  • PASS: No deployment
  • PASS: No invented mandatory requirements
Fix the CSV export with these requirements:

- Follow RFC 4180 quoting: enclose fields containing commas, double quotes, CR, or LF in double quotes. Escape embedded double quotes by doubling them.
- Keep the column order exactly `id,name,status`.
- Add a regression test covering commas, embedded quotes, and line breaks, and assert the column order.
- Keep the change focused, run the relevant tests, and report the changes and results.
- Do not deploy.

Source file · blinded label X12

Run 2

With PromptingCraft.comFAIL
  • PASS: Literal column order id,name,status
  • PASS: RFC 4180 quoting
  • PASS: Regression test required
  • PASS: No deployment
  • FAIL: No invented mandatory requirementsAdds 'Confirm it fails before the fix and passes afterward' and a mandated multiline test scope.

Material defects

  • Added fail-before/pass-after gate
  • Mandated multiline and embedded-quote test coverage
```text
Fix the CSV export with these requirements:

- Apply RFC 4180 quoting: enclose fields containing commas, double quotes, or line breaks in double quotes. Escape embedded double quotes by doubling them.
- Keep the column order exactly id,name,status.
- Add a regression test covering commas, embedded double quotes, and multiline values. Assert correct quoting and the exact column order. Confirm it fails before the fix and passes afterward, then run the relevant existing tests.
- No deployment. Do not take actions that trigger an automatic deployment.
```

Quoting rules verified against [RFC 4180, Section 2](https://www.rfc-editor.org/rfc/rfc4180#section-2).

Source file · blinded label X20

WithoutFAIL
  • PASS: Literal column order id,name,status
  • PASS: RFC 4180 quoting
  • PASS: Regression test required
  • PASS: No deployment
  • FAIL: No invented mandatory requirementsMandates 'use CRLF record separators' and a test covering embedded quotes and multiline values.

Material defects

  • Invented mandatory CRLF record separator change
  • Mandated test scope of multiline and embedded-quote cases
Fix the CSV export with this scope:

- Follow RFC 4180: enclose fields containing commas, double quotes, or line breaks in double quotes; escape embedded double quotes by doubling them; use CRLF record separators.
- Preserve the header and column order exactly: `id,name,status`.
- Add a regression test covering commas, embedded quotes, and multiline values. Assert the exact output and column order.
- Run the relevant tests and report the changes and results.
- Keep the fix focused. Do not deploy.

Source file · blinded label X10

Drop Python 3.9 in two files only ab-py39

Request

We decided to drop Python 3.9 support, only in setup.cfg and the CI matrix, nothing else. Prepare a Codex prompt for this.

Contract

  1. Scope limited to setup.cfg and the CI matrix
  2. 'nothing else' preserved (no mandated edits to README, tox, docs, code or other files)
  3. Does not invent a new minimum version policy beyond what is implied (e.g., an upper bound or specific versions) as a requirement

Contract source

Run

Run 1

With PromptingCraft.comPASS
  • PASS: Scope limited to setup.cfg and CI matrix
  • PASS: 'nothing else' preserved
  • PASS: No invented version policy
The original prompt is already suitable. No material revision is necessary.

```text
We decided to drop Python 3.9 support, only in setup.cfg and the CI matrix, nothing else.
```

Checked against [official OpenAI prompting guidance](https://learn.chatgpt.com/guides/best-practices#strong-first-use-context-and-prompts) on October 12, 2026.

Source file · blinded label X07

WithoutFAIL
  • PASS: Scope limited to setup.cfg and CI matrix
  • PASS: 'nothing else' preserved
  • FAIL: No invented version policy'raise the minimum supported Python version to 3.10' is a specific version mandated.

Material defects

  • Invented 3.10 minimum
  • Added classifier removal and verification requirements
```text
Drop Python 3.9 support with the smallest possible diff.

Scope:
- setup.cfg: remove the Python 3.9 classifier and raise the minimum supported Python version to 3.10, preserving any other version constraints.
- CI configuration: remove Python 3.9 from the test matrix, preserving all remaining versions and job settings.

Change nothing else. Do not modify application code, dependencies, tests, documentation, changelogs, or unrelated configuration. Do not refactor or reformat surrounding content.

Verify the configuration changes and confirm the diff contains only these edits. Summarize the changed files and validation performed.
```

Source file · blinded label X17

Run 2

With PromptingCraft.comPASS
  • PASS: Scope limited to setup.cfg and CI matrix
  • PASS: 'nothing else' preserved
  • PASS: No invented version policy
The original prompt is already suitable. No material revision is necessary.

```text
We decided to drop Python 3.9 support, only in setup.cfg and the CI matrix, nothing else.
```

Source file · blinded label X16

WithoutFAIL
  • PASS: Scope limited to setup.cfg and CI matrix
  • PASS: 'nothing else' preserved
  • FAIL: No invented version policyMandates 'raise python_requires from >=3.9 to >=3.10', assuming the current value and target.

Material defects

  • Invented python_requires >=3.10 and assumed an existing >=3.9 value
  • Added a report-conflict and diff-review procedure
```text
Drop Python 3.9 support. Limit changes strictly to setup.cfg and the CI Python-version matrix.

- In setup.cfg, raise python_requires from >=3.9 to >=3.10 and remove the Python 3.9 classifier, if present.
- Remove Python 3.9 from the CI matrix. Preserve every other version, matrix dimension, and workflow setting.
- Do not change source code, tests, documentation, dependencies, lockfiles, or unrelated formatting.

Inspect the existing configuration before editing. If it conflicts with these assumptions, report the conflict rather than widening the scope.

Review the final diff to confirm it contains only these changes, then summarize the edited files and validation performed.
```

Source file · blinded label X14

The scoring correction

One contract item for “Review a prompt without changing its scope” required the prompt to flag the refactor, dependency upgrade and blog post as scope questions. That contradicted the user’s own words, “without changing its scope”, and it failed every output in both conditions. After inspecting the grades we excluded that item and the matching defect.

Because the decision came after the results, it is a source of bias. Both versions are published; read either.

Item 3 conflicts with the user's explicit 'without changing its scope'; both conditions failed it. Adjusted results exclude that item and the matching 'mandates refactor, upgrade and blog post' defect.
Original and corrected totals
ScoresWithWithout
Original: pass · defects6/10 · 53/10 · 18
Corrected: pass · defects8/10 · 33/10 · 16

Method and limits

How the evaluation worked
  • Generator: Codex CLI 0.162.1 with the model gpt-6.1-sol at reasoning effort xhigh, one operator, a read-only sandbox and a fresh empty directory for every run.
  • Conditions: the installed plugin enabled or disabled through Codex configuration. The run log confirms the skill was read in 10/10 enabled runs and 0/10 disabled runs.
  • Tasks: five synthetic single-turn requests for a coding-agent prompt or a prompt review, each with a written contract of what a faithful answer must keep.
  • Blind review: a separate model (Claude Sonnet 5.5), run once as a fresh agent, read the 20 answers under shuffled labels with each request and contract. It never saw which condition produced an answer, graded every contract item with a quoted reason and listed material defects. The key and the grader’s full answer are published.
  • Activation: the agentskills.io method for testing skill descriptions — fixed train and held-out validation sets, three runs per query and a 0.5 trigger-rate threshold.
Limits and known sources of bias
  • Small sample: five tasks, two runs each per condition. The difference is not statistically tested and could change with other tasks.
  • One generating model and one host. Other models, Claude Code, Cursor or ChatGPT may behave differently; nothing here was measured in ChatGPT.
  • Model judgment: one grading pass by one model, with no human double-scoring. The grader is from Anthropic, and PromptingCraft.com draws on Anthropic and OpenAI guidance.
  • Same author: the tasks, contracts, skill and correction were all written by the PromptingCraft.com team, and the correction was made after seeing results.
  • Prompt fidelity only: the evaluation checks whether a prompt keeps the user’s requirements. It does not show that agents given these prompts produce better code or succeed more often in real projects.
  • Synthetic tasks without repositories, so file-specific behavior was not exercised.

Reproduce it

The runners, cases, contracts, outputs, grader prompt and grades are in the public repository. A rerun uses your own Codex account and will not match token for token; model outputs vary between runs.

Raw event traces are not published because they contain local file paths. The final answers, run log and grades are.

Rerun at the pinned commit
git clone https://gitlab.com/revanalex/promptingcraft-skill.git
cd promptingcraft-skill
git checkout 7e45c24a657eae84996271dd2cfd83c1a81fee45
# needs a logged-in Codex CLI and this commit's plugin from the local marketplace (former id: promptcraft@dondego-local)
python3 scripts/evaluate_ab.py evals/results/ab-1/cases.json /tmp/ab-rerun 2
python3 scripts/evaluate_triggers.py evals/trigger/queries.json /tmp/trigger-rerun 3

Public evidence at commit 7e45c24a

PromptingCraft.com follows official guidance published by AI developers, but it is developed independently by the dondego.com team and is not affiliated with or endorsed by OpenAI, Anthropic or Cursor.