With PromptingCraft.com
8/10
prompts passed their contract
3 observed defects
Benchmarks · Research note
Promising results in a small, transparent evaluation. With PromptingCraft.com, the same model wrote prompts that kept the user’s requirements more often and added fewer requirements nobody asked for.
Every number on this page is computed from public files at commit 7e45c24a of the skill repository. Runs: 2026-10-12.
With PromptingCraft.com
8/10
prompts passed their contract
3 observed defects
Without PromptingCraft.com
3/10
prompts passed their contract
16 observed defects
Activation
60/60
valid runs behaved as expected
30/30 loaded when asked · 0/30 loaded when not
Scoring correction
Before one erroneous contract item was removed, the totals were 6/10 with 5 defects versus 3/10 with 18. The correction raised only the with-skill pass count, and it was decided after the grades were seen. Both versions
Sample limits
Ten prompts per condition, one generating model, one model grader, tasks written by the same team, no statistical test. This measures prompt fidelity, not better code or real-project success. All limits
Corrected score as a bar; the original score as a red tick.
One dot per defect. Hollow dots were removed by the scoring correction.
| Task | With PromptingCraft.com | Without |
|---|---|---|
| Review a prompt without changing its scope | 0 + 2 excluded | 5 + 2 excluded |
| Keep a suitable prompt unchanged | 0 | 0 |
| Leave an undecided repository open | 0 | 5 |
| Delegate a CSV fix with four stated rules | 3 | 2 |
| Drop Python 3.9 in two files only | 0 | 4 |
| Total | 3 (original 5) | 16 (original 18) |
A defect is a dropped or weakened requirement, an open choice decided by guessing, a lost literal, or a mandatory requirement added with no source in the request. Without the skill, the model most often added gates, procedures or assumed values. With it, the model lost the CSV task twice by adding a “fails before, passes after” check nobody asked for; without it, one of the two CSV runs also failed.
20 realistic requests — 10 asking for a prompt, 10 near misses such as coding tasks or questions about Codex — ran three times each in Codex, in a fresh empty directory. A run counted as loaded only when the host actually read the installed SKILL.md; the model’s own claims were ignored.
65 attempts were made in Codex. 5 failed on the host side (3 transport failures, 2 capacity responses) and are excluded, leaving 60 valid runs. Train 12/12, held-out validation 8/8. Every attempt.
In the Cursor CLI the result was different: with Auto (reported as Composer 2.5 Fast) the skill was discovered but loaded for 0/30 prompt requests on its own, and 2/2 when named. Cursor results are outside the snapshot used here.
Should load
Should not load
| Query | Set | Expected | Valid runs loaded |
|---|---|---|---|
| ok so we settled on: keep the /v1/orders response shape, add cursor pagination with the existing helper, page size 50, no new deps. write this up as a task for codex | train | load | 3/3 · 2 excluded |
| We figured out the flaky login test is a race between the session cookie and the redirect; the fix should await the cookie write and must not change the public auth API. Can you turn that into instructions I can paste into Claude Code? | train | load | 3/3 |
| here's my claude code prompt, it feels bloated, can you tighten it without changing what it asks for? Fix the null check in parseUser in src/users.ts. Be careful. Be very careful. Make sure to be careful. Don't change the exported types. | train | load | 3/3 |
| I need a handoff for a coding agent: move the nightly cleanup from node-cron to our existing queue worker, staging only, do not touch production config, keep the same schedule. | train | load | 3/3 |
| draft a codex task from this bug report: Safari users get a blank settings page since release 2.3, we suspect the optional chaining in settings.js, and the fix needs a test that runs on Safari 15 | train | load | 3/3 |
| Is this good enough to hand to Codex or should I change it? Add a --dry-run flag to scripts/cleanup.sh that prints the files it would delete instead of deleting them. Keep all existing flags working. | train | load | 3/3 |
| pls make an execution-ready prompt for claude code: rename getUserById to findUserById across the api package and keep the old name as a deprecated alias for one release | validation | load | 3/3 |
| My manager wants me to delegate the CSV export fix to an AI coding agent. Write the instructions it should get: quote fields per RFC 4180, keep the column order id,name,status, add a regression test, no deployment. | validation | load | 3/3 · 1 excluded |
| review this agent prompt for scope creep before I send it to codex: 'Fix the typo in README.md and also redesign the landing page and switch the site to Tailwind.' | validation | load | 3/3 · 1 excluded |
| We decided to drop Python 3.9 support, only in setup.cfg and the CI matrix, nothing else. Prepare a Codex prompt for this. | validation | load | 3/3 · 1 excluded |
| What's the practical difference between Codex and Claude Code for a small team? | train | stay out | 0/3 |
| Write a Python function that quotes CSV fields correctly per RFC 4180. | train | stay out | 0/3 |
| Explain what prompt injection is and how coding agents should defend against it, in one short paragraph. | train | stay out | 0/3 |
| my React useEffect runs twice in dev mode, why? useEffect(() => { fetchData() }, []) | train | stay out | 0/3 |
| Write a system prompt for a customer-support chatbot for a small bakery. | train | stay out | 0/3 |
| Summarize in three bullets the trade-offs between cursor and offset pagination. | train | stay out | 0/3 |
| Translate this commit message into Spanish: 'fix: quote names containing commas in CSV export' | validation | stay out | 0/3 |
| How do I install a local plugin in the Codex CLI? | validation | stay out | 0/3 |
| Give me an image-generation prompt for a cyberpunk city at night in the rain. | validation | stay out | 0/3 |
| Write unit tests for this function: def add(a, b): return a + b | validation | stay out | 0/3 |
Unedited final answers side by side, with the blinded grader’s verdict for every contract item. Highlighted text is what the grader quoted when it failed an item.
Request
Review and improve this Claude Code prompt without changing its scope: Fix the failing date-parsing test in utils/dates.test.ts. Also refactor the whole utils folder, upgrade all dependencies, and write a blog post about it. Be extremely careful. Be extremely careful. Do not change the public API of formatDate.
Contract
Corrected: Item 3 conflicts with the user's explicit 'without changing its scope'; both conditions failed it. Adjusted results exclude that item and the matching 'mandates refactor, upgrade and blog post' defect.
Run 1
Originally graded FAIL; changed by the correction.
Material defects
The original specifies four tasks but repeats a vague caution. This rewrite preserves the scope and replaces that repetition with concrete verification. Upgrade versions and the blog’s destination remain unspecified. ```text Complete all four tasks: 1. Fix the failing date-parsing test in utils/dates.test.ts by investigating and correcting its cause. 2. Refactor the whole utils folder. 3. Upgrade all dependencies. 4. Write a blog post about the changes. Do not change the public API of formatDate. Reproduce the date-parsing failure, then verify that the fix resolves it. Run the relevant existing tests and checks after the refactor and dependency upgrades. Base the blog post on the completed changes and observed results. ``` Verification guidance checked against [Claude Code’s official documentation](https://code.claude.com/docs/en/best-practices) via Context7 on October 12, 2026.
Source file · blinded label X19
Material defects
The scope contains four tasks. Keep them explicit, and replace repeated caution with concrete verification steps. ```text Complete all four tasks: 1. Diagnose and fix the failing date-parsing test in utils/dates.test.ts. Address the underlying cause; do not skip or weaken the test to make it pass. 2. Refactor the entire utils folder. 3. Upgrade all dependencies and resolve any resulting compatibility issues. 4. Write a blog post explaining the date-parsing fix, the utils refactor, and the dependency upgrades. Hard constraint: Do not change the public API of formatDate. Before editing, inspect the relevant code and reproduce the failing test. Make changes in separate, reviewable steps so failures can be traced to their cause. Verify the date-parsing fix with the focused test, run the utils tests after the refactor, and run the full test suite and available project checks after the dependency upgrades. Base the blog post on the completed work and actual verification results. Finish with a concise summary of changes, checks performed, any unresolved issues, and the blog post’s location. ```
Source file · blinded label X01
Run 2
Originally graded FAIL; changed by the correction.
Material defects
The prompt contains four distinct tasks. Repeated “Be extremely careful” adds little; concrete verification makes it actionable. Dependency version targets and blog-post details remain unspecified. ```text Complete all four tasks: 1. Reproduce and fix the failing date-parsing test in utils/dates.test.ts. 2. Refactor the whole utils folder. 3. Upgrade all dependencies. 4. Write a blog post about the completed work, grounded in the actual changes and verified results. Do not change the public API of formatDate. Work carefully: run the relevant existing tests and checks to verify the fix and check for regressions from the refactor and dependency upgrades. Unspecified details: dependency target versions and the blog post’s audience, length, and destination. ``` This preserves the scope and replaces vague emphasis with verification, consistent with [Claude Code’s prompting guidance](https://code.claude.com/docs/en/prompt-library), retrieved through Context7 on October 12, 2026.
Source file · blinded label X18
Material defects
The prompt has four distinct tasks, but “be extremely careful” provides no actionable guidance. This version preserves all four tasks and makes the constraints and verification explicit: ```text Complete these tasks: 1. Fix the failing date-parsing test in utils/dates.test.ts. 2. Refactor the entire utils folder. 3. Upgrade all project dependencies. 4. Write a blog post explaining the changes. Constraint: Do not change the public API of formatDate. Approach: - Inspect the failing test and its implementation, identify the cause, and fix it without weakening the test. - Refactor utils while preserving behavior except for the intended date-parsing correction. - Update dependency manifests and lockfiles consistently, and make compatibility changes required by the upgrades. - Follow existing repository conventions for the blog post. Describe only changes and results actually verified. Verification: - Run the date-parsing tests after the fix. - Run the relevant tests and available lint, type-check, and build checks after the refactor and upgrades. - Review the final diff for unintended changes and confirm that formatDate’s public API is unchanged. Finish with a concise summary of changes, verification results, and any unresolved failures or blockers. Do not claim checks passed unless they were run successfully. ```
Source file · blinded label X09
Request
Review this complete Codex task. If it is already suitable, keep it unchanged: Fix the CSV export so names containing commas are quoted per RFC 4180. Keep the columns and order id,name,status. Add a regression test for a name containing a comma. Do not add dependencies. Implementation is approved; do not deploy.
Contract
Run 1
The original prompt is already suitable. No material revision is necessary. ```text Fix the CSV export so names containing commas are quoted per RFC 4180. Keep the columns and order id,name,status. Add a regression test for a name containing a comma. Do not add dependencies. Implementation is approved; do not deploy. ```
Source file · blinded label X04
Suitable as written; unchanged: > Fix the CSV export so names containing commas are quoted per RFC 4180. Keep the columns and order id,name,status. Add a regression test for a name containing a comma. Do not add dependencies. Implementation is approved; do not deploy.
Source file · blinded label X08
Run 2
The original prompt is already suitable. No material revision is necessary. ```text Fix the CSV export so names containing commas are quoted per RFC 4180. Keep the columns and order id,name,status. Add a regression test for a name containing a comma. Do not add dependencies. Implementation is approved; do not deploy. ```
Source file · blinded label X05
Fix the CSV export so names containing commas are quoted per RFC 4180. Keep the columns and order id,name,status. Add a regression test for a name containing a comma. Do not add dependencies. Implementation is approved; do not deploy.
Source file · blinded label X03
Request
Prepare a Claude Code prompt. Investigate why the billing export drops rows. Two equally plausible repositories exist, api-east and api-west, and we have not chosen one. This is investigation only; no edits.
Contract
Run 1
[Promptcraft](<redacted local plugin cache path>/skills/promptcraft/SKILL.md) requires “ask for the deciding identity instead,” so repository selection remains open. ```text Investigate why the billing export drops rows. This is investigation only; no edits. Two equally plausible repositories exist: api-east and api-west. We have not chosen one. First ask me which repository to investigate and wait for my answer. Do not choose one yourself. Once I choose, trace the billing export and identify where rows are excluded or lost. Report the findings with supporting file and line references and any available reproduction evidence. Distinguish confirmed causes from hypotheses. If the cause remains unresolved, explain what evidence is missing. ``` [Claude Code guidance](https://code.claude.com/docs/en/how-claude-code-works) checked through Context7 on 2026-10-12.
Source file · blinded label X15
Material defects
```text Investigate why the billing export drops rows. This is investigation only; no edits. The target repository is unresolved: api-east and api-west are equally plausible, and neither has been selected. First ask me which repository to investigate, or whether to compare both. Do not silently choose a repository or begin repository-specific diagnosis before I answer. Constraints: - Do not modify files, configuration, data, dependencies, or Git state. - Do not create plans, reports, temporary files, or other artifacts on disk. - Use read-only inspection. Avoid running exports, tests, or other commands that write files or mutate external systems. - Do not expose credentials or sensitive billing records. After the repository scope is confirmed: 1. Establish the export entry point, output format, relevant version/environment, expected row count, and observed row count. Request missing evidence when it prevents a reliable diagnosis. 2. Trace rows from source selection through the final exported output. Inspect filters, joins, pagination, batching, deduplication, date/time boundaries, validation, serialization, and error handling. Distinguish intended exclusions from unintended losses. 3. Where evidence permits, trace a missing row and a comparable included row through the same path. Identify the first stage where the missing row disappears and the exact predicate or failure responsible. Use sanitized identifiers. 4. Check existing tests and available logs for evidence supporting or contradicting each hypothesis. Do not infer the cause from an error name or a suspicious code fragment alone. 5. If both repositories are authorized, keep findings separate and establish which implementation handles the affected export before attributing the defect. Return an inline report containing: - Repository and version investigated. - Confirmed findings, with file:line references and available count/log evidence. - The first proven row-loss stage and mechanism, if established. - Remaining hypotheses, missing evidence, and the smallest next read-only diagnostic step. Do not implement a fix. If the cause cannot be established, state that clearly. ```
Source file · blinded label X11
Run 2
The prompt leaves repository selection open and asks Claude Code to resolve it before investigating. ```text Investigate why the billing export drops rows. This is investigation only; no edits. Two equally plausible repositories exist: api-east and api-west. Neither has been selected. First ask me: “Which repository should I investigate: api-east or api-west?” Wait for my answer before repository-specific investigation. Do not choose one yourself or assume both are in scope. Once I select the repository, use read-only inspection to trace rows from the source through the export and identify where and why they disappear. Examine relevant code and available examples, tests, or logs without changing files or implementing a fix. Report the findings with file-and-line references and supporting evidence. Distinguish confirmed causes from hypotheses. If the available evidence cannot establish the cause, explain what remains unknown and what specific information would resolve it. Do not claim reproduction or verification that you have not performed. ``` Guidance checked October 12, 2026: [Claude Code best practices](https://code.claude.com/docs/en/best-practices).
Source file · blinded label X06
Material defects
```text Investigate why the billing export drops rows. Investigation only; no edits or fixes. Repository selection is unresolved: api-east and api-west are equally plausible. Your first response must ask: “Which repository should I investigate: api-east or api-west, and what is its checkout path?” Then wait for the answer. Do not inspect either repository, choose based on the current directory, or combine evidence from both. After I select the repository: 1. Confirm the checkout path, branch, and commit. Read applicable repository instructions and preserve all existing work. 2. Use read-only inspection. Do not modify files, install dependencies, create artifacts, commit, deploy, trigger export jobs, or change database or external state. Run existing checks only when their lack of side effects is established. 3. Establish the expected export population and observed omissions from available evidence. If necessary, request a sanitized failing example with export parameters, expected rows, and actual output. 4. Trace the export from source query through pagination, filtering, transformation, serialization, and delivery. Find the earliest stage where an expected row disappears. 5. Examine relevant mechanisms such as joins, date/timezone boundaries, pagination termination, deduplication, null handling, swallowed errors, concurrency, and output truncation. Treat these as hypotheses until supported. 6. Compare a missing row with a retained row and, where available, a working sibling code path. Distinguish intentional exclusions from defects. Return: - Repository and revision investigated. - Findings with file:line references and observed evidence. - Confirmed cause, or ranked hypotheses with supporting and contradictory evidence. - Remaining unknowns and the smallest read-only check needed to resolve them. If the cause is unproven, say so. Stop after the investigation report; implementation requires a separate instruction. ```
Source file · blinded label X02
Request
My manager wants me to delegate the CSV export fix to an AI coding agent. Write the instructions it should get: quote fields per RFC 4180, keep the column order id,name,status, add a regression test, no deployment.
Contract
Run 1
Material defects
```text Fix the CSV export: - Quote fields per RFC 4180: wrap fields containing commas, double quotes, or line breaks in double quotes, and escape embedded double quotes by doubling them. - Keep the column order exactly id,name,status. - Add a regression test covering quoting, escaping, and column order. Confirm it fails before the fix and passes afterward; run relevant existing tests. - Do not deploy or trigger deployment through another action. ``` Quoting details follow [RFC 4180, section 2](https://www.rfc-editor.org/info/rfc4180/), checked October 12, 2026.
Source file · blinded label X13
Fix the CSV export with these requirements: - Follow RFC 4180 quoting: enclose fields containing commas, double quotes, CR, or LF in double quotes. Escape embedded double quotes by doubling them. - Keep the column order exactly `id,name,status`. - Add a regression test covering commas, embedded quotes, and line breaks, and assert the column order. - Keep the change focused, run the relevant tests, and report the changes and results. - Do not deploy.
Source file · blinded label X12
Run 2
Material defects
```text Fix the CSV export with these requirements: - Apply RFC 4180 quoting: enclose fields containing commas, double quotes, or line breaks in double quotes. Escape embedded double quotes by doubling them. - Keep the column order exactly id,name,status. - Add a regression test covering commas, embedded double quotes, and multiline values. Assert correct quoting and the exact column order. Confirm it fails before the fix and passes afterward, then run the relevant existing tests. - No deployment. Do not take actions that trigger an automatic deployment. ``` Quoting rules verified against [RFC 4180, Section 2](https://www.rfc-editor.org/rfc/rfc4180#section-2).
Source file · blinded label X20
Material defects
Fix the CSV export with this scope: - Follow RFC 4180: enclose fields containing commas, double quotes, or line breaks in double quotes; escape embedded double quotes by doubling them; use CRLF record separators. - Preserve the header and column order exactly: `id,name,status`. - Add a regression test covering commas, embedded quotes, and multiline values. Assert the exact output and column order. - Run the relevant tests and report the changes and results. - Keep the fix focused. Do not deploy.
Source file · blinded label X10
Request
We decided to drop Python 3.9 support, only in setup.cfg and the CI matrix, nothing else. Prepare a Codex prompt for this.
Contract
Run 1
The original prompt is already suitable. No material revision is necessary. ```text We decided to drop Python 3.9 support, only in setup.cfg and the CI matrix, nothing else. ``` Checked against [official OpenAI prompting guidance](https://learn.chatgpt.com/guides/best-practices#strong-first-use-context-and-prompts) on October 12, 2026.
Source file · blinded label X07
Material defects
```text Drop Python 3.9 support with the smallest possible diff. Scope: - setup.cfg: remove the Python 3.9 classifier and raise the minimum supported Python version to 3.10, preserving any other version constraints. - CI configuration: remove Python 3.9 from the test matrix, preserving all remaining versions and job settings. Change nothing else. Do not modify application code, dependencies, tests, documentation, changelogs, or unrelated configuration. Do not refactor or reformat surrounding content. Verify the configuration changes and confirm the diff contains only these edits. Summarize the changed files and validation performed. ```
Source file · blinded label X17
Run 2
The original prompt is already suitable. No material revision is necessary. ```text We decided to drop Python 3.9 support, only in setup.cfg and the CI matrix, nothing else. ```
Source file · blinded label X16
Material defects
```text Drop Python 3.9 support. Limit changes strictly to setup.cfg and the CI Python-version matrix. - In setup.cfg, raise python_requires from >=3.9 to >=3.10 and remove the Python 3.9 classifier, if present. - Remove Python 3.9 from the CI matrix. Preserve every other version, matrix dimension, and workflow setting. - Do not change source code, tests, documentation, dependencies, lockfiles, or unrelated formatting. Inspect the existing configuration before editing. If it conflicts with these assumptions, report the conflict rather than widening the scope. Review the final diff to confirm it contains only these changes, then summarize the edited files and validation performed. ```
Source file · blinded label X14
One contract item for “Review a prompt without changing its scope” required the prompt to flag the refactor, dependency upgrade and blog post as scope questions. That contradicted the user’s own words, “without changing its scope”, and it failed every output in both conditions. After inspecting the grades we excluded that item and the matching defect.
Because the decision came after the results, it is a source of bias. Both versions are published; read either.
Item 3 conflicts with the user's explicit 'without changing its scope'; both conditions failed it. Adjusted results exclude that item and the matching 'mandates refactor, upgrade and blog post' defect.
| Scores | With | Without |
|---|---|---|
| Original: pass · defects | 6/10 · 5 | 3/10 · 18 |
| Corrected: pass · defects | 8/10 · 3 | 3/10 · 16 |
The runners, cases, contracts, outputs, grader prompt and grades are in the public repository. A rerun uses your own Codex account and will not match token for token; model outputs vary between runs.
Raw event traces are not published because they contain local file paths. The final answers, run log and grades are.
git clone https://gitlab.com/revanalex/promptingcraft-skill.git
cd promptingcraft-skill
git checkout 7e45c24a657eae84996271dd2cfd83c1a81fee45
# needs a logged-in Codex CLI and this commit's plugin from the local marketplace (former id: promptcraft@dondego-local)
python3 scripts/evaluate_ab.py evals/results/ab-1/cases.json /tmp/ab-rerun 2
python3 scripts/evaluate_triggers.py evals/trigger/queries.json /tmp/trigger-rerun 3PromptingCraft.com follows official guidance published by AI developers, but it is developed independently by the dondego.com team and is not affiliated with or endorsed by OpenAI, Anthropic or Cursor.