benchmark-paper
Scores a draft against the measurable patterns of recent award-winning and top-cited papers at a target venue, producing a shareable venue-fit scorecard. Use when a researcher asks "score my paper", "how does my paper compare to best papers at SIGSPATIAL/NeurIPS/...", "is my paper good enough for this venue", "benchmark my draft against award winners", "rate my paper out of 10", or wants a readiness scorecard before submitting. Builds an exemplar corpus via study-exemplars, extracts comparable structural and rhetorical features from the draft, and reports a per-dimension scorecard plus an overall venue-fit index — explicitly a conformance measure (does the draft match the form of strong work at this venue), never a prediction of acceptance, award, or scientific quality. Trigger words - score my paper, rate my paper, benchmark, scorecard, compare to best papers, award winners, venue fit, am I ready to submit.
How do I install this agent skill?
npx skills add https://github.com/shaishavmaisuria/research-paper-lifecycle-skills --skill benchmark-paperIs this agent skill safe to install?
- Gen Agent Trust Hubpass
The benchmark-paper skill assesses research paper drafts for conformance with venue-specific patterns of successful publications. It extracts structural and rhetorical features and uses a local, deterministic Python script to generate a scorecard. It maintains local state in a .paper-memory directory to track progress. No security risks or malicious patterns were identified.
- Socketpass
No alerts
- Snykwarn
Risk: MEDIUM · 1 issue
What does this agent skill do?
Benchmark Paper
Produces a venue-fit scorecard: how closely a draft matches the measurable patterns of recent award-winning and top-cited papers at the target venue. This is a conformance gauge to help an author find gaps before submitting — it is not a prediction of acceptance, a best-paper forecast, or a judgment of scientific merit.
Pairs naturally with simulate-reviewers (which red-teams content quality) and study-exemplars (which it calls to build the comparison corpus). Run preflight-check first — a desk-reject defect makes any score moot.
When to use
- The author wants a single, shareable readiness signal before submitting.
- The author asks how their draft stacks up against the venue's strongest recent papers.
- The author wants to know which dimensions are weakest relative to exemplars, ranked by fixability.
When NOT to use it (say this plainly to the user)
- It cannot judge whether the science is novel, correct, or important — the things that actually win awards. Use
simulate-reviewersfor content critique. - It cannot predict acceptance or a best-paper award. Anyone who claims a tool can is selling false precision.
- A high score on a flawed paper is meaningless. Conformance to form ≠ quality of substance.
Inputs
- The draft: a
.texfile (or compiled PDF / markdown), with.bibif available. - The target venue id (e.g.
sigspatial-2026) → its profile invenues/. - Optional: a corpus size N (default 8 exemplars) and a recency window (default last 5 years).
Process
- Resolve the venue profile from
venues/conferences/<venue>.ymland itsfamily:profile invenues/families/<family>.yml. If missing, ask the user or haveadd-venue-profilecreate it. Re-verify the venue is correct before scoring. Apply the staleness gate: profiles are year-versioned (verified.valid_window,verified.last_verified_against_cfp). Do not assert a hard format constraint (page limit, column count, mandatory section, deadline) from a profile whosevalid_windowdoes not include the target cycle without a fresh CFP check first; if you cannot check, mark itneeds-verificationand disclose that the basis was a year-mismatched profile (see references/scoring-rubric.md "Staleness gate"). A year-mismatched profile may still inform priors (the exemplar distribution, the modal skeleton). - Build the on-family exemplar corpus. The distribution dimensions (section architecture, citation density, abstract structure, figure/table conventions) must be scored against an on-family distribution — same venue family — never an off-family proxy. Resolve it in priority order: (a) a live
study-exemplarscorpus for the target venue — preferred; invokestudy-exemplarsto fetch (on demand, legally, transiently) N recent best-paper awardees and top-cited papers at the venue and extract their feature profile, never bundling or storing paper text; (b) the family profile'sexemplar_distributionblock as a fallback prior (disclose it is from the family prior, carry its confidence); (c) if neither exists for this family, reduce N and discloseno on-family exemplar distributionon those dimensions — do not borrow another family's numbers. If award lists are unavailable, fall back to top-cited and say so. The corpus basis (live vs family-prior vs none) must be disclosed. - Extract the draft's comparable features along the dimensions in references/scoring-rubric.md (section architecture, contribution framing, evaluation rigor signals, claim/citation density, abstract structure, figure/table conventions, reproducibility artifacts). Use
verify-citationsoutput if present so the citation-integrity dimension is grounded. Detect the realization level of each dimension (absent/planned/drafted/complete): a dimension whose evidence is dominated by honest[RESULT]/[TBD]/\todoplaceholders isplanned, not weak. Record it in the features JSONrealizationfield. - Score each dimension as conformance to the exemplar distribution, with an explicit basis for every number (what was measured, what the exemplar range was). For
planned/absentdimensions, score the completeness and specificity of the design (named baselines, datasets, ablation list, matched-budget protocol, variance/significance policy) — never realized numbers — and mark deferred[RESULT]slots as deferred-but-specified in the basis. Runpython3 scripts/scorecard.py features.json --venue <id>to compute and render deterministically — do not eyeball the aggregate. The script caps planned dimensions below the "within range" band and relabels the index plan-conformance when any dimension is unrealized, so it is never silently compared against an executed paper. - Report: the scorecard (per-dimension 0–10 + venue-fit/plan-conformance index), the 3 weakest dimensions ranked by fixability, concrete fixes tied to exemplar patterns, and the mandatory caveat block. For a plan, frame the gap as experiments not run (path: specify → run), not design is weak. Offer the one-line shareable summary the user can screenshot.
Output
A scorecard.md containing:
- Venue-fit index (0–10) with a one-line plain-English band (e.g. "structurally in line with recent {venue} winners; evaluation section is the gap"). For a plan/outline the index is labeled plan-conformance and must not be compared against executed papers.
- Per-dimension table: score, realization level, exemplar range, your value, basis.
- Top fixes ranked by impact × ease.
- Corpus disclosure: which papers formed the basis, award-vs-cited, recency, and whether the distribution was a live on-family corpus, a family-profile prior, or absent (
no on-family exemplar distribution). - Format-basis disclosure when any hard constraint came from a year-mismatched profile (flag it
needs-verificationuntil checked against the live CFP). - Caveats (always): conformance not quality; not an acceptance/award prediction; human judgment required.
Guardrails
- Every score states what was measured and the exemplar range it was compared against. No bare numbers.
- Never present the index as a probability of acceptance or an award. Refuse to, if asked — explain why.
- Corpus is fetched on demand from open-access sources, processed transiently, never stored or committed (see
study-exemplars). - Never fabricate exemplar data to fill the corpus; if you can't reach N papers, score against fewer and disclose it.
- This is a copilot signal, not a verdict. The author decides.
Memory
Uses the shared .paper-memory/ convention in the user's paper directory
(full spec:
paper-memory-convention.md).
- At start: read
.paper-memory/profile.yml(vertical, venue tier) to pick exemplar emphasis, andlessons.mdto recall which dimensions were weak last run and anyrecurringgaps for this author — lead with them. - At end: append durable findings in the shared format
- [YYYY-MM-DD] (benchmark-paper | <scope>) weak-dimension -> recommendation(viareflect-and-improve'sreflect_log.py append, which dedupes and dates). A dimension that lags exemplars across drafts isrecurring; a one-time gap isthis-paper. Do not log the full scorecard, only the lasting takeaways. - Create
.paper-memory/on demand if absent and offer to add it to the project.gitignore. It is local-only; never upload it or copy it into this repo.
How can the creator link this skill?
Add the canonical catalog link to the repository README so users can inspect current installs and available audits. The publishing guide covers the complete discovery path.
<a href="https://skillzs.dev/skills/shaishavmaisuria/research-paper-lifecycle-skills/benchmark-paper">View benchmark-paper on skillZs</a>