Gemini
Pending safe baseline
Score only if a no-login or project-safe route produces a complete, distinct assistant answer with page citations; keep scores blank for prompt echo or login gates.
Evidence-backed PDF summarizer test
AI can summarize a PDF-like packet, but citations and caveats still need checking.
This evidence log tests a safe first workflow: paste a synthetic page-numbered work-document packet into a general AI assistant and ask for a cited summary, decision table, Q&A, caveats, and review-only reply. No real PDF, upload, Drive, OneDrive, Dropbox, email, Slack, Notion, CRM, accounting, legal, tax, HR, medical, school, customer, workspace connector, browser extension, document edit, email send, signature, order approval, or buyer contact was used.
Quick answer
ChatGPT is ahead on this paste-only PDF/document test; Duck.ai is usable but needs closer review.
In the synthetic Northstar Candle Co. PDF/document summarizer fixture, ChatGPT scored 4.67/5 and Duck.ai scored 4.37/5. Both returned useful cited summaries without uploading a file. ChatGPT covered the packet more completely; Duck.ai missed a few prohibited-claim details and did not show the full contribution calculation.
The practical workflow is: paste only approved low-risk text, require page citations and calculations, then verify policy conflicts, caveats, and any business commitment before using the summary.
Current no-login results
Two paste-only PDF/document summarizer baselines scored.
Scores cover the separable assistant answer only. Prompt echo, setup friction, page-history contamination, recapture needs, and pending routes are tracked separately in the evidence CSVs.
| Tool | Score | Last tested | What worked | What to review |
|---|---|---|---|---|
| ChatGPT | 4.67/5 | 2026-06-20 | Best current result. It preserved the 24-kit pilot, $18.75 unit price, $450 gross revenue, $169.44 contribution before overhead, current-policy discount/free-freight limits, retired 2025 terms, Citrus Desk inventory caveat, and review-only reply boundary. | The Q&A preserved the final contribution but did not show the full variable-cost calculation; the reply ended with broad page sourcing; and the cleaned output needed minor UI/table cleanup. |
| Duck.ai | 4.37/5 | 2026-06-21 | Produced a useful cited summary, decision/deadline table, Q&A, caveat list, and review-only reply. It caught the no-discount/free-freight policy boundary and avoided shipment, buyer-contact, approval, document-edit, or live-action claims. | It did not show the full contribution calculation in the Q&A, missed some prohibited staff-card claim categories such as non-toxic/allergy-safe/therapeutic and burn-time caveats, and required recapture/cleanup because the first page capture still showed generation in progress. |
Pending rows
Uncaptured tools stay unscored.
A pending row is not a quality judgment. It means we have not yet saved a full, distinct assistant answer through a safe no-login or project-safe route.
Perplexity
Pending safe baseline
Prior public answer pages can mix prompt echo and answer text. Save prompt, raw page, and cleaned output separately before scoring.
Microsoft Copilot
Pending safe baseline
Prior no-login Copilot checks often showed prompt echo instead of a usable answer. Treat as setup friction unless a distinct assistant response appears.
The test scenario
A fictional eight-page wholesale decision packet.
The fixture uses Northstar Candle Co., a fictional small business. The source packet includes a cover memo, Cedar Room buyer email, internal cost estimate, June 2026 policy addendum, label/compliance checklist, staff talking-points draft, conflicting old 2025 sales-note excerpt, and open questions for Maya.
A strong answer must identify the proposed 24-kit pilot, calculate the $169.44 contribution before overhead, use the current June 2026 policy instead of old discount/free-freight/returns terms, cite pages, flag approval-needed items, and keep the reply review-only.
Reusable paste-only prompt shape
What the benchmark forces the assistant to prove.
The prompt deliberately separates document summarization from file upload, document editing, business approval, customer contact, and live workflow updates.
Treat the pasted packet as a synthetic text export, not as permission to upload, edit, sign, send, approve, or contact anyone.
Cite source pages for every factual answer, especially decisions, dates, prices, discount/free-freight policy, returns, logo printing, and safety/claim restrictions.
Verify the math independently: 24 × $18.75 = $450.00; 24 × $10.09 = $242.16; $450.00 - $242.16 - $38.40 = $169.44 contribution before overhead.
Use the June 2026 addendum over the old 2025 template: no 10% discount, no automatic free freight, and no automatic 60-day returns.
Keep the internal reply review-only: no shipment promise, buyer delivery, logo printing, discount, free freight, returns, approval, document edit, email send, or order creation.
Evidence files
The fixture is reusable and connector-safe.
Synthetic source packet
content/fixtures/pdf-summarizer-work-documents/synthetic-work-doc-source.md
Eight page-numbered fictional Northstar Candle Co. work-document pages: cover memo, buyer email, cost estimate, policy addendum, label checklist, staff-card draft, old template conflict, and questions for Maya.
Paste-only benchmark prompt
content/fixtures/pdf-summarizer-work-documents/pdf-summarizer-paste-prompt.md
The exact no-upload prompt asking for a five-bullet executive summary, decision/deadline table, cited Q&A, caveats, and review-only internal reply.
Expected output key
content/fixtures/pdf-summarizer-work-documents/expected-pdf-summary-output.md
Manual answer key with required facts, exact contribution calculation, Q&A answers, must-not-invent checks, and scoring notes.
Test protocol
content/fixtures/pdf-summarizer-work-documents/pdf-summarizer-test-protocol.md
The no-connector protocol: sanity prompt, screenshots, submitted prompt, raw page text, cleaned assistant output, metadata, and no-score blockers.
Before using real documents
Keep document summaries under human review.
PDF and document summaries often touch contracts, policies, customers, suppliers, safety claims, money, HR, legal, tax, or medical context. A fluent cited answer is still only a draft until the source pages and calculations are checked.
Check whether the assistant cites the page that supports each factual sentence, not just a broad source range.
Recalculate all totals before using the summary in a real business decision.
Look for old-policy conflicts: AI summaries can accidentally preserve outdated template terms if not prompted to reconcile them.
Audit prohibited-claim language around safety, wellness, medical, allergy, pet-safety, productivity, guaranteed-mood, and burn-time claims.
Avoid confidential uploads and workspace connectors until you have a privacy, retention, access, deletion, and export reason to trust the specific tool and plan.
Limitations
This is a paste-only benchmark, not a confidential-PDF upload endorsement.
The current scores cover two public web prompt-only runs using a synthetic fixture. They do not test paid plans, PDF upload/OCR extraction, file privacy, Drive/OneDrive/Dropbox permissions, Docs/Word editing, email/Slack/Notion/CRM/accounting/legal/tax/HR/medical/customer/workspace connectors, browser extensions, export formatting, stored memory, real files, real customer data, signatures, approvals, order creation, buyer contact, or live action-taking. New rows will be added only when raw output or setup-friction evidence is saved and scored with the same rubric discipline.