databricks-unstructured-pdf-generation
Build RAG / unstructured-document evaluation datasets and demo documents (e.g. for a Genie Agent with an attached volume) on Databricks: generate synthetic PDFs locally, upload to Unity Catalog volumes, and pair each document with test questions for retrieval evaluation.
How do I install this agent skill?
npx skills add https://github.com/databricks/databricks-agent-skills --skill databricks-unstructured-pdf-generationIs this agent skill safe to install?
- Gen Agent Trust Hubpass
This skill provides a utility for generating synthetic PDF documents from HTML files and uploading them to Databricks Unity Catalog volumes for evaluation purposes. It uses a local Python script and the official Databricks CLI for file operations.
- Socketpass
No alerts
- Snykpass
Risk: LOW · No issues
What does this agent skill do?
Unstructured-Document for Demos and Eval Datasets on Databricks
Workflow for producing synthetic PDF documents + paired test questions as a Unity Catalog-resident dataset for Demos and RAG / unstructured-document retrieval evaluation on Databricks. The PDF-generation step uses standard local HTML → PDF tooling; the Databricks-specific value is the workflow shape — UC volume layout, paired question files, and integration with downstream Databricks retrieval / ai_extract / ai_parse_document evaluation.
Workflow
- Write HTML files to
./raw_data/html/(write multiple files in parallel for speed) — domain-shaped to match the documents your retrieval pipeline will see in production. - Convert HTML → PDF using
<SKILL_ROOT>/scripts/pdf_generator.py(parallel conversion, wrapsplutoprint). - Upload PDFs to a Unity Catalog volume via
databricks fs cp— same volume shape your production pipeline will read from. - Generate
./raw_data/pdf/pdf_eval_questions.jsonpairing each document with retrieval-eval questions; this becomes the gold dataset formlflow.genai.evaluate()or comparable retrieval-quality scorers.
If you only need ad-hoc PDFs (no Databricks workflow), any HTML → PDF tool (
weasyprint,wkhtmltopdf,playwright pdf,plutoprint) works directly — this skill exists for the synthetic-dataset-on-UC end-to-end shape, not as a general PDF generator.
Path convention:
<SKILL_ROOT>below = the directory containing this SKILL.md. Resolve to the absolute install path (e.g.~/.claude/skills/databricks-unstructured-pdf-generation)../raw_data/...paths are relative to your own project cwd.
Dependencies
uv pip install plutoprint
Step 1: Write HTML Files
mkdir -p ./raw_data/html
Write HTML documents to ./raw_data/html/filename.html. Use subdirectories to organize (structure is preserved).
Step 2: Convert to PDF
# Convert entire folder (parallel, 4 workers)
python <SKILL_ROOT>/scripts/pdf_generator.py convert --input ./raw_data/html --output ./raw_data/pdf
Skips files where PDF exists and is newer than HTML. Use --force to reconvert all.
Step 3: Upload to Volume
databricks fs requires the dbfs: scheme prefix even for UC Volume paths. -r copies the contents of the source directory into the target (the source directory name is not preserved), so name the target raw_data/pdf explicitly to keep the PDFs in their own folder on the volume. They land under raw_data/pdf/ — i.e. dbfs:/Volumes/my_catalog/my_schema/raw_data/pdf/report.pdf — so an ingest pipeline can point at that single folder. For a Genie Agent, upload to a dedicated managed volume instead (e.g. dbfs:/Volumes/my_catalog/my_schema/docs/), because Genie indexes the whole volume, not a subfolder (see databricks-genie-agents § Documents).
databricks fs cp -r --overwrite ./raw_data/pdf dbfs:/Volumes/my_catalog/my_schema/raw_data/pdf
Step 4: Generate Test Questions
Create ./raw_data/pdf/pdf_eval_questions.json with questions for evaluating a Genie Agent (or a legacy Knowledge Assistant) over the PDFs. Keep it out of a volume attached to a Genie Agent: the whole volume is indexed, so the answers would leak into the agent's context (upload the PDFs before writing it, or databricks fs rm it from that volume):
{
"api_errors_guide.pdf": {
"question": "What is the solution for error ERR-4521?",
"expected_fact": "Call /api/v2/auth/refresh with refresh_token before the 3600s TTL expires"
},
"installation_manual.pdf": {
"question": "What port does the service use by default?",
"expected_fact": "Port 8443 for HTTPS, configurable via CONFIG_PORT environment variable"
}
}
Use it to build Agent-mode test cases (or Genie benchmarks) and validate retrieval accuracy.
Document Content Guidelines
When generating documents for Genie Agent (document Q&A) testing or demos:
- Multi-page documents: Each PDF should be several pages with substantial content
- Specific error codes and solutions: Include product-specific error codes, causes, and resolution steps
- Technical details: API endpoints, configuration parameters, version numbers, specific commands
- Simple CSS: Keep styling minimal for fast HTML creation and reliable PDF conversion
- Queryable facts: Include details the agent must read the document to answer (not general knowledge)
Good document types:
- Product user manuals with troubleshooting sections
- API error reference guides (error codes, causes, solutions)
- Installation/configuration guides with specific steps
- Technical specifications with version-specific details
Example content: Instead of generic "Connection failed" errors, write:
- "Error ERR-4521: OAuth token expired. Cause: Token TTL exceeded 3600s default. Solution: Call
/api/v2/auth/refreshwith your refresh_token before expiration. See Section 4.2 for token lifecycle management."
CLI Reference
python <SKILL_ROOT>/scripts/pdf_generator.py convert [OPTIONS]
--input, -i Input HTML file or folder (required)
--output, -o Output folder for PDFs (required)
--force, -f Force reconvert (ignore timestamps)
--workers, -w Parallel workers (default: 4)
Folder Structure
Subfolder structure is preserved:
./raw_data/html/ ./raw_data/pdf/
├── report.html → ├── report.pdf
├── quarterly/ ├── quarterly/
│ └── q1.html → │ └── q1.pdf
└── legal/ └── legal/
└── terms.html → └── terms.pdf
Bundled Script
This skill ships one helper script:
| File | Description |
|---|---|
| scripts/pdf_generator.py | HTML → PDF converter (wraps plutoprint); parallel folder conversion with timestamp-skip. Referenced by Step 2 and the CLI Reference. |
The script ships at <SKILL_ROOT>/scripts/pdf_generator.py. If it is absent, recreate it from the CLI Reference above (a convert subcommand taking --input/--output/--force/--workers, wrapping plutoprint for HTML → PDF).
Troubleshooting
| Issue | Solution |
|---|---|
| "plutoprint not installed" | uv pip install plutoprint |
| PDF looks wrong | Check HTML/CSS syntax |
| "Volume does not exist" | databricks volumes create CATALOG SCHEMA VOLUME_NAME MANAGED (four separate positional args, not catalog.schema.volume) |
How can the creator link this skill?
Add the canonical catalog link to the repository README so users can inspect current installs and available audits. The publishing guide covers the complete discovery path.
<a href="https://skillzs.dev/skills/databricks/databricks-agent-skills/databricks-unstructured-pdf-generation">View databricks-unstructured-pdf-generation on skillZs</a>