databricks-ai-functions
Use Databricks built-in AI Functions (ai_classify, ai_decide, ai_extract, ai_summarize, ai_mask, ai_translate, ai_fix_grammar, ai_gen, ai_analyze_sentiment, ai_similarity, ai_parse_document, ai_prep_search, ai_query, ai_forecast) to add AI capabilities directly to SQL and PySpark pipelines without managing model endpoints. Also covers document parsing and building custom RAG pipelines (parse → prep_search → index → query).
How do I install this agent skill?
npx skills add https://github.com/databricks/databricks-agent-skills --skill databricks-ai-functionsIs this agent skill safe to install?
- Gen Agent Trust Hubpass
This skill provides instructions for utilizing Databricks AI functions to process data in SQL and Spark. It contains a surface for indirect prompt injection because its core purpose involves processing external files (PDFs, images, Office documents) that could contain malicious instructions. References to external code and documentation are limited to official Databricks resources.
- Socketpass
No alerts
- Snykwarn
Risk: MEDIUM · 1 issue
What does this agent skill do?
Databricks AI Functions
Official Docs: https://docs.databricks.com/large-language-models/ai-functions Individual function reference: https://docs.databricks.com/sql/language-manual/sql-ref-functions-builtin
Overview
Databricks AI Functions are built-in SQL and PySpark functions that call Foundation Model APIs directly from your data pipelines — no model endpoint setup, no API keys, no boilerplate. They operate on table columns as naturally as UPPER() or LENGTH(), and are optimized for batch inference at scale.
Always prefer a task-specific function over ai_query. Reach for ai_query only when no task function fits (custom/external endpoints, multimodal, or JSON beyond ai_extract's limits). Every function below shares a baseline: DBR 15.1+ (notebooks) / 15.4 ML LTS (batch), not on SQL Warehouse Classic, and region must support AI Functions — the Prereqs column lists only what's additional.
Cost & speed — each call is an LLM inference (slow and billed per token). Run a function once per row and persist the result to a Delta table; never re-invoke it on every downstream query. In demos, avoid generating tables with millions of rows — sample the input when needed so the demo runs quickly. Materialize once, then query the cheap Delta output.
The Function column links to the in-repo deep reference (full options, schemas, examples); Docs links to the official page.
| Function | Task | Input | Output | Extra prereqs | Docs |
|---|---|---|---|---|---|
ai_analyze_sentiment | Sentiment scoring | content STRING | STRING — positive/negative/neutral/mixed, or NULL | — | ↗ |
ai_classify | Fixed-label routing | content STRING|VARIANT, labels (2–500), [options MAP] | v2.1 VARIANT — {response:[{value:label}], error_message, metadata} | — | ↗ |
ai_decide | Evaluate questions (boolean / choice / score) over text or structured data | state STRING|VARIANT, questions STRING (JSON), [options MAP] | VARIANT — {response:{answers}, metadata, error_message} | Beta; admin-gated via Previews | ↗ |
ai_extract | Entity / field extraction | content STRING|VARIANT, schema STRING (JSON), [options MAP] | v2.1 VARIANT — {response:{field:{value:…}}, error_message, metadata} | ≤256 fields, ≤12 nesting levels | ↗ |
ai_fix_grammar | Grammar correction | content STRING | STRING (corrected) | — | ↗ |
ai_gen | Free-form generation | prompt STRING | STRING | — | ↗ |
ai_mask | PII redaction | content STRING, labels ARRAY<STRING> | STRING (entities → [MASKED]) | — | ↗ |
ai_similarity | Semantic similarity | expr1 STRING, expr2 STRING | FLOAT (0.0–1.0) | — | ↗ |
ai_summarize | Summarization | content STRING, [max_words INT] (0 = uncapped) | STRING | Public Preview; English-tuned | ↗ |
ai_translate | Translation | content STRING, to_lang STRING | STRING | Langs: en, fr, de, hi, it, pt, es, th | ↗ |
ai_parse_document | Parse PDF / Office / images | content BINARY, [Map('version','2.0', …)] | VARIANT — pages, elements, error_status | DBR 17.3+; ≤500 pages / 100 MB | ↗ |
ai_prep_search | RAG chunking from parsed docs or text | parsed VARIANT|STRING, [options MAP] | VARIANT — {document:{contents, pages, source_uri}, error_status} | DBR 18.2+ (serverless env v3+) | ↗ |
ai_query | Hosted model service or custom serving endpoint, multimodal, complex JSON (last resort) | endpoint STRING, request STRING|STRUCT, [returnType], [failOnError BOOL], [modelParameters STRUCT], [responseFormat STRING], [files] | Parsed response; with failOnError => false a STRUCT{response, errorMessage} | Pro/Serverless warehouse; EXECUTE on model service or CAN QUERY on serving endpoint | ↗ |
ai_forecast | Time series forecasting (table-valued) | observed TABLE, horizon, time_col, value_col, [group_col], [prediction_interval_width], [frequency], [seed], [parameters] | Rows: time/group cols + per value {v}_forecast, {v}_upper, {v}_lower (DOUBLE) | Pro/Serverless warehouse; Public Preview | ↗ |
Models run under Apache 2.0 or LLAMA 3.3 Community License — you are responsible for compliance.
Patterns
Pin extraction and classification to v2.1 in new code. Every extracted scalar and returned label has a value field, even when citations, confidence scores, and rationales are disabled. Existing v2.0 queries remain supported; update downstream projections to read value when upgrading. See the task-function reference for the version-specific contracts.
Chain task functions to enrich a column in one pass. ai_classify/ai_extract return a VARIANT — read it with the colon operator (:response):
SELECT id,
ai_analyze_sentiment(content) AS sentiment,
ai_summarize(content, 30) AS summary,
ai_classify(content, '["technical","billing","other"]', map('version','2.1')):response[0].value::STRING AS category,
ai_extract(content, '["product","error_code","date"]', map('version','2.1')):response:product:value::STRING AS product,
ai_fix_grammar(content) AS content_clean
FROM raw_feedback;
In PySpark, call any of these inside expr(...): df.withColumn("category", expr("ai_classify(content, '[\"a\",\"b\"]', map('version','2.1')):response[0].value::STRING")) — and read extracted scalar fields via selectExpr("col:response:field:value::STRING AS field").
PII redaction before storage — ai_mask(content, ARRAY(entity_types)) returns text with entities → [MASKED].
SELECT ai_mask(message, array('person','email','phone','address')) AS message_safe FROM raw_messages;
Semantic matching / dedup — ai_similarity returns 0–1; self-join and threshold:
SELECT a.id, b.id, ai_similarity(a.name, b.name) AS score
FROM companies a JOIN companies b ON a.id < b.id
WHERE ai_similarity(a.name, b.name) > 0.85;
Forecasting — table-valued; one row per future period (+ per group). Full param/group/interval forms → 3-ai-forecast.md:
SELECT * FROM ai_forecast(
observed => TABLE(SELECT date, sales FROM daily_sales),
horizon => '2026-12-31', time_col => 'date', value_col => 'sales');
-- Returns: date, sales_forecast, sales_upper, sales_lower
Nested field extraction — ai_extract supports arrays of objects, including invoice line items. See extraction schemas and versioned outputs.
Document parsing (ai_parse_document) and RAG chunking (ai_prep_search) get their own staged pipeline below.
Document Processing Pipeline
Chain AI Functions stage-by-stage into Delta tables for batch document processing. The example is written as a Spark Declarative Pipeline (SDP / Lakeflow / DLT) — CREATE OR REFRESH STREAMING TABLE with STREAM(...) sources. To run the same logic standalone in a notebook / SQL warehouse, swap each CREATE OR REFRESH STREAMING TABLE x AS for CREATE OR REPLACE TABLE x AS and drop the STREAM(...) wrappers. In SDP Python it's @dp.table with from pyspark import pipelines as dp.
Pass the parsed VARIANT directly into extraction and classification to preserve layout and source context. For plain-text or Markdown files, skip parsing: use read_files(..., format => 'text', wholetext => true) and pass its value column. Without wholetext => true, text input is read one line per row.
-- Stage 1 — parse supported PDF, image, or Office documents, filter parse errors
CREATE OR REFRESH STREAMING TABLE raw_parsed AS
SELECT path,
parsed AS parsed_content,
parsed:error_status AS parse_error
FROM (
SELECT path, ai_parse_document(content, map('version','2.0')) AS parsed
FROM STREAM read_files('/Volumes/my_catalog/doc_processing/landing/', format => 'binaryFile')
)
WHERE is_variant_null(parsed:error_status);
-- Stage 2 — classify document type (cheap, no endpoint selection)
CREATE OR REFRESH STREAMING TABLE classified_docs AS
SELECT *,
ai_classify(parsed_content, '["invoice","purchase_order","receipt","contract","other"]', map('version','2.1')):response[0].value::STRING AS doc_type
FROM STREAM raw_parsed;
-- Stage 3 — extract fields; ai_extract returns a VARIANT, read fields with `:`
CREATE OR REFRESH STREAMING TABLE extracted AS
SELECT path, doc_type,
result:response:invoice_number:value::STRING AS invoice_number,
result:response:vendor_name:value::STRING AS vendor_name,
result:response:total_amount:value::DOUBLE AS total_amount,
result:error_message::STRING AS extract_error
FROM (
SELECT *, ai_extract(parsed_content,
'{"invoice_number":{"type":"string"},"vendor_name":{"type":"string"},"total_amount":{"type":"number"}}',
map('version','2.1')) AS result
FROM STREAM classified_docs WHERE doc_type = 'invoice' AND parsed_content IS NOT NULL
);
In a batch job, retain per-row errors for inspection or route them to a sidecar table: check ai_extract's result:error_message with is_variant_null for JSON null on success, and for ai_query pass failOnError => false and check ai_response.errorMessage (a STRUCT field, dot-accessed). SQL IS NULL does not recognize a JSON null inside VARIANT. See 2-ai-query.md.
Custom RAG Pipeline — Parse → Prep → Index
For retrieval rather than field extraction: ai_parse_document → ai_prep_search (semantic chunking + context enrichment, DBR 18.2+) → Vector Search Delta Sync index. ai_prep_search returns chunk_id, chunk_to_retrieve, and chunk_to_embed (enriched with title/headers/page) — embed chunk_to_embed, return chunk_to_retrieve to the LLM. Shown standalone; in an SDP swap CREATE OR REPLACE TABLE for CREATE OR REFRESH STREAMING TABLE + STREAM read_files(...).
CREATE OR REPLACE TABLE parsed_chunks AS
WITH prepped AS (
SELECT path AS source_path, ai_prep_search(ai_parse_document(content, map('version','2.0'))) AS prep
FROM read_files('/Volumes/my_catalog/doc_processing/docs/', format => 'binaryFile')
)
SELECT
variant_get(chunk, '$.chunk_id', 'STRING') AS chunk_id,
variant_get(chunk, '$.chunk_to_retrieve', 'STRING') AS chunk_to_retrieve,
variant_get(chunk, '$.chunk_to_embed', 'STRING') AS chunk_to_embed,
source_path
FROM prepped LATERAL VIEW explode(variant_get(prep, '$.document.contents', 'ARRAY<VARIANT>')) c AS chunk;
Then enable CDF (ALTER TABLE parsed_chunks SET TBLPROPERTIES (delta.enableChangeDataFeed = true)) and use the databricks-vector-search skill to build a Delta Sync index: PK chunk_id, embedding source chunk_to_embed, return chunk_to_retrieve.
Beyond batch:
- Ask questions over the output — point a Genie Agent at the resulting Delta table for natural-language querying instead of hand-written SQL; see the databricks-genie-agents skill.
- Low-latency / serving — to expose this as a real-time, governed endpoint (e.g. register a model to Unity Catalog and serve it), use the databricks-model-serving skill.
- Production incremental ingestion — for a runnable end-to-end streaming
ai_parse_documentjob (checkpoints,trigger(availableNow=True)), see databricks/bundle-examples · job_with_ai_parse_document.
Reference Files
- 1-task-functions.md — Deep reference for every task-specific function: full options/schemas (e.g.
ai_extractv2.1 citations + confidence scores,ai_classifymultilabel,ai_parse_documentoptions + output schema,ai_prep_searchchunk schema) and non-trivial examples. The Overview table above links to each function's section directly. - 2-ai-query.md —
ai_querycomplete reference: all parameters, structured output withresponseFormat, multimodalfiles =>, UDF patterns, and error handling - 3-ai-forecast.md —
ai_forecastparameters, single-metric, multi-group, multi-metric, and confidence interval patterns
Common Issues
| Issue | Solution |
|---|---|
ai_parse_document not found | Requires DBR 17.3+. Check cluster runtime. |
ai_prep_search not found | Requires DBR 18.2+ (serverless env v3+). |
ai_decide not found / unavailable | Beta — a workspace admin must enable it on the Previews page. Requires Databricks SQL (not SQL Classic) or DBR 15.4 LTS+. |
explode() fails on a VARIANT | explode needs ARRAY — cast first: explode(variant_get(prep, '$.document.contents', 'ARRAY<VARIANT>')). |
| Embedding the wrong RAG column | Embed chunk_to_embed (context-enriched); return chunk_to_retrieve to the LLM. |
ai_forecast fails | Requires Pro or Serverless SQL warehouse — not available on Classic or Starter. |
| All functions return NULL | Input column is NULL. Filter with WHERE col IS NOT NULL before calling. |
ai_translate fails for a language | Supported (8): English (en), French (fr), German (de), Hindi (hi), Italian (it), Portuguese (pt), Spanish (es), Thai (th) — to_lang takes the code or the full name (see references/1-task-functions.md). Use ai_query with a multilingual model for others. |
ai_classify returns unexpected labels | Use clear, mutually exclusive label names. Fewer labels (2–5) produces more reliable results. |
ai_query raises on some rows in a batch job | Add failOnError => false — returns a STRUCT with .response and .errorMessage (dot-accessed) instead of raising. |
ai_query fails with 403 "no longer available" | Discover the exact replacement system.ai model service using the AI Gateway listing; legacy names do not always have a matching model service. Custom serving endpoints keep their existing names. See 2-ai-query.md. |
| Batch job runs slowly | Use DBR 15.4 ML LTS cluster (not serverless or interactive) for optimized batch inference throughput. |
How can the creator link this skill?
Add the canonical catalog link to the repository README so users can inspect current installs and available audits. The publishing guide covers the complete discovery path.
<a href="https://skillzs.dev/skills/databricks/databricks-agent-skills/databricks-ai-functions">View databricks-ai-functions on skillZs</a>