skillZs
★ LIVE SKILL TAGS ★
>>> LIVE SKILLS INDEX <<<
* OPEN SOURCE *
NO LOGIN, NO TRACKING
※ REAL INSTALL DATA ※
← back to all skills
databricks/databricks-agent-skills387 installs

databricks-ai-functions

Use Databricks built-in AI Functions (ai_classify, ai_decide, ai_extract, ai_summarize, ai_mask, ai_translate, ai_fix_grammar, ai_gen, ai_analyze_sentiment, ai_similarity, ai_parse_document, ai_prep_search, ai_query, ai_forecast) to add AI capabilities directly to SQL and PySpark pipelines without managing model endpoints. Also covers document parsing and building custom RAG pipelines (parse → prep_search → index → query).

How do I install this agent skill?

npx skills add https://github.com/databricks/databricks-agent-skills --skill databricks-ai-functions
view source ↗

Is this agent skill safe to install?

  • Gen Agent Trust Hubpass

    This skill provides instructions for utilizing Databricks AI functions to process data in SQL and Spark. It contains a surface for indirect prompt injection because its core purpose involves processing external files (PDFs, images, Office documents) that could contain malicious instructions. References to external code and documentation are limited to official Databricks resources.

  • Socketpass

    No alerts

  • Snykwarn

    Risk: MEDIUM · 1 issue

What does this agent skill do?

Databricks AI Functions

Official Docs: https://docs.databricks.com/large-language-models/ai-functions Individual function reference: https://docs.databricks.com/sql/language-manual/sql-ref-functions-builtin

Overview

Databricks AI Functions are built-in SQL and PySpark functions that call Foundation Model APIs directly from your data pipelines — no model endpoint setup, no API keys, no boilerplate. They operate on table columns as naturally as UPPER() or LENGTH(), and are optimized for batch inference at scale.

Always prefer a task-specific function over ai_query. Reach for ai_query only when no task function fits (custom/external endpoints, multimodal, or JSON beyond ai_extract's limits). Every function below shares a baseline: DBR 15.1+ (notebooks) / 15.4 ML LTS (batch), not on SQL Warehouse Classic, and region must support AI Functions — the Prereqs column lists only what's additional.

Cost & speed — each call is an LLM inference (slow and billed per token). Run a function once per row and persist the result to a Delta table; never re-invoke it on every downstream query. In demos, avoid generating tables with millions of rows — sample the input when needed so the demo runs quickly. Materialize once, then query the cheap Delta output.

The Function column links to the in-repo deep reference (full options, schemas, examples); Docs links to the official page.

FunctionTaskInputOutputExtra prereqsDocs
ai_analyze_sentimentSentiment scoringcontent STRINGSTRING — positive/negative/neutral/mixed, or NULL—↗
ai_classifyFixed-label routingcontent STRING|VARIANT, labels (2–500), [options MAP]v2.1 VARIANT — {response:[{value:label}], error_message, metadata}—↗
ai_decideEvaluate questions (boolean / choice / score) over text or structured datastate STRING|VARIANT, questions STRING (JSON), [options MAP]VARIANT — {response:{answers}, metadata, error_message}Beta; admin-gated via Previews↗
ai_extractEntity / field extractioncontent STRING|VARIANT, schema STRING (JSON), [options MAP]v2.1 VARIANT — {response:{field:{value:…}}, error_message, metadata}≤256 fields, ≤12 nesting levels↗
ai_fix_grammarGrammar correctioncontent STRINGSTRING (corrected)—↗
ai_genFree-form generationprompt STRINGSTRING—↗
ai_maskPII redactioncontent STRING, labels ARRAY<STRING>STRING (entities → [MASKED])—↗
ai_similaritySemantic similarityexpr1 STRING, expr2 STRINGFLOAT (0.0–1.0)—↗
ai_summarizeSummarizationcontent STRING, [max_words INT] (0 = uncapped)STRINGPublic Preview; English-tuned↗
ai_translateTranslationcontent STRING, to_lang STRINGSTRINGLangs: en, fr, de, hi, it, pt, es, th↗
ai_parse_documentParse PDF / Office / imagescontent BINARY, [Map('version','2.0', …)]VARIANT — pages, elements, error_statusDBR 17.3+; ≤500 pages / 100 MB↗
ai_prep_searchRAG chunking from parsed docs or textparsed VARIANT|STRING, [options MAP]VARIANT — {document:{contents, pages, source_uri}, error_status}DBR 18.2+ (serverless env v3+)↗
ai_queryHosted model service or custom serving endpoint, multimodal, complex JSON (last resort)endpoint STRING, request STRING|STRUCT, [returnType], [failOnError BOOL], [modelParameters STRUCT], [responseFormat STRING], [files]Parsed response; with failOnError => false a STRUCT{response, errorMessage}Pro/Serverless warehouse; EXECUTE on model service or CAN QUERY on serving endpoint↗
ai_forecastTime series forecasting (table-valued)observed TABLE, horizon, time_col, value_col, [group_col], [prediction_interval_width], [frequency], [seed], [parameters]Rows: time/group cols + per value {v}_forecast, {v}_upper, {v}_lower (DOUBLE)Pro/Serverless warehouse; Public Preview↗

Models run under Apache 2.0 or LLAMA 3.3 Community License — you are responsible for compliance.

Patterns

Pin extraction and classification to v2.1 in new code. Every extracted scalar and returned label has a value field, even when citations, confidence scores, and rationales are disabled. Existing v2.0 queries remain supported; update downstream projections to read value when upgrading. See the task-function reference for the version-specific contracts.

Chain task functions to enrich a column in one pass. ai_classify/ai_extract return a VARIANT — read it with the colon operator (:response):

SELECT id,
  ai_analyze_sentiment(content)                                                   AS sentiment,
  ai_summarize(content, 30)                                                       AS summary,
  ai_classify(content, '["technical","billing","other"]', map('version','2.1')):response[0].value::STRING AS category,
  ai_extract(content, '["product","error_code","date"]', map('version','2.1')):response:product:value::STRING AS product,
  ai_fix_grammar(content)                                                         AS content_clean
FROM raw_feedback;

In PySpark, call any of these inside expr(...): df.withColumn("category", expr("ai_classify(content, '[\"a\",\"b\"]', map('version','2.1')):response[0].value::STRING")) — and read extracted scalar fields via selectExpr("col:response:field:value::STRING AS field").

PII redaction before storage — ai_mask(content, ARRAY(entity_types)) returns text with entities → [MASKED].

SELECT ai_mask(message, array('person','email','phone','address')) AS message_safe FROM raw_messages;

Semantic matching / dedup — ai_similarity returns 0–1; self-join and threshold:

SELECT a.id, b.id, ai_similarity(a.name, b.name) AS score
FROM companies a JOIN companies b ON a.id < b.id
WHERE ai_similarity(a.name, b.name) > 0.85;

Forecasting — table-valued; one row per future period (+ per group). Full param/group/interval forms → 3-ai-forecast.md:

SELECT * FROM ai_forecast(
    observed => TABLE(SELECT date, sales FROM daily_sales),
    horizon => '2026-12-31', time_col => 'date', value_col => 'sales');
-- Returns: date, sales_forecast, sales_upper, sales_lower

Nested field extraction — ai_extract supports arrays of objects, including invoice line items. See extraction schemas and versioned outputs.

Document parsing (ai_parse_document) and RAG chunking (ai_prep_search) get their own staged pipeline below.

Document Processing Pipeline

Chain AI Functions stage-by-stage into Delta tables for batch document processing. The example is written as a Spark Declarative Pipeline (SDP / Lakeflow / DLT) — CREATE OR REFRESH STREAMING TABLE with STREAM(...) sources. To run the same logic standalone in a notebook / SQL warehouse, swap each CREATE OR REFRESH STREAMING TABLE x AS for CREATE OR REPLACE TABLE x AS and drop the STREAM(...) wrappers. In SDP Python it's @dp.table with from pyspark import pipelines as dp.

Pass the parsed VARIANT directly into extraction and classification to preserve layout and source context. For plain-text or Markdown files, skip parsing: use read_files(..., format => 'text', wholetext => true) and pass its value column. Without wholetext => true, text input is read one line per row.

-- Stage 1 — parse supported PDF, image, or Office documents, filter parse errors
CREATE OR REFRESH STREAMING TABLE raw_parsed AS
SELECT path,
  parsed AS parsed_content,
  parsed:error_status AS parse_error
FROM (
  SELECT path, ai_parse_document(content, map('version','2.0')) AS parsed
  FROM STREAM read_files('/Volumes/my_catalog/doc_processing/landing/', format => 'binaryFile')
)
WHERE is_variant_null(parsed:error_status);

-- Stage 2 — classify document type (cheap, no endpoint selection)
CREATE OR REFRESH STREAMING TABLE classified_docs AS
SELECT *,
  ai_classify(parsed_content, '["invoice","purchase_order","receipt","contract","other"]', map('version','2.1')):response[0].value::STRING AS doc_type
FROM STREAM raw_parsed;

-- Stage 3 — extract fields; ai_extract returns a VARIANT, read fields with `:`
CREATE OR REFRESH STREAMING TABLE extracted AS
SELECT path, doc_type,
  result:response:invoice_number:value::STRING AS invoice_number,
  result:response:vendor_name:value::STRING    AS vendor_name,
  result:response:total_amount:value::DOUBLE   AS total_amount,
  result:error_message::STRING           AS extract_error
FROM (
  SELECT *, ai_extract(parsed_content,
    '{"invoice_number":{"type":"string"},"vendor_name":{"type":"string"},"total_amount":{"type":"number"}}',
    map('version','2.1')) AS result
  FROM STREAM classified_docs WHERE doc_type = 'invoice' AND parsed_content IS NOT NULL
);

In a batch job, retain per-row errors for inspection or route them to a sidecar table: check ai_extract's result:error_message with is_variant_null for JSON null on success, and for ai_query pass failOnError => false and check ai_response.errorMessage (a STRUCT field, dot-accessed). SQL IS NULL does not recognize a JSON null inside VARIANT. See 2-ai-query.md.

Custom RAG Pipeline — Parse → Prep → Index

For retrieval rather than field extraction: ai_parse_document → ai_prep_search (semantic chunking + context enrichment, DBR 18.2+) → Vector Search Delta Sync index. ai_prep_search returns chunk_id, chunk_to_retrieve, and chunk_to_embed (enriched with title/headers/page) — embed chunk_to_embed, return chunk_to_retrieve to the LLM. Shown standalone; in an SDP swap CREATE OR REPLACE TABLE for CREATE OR REFRESH STREAMING TABLE + STREAM read_files(...).

CREATE OR REPLACE TABLE parsed_chunks AS
WITH prepped AS (
  SELECT path AS source_path, ai_prep_search(ai_parse_document(content, map('version','2.0'))) AS prep
  FROM read_files('/Volumes/my_catalog/doc_processing/docs/', format => 'binaryFile')
)
SELECT
  variant_get(chunk, '$.chunk_id',          'STRING') AS chunk_id,
  variant_get(chunk, '$.chunk_to_retrieve', 'STRING') AS chunk_to_retrieve,
  variant_get(chunk, '$.chunk_to_embed',    'STRING') AS chunk_to_embed,
  source_path
FROM prepped LATERAL VIEW explode(variant_get(prep, '$.document.contents', 'ARRAY<VARIANT>')) c AS chunk;

Then enable CDF (ALTER TABLE parsed_chunks SET TBLPROPERTIES (delta.enableChangeDataFeed = true)) and use the databricks-vector-search skill to build a Delta Sync index: PK chunk_id, embedding source chunk_to_embed, return chunk_to_retrieve.

Beyond batch:

  • Ask questions over the output — point a Genie Agent at the resulting Delta table for natural-language querying instead of hand-written SQL; see the databricks-genie-agents skill.
  • Low-latency / serving — to expose this as a real-time, governed endpoint (e.g. register a model to Unity Catalog and serve it), use the databricks-model-serving skill.
  • Production incremental ingestion — for a runnable end-to-end streaming ai_parse_document job (checkpoints, trigger(availableNow=True)), see databricks/bundle-examples · job_with_ai_parse_document.

Reference Files

  • 1-task-functions.md — Deep reference for every task-specific function: full options/schemas (e.g. ai_extract v2.1 citations + confidence scores, ai_classify multilabel, ai_parse_document options + output schema, ai_prep_search chunk schema) and non-trivial examples. The Overview table above links to each function's section directly.
  • 2-ai-query.md — ai_query complete reference: all parameters, structured output with responseFormat, multimodal files =>, UDF patterns, and error handling
  • 3-ai-forecast.md — ai_forecast parameters, single-metric, multi-group, multi-metric, and confidence interval patterns

Common Issues

IssueSolution
ai_parse_document not foundRequires DBR 17.3+. Check cluster runtime.
ai_prep_search not foundRequires DBR 18.2+ (serverless env v3+).
ai_decide not found / unavailableBeta — a workspace admin must enable it on the Previews page. Requires Databricks SQL (not SQL Classic) or DBR 15.4 LTS+.
explode() fails on a VARIANTexplode needs ARRAY — cast first: explode(variant_get(prep, '$.document.contents', 'ARRAY<VARIANT>')).
Embedding the wrong RAG columnEmbed chunk_to_embed (context-enriched); return chunk_to_retrieve to the LLM.
ai_forecast failsRequires Pro or Serverless SQL warehouse — not available on Classic or Starter.
All functions return NULLInput column is NULL. Filter with WHERE col IS NOT NULL before calling.
ai_translate fails for a languageSupported (8): English (en), French (fr), German (de), Hindi (hi), Italian (it), Portuguese (pt), Spanish (es), Thai (th) — to_lang takes the code or the full name (see references/1-task-functions.md). Use ai_query with a multilingual model for others.
ai_classify returns unexpected labelsUse clear, mutually exclusive label names. Fewer labels (2–5) produces more reliable results.
ai_query raises on some rows in a batch jobAdd failOnError => false — returns a STRUCT with .response and .errorMessage (dot-accessed) instead of raising.
ai_query fails with 403 "no longer available"Discover the exact replacement system.ai model service using the AI Gateway listing; legacy names do not always have a matching model service. Custom serving endpoints keep their existing names. See 2-ai-query.md.
Batch job runs slowlyUse DBR 15.4 ML LTS cluster (not serverless or interactive) for optimized batch inference throughput.

Add the canonical catalog link to the repository README so users can inspect current installs and available audits. The publishing guide covers the complete discovery path.

<a href="https://skillzs.dev/skills/databricks/databricks-agent-skills/databricks-ai-functions">View databricks-ai-functions on skillZs</a>