databricks-execution-compute
Execute code and manage compute on Databricks: run Python/Scala/SQL/R via serverless, classic, or interactive clusters, and create/resize/delete clusters and SQL warehouses.
How do I install this agent skill?
npx skills add https://github.com/databricks/databricks-agent-skills --skill databricks-execution-computeIs this agent skill safe to install?
- Gen Agent Trust Hubpass
The skill provides a set of instructions and patterns for managing Databricks compute resources and executing code via the Databricks CLI. It includes legitimate workflows for local Spark execution using Databricks Connect, as well as remote execution via Serverless Jobs and Interactive Clusters.
- Socketpass
No alerts
- Snykpass
Risk: LOW · No issues
What does this agent skill do?
Databricks Execution & Compute
Run code on Databricks. Four execution modes—choose based on workload. Most examples below use the Databricks CLI (Databricks Sandbox is SDK-driven); see the databricks-core skill for install and authentication.
Execution Mode Decision Matrix
| Aspect | Databricks Connect ⭐ | Serverless Job | Interactive Cluster | Databricks Sandbox |
|---|---|---|---|---|
| Use for | Spark code (ETL, data gen) | Heavy processing (ML) | State across tool calls, Scala/R | Untrusted / agent-generated code, isolated env |
| Startup | Instant | ~25-50s cold start | ~5min if stopped | ~1s (warm start) |
| State | Within Python process | None | Via context_id | Home dir persisted across stop/restart |
| Languages | Python (PySpark) | Python, SQL | Python, Scala, SQL, R | Any (general Linux shell) |
| Dependencies | withDependencies() | CLI with environments spec | Install on cluster | Install in sandbox, or spec.environment.image_uri (UC container artifact) for a custom image |
Decision Flow
Main decision point: if you're using Declarative Automation Bundles (DABs) then follow the instructions of the databricks-dabs skill first. In short, you can use databricks bundle run to run code associated with jobs, pipelines, and other resources. This can be recognized by looking for a databricks.yml file in the project root. If these resources don't exist, or if you're not using DABs, then proceed with the below.
Prefer Databricks Connect for all spark-based workload, then serverless.
Spark-based code? → Databricks Connect (fastest)
└─ Python 3.12 missing? → Install it + databricks-connect
└─ Install fails? → Ask user (don't auto-switch modes)
Heavy/long-running (ML)? → Serverless Job (independent)
Need state across calls? → Interactive Cluster (list and ask which one to use)
Scala/R? → Interactive Cluster (list and ask which one to use)
Untrusted / agent-generated code, or need an isolated Linux env? → Databricks Sandbox
How to Run Code
Read the reference file for your chosen mode before proceeding.
Databricks Connect (run locally, prefer when it's pure spark code) → reference
from databricks.connect import DatabricksSession
...
spark = DatabricksSession.builder.profile("my-local-profile").serverless(True).getOrCreate()
python my_spark_script.py
Serverless Job → reference
Pure CLI flow: upload a local file as a workspace notebook, fire a one-time run with databricks jobs submit (create + run in one call, ephemeral — no Jobs UI entry, no retry), then poll + fetch the result. The local file must be a Databricks source notebook — top line # Databricks notebook source (Python) or -- Databricks notebook source (SQL).
1. Upload the local file as a workspace notebook. TARGET_PATH is positional; --file is the local path.
databricks workspace import /Workspace/Users/<user>/.ai_dev_kit/train --file /local/path/to/train.py --format SOURCE --language PYTHON --overwrite
2. Submit the run. Use --no-wait to get {"run_id": N} back immediately; drop it to block until terminated. "client": "4" is required for dependencies to install ("1" silently ignores them).
databricks jobs submit --no-wait --json @submit.json
{
"run_name": "train-run",
"tasks": [{
"task_key": "main",
"notebook_task": {"notebook_path": "/Workspace/Users/<user>/.ai_dev_kit/train"},
"environment_key": "ml_env"
}],
"environments": [{
"environment_key": "ml_env",
"spec": {"client": "4", "dependencies": ["scikit-learn==1.5.2", "mlflow==2.22.0"]}
}]
}
3. Check state / wait for completion. Life-cycle: PENDING → RUNNING → TERMINATED (or SKIPPED / INTERNAL_ERROR). Only read .state.result_state (SUCCESS / FAILED / CANCELED) once life-cycle is TERMINATED.
databricks jobs get-run <RUN_ID> | jq '{state: .state.life_cycle_state, result: .state.result_state, duration_ms: .execution_duration, url: .run_page_url, task_run_id: .tasks[0].run_id}'
4. Fetch the output / error. Gotcha: get-run-output takes the task run_id (.tasks[0].run_id), NOT the parent run_id from submit. notebook_output.result is the string passed to dbutils.notebook.exit().
databricks jobs get-run-output <TASK_RUN_ID> | jq '{result: .notebook_output.result, error, error_trace}'
Always use dbutils.notebook.exit(<string>) in the notebook — print() is not captured by get-run-output. For JSON results: dbutils.notebook.exit(json.dumps({...})) then parse .notebook_output.result client-side.
Interactive Cluster → reference
Avoid by default — prefer Serverless Job. Only use an interactive cluster when:
- you have an existing classic cluster already running and available, or
- you need live, stateful execution across multiple calls (debugging via an execution context), or
- the user explicitly asks for it.
Interactive clusters are slow to start (3-8 min) and cost money while running. Don't start one implicitly.
Databricks Sandbox → skill
Isolated serverless environment for running untrusted or agent-generated code, driven through the Databricks SDK. Prefer it for agent tool-execution and ephemeral build/test environments rather than Spark workloads. See the databricks-sandbox skill for lifecycle, command execution, and file operations.
CLI Command Map
All compute lifecycle and code-execution actions go through the Databricks CLI. Headline commands:
| Action | Command |
|---|---|
| Upload local file as workspace notebook | databricks workspace import <WORKSPACE_PATH> --file <LOCAL> --format SOURCE --language PYTHON --overwrite |
| Run serverless code (upload + submit + wait) | databricks jobs submit --json @submit.json (see Serverless Job section above; with --no-wait for async) |
| Get run state / wait | databricks jobs get-run <RUN_ID> (poll .state.life_cycle_state) |
| Fetch run output | databricks jobs get-run-output <TASK_RUN_ID> |
| List clusters | databricks clusters list --output json |
| Get cluster details | databricks clusters get <CLUSTER_ID> |
| Start / restart / terminate cluster | databricks clusters start/restart/delete <CLUSTER_ID> |
| Permanently delete cluster | databricks clusters permanent-delete <CLUSTER_ID> |
| Create cluster | databricks clusters create --json '{...}' (see 3-interactive-cluster.md) |
| List node types / Spark versions | databricks clusters list-node-types / databricks clusters spark-versions |
| Execute code on a running cluster | databricks api post /api/1.2/contexts/create + databricks api post /api/1.2/commands/execute (see 3-interactive-cluster.md) |
| SQL warehouses | databricks warehouses create/list/get/start/stop/edit/delete (see SQL Warehouses below) |
SQL Warehouses
All ID-taking commands use positional arg (no --id flag). Use databricks warehouses list to find an ID.
# Create a serverless SQL warehouse. min_num_clusters + max_num_clusters are REQUIRED
# (the server rejects the default 0). Keep the aidevkit_project tag for resource tracking.
databricks warehouses create --json '{
"name": "my-warehouse",
"cluster_size": "Small",
"enable_serverless_compute": true,
"auto_stop_mins": 10,
"min_num_clusters": 1,
"max_num_clusters": 1,
"tags": {"custom_tags": [{"key": "aidevkit_project", "value": "ai-dev-kit"}]}
}'
# List / find — trim to id, name, state with jq
databricks warehouses list -o json | jq '.[] | {id, name, state, size: .cluster_size}'
# Find by name
databricks warehouses list -o json | jq '.[] | select(.name == "my-warehouse")'
# Get one warehouse's full config
databricks warehouses get <WAREHOUSE_ID>
# Start / stop (both are LROs; add --no-wait to return immediately)
databricks warehouses start <WAREHOUSE_ID>
databricks warehouses stop <WAREHOUSE_ID>
# Resize / reconfigure — pass the FULL desired config (omitted fields revert to defaults,
# so always re-state min_num_clusters/max_num_clusters). Use --no-wait if the warehouse
# is STOPPED, otherwise edit blocks trying to reach RUNNING and errors out (the mutation
# itself still applies). When the warehouse is already RUNNING, --no-wait is optional.
databricks warehouses edit <WAREHOUSE_ID> --no-wait --json '{
"name": "my-warehouse",
"cluster_size": "Medium",
"enable_serverless_compute": true,
"auto_stop_mins": 15,
"min_num_clusters": 1,
"max_num_clusters": 1
}'
# Delete (irreversible)
databricks warehouses delete <WAREHOUSE_ID>
Sizes: 2X-Small, X-Small, Small, Medium, Large, X-Large, 2X-Large, 3X-Large, 4X-Large. Types: set "warehouse_type": "PRO" (default) or "CLASSIC" in the JSON body.
Related Skills
- databricks-synthetic-data-gen — Data generation using Spark + Faker
- databricks-jobs — Production job orchestration
- databricks-dbsql — SQL warehouse and AI functions
- databricks-sandbox — Isolated serverless sandboxes for untrusted / agent-generated code
How can the creator link this skill?
Add the canonical catalog link to the repository README so users can inspect current installs and available audits. The publishing guide covers the complete discovery path.
<a href="https://skillzs.dev/skills/databricks/databricks-agent-skills/databricks-execution-compute">View databricks-execution-compute on skillZs</a>