skillZs
★ LIVE SKILL TAGS ★
>>> LIVE SKILLS INDEX <<<
* OPEN SOURCE *
NO LOGIN, NO TRACKING
※ REAL INSTALL DATA ※
← back to all skills
alexanderstephenthompson/claude-hub48 installs

data-python

Python patterns for data processing - pandas, polars, pyspark

How do I install this agent skill?

npx skills add https://github.com/alexanderstephenthompson/claude-hub --skill data-python
view source ↗

Is this agent skill safe to install?

  • Gen Agent Trust Hubpass

    This skill is a documentation-only resource providing best practices for data processing in Python. It contains no executable scripts, malicious commands, or security vulnerabilities. It focuses on memory efficiency, vectorization, and schema validation using standard libraries like pandas and polars.

  • Socketpass

    No alerts

  • Snykpass

    Risk: LOW · No issues

  • Runlayerpass

    4 files scanned · No issues

  • ZeroLeakspass

    Score: 93/100 · 2 sections analyzed

What does this agent skill do?

Data Python Skill

Version: 1.0 Stack: Python (pandas, polars, pyspark)

Python makes it easy to write data processing code that works on sample data and fails on real data. iterrows() takes 30 seconds on 10K rows and 30 minutes on 10M. A DataFrame without explicit dtypes uses 8x the memory it needs. Chained indexing creates silent copies that lose your changes. These aren't edge cases — they're the default behavior of pandas when you write it like regular Python.

Vectorized operations, explicit schemas, and proper dtypes mean your code scales from prototype to production without rewriting.


Scope and Boundaries

This skill covers:

  • DataFrame patterns (pandas, polars, pyspark)
  • Data type handling and validation
  • Memory-efficient processing
  • Vectorized operations over loops
  • Method chaining patterns
  • Error handling for data pipelines

Defers to other skills:

  • code-quality: General code structure, testing, naming
  • data-sql: Query patterns when using SQL interfaces
  • data-pipelines: Orchestration and ETL architecture

Use this skill when: Writing Python code that processes data.


Core Principles

  1. Vectorize, Don't Loop — Use DataFrame operations, not row iteration.
  2. Fail Fast on Bad Data — Validate early, reject invalid data at boundaries.
  3. Memory Awareness — Know your data size, use appropriate dtypes.
  4. Immutable Transforms — Chain operations, don't mutate in place.
  5. Explicit Schemas — Define expected columns and types upfront.

Patterns

Method Chaining

# Good - readable pipeline
result = (
    df
    .query("status == 'active'")
    .assign(total=lambda x: x["quantity"] * x["price"])
    .groupby("category")
    .agg({"total": "sum"})
    .sort_values("total", ascending=False)
)

# Bad - intermediate variables obscure flow
filtered = df[df["status"] == "active"]
filtered["total"] = filtered["quantity"] * filtered["price"]
grouped = filtered.groupby("category")
result = grouped.agg({"total": "sum"})
result = result.sort_values("total", ascending=False)

Schema Validation

EXPECTED_COLUMNS = {"id", "name", "value", "timestamp"}
REQUIRED_COLUMNS = {"id", "value"}

def validate_schema(df: pd.DataFrame) -> pd.DataFrame:
    missing = REQUIRED_COLUMNS - set(df.columns)
    if missing:
        raise ValueError(f"Missing required columns: {missing}")
    return df[list(EXPECTED_COLUMNS & set(df.columns))]

Type Optimization

def optimize_dtypes(df: pd.DataFrame) -> pd.DataFrame:
    """Downcast numeric types to reduce memory."""
    for col in df.select_dtypes(include=["int"]).columns:
        df[col] = pd.to_numeric(df[col], downcast="integer")
    for col in df.select_dtypes(include=["float"]).columns:
        df[col] = pd.to_numeric(df[col], downcast="float")
    return df

Anti-Patterns

Anti-PatternProblemFix
for row in df.iterrows()Slow, defeats vectorizationUse vectorized operations
df["col"] = df.apply(...)Usually slower than vectorizedUse np.where or df.assign
Chained indexing df["a"]["b"]SettingWithCopyWarningUse df.loc[:, "a"]
Loading entire file to check schemaWastes memoryUse nrows=100 or chunking
Ignoring dtypes on readMemory bloatSpecify dtype= parameter

Checklist

  • No iterrows() or itertuples() for computation
  • Explicit dtypes on file reads
  • Schema validation at boundaries
  • Method chaining for transforms
  • Memory profiled for large datasets

References

  • references/vectorization.md — Vectorized operations and performance
  • references/memory-optimization.md — Memory optimization techniques

Assets

  • assets/pandas-cheatsheet.md — Quick reference for pandas operations

Add the canonical catalog link to the repository README so users can inspect current installs and available audits. The publishing guide covers the complete discovery path.

<a href="https://skillzs.dev/skills/alexanderstephenthompson/claude-hub/data-python">View data-python on skillZs</a>