skillZs
★ LIVE SKILL TAGS ★
>>> LIVE SKILLS INDEX <<<
* OPEN SOURCE *
NO LOGIN, NO TRACKING
※ REAL INSTALL DATA ※
← back to all skills
alterlab-ieu/alterlab-academic-skills68 installs

alterlab-vaex

Out-of-core tabular analytics with Vaex — memory-mapped HDF5/Arrow/Parquet via vaex.open, lazy virtual columns, delayed single-pass aggregations on billion-row tables, binned histograms/heatmaps, and vaex.ml transformers on one machine. Vaex is in minimal-maintenance mode (vaex-core 4.19 from Sep 2025 supports only Python 3.9–3.12 and pandas < 3), so use it to run, fix, or migrate existing Vaex code; for new larger-than-RAM work prefer Polars lazy/streaming queries (alterlab-polars) or Dask (alterlab-dask). Use when a project already depends on Vaex, when opening memory-mapped HDF5/Arrow files, batching aggregations with delay=True, or porting a Vaex pipeline to Polars. Part of the AlterLab Academic Skills suite.

How do I install this agent skill?

npx skills add https://github.com/alterlab-ieu/alterlab-academic-skills --skill alterlab-vaex
view source ↗

Is this agent skill safe to install?

  • Gen Agent Trust Hubpass

    The skill provides comprehensive documentation and implementation patterns for the Vaex library to handle billion-row datasets. It covers out-of-core data loading, lazy evaluation, performance optimization, and machine learning integration. Standard high-performance features such as JIT compilation and cloud storage access are documented. No malicious patterns or security risks were identified beyond the expected attack surface of a data science library.

  • Socketpass

    No alerts

  • Snykpass

    Risk: LOW · No issues

What does this agent skill do?

Vaex

Overview

Vaex is a Python library for lazy, out-of-core DataFrames: it memory-maps HDF5/Arrow files, evaluates expressions lazily, and computes statistics on N-dimensional grids fast enough for interactive work on billions of rows on a single machine.

Maintenance Status (read first)

Vaex is effectively in minimal-maintenance mode. In the project's "Is Vaex active?" discussion (vaexio/vaex #2363, June 2024), one core developer stated he is no longer involved and the other committed only to "do my best to at least get vaex releases out". Releases have slowed to roughly one per year (vaex-core 4.18 in Oct 2024, 4.19 in Sep 2025), and 4.19 still supports only Python 3.9–3.12 with pandas<3; the 2-D matplotlib plots break on matplotlib ≥ 3.9. Practical consequences:

  • Existing Vaex code: keep it running in a pinned Python 3.12 environment (below), and plan a migration.
  • New projects: default to Polars (alterlab-polars) — pl.scan_parquet(...) plus collect(engine="streaming") covers most larger-than-RAM aggregation work — or Dask (alterlab-dask) for clusters. Recommend Vaex for new work only when its specific strengths (memory-mapped HDF5, fast binned statistics/visualization of huge tables) matter and the user accepts the environment constraints.

When to Use This Skill

Use this skill when the user wants to:

  • Run, debug, or extend an existing Vaex pipeline (vaex.open, virtual columns, selections, vaex.ml).
  • Open memory-mapped HDF5/Arrow files that exceed RAM and compute fast aggregations or binned statistics.
  • Batch several aggregations into one pass with delay=True / df.execute().
  • Convert large CSV to HDF5/Arrow/Parquet, or port a Vaex workflow to Polars.

Does NOT Trigger

ScenarioUse Instead
New larger-than-RAM analytics project with no existing Vaex code, or Python 3.13+ / pandas 3 environmentsalterlab-polars
Data fits in memory and the goal is the fastest in-memory groupby/joinsalterlab-polars
Multi-node cluster execution, task graphs, or a distributed scheduler dashboardalterlab-dask
Chunked N-D scientific arrays (not tables) on disk or cloud object storesalterlab-zarr

Installation

# Dedicated environment — vaex 4.19 has no wheels for Python 3.13+ and requires pandas < 3
uv venv --python 3.12 .venv-vaex
uv pip install --python .venv-vaex "vaex==4.19.*"
uv pip install --python .venv-vaex "matplotlib<3.9"   # only if you need df.viz.heatmap

Core Capabilities

Vaex provides six capability areas, each documented in references/:

1. DataFrames and Data Loading

Open HDF5/Arrow/Parquet/CSV files, convert from pandas/NumPy/Arrow, and inspect structure — references/core_dataframes.md.

2. Data Processing and Manipulation

Filtering, named selections, virtual columns and expressions, groupby/aggregations, strings, datetimes, and missing data — references/data_processing.md.

3. Performance and Optimization

Lazy evaluation, delay=True batching, materialization, caching, and async execution — references/performance.md.

4. Data Visualization

Binned histograms and heatmaps of huge tables via the df.viz accessor, selections, and widgets — references/visualization.md.

5. Machine Learning Integration

vaex.ml scalers, encoders, PCA, k-means, and scikit-learn/XGBoost/LightGBM/CatBoost wrappers — references/machine_learning.md. The booster and scikit-learn wrappers materialize the training features in memory; only the transformers and prediction columns stay lazy.

6. I/O Operations

Format recommendations, export strategies, Arrow interop, chunked CSV conversion, and remote data — references/io_operations.md.

Quick Start Pattern

import vaex

# 1. Open or create a DataFrame
df = vaex.open('large_file.hdf5')  # or .arrow, .parquet; CSV is read (and optionally converted)
# OR
df = vaex.from_pandas(pandas_df)

# 2. Explore
print(df)          # first/last rows and column info
df.describe()      # statistical summary

# 3. Virtual columns (no memory overhead)
df['new_column'] = df.x ** 2 + df.y

# 4. Filter
df_filtered = df[df.age > 25]

# 5. Statistics (lazy, computed in passes over the data)
mean_val = df.x.mean()
stats = df.groupby('category').agg({'value': 'sum'})

# 6. Visualize (df.plot1d / df.plot are deprecated aliases of these)
df.viz.histogram(df.x, limits=[0, 100])
df.viz.heatmap(df.x, df.y, limits='99.7%')   # needs matplotlib < 3.9

# 7. Export
df.export_hdf5('output.hdf5')

Best Practices

  1. Use HDF5 or Apache Arrow for memory-mapped access to large datasets; Parquet is the most portable choice if a Polars migration is likely.
  2. Prefer virtual columns to materializing data.
  3. Batch operations with delay=True when computing several statistics.
  4. Convert CSV once to HDF5/Arrow/Parquet rather than re-reading it.
  5. Check memory with df.byte_size() and inspect data with df.describe() before optimizing.
  6. Pin the environment (Python 3.12, vaex==4.19.*) and record it with results so analyses stay reproducible.

Common Patterns

Pattern: Converting Large CSV to HDF5

import vaex

# convert=True streams the CSV in chunks to an HDF5 file next to it
df = vaex.from_csv('large_file.csv', convert=True, chunk_size=5_000_000)

# Future loads are memory-mapped and near-instant
df = vaex.open('large_file.csv.hdf5')

Pattern: Efficient Aggregations

# delay=True batches several aggregations into ONE pass over the data.
# Each delayed call returns a promise; df.execute() runs them together.
mean_x = df.mean(df.x, delay=True)
std_y = df.std(df.y, delay=True)
sum_z = df.sum(df.z, delay=True)

df.execute()
print(mean_x.get(), std_y.get(), sum_z.get())

Pattern: Virtual Columns for Feature Engineering

# No memory overhead — computed on the fly
df['age_squared'] = df.age ** 2
df['full_name'] = df.first_name + ' ' + df.last_name
df['is_adult'] = df.age >= 18

Pattern: Migrating a Vaex Query to Polars

# Vaex: df = vaex.open('big.parquet'); df['z'] = df.x ** 2 + df.y
#       df[df.x > 0.5].groupby('category').agg({'z': 'mean'})
import polars as pl

result = (
    pl.scan_parquet('big.parquet')                      # lazy, like vaex.open
    .with_columns((pl.col('x') ** 2 + pl.col('y')).alias('z'))   # virtual column
    .filter(pl.col('x') > 0.5)                          # filter
    .group_by('category').agg(pl.col('z').mean())       # groupby/agg
    .collect(engine='streaming')                        # out-of-core execution
)

Polars cannot read Vaex's HDF5 layout directly — export once with df.export_parquet(...) or df.export_arrow(...). Several delay=True statistics become one select(...) in a single lazy query. See alterlab-polars for the rest of the Polars API.

Resources

  • references/core_dataframes.md — DataFrame creation, loading, and basic structure
  • references/data_processing.md — filtering, expressions, aggregations, and transformations
  • references/performance.md — optimization strategies and lazy evaluation
  • references/visualization.md — plotting and interactive visualizations
  • references/machine_learning.md — ML pipelines and model integration
  • references/io_operations.md — file formats and data import/export

Part of the AlterLab Academic Skills suite.

Add the canonical catalog link to the repository README so users can inspect current installs and available audits. The publishing guide covers the complete discovery path.

<a href="https://skillzs.dev/skills/alterlab-ieu/alterlab-academic-skills/alterlab-vaex">View alterlab-vaex on skillZs</a>