Setup & Installation
Or with the ClawHub CLI, for registry-managed skill folders outside a full OpenClaw workspace:
Version History
v2.1.1: optimization, reference splits
What This Skill Does
Covers production-grade data pipeline design, ETL/ELT implementation, and data infrastructure decisions. Handles batch and streaming architectures using Python, SQL, Spark, Airflow, dbt, and Kafka. Includes data modeling, quality validation, and DataOps practices.
Combines architecture decision frameworks with ready-to-run scripts for orchestration, quality validation, and performance analysis, covering the full modern data stack in one place.
When to use it
- Building a daily batch ETL from Postgres to Snowflake
- Setting up Airflow DAGs for scheduled data ingestion
- Validating data quality on freshness, completeness, and uniqueness
- Choosing between Lambda and Kappa pipeline architectures
- Optimizing a slow Spark job processing large parquet files
Example Workflow
Here's how your AI assistant might use this skill in practice.
User asks: Design a pipeline to ingest sales data from Postgres into Snowflake daily
- 1Evaluates latency and volume requirements to confirm batch processing is appropriate
- 2Generates an Airflow DAG config with a scheduled daily run targeting Snowflake
- 3Defines dbt models for transformation and dimensional modeling in the warehouse
- 4Adds data quality checks for freshness, completeness, and uniqueness on the sales dataset
Airflow DAG, dbt model files, and a data quality validation script ready for deployment