Data Engineering Agent Skills

Data engineering workflows for your agent: spec, plan, build, validate and ship pipelines, with safe backfills and quality gates.

Pack · 74 skills

About

Ratings and reviews

What Data Engineering Agent Skills does

About

Data Engineering Agent Skills is a community skill pack by Viquar Khan that gives your agent operating procedures for data engineering, rather than generic prompts. Each skill sets out when to use it, a step-by-step workflow, red flags and how to verify the result. What you can do with it - Work through a spec-first lifecycle: define the data product, plan small tasks, build in slices, validate, review, backfill safely and ship - Design orchestration in Airflow, MWAA, Cloud Composer, Azure Data Factory, Step Functions or Databricks Workflows, with explicit retries and backfills - Build ingestion, CDC with Debezium and Kafka Connect, streaming with Kafka and Spark, and lakehouse tables in Delta, Iceberg or Hudi - Model warehouses and tune their cost on BigQuery, Snowflake, ClickHouse, Trino and DuckDB, and run analytics engineering with dbt - Put quality and contracts in place with Great Expectations, Deequ, schema registries and reconciliation checks - Handle governance and compliance: lineage, PII, retention and right to delete, GDPR and data residency, Unity Catalog, Purview, Dataplex and Lake Formation - Plan migrations, mainframe offload, incident triage and disaster recovery for data platforms How it works - Start with using-data-engineering-agent-skills, which classifies the work, picks a platform preset and suggests the safest next command - Platform presets cover AWS, Azure, GCP, Databricks, Snowflake, Alibaba Cloud, Informatica, Talend and Apache-first stacks, with starter packs, templates and example scaffolds - Install from the repo with scripts/install.sh, or as the maker's VS Code or JetBrains extension Good to know - The skills are guidance and workflows; your agent still runs the tools and platforms you already use - MIT licensed

Things to ask it

Once it’s added

“Write a spec for this new orders data product: contract, SLA, lineage and retention”
“Plan a safe backfill of the last 90 days without double-writing”
“Design the Airflow DAG for this ingestion, with retries and SLAs”
“Add data quality checks and a contract test before we publish this table”
“Set up CDC from Postgres with Debezium and Kafka Connect”
“Review this pipeline change for reliability, governance and cost”

What it can do

Before you add it

Reads only

Your pipeline code, schemas and platform configuration

Makes changes · your agent can ask first

Specs, plans, pipeline code and quality checks in your project

Skills in Data Engineering Agent Skills

Pack · 74 skills, each with its own page

airflow-and-workflow-orchestrationSecurity scan passedGuides agents through workflow orchestration design and operation across Airflow-style DAGs, cloud-native schedulers, a…
apache-beam-unified-batch-and-streamSecurity scan passedGuides agents through Apache Beam pipelines that unify batch and streaming logic.
apache-hudi-lakehouseSecurity scan passedGuides agents through Apache Hudi lakehouse design.
api-and-saas-ingestion-patternsSecurity scan passedGuides agents through API and SaaS ingestion workflows.
avro-protobuf-json-schema-registrySecurity scan passedGuides agents through schema-registry-backed event contracts.
bigquery-and-dataform-platform-engineeringSecurity scan passedGuides agents through BigQuery- and Dataform-centered data engineering workflows.
cdc-and-incremental-loadingSecurity scan passedGuides agents through change data capture and incremental load design.
clickhouse-real-time-analyticsSecurity scan passedGuides agents through ClickHouse-based real-time analytics design.
data-catalog-and-discoverySecurity scan passedGuides agents through data catalog, discovery, and metadata quality workflows.
data-contract-testing-with-schema-registrySecurity scan passedGuides agents through data-contract testing using schema registries and compatibility checks.
data-lake-and-zone-architectureSecurity scan passedGuides agents through data lake and zone architecture design.
data-mesh-and-domain-oriented-designSecurity scan passedGuides agents through domain-oriented data product and data mesh design.
data-migration-and-platform-cutoverSecurity scan passedGuides agents through data migration and platform cutover workflows.
data-observability-and-sla-managementSecurity scan passedGuides agents through data observability and service-level management.
data-platform-ci-cd-and-release-managementSecurity scan passedGuides agents through CI/CD and release management for data platforms.
data-platform-disaster-recovery-and-business-continuitySecurity scan passedGuides agents through disaster recovery and business continuity planning for data platforms.
data-platform-operating-model-and-service-ownershipSecurity scan passedGuides agents through data platform operating model and ownership design.
dataplex-and-bigquery-governanceSecurity scan passedGuides agents through GCP-native data governance workflows with Dataplex and BigQuery.
data-quality-and-contract-testingSecurity scan passedDrives data implementation with contracts, assertions, and validation evidence.
data-quality-platforms-and-rule-managementSecurity scan passedGuides agents through data-quality operating models and tool selection.
data-reconciliation-and-financial-controlsSecurity scan passedGuides agents through reconciliation and control design for business-critical data.
data-resiliency-testing-and-failure-injectionSecurity scan passedGuides agents through resiliency testing for data platforms.
data-security-compliance-and-regulated-dataSecurity scan passedGuides agents through regulated-data security and compliance workflows for PII, PCI, HIPAA, PHI, and similar obligation…
data-sharing-and-publishing-contractsSecurity scan passedGuides agents through publishing data products for internal or external consumers.
data-specificationSecurity scan passedCreates structured specifications for data products and pipeline changes.
dbt-and-analytics-engineeringSecurity scan passedGuides agents through analytics engineering workflows with dbt.
debezium-and-kafka-connect-cdcSecurity scan passedGuides agents through Debezium and Kafka Connect CDC workflows.
delta-lake-and-medallion-architectureSecurity scan passedGuides agents through Delta Lake and medallion-style lakehouse design.
duckdb-local-analytics-and-devSecurity scan passedGuides agents through DuckDB-based local analytics and development workflows.
enterprise-etl-and-data-integration-modernizationSecurity scan passedGuides agents through operating, hardening, and modernizing enterprise ETL and integration stacks such as Informatica,…
esg-and-sustainability-regulatory-reportingSecurity scan passedGuides agents through ESG, sustainability, and regulatory reporting data products.
etl-elt-and-modernization-strategySecurity scan passedGuides agents through ETL, ELT, and transformation-modernization decisions.
feature-store-and-ml-data-pipelinesSecurity scan passedGuides agents through machine-learning data pipelines and feature serving workflows.
file-and-partner-feed-ingestionSecurity scan passedGuides agents through file-based and partner-feed ingestion workflows.
glue-data-catalog-and-lake-formation-governanceSecurity scan passedGuides agents through AWS-native data catalog and lake governance workflows.
great-expectations-deequ-and-cualleeSecurity scan passedGuides agents through data-quality frameworks such as Great Expectations, Deequ, and Cuallee.
incident-triage-and-pipeline-recoverySecurity scan passedGuides agents through production data incidents.
java-data-engineering-and-integration-servicesSecurity scan passedGuides agents through Java-based data engineering services and processors.
kafka-resilience-and-schema-evolutionSecurity scan passedEnforces production Kafka guardrails including non-breaking schema evolution, dead-letter queues for poison messages, a…
lakefs-and-data-versioningSecurity scan passedGuides agents through data versioning workflows using lakeFS or similar systems.
lakehouse-table-format-engineeringSecurity scan passedGuides agents through lakehouse table design and open table format decisions.
lineage-pii-and-governanceSecurity scan passedApplies governance, lineage, ownership, and sensitive-data controls to data changes.
lower-environment-data-masking-and-obfuscationSecurity scan passedGuides agents through masking, obfuscating, and safely promoting production-like data into lower environments.
mainframe-modernization-and-data-offloadSecurity scan passedGuides agents through mainframe data modernization and offload workflows.
master-data-and-entity-resolutionSecurity scan passedGuides agents through master data and entity resolution workflows.
mcp-data-observability-integrationSecurity scan passedGuides agents to wire Model Context Protocol servers for live data platform observability including Spark execution pla…
microsoft-purview-and-azure-data-governanceSecurity scan passedGuides agents through Microsoft Purview and Azure-native data governance workflows.
notebook-to-production-hardeningSecurity scan passedGuides agents through converting exploratory notebooks into production-ready data jobs.
openmetadata-datahub-and-openlineageSecurity scan passedGuides agents through metadata platform and lineage workflows using OpenMetadata, DataHub, or OpenLineage-compatible sy…
operational-datastore-selection-relational-and-nosqlSecurity scan passedGuides agents through choosing relational operational stores such as MySQL versus NoSQL options such as document, key-v…
orchestration-and-backfillsSecurity scan passedDesigns scheduling, reruns, and backfills safely for data systems.
pipeline-planning-and-task-breakdownSecurity scan passedBreaks approved data specifications into safe, verifiable implementation tasks.
privacy-retention-and-right-to-deleteSecurity scan passedGuides agents through privacy, retention, and deletion workflows in data systems.
python-data-engineering-and-pipeline-packagingSecurity scan passedGuides agents through Python-based data engineering implementation.
regional-data-compliance-and-sovereigntySecurity scan passedGuides agents through region-specific data compliance, residency, sovereignty, and transfer design.
reverse-etl-and-operational-data-servingSecurity scan passedGuides agents through reverse ETL and operational data serving workflows.
safe-backfill-and-replay-orchestrationSecurity scan passedForces replay-safe rollout plans, reconciliation gates, and rollback paths before executing any data backfill or histor…
scala-data-engineering-on-jvm-runtimesSecurity scan passedGuides agents through Scala-based data engineering on JVM runtimes.
schema-evolution-and-contract-migrationsSecurity scan passedGuides agents through schema changes and contract migrations.
semantic-layer-and-metric-governanceSecurity scan passedGuides agents through semantic layer and shared metric design.
snowflake-native-pipelines-and-governanceSecurity scan passedGuides agents through Snowflake-native pipeline and governance workflows.
source-reliability-and-extraction-resilienceSecurity scan passedGuides agents through source reliability and extraction resilience.
spark-and-distributed-processingSecurity scan passedGuides agents through batch and distributed data processing design.
spark-serverless-reliability-and-state-managementSecurity scan passedEnforces timeout-aware rollbacks, resumable checkpoints, and orphan cleanup for serverless Spark workloads on AWS Lambd…
streaming-and-messaging-systemsSecurity scan passedGuides agents through event streaming and real-time data pipeline design.
superset-and-metrics-servingSecurity scan passedGuides agents through Superset and metrics-serving workflows.
terraform-and-data-platform-infrastructureSecurity scan passedGuides agents through infrastructure as code for data platforms.
test-data-preparation-and-synthetic-dataSecurity scan passedGuides agents through preparing test data, masked fixtures, and synthetic datasets for development, QA, and lower envir…
trino-presto-federated-querySecurity scan passedGuides agents through Trino and Presto federated query design.
unity-catalog-and-lakehouse-governanceSecurity scan passedGuides agents through Unity Catalog governance workflows for Databricks lakehouses.
using-data-agent-skillsSecurity scan passedProvides a compatibility alias for the main data engineering entry skill.
using-data-engineering-agent-skillsSecurity scan passedHelps agents classify data engineering work, choose the right preset and skill bundle, and pick the safest next command.
warehouse-and-schema-designSecurity scan passedGuides agents through data warehouse and schema design.
warehouse-performance-and-cost-optimizationSecurity scan passedGuides agents through warehouse performance and cost decisions.

Install Data Engineering Agent Skills

Your agent

Runs on your machine.

In ChatGPTDownload the data-engineering-agent-skills folder and zip it, then Plugins → Skills → Create → Upload from your computer. Business and Enterprise plans.

View the skills on GitHub

Runs on your machine.