Workload

ETL / pipeline migration

ETL / pipeline migration

ETL / pipeline migration

Workload scope

ETL and pipeline migration covers scheduled transformation logic, incremental loads, orchestration dependencies, restart behavior, and late-arriving data handling across source-to-target migrations.

Use this hub to find the source-to-target pages that already include workload-specific conversion risks, supported patterns, validation gates, examples, and cutover criteria. The hub is intentionally tied to the migrated child-page corpus so missing child content remains visible during parity review.

  • Databricks to BigQuery - Move Databricks/Delta pipelines to BigQuery with parity for incremental loads, MERGE/upserts, and reprocessing windows, validated with reconciliation gates.
  • Databricks to Snowflake - Move Databricks/Delta ETL pipelines to Snowflake with parity for incremental loads, MERGE/upserts and reprocessing windows, validated with reconciliation gates.
  • Hadoop legacy cluster to BigQuery - Move Hadoop ETL (Hive/Impala/Spark/Oozie) to BigQuery with parity for partition processing, incremental semantics, and restartability with reconciliation gates.
  • Hive to BigQuery - Move Hive ETL pipelines to BigQuery with parity for incrementals, dedupe rules, and orchestration, validated with reconciliation and pruning governance.
  • Impala to BigQuery - Move Impala/Hadoop ETL pipelines to BigQuery with parity for partition loads, incremental semantics, and orchestration, validated with reconciliation gates.
  • Impala to Snowflake - Move Impala ETL pipelines to Snowflake with parity for partition windows, dedupe, orchestration, and reconciliation gates to prevent drift and credit spikes.
  • Netezza to BigQuery - Move Netezza ETL pipelines to BigQuery with parity for incrementals, upsert behavior, and orchestration, validated with reconciliation and pruning governance.
  • Oracle to BigQuery - Move Oracle ETL pipelines to BigQuery with parity for incremental loads, CDC/apply semantics, and data quality controls, validated with reconciliation gates.
  • Redshift to BigQuery - Migrate Redshift ETL to BigQuery with parity for incremental loads, MERGE/UPSERT patterns, orchestration, and cost governance, validated with cutover gates.
  • Redshift to Snowflake - Move Redshift ETL pipelines to Snowflake with parity for incremental loads, staging/upsert patterns, orchestration, and reconciliation gates with stable credit.
  • Snowflake to BigQuery - Migrate Snowflake ETL pipelines to BigQuery with parity for incremental loads, SCD logic, orchestration, and cost control, validated by reconciliation gates.
  • Spark SQL to BigQuery - Move Spark SQL pipelines to BigQuery with parity for incremental loads, dedupe rules, late-data windows, and orchestration with reconciliation gates.
  • Teradata to BigQuery - Move Teradata ETL pipelines to BigQuery with parity for incremental loads, restartability, SCD logic, and orchestration, validated with reconciliation gates.
  • Teradata to Snowflake - Move Teradata ETL pipelines to Snowflake with parity for incremental loads, staging/MERGE patterns, orchestration, and reconciliation gates with stable credits.
  • Vertica to BigQuery - Move Vertica ETL pipelines to BigQuery with parity for incrementals and MERGE/upserts—validated with reconciliation gates and pruning-aware governance.

Review points

  • Incremental and full-refresh semantics
  • Late-arriving data and rerun behavior
  • Scheduler, dependency, and ownership changes
  • Idempotency, dedupe, and restart controls

Acceptance criteria

A etl / pipeline migration page is complete when the migration team can identify source assets, understand conversion assumptions, review unsupported or ambiguous constructs, run validation checks, and decide whether the workload is ready for cutover.

For each related path, confirm that the child page explains what changes, how conversion works, which patterns are supported, what can drift, and which evidence is required before production traffic moves to the target platform.