Workload

Validation & reconciliation for Hadoop legacy cluster → BigQuery

Turn “it runs” into a measurable parity contract. We prove correctness *and* pruning posture with golden queries, KPI diffs, and replayable integrity simulations—then gate cutover with rollback-ready criteria.

Book assessment Explore workloads

Quick answer

Turn “it runs” into a measurable parity contract. We prove correctness *and* pruning posture with golden queries, KPI diffs, and replayable integrity simulations—then gate cutover with rollback-ready criteria.

Back to pair page

Context

Why this breaks

Hadoop migrations fail late because correctness and performance were enforced by convention: always filter partitions, accept implicit casts, and rely on orchestrator scripts for reruns and backfills. BigQuery will compile many translated jobs—but drift and cost spikes appear when partition/pruning behavior, typing/NULL semantics, and time conversions aren’t made explicit and validated under stress. Common drift drivers in Hadoop legacy cluster → BigQuery:

  • Pruning contract lost: partition filters don’t translate, or filters defeat pruning → scan bytes explode
  • Implicit casts & NULL semantics: CASE/COALESCE branches and join keys behave differently
  • Window/top-N ambiguity: missing tie-breakers changes winners under parallelism
  • Epoch/time conversions: timezone intent missing → boundary-day drift
  • Operational behavior: reruns/backfills differ when overwrite/reprocessing conventions weren’t recreated Validation must treat this as an operational system and include pruning/cost posture as a first-class cutover gate.

Approach

How conversion works

  • Define the parity contract: what must match (facts/dims, KPIs, dashboards) and what tolerances apply. - Define the pruning/cost contract: which workloads must prune and what scan-byte/slot thresholds are acceptable. - Build validation datasets: golden inputs, edge cohorts (ties, null-heavy segments), and representative windows (including boundary dates). - Run readiness + execution gates: schemas/types align, dependencies deployed, and jobs run reliably. - Run layered parity gates: counts/profiles → KPI diffs → targeted row-level diffs where needed. - Validate operational integrity where applicable: idempotency reruns, restart simulations, and backfill/late-data injections. - Gate cutover: pass/fail thresholds, canary strategy, rollback triggers, and post-cutover monitors.

Coverage

Supported constructs

Representative validation and reconciliation mechanisms we apply in Hadoop legacy cluster → BigQuery migrations.

SourceTargetNotes
Golden dashboards/queriesGolden query harness + repeatable parameter setsCodifies business sign-off into runnable tests.
Partition/pruning expectationsPruning verification + scan-byte thresholdsTreat pruning as part of correctness in BigQuery.
Counts and profilesPartition-level counts + null/min/max/distinct profilesCheap early drift detection before deep diffs.
KPI validationAggregate diffs by key dimensions + tolerance thresholdsAligns validation with business meaning.
Row-level diffsTargeted sampling diffs + edge cohort testsUse deep diffs only where aggregates signal drift.
Reruns/backfills in orchestratorsOperational integrity simulationsProves behavior under operational stress.

Compare

How workload changes

TopicHadoop legacy clusterBigQueryNotes
Performance contractPartition predicates are mandatory to avoid HDFS scansBytes scanned is the cost driver; pruning must be explicitValidation adds pruning and scan thresholds as gates.
Drift driversImplicit casts and time conversions often toleratedExplicit casts and timezone intent requiredEdge cohorts (ties/null-heavy/boundary days) are mandatory test cases.
Operational sign-offOften based on “looks right” report checksEvidence-based gates + rollback triggersCutover becomes measurable, repeatable, dispute-proof.

Examples

Examples

Illustrative parity and pruning checks in BigQuery. Replace datasets, keys, and KPI definitions to match your migration.

01_partition_row_counts.sql
-- Row counts by partition/window
SELECT
  event_date AS d,
  COUNT(*) AS rows
FROM `proj.mart.events`
WHERE event_date BETWEEN @start_date AND @end_date
GROUP BY 1
ORDER BY 1;
02_kpi_aggregate_diff.sql
-- KPI aggregate comparison (example)
WITH src AS (
  SELECT event_date d, country, SUM(metric_value) v
  FROM `proj.compare.src_metrics`
  WHERE event_date BETWEEN @start_date AND @end_date
  GROUP BY 1,2
), tgt AS (
  SELECT event_date d, country, SUM(metric_value) v
  FROM `proj.compare.tgt_metrics`
  WHERE event_date BETWEEN @start_date AND @end_date
  GROUP BY 1,2
)
SELECT
  COALESCE(src.d, tgt.d) AS d,
  COALESCE(src.country, tgt.country) AS country,
  src.v AS src_v,
  tgt.v AS tgt_v,
  (tgt.v - src.v) AS diff,
  SAFE_DIVIDE((tgt.v - src.v), NULLIF(src.v, 0)) AS diff_pct
FROM src
FULL OUTER JOIN tgt
USING (d, country)
ORDER BY d, country;
03_checksum_aggregate.sql
-- Checksum-style aggregate (approximate)
SELECT
  event_date AS d,
  COUNT(*) AS rows,
  SUM(ABS(FARM_FINGERPRINT(CONCAT(CAST(id AS STRING), '|', CAST(status AS STRING))))) AS fp_sum
FROM `proj.mart.events`
WHERE event_date BETWEEN @start_date AND @end_date
GROUP BY 1
ORDER BY 1;
04_idempotency_rerun_assertion.sql
-- Idempotency gate: compare snapshot metrics before/after rerun
WITH snap AS (
  SELECT
    COUNT(*) c,
    SUM(ABS(FARM_FINGERPRINT(CAST(id AS STRING)))) fp
  FROM `proj.mart.events`
  WHERE event_date = @d
)
SELECT
  s1.c AS before_count, s2.c AS after_count,
  s1.fp AS before_fp, s2.fp AS after_fp,
  IF(s1.c = s2.c AND s1.fp = s2.fp, 'PASS', 'FAIL') AS verdict
FROM snap s1, snap s2;

Workload Assessment

Validate parity and scan-cost before cutover

We define parity + pruning contracts, build golden queries, and implement layered reconciliation gates—so Hadoop→BigQuery cutover is gated by evidence and scan-cost safety.

Book assessment

Avoid

Common pitfalls

  • Spot-check validation: a few samples miss drift in ties and edge cohorts.
  • No pruning gate: scan-cost regressions slip through because bytes scanned isn’t validated.
  • Filters that defeat pruning: wrapping partition columns in functions/casts in WHERE.
  • No tolerance model: teams argue about diffs because thresholds weren’t defined upfront.
  • Wrong comparison level: comparing raw rows when business cares about rollups (or vice versa).
  • Ignoring reruns/backfills: parity looks fine once but fails under retries and historical replays.
  • Cost-blind diffs: exhaustive row-level diffs can be expensive; use layered gates (cheap→deep).

Proof

Validation approach

### Gate set (layered) Gate 0 — Readiness - Datasets, permissions, and target schemas exist - Dependent assets deployed (UDFs/routines, reference data, control tables) Gate 1 — Execution - Converted jobs compile and run reliably - Deterministic ordering + explicit casts enforced Gate 2 — Structural parity - Row counts by partition/window - Null/min/max/distinct profiles for key columns Gate 3 — KPI parity - KPI aggregates by key dimensions - Rankings and top-N validated on tie/edge cohorts Gate 4 — Pruning & cost posture (mandatory) - Partition filters prune as expected on representative parameters - Bytes scanned and slot time remain within agreed thresholds - Regression alerts defined for scan blowups Gate 5 — Operational integrity (when applicable) - _Idempotency:_ rerun same window → no net change - _Restart simulation:_ fail mid-run → resume → correct final state - _Backfill:_ historical windows replay without drift - _Late-arrival:_ inject late corrections → only expected rows change Gate 6 — Cutover & monitoring

  • Canary criteria + rollback triggers - Post-cutover monitors: latency, scan bytes/slot time, failures, KPI sentinels

Execution

Migration steps

A practical sequence for making validation repeatable and scan-cost safe.

  1. 01

    Define parity and cost/pruning contracts

    Decide what must match (tables, dashboards, KPIs) and define tolerances. Identify workloads where pruning is mandatory and set scan-byte/slot thresholds.

  2. 02

    Create validation datasets and edge cohorts

    Select representative windows and cohorts that trigger edge behavior (ties, null-heavy segments, boundary dates, epoch conversions).

  3. 03

    Implement layered gates

    Start with cheap checks (counts/profiles), then KPI diffs, then deep diffs only where needed. Add pruning verification and baseline capture for top workloads.

  4. 04

    Validate operational integrity

    Run idempotency reruns, restart simulations, backfill windows, and late-arrival injections where applicable. These scenarios typically break if not tested.

  5. 05

    Gate cutover and monitor

    Establish canary/rollback criteria and post-cutover monitors for KPIs and scan-cost sentinels (bytes/slot), plus latency and failures.

FAQ

Frequently asked questions

Why is pruning part of validation for Hadoop migrations? +

Because Hadoop performance relied on partition discipline. If pruning is lost in BigQuery, costs can explode even when results match. We validate both semantic parity and scan-cost posture before cutover.

Do we need row-level diffs for everything? +

Usually no. A layered approach is faster and cheaper: start with counts/profiles and KPI diffs, then do targeted row-level diffs only where aggregates signal drift or for critical entities.

What if our pipelines rely on reruns and backfills? +

Then validation must include replay simulations: rerun the same window, replay backfill partitions, and inject late updates. These gates prove the migrated system behaves correctly under operational stress.

How does validation tie into cutover? +

We convert gates into cutover criteria: pass/fail thresholds, canary rollout, rollback triggers, and post-cutover monitors. Cutover becomes evidence-based and dispute-proof.

Cutover Readiness

Gate cutover with evidence and rollback criteria

Get a validation plan, runnable gates, and sign-off artifacts (diff reports, thresholds, pruning baselines, monitors) so Hadoop→BigQuery cutover is controlled and dispute-proof.

Book assessment
Book assessment