Data Engineering Interview Questions and Answers (2026)

ETL transforms data *before* loading it into the warehouse; ELT loads raw data first and transforms it *inside* the warehouse. Modern cloud warehouses (Snowflake, BigQuery, Redshift) have cheap storage and elastic compute, so ELT is now the default: you keep raw data for reprocessing and express transformations in SQL (often with dbt). ETL still makes sense when data must be cleaned or masked before it lands — for example removing PII — or when the target can't do heavy compute.

A warehouse stores structured, modelled data optimised for SQL analytics. A lake stores raw files of any format cheaply (object storage) but without strong guarantees. A lakehouse puts a table layer (Delta Lake, Apache Iceberg, Apache Hudi) over lake storage to add ACID transactions, schema enforcement and time travel, so one copy of data serves both BI and ML.

Batch processes bounded data on a schedule — simpler, cheaper, easier to reason about and re-run. Streaming processes unbounded data continuously for low latency. Many "real-time" requirements are satisfied by micro-batches every few minutes; ask what latency the business actually needs before paying the complexity cost of streaming.

An idempotent job produces the same result no matter how many times it runs on the same input. Pipelines fail and get retried or backfilled constantly, so a job that appends duplicates on re-run is a bug. Achieve it with overwrite-by-partition, MERGE/upsert on a business key, or deduplication on a unique event ID.

Reprocessing historical data after a bug fix or logic change. Make the job parameterised by date range, idempotent, and run it in partitions so you can throttle load and resume after failure. Compare backfilled output with the old output on a sample before swapping it in.

A star schema has a central fact table (events or measurements: orders, clicks) joined to denormalised dimension tables (customer, product, date). A snowflake schema normalises the dimensions into further tables. Star schemas are simpler and faster for analytics queries; snowflaking saves storage and reduces duplication at the cost of more joins.

The grain defines exactly what one row represents — "one line item per order", not "orders". Deciding it first prevents double counting and makes every measure in the table consistent. Mixing grains in one table is a common modelling error.

How you handle dimension attributes that change over time. Type 1 overwrites (no history). Type 2 adds a new row with validity dates (valid_from, valid_to, is_current), preserving history — the most common for analytics. Type 3 keeps a limited previous value in an extra column.

Normalised models (3NF) suit transactional systems: less redundancy, safe updates. Denormalised models suit analytics: fewer joins, faster reads, at the cost of redundancy. Warehouses are typically denormalised or dimensional.

A driver builds a DAG of transformations; executors run tasks in parallel on partitions of data. Transformations are lazy — nothing runs until an action (count, write) triggers execution, which lets Spark optimise the whole plan (Catalyst optimiser).

Narrow transformations (map, filter) work within a single partition. Wide ones (groupBy, join, distinct) need data redistributed across the cluster — a shuffle, which means network and disk I/O and is usually the main cost in a Spark job. Reducing shuffles is the core of Spark tuning.

When one key has far more rows than others, one task does most of the work while the rest wait. Fixes: broadcast the small side of a join, salt the hot key (append a random suffix and aggregate in two stages), filter out null/default keys that often cause it, or enable adaptive query execution (AQE) skew handling.

repartition(n) does a full shuffle and can increase or decrease partitions with even sizing. coalesce(n) only reduces partitions by merging existing ones without a full shuffle — cheaper, but possibly uneven. Use coalesce before writing to avoid many tiny files.

Thousands of tiny files make listing and opening them dominate runtime and strain the metastore. Compact files to roughly 128 MB–1 GB, tune partition counts before writing, and use table-format compaction (OPTIMIZE in Delta, rewrite_data_files in Iceberg).

Partitioning splits data into directories by a column's value (usually date) so queries can skip irrelevant data — choose low-to-medium cardinality columns. Bucketing hashes a column into a fixed number of files to speed up joins and aggregations on that key. Over-partitioning on a high-cardinality column creates the small-files problem.

Parquet and ORC are columnar: great compression and fast analytical scans because only needed columns are read. Avro is row-based with embedded schema, good for streaming and schema evolution. CSV/JSON are human-readable but slow, large and weakly typed — fine for ingestion, rarely for storage.

A distributed, durable, append-only log. Producers write to topics, which are split into partitions; ordering is guaranteed only within a partition. Consumer groups share partitions among consumers, tracking progress with offsets. Messages are retained for a configured time regardless of consumption, so consumers can replay.

At-most-once may lose messages; at-least-once never loses but may duplicate; exactly-once means each message's effect happens once. True end-to-end exactly-once needs transactional or idempotent writes (Kafka transactions, idempotent sinks) — in practice most systems use at-least-once delivery plus idempotent processing.

In event-time streaming, a watermark is the system's estimate of how far event time has progressed, used to decide when a window is complete. Events arriving after the watermark are "late"; you choose to drop them, side-output them, or allow a lateness window and update results.

It schedules and runs workflows defined as DAGs of tasks, handles dependencies, retries, backfills, alerting and logging. Good practice: keep tasks small and idempotent, keep heavy processing in Spark/warehouse (not inside Airflow workers), avoid top-level code in DAG files that slows parsing, and pass references to data, not the data itself, between tasks.

Validate at each stage: schema checks, null and uniqueness checks, accepted ranges, referential integrity, row-count and freshness checks against expectations. Tools: dbt tests, Great Expectations, Soda. Decide per check whether failure blocks the pipeline or only alerts, and quarantine bad records rather than silently dropping them.

Capturing row-level inserts, updates and deletes from a source database's transaction log (for example Debezium reading the MySQL binlog or Postgres WAL) and streaming them downstream. It avoids heavy full-table extracts and preserves deletes, which timestamp-based incremental loads often miss.

Prefer additive, backward-compatible changes (new nullable columns). Use a schema registry for streaming, table formats that support evolution, and data contracts with upstream owners. Alert on unexpected schema drift instead of letting a renamed column quietly produce nulls.

Extract incrementally (CDC or a reliable updated_at watermark) into a raw landing zone in object storage, load to a raw/staging schema in the warehouse, then transform with SQL/dbt into cleaned staging models and finally dimensional marts. Make each step idempotent, partition by date, add data-quality tests between layers, monitor freshness and volumes, and document how to backfill. Mention cost, retries and what happens when the source is late.

Check what changed: data volume, a new skewed key, a schema or code change, resource contention, or an upstream delay. Use the Spark UI or query profile to find the slow stage, look for skew, spills to disk, excessive shuffles or small files, and compare against an earlier run. Fix the root cause, then add a runtime alert so you notice the next regression earlier.

Pin down each definition (filters, time zones, grain, dedup rules), trace both back to source, and diff row counts layer by layer to find where they diverge. Then fix it by creating one governed definition — a shared metric layer or semantic model — rather than patching individual reports.

Strong SQL, Python, solid data modelling, one distributed engine (usually Spark), a cloud platform (AWS, Azure or GCP), orchestration (Airflow or similar) and an understanding of data quality and cost. Streaming experience is a strong differentiator.

For many roles, yes — it remains the most common large-scale processing engine. Some warehouse-centric roles lean on SQL and dbt instead. Check the job description and prepare accordingly.

Data engineers build and maintain the reliable pipelines and platforms that deliver data; data scientists analyse it and build models on top. In interviews that means engineering questions focus on systems, scale and reliability rather than statistics.

Yes. Our Python job support and AWS job support cover pipeline code, cloud data services and debugging in your actual environment.