When a CV arrives with "Data Engineer" at the top, the title tells me very little. The tooling underneath tells me nearly everything. Data engineering has moved through three distinct eras in twenty years, and the stack an engineer lists dates them almost to the year: Hadoop and on-prem ETL first, then Spark and streaming, then the cloud warehouse with Airflow and dbt.

Keyword-matching cannot separate them. A search for "data engineer" returns all three eras in one pile, plus a crowd of adjacent roles wearing the same badge. Reading the stack, and the trajectory across stacks, is the instrument I use before anyone hires a data engineer on my shortlist.

TL;DR: "Data engineer" spans three tooling eras, and each demands a different screen. Date the stack first, then probe the migrations between eras, because trajectory reveals more than any single logo. And before you brief the search at all, check whether the role you need is an analytics engineer.

Why hiring a data engineer trips up a generalist recruiter

Data engineers made up just 1.9% of respondents to the Stack Overflow Developer Survey in 2024, and 1.7% in the 2025 edition. The survey restructured its role categories between editions, so read that as one steady signal rather than a decline: consistently under 2% of the developer population.

Scarcity is only half the problem. The other half is that nearly everyone adjacent to data claims the title. BI developers who maintain dashboards, ETL contractors who configure a vendor tool, analysts who once scheduled a Python script: all of them can plausibly write "data engineer" on a CV, and many do.

A generalist recruiter matching on the keyword has no instrument for telling these apart. The data briefs that cross my desk almost never specify an era either; they list a wishlist of tools drawn from all three, which guarantees a shortlist that fits none of them. The fix is to date the stack itself, era by era.

Three eras of data engineering, and how to read them on a CV

Each era below comes with its years, what it means about how the engineer thinks, and one screening question a non-specialist can ask without bluffing.

• The warehouse-ETL and Hadoop era (roughly 2006 to 2014). Apache Hadoop v1.0 was announced in January 2012 after what the Apache Software Foundation described as six years of development, which anchors the era's start around 2006. This was batch MapReduce on hardware you owned, and at serious scale: the same announcement notes Yahoo running a production environment of more than 42,000 Hadoop nodes. An engineer formed here thinks in batch windows, cluster capacity and overnight job schedules. The question to ask: what did you migrate that estate to, and what broke on the way?
• The big-data era (roughly 2014 to 2019). Apache Spark, created in 2009 at UC Berkeley's AMPLab, graduated to Apache top-level status in February 2014, positioned as up to 100x faster than Hadoop MapReduce for in-memory workloads. This is the era of Spark clusters, streaming ambitions and Kafka-adjacent architectures. The question to ask: where did Spark genuinely earn its keep on your projects, and where would plain SQL have done the job? Engineers who lived this era answer honestly; engineers wearing it as decoration reach for buzzwords.
• The cloud-warehouse era (2015 to present). The Snowflake Elastic Data Warehouse went generally available in June 2015, billed in Snowflake's 2015 launch announcement as the first data warehouse designed from the start for the cloud. Apache Airflow began at Airbnb in 2014 and became a top-level project in January 2019, by then used by more than 200 organisations; the official description calls it a platform to programmatically author, schedule and monitor workflows. On top of the warehouse sits dbt, which per dbt's documentation transforms raw warehouse data by letting engineers write SQL select statements while the tool handles modular, maintainable data models. The question to ask: how do you structure your dbt models and tests, and what does your Airflow actually orchestrate?

In the screens I run, which tools appear matters less than whether the candidate can explain why their team moved from one era's tooling to the next, in terms of cost, latency or team size, without reciting a vendor pitch.

Legacy stack, not legacy engineer

Hadoop is still shipping: release 3.5.0 landed in April 2026, per the Apache Hadoop project site. That matters for hiring, because it means enterprise Hadoop estates are maintained realities, and the engineers who run them are working professionals with current, production-grade responsibilities.

The prize candidate here is the migration engineer. Someone who has taken a Hadoop estate and moved it onto the cloud-warehouse stack has felt both eras under load: the capacity planning and failure modes of the old world, and the cost model and modelling discipline of the new one. In my experience that profile screens stronger than an engineer who has only ever known Snowflake.

This is where the whole framework earns its keep. A toolset timestamps a CV; the movement between toolsets reveals the engineer. A snapshot of tools shows when someone was trained. The movement between snapshots shows whether they can learn the next era too, which is what you are paying for.

The fork nobody briefs for: data engineer or analytics engineer?

The analytics engineer is a genuinely distinct role, and dbt Labs treats it as one: its own analytics engineer certification evaluates building, testing and maintaining models for data accessibility, applying engineering principles to analytics infrastructure through modelling, testing, governance and version control. That is a discipline, with its own credential, sitting one layer above pipeline engineering.

The scale of the ecosystem explains why. dbt positions itself as the modern standard for data transformation, citing a community of more than 100,000 members and adopters including Nasdaq, Canva, Duolingo and GitLab; vendor-reported figures, but the direction is unmistakable. SQL-based transformation inside the warehouse has become its own career.

So here is the honest advice, the one I give even when it shrinks my own brief. If your "data engineering" need is actually SQL modelling inside a warehouse you already have, you need an analytics engineer, or a strong analyst plus dbt, not a premium distributed-systems engineer. Hiring the latter for the former overpays for skills the role will never use, and the engineer often leaves early out of boredom. Briefing the fork correctly costs one conversation and saves a mishire.

The modern stack won, and it is already consolidating

The cloud-warehouse era is no longer an emerging bet. Snowflake reports more than 13,300 customers and Q4 FY26 product revenue of 1.23 billion dollars, up 30% year over year, in its fiscal 2026 results. Databricks reports a revenue run-rate above 5.4 billion dollars and more than 20,000 organisations in a recent company announcement. Both figures are vendor-reported, and both point the same way.

Consolidation has already started. Fivetran and dbt Labs completed their merger on 1 June 2026, together supporting a community of more than 100,000 data teams. The tools your team standardises on this quarter may be one company's product line by next year.

The hiring lesson follows directly. Hire for the ability to cross eras rather than for this quarter's logo soup. An engineer who moved from Hadoop to Spark to Snowflake will handle whatever the consolidated stack becomes; an engineer who only pattern-matches the current logos will need rescuing at the next platform shift.

How to screen a data engineer before you hire

1. Date the stack against the three eras. Place every tool on the CV into the Hadoop, Spark or cloud-warehouse era before the first call. Ten minutes of sorting tells you which engineer you are meeting.
2. Ask the migration question. What did you move, from what, to what, and what broke? Candidates who have crossed an era boundary answer in specifics. Candidates who have watched one from a distance answer in generalities.
3. Probe SQL and data-modelling depth in prose. Ask them to talk through an idempotent pipeline they built, how they structure incremental models, and what their testing covers. You are listening for reasoning about failure and re-runs, and you do not need to be technical to hear its absence.
4. Run the fork check honestly. Describe the first six months of real work, then ask the candidate whether that sounds like data engineering or analytics engineering to them. Strong candidates will tell you the truth, and their answer audits your own brief for free.
5. Weight trajectory over tool count. Two eras crossed outweighs five tools listed. This era-dating instrument is the same one I use for iOS hiring, where the platform's API generations date a CV just as precisely, and it anchors how I screen across the whole data and analytics practice.

Common questions about hiring data engineers

What is the difference between a data engineer and an analytics engineer?

A data engineer builds and operates the pipelines that move raw data into the warehouse. An analytics engineer works inside the warehouse, applying engineering discipline to SQL models: testing, governance and version control, as dbt Labs' own certification syllabus defines the role. Brief the wrong one and both sides lose a year.

Is Hadoop experience still valuable in 2026?

Yes, and more than most briefs assume. Hadoop remains actively maintained, with release 3.5.0 shipped in April 2026 according to the project itself, and enterprise estates still run on it. An engineer who has migrated a Hadoop estate to the cloud stack is often the strongest candidate on the shortlist.

Which tools should a modern data engineer know?

The current-era core is a cloud warehouse such as Snowflake, generally available since its June 2015 launch, orchestration with Apache Airflow, and transformation with dbt. Depth in SQL and data modelling matters more than breadth across any particular vendor list.

Do I need a data engineer, or an analyst with dbt?

If the work is transforming and modelling data already landing in your warehouse, a strong analyst working with dbt's SQL-based workflow may cover it at a fraction of the cost. Reserve the data-engineer search for genuine pipeline, infrastructure and scale problems that sit upstream of the warehouse.

---

Data engineering searches sit alongside the BI side of my practice; if your need leans toward dashboards and Qlik estates, the sister piece on hiring Qlik developers covers that fork in detail.

I keep two to four active searches at a time, which is why every brief starts with the era conversation above. Three CVs that fit beats thirty that don't. If you're hiring on the data side, tell me about the role and I'll tell you honestly which engineer, or which fork, you actually need.