AI, data & engineering
Data Engineer
Data Engineer is an engineer who builds the data pipelines and warehouses that analytics, ML models and the product itself rely on.
Also known as: Big Data Engineer, Data Platform Engineer, ETL Developer, Data Infrastructure Engineer.
What does a Data Engineer do?
A Data Engineer is an engineer who builds and runs the infrastructure that data moves through in a company: from in-app events and service databases to the warehouse where analysts and models work with it. If data arrives late, goes missing or does not add up, every decision based on it is wrong, so their work goes unnoticed as long as everything is fine. Typical tasks:
- collect data from services, product events and external sources;
- build batch and streaming pipelines and transformations;
- design the warehouse or lakehouse and the data models inside it;
- set up data quality checks and alerts so breakages are caught before analysts notice;
- manage access and the cost of compute and storage.
When does a company need a Data Engineer?
A company needs a Data Engineer when there is more data than can be collected by hand and analysts spend half their time on exports and reconciling numbers. Common triggers: different reports show different revenue, queries against the product database have started to slow down the product itself, a company preparing to launch ML discovers that its data is not usable for training. Another reason is a move to a new warehouse or to the cloud. If there is little data and every question can be answered with one database and a BI tool, it is better to hire a Product Analyst first, who can set up simple exports themselves.
Data Engineer vs Data Scientist: what's the difference?
A Data Engineer differs from a Data Scientist in building the system that delivers data rather than drawing conclusions from it. A Data Engineer is responsible for data being complete, fresh and correct; a Data Scientist uses it to answer business questions and build models. An Analytics Engineer sits between them: they work with data models and metrics inside the warehouse but do not build the infrastructure underneath.
Data Engineer vs related roles: comparison
| Data Engineer | Data Scientist | ML Engineer | |
|---|---|---|---|
| Responsible for | Data delivery and quality | Conclusions and models from data | Models in production |
| Main result | Reliable pipelines and warehouse | A decision based on data | A working ML service |
| Works with | Backend teams, analysts, ML | Business, Product Manager | Data Scientists, backend team |
Why is it hard to hire a Data Engineer?
Data Engineers are hard to hire because one title covers engineers with very different scale of experience, and the difference only shows in a deep technical assessment.
- Many Data Engineers have maintained ready-made pipelines in a well-known tool but have never designed a platform from scratch.
- Engineers who have built streaming at high volumes are noticeably fewer than those who have only worked with batch processing.
- The stack varies a lot between companies, and experience with one cloud or orchestrator is often mistaken for overall level.
- Strong candidates with experience of high-volume data platforms are actively retained and rarely apply to job posts.
What to decide before hiring a Data Engineer?
- What is needed: build from scratch, migrate, or fix? Each task calls for different experience, and someone who maintains a system well may struggle with a migration.
- Batch, streaming or both, and at what volumes? This determines which pool to search and how narrow it is.
- Which parts of the stack are fixed and which can change? A hard requirement for a specific tool often narrows the search for no real benefit.
- Who are the main data consumers: analysts, ML or the product itself? If data feeds the product directly, reliability requirements are closer to backend development.
- Is this the first Data Engineer or a hire into an existing team? The first one will make architectural decisions alone and needs a more senior profile.
Where to find a Data Engineer?
Data Engineers most often come from fintech, e-commerce, marketplaces, adtech and product companies with large data volumes, and from Backend Developers who moved into data. A strong profile signal is a migration or platform the person designed themselves, with a clear scale and result, or contributions to open-source data tools. NEWHR finds such candidates by the results of their work, not by applications to a job post.
How to assess a Data Engineer in an interview?
Data Engineers are assessed with SQL and coding tasks, followed by a system design case based on the company's situation. In the case, the candidate builds a pipeline for specific sources and volumes, decides what to do with late and duplicate events, and explains how to find out about a breakage before the analysts do. On the company side, the Head of Data or a lead Data Engineer and someone from the main data consumers take part.
Strong signals:
- describes what broke in their systems in production and what had to be redone;
- thinks about compute and storage cost, not only speed;
- asks who will use the data and how.
Red flags:
- knows the tools but cannot explain why they chose them;
- does not think about data quality until users report an error;
- proposes streaming where a daily batch load would do.
How long does it take to hire a Data Engineer?
We present the first candidates for the Data Engineer role within 5–7 days of the brief.
Related roles
- Data Scientist — uses data for conclusions and models rather than building its delivery.
- ML Engineer — takes models to production and depends on the data a Data Engineer prepares.
- Product Analyst — answers product questions using data from the warehouse.
- Crawling Engineer — extracts data from the web that nobody provides through an API.
- CDO (Chief Data Officer) — owns the data strategy and leads the data teams.