Summary
Highlights
The video addresses the overwhelming number of data engineering tools and aims to categorize them by learning effort and industry usage for 2025. The tools are divided into five buckets: low effort/high usage (must-learn and trending), high effort/low usage, and dead tools. The speaker will share insights based on personal experience in data engineering.
Data engineering focuses on building pipelines to move and transform raw data for various applications. The most fundamental tools include Python (low effort, high usage for data science and engineering), SQL (low effort, high usage for database interaction), and Excel (low effort, high usage for simple data analysis). Pandas and NumPy (low effort, high usage) are crucial Python libraries, while Matplotlib is considered low effort and low usage for data engineers. R and SAS are largely considered 'dead' or high effort/low usage, with Python being the preferred alternative.
Working with terabytes of data requires distributed processing. Spark is highlighted as a critical tool (high effort, very high usage), with PySpark being the easiest way to learn and interact with it. Older tools like Hadoop and Hive are considered almost 'dead' in 2025. Scala, the language Spark is built on, is rated as high effort and low usage compared to Python.
Data processing can be batch (processed in chunks, more time-consuming) or real-time (faster). Kafka and Flink are open-source tools for real-time processing, rated as high effort and high usage. Learning these provides a foundation for understanding similar tools on cloud platforms.
Orchestration tools manage the sequential execution of various data processing scripts. Apache Airflow is the most popular open-source option (high effort, high usage). Cloud platforms like AWS (EC2, Lambda, Glue), Azure (Data Factory), and GCP (GCS, Dataflow) offer their own orchestration services. Azure is noted as the easiest to learn among cloud platforms for understanding ETL processes.
Data warehouses (like BigQuery, Redshift, Snowflake) handle structured data, while data lakehouses accommodate various data formats (structured, unstructured, semi-structured). Data Lakehouse Intelligence Platforms, like Delta Lake and Apache Iceberg, built on Apache Spark, allow for easier data migration between cloud providers. Databricks, which integrates with Delta Lake, is recommended for learning how to work with this framework (high effort, high usage), as is Snowflake for data modeling and Apache Iceberg.
Data engineers, especially in smaller companies, are increasingly expected to manage the entire data engineering lifecycle, including DevOps aspects like model deployment, automation, and CI/CD pipelines. Recommended DevOps tools include Docker and Kubernetes (high effort, high usage), CI/CD (low effort, high usage), and Git for version control (low effort, high usage). Terraform is mentioned as beneficial but high effort/low usage for beginners.
The key strategy is not to learn every tool, but to master the basics of one tool in each category to understand underlying concepts, as knowledge is often transferable across similar tools. The advice is to start with 'green zone' tools (must-learn essentials) and then focus on 'blue zone' tools (trending and highly used). The video concludes by encouraging viewers to check out other tutorials for building projects with these tools and to subscribe.