2024-10-03 01:02
Route to Data Engineer 2024
Fundamentals:
- Data querying languages (SQL)
- Pogramming language tools (Python, Scala, Java)
- Processing: Batch (Spark), Stream (Fink, Spark)
- Storage: Databases (SQL, NoSQL)
- Data Warehouses(Hive, Snowflake)
- Messaging: Message Queues (Kafka)
- Cloud: Azure, Amazon, GCP
- Workflow Management: Airflow
- Resource Management: Yarn
- Fast Ingestion: Druid
- Visualisation: Create visuals with Power BI, tableau, Kibana.