Data Engineer

Job Summary Design, build, and maintain scalable, fault-tolerant data pipelines using Python, SQL, and orchestration tools. Manage package dependencies, ensure security compliance, and optimize code for production data workflows. Responsibilities Develop modular Python code with robust error handling and logging for data pipelines Write advanced SQL queries using joins, window functions, and optimization techniques to support data processing Process and transform data using Pandas and integrate streaming data with Kafka producers and consumers Design, build, and maintain Directed Acyclic Graphs (DAGs) and flows using orchestration tools such as Apache Airflow and Prefect Ensure pipelines are idempotent, scalable, and fault-tolerant to support reliable data workflows Implement logging, monitoring, and alerting mechanisms to maintain pipeline observability and operational health Manage Python package installations, upgrades, and dependency resolution across development, UAT, and producti...