As a Cloud Data Engineer, you will play a critical role in designing and implementing data pipelines and systems that meet the needs of our business. You will work with cross-functional teams to understand business requirements and develop technical solutions.
- Implementing robust, secure, and efficient data pipelines to handle big data workloads
- Developing data pipelines for validating ITS data and computing derived data using metadata-driven expression language
- Collaborating with cross-functional teams to understand business requirements and translate them into technical solutions
- Monitoring system performance and implementing improvements to ensure optimal efficiency
- Staying current with emerging trends and best practices in cloud computing and data engineering
- Strong knowledge of AWS services, including S3, Athena, and Glue
- Strong background in SQL in analytical environments
- Strong experience with Python, with proficiency in Java as a plus
- Experience with distributed big data technologies such as Hadoop and Apache Spark
- Experience with relational and dimensional database modeling
- Minimum 3 years of experience in data engineering or a related field
- Experience with single-node computing analytics libraries like Pandas/Polars
- Knowledge of cloud-based data warehousing and data lake architecture
- Experience with data pipeline orchestration tools such as Apache Airflow
- Familiarity with containerization using Docker and Kubernetes
- Advanced degree in Computer Science, Data Science, or a related field