BIG DATA ENGINEER

Job Spec/Qualifications:Bachelor's or Master's degree in Computer Science, Data Science, or a related field.Proven at least 5 years of working experience as a Big Data Engineer, preferably working with Cloudera and Spark technologies.Strong proficiency in programming languages such as Scala, Python, or Java.Strong knowledge of API architectures preferably developed with Spring Boot.In-depth knowledge of distributed computing principles and frameworks, including Hadoop and Spark.Hands-on experience with Cloudera distribution and related tools like HDFS, Hive, Impala, and HBase.Proficiency in designing, implementing, and optimizing data pipelines using Spark Streaming, Spark SQL, or other Spark modules.Expertise in data modeling and database design principles, including schema design, partitioning, and indexing.Familiarity with data ingestion techniques and tools, such as Kafka, Flume, Sqoop, or Nifi.Strong understanding of SQL and NoSQL databases, data warehousing concepts, and ETL processes.Experience with cloud platforms like AWS or Azure, and knowledge of containerization technologies like Docker or Kubernetes is a plus.Understanding of data governance, data privacy, and security practices, particularly in an international context.Excellent problem-solving and analytical skills, with the ability to troubleshoot and optimize data processing workflows.Strong communication and collaboration skills to work effectively in an international team setting.Ability to prioritize tasks, manage multiple projects simultaneously, and deliver high-quality results within defined timelines.Continuous learning mindset to stay updated with emerging technologies, industry trends, and best practices in big data engineering.The ability to thrive in a fast-paced environment and take on new responsibilities quickly. Duties / Responsibilities: Develop and maintain scalable and efficient data processing pipelines using Cloudera and Spark technologies among others to handle large volumes of data from diverse sources.Design and implement data ingestion, transformation, and storage processes to ensure data quality, integrity, and availability across international locations.Develop and maintain scalable API solutions to provide data to both: external and internal customers. Collaborate with cross-functional teams to understand business requirements and translate them into technical specifications for data processing and analysis.Optimize data workflows and performance of Spark jobs to meet strict latency and throughput requirements in processing massive datasets.Implement data governance practices and ensure compliance with data privacy and security regulations across different regions.Conduct troubleshooting and performance tuning of the Cloud or On-premises infrastructure to identify and resolve performance bottlenecks and optimize resource utilization.Collaborate with data scientists and analysts to provide timely and reliable data sets for advanced analytics and machine learning models.Stay updated with emerging technologies and industry trends related to big data processing, Cloudera, and Spark, and propose innovative solutions to enhance data processing capabilities.Provide technical leadership, mentorship, and guidance to junior members of the team, fostering a collaborative and learning culture within the international team.Participate in agile development practices, including sprint planning, code reviews, and continuous integration and deployment, to ensure high-quality software delivery.