Summary:
As a Data Engineer based out of our BMS Hyderabad you are part of the Data Platform team along with supporting the larger Data Engineering community, that delivers data and analytics capabilities for Data Platforms and Data Engineering Community. The ideal candidate will have a strong background in data engineering, DataOps, cloud native services, and will be comfortable working with both structured and unstructured data.
Key Responsibilities
- The Data Engineer will be responsible for developing and maintaining ETL/ELT pipelines for ingesting data from various sources into our data warehouse.
- Work with an end-to-end ownership mindset, innovate and drive initiatives through completion.
- Optimize data storage and retrieval to ensure efficient performance and scalability
- Collaborate with data architects, data analysts and data scientists to understand their data needs and ensure that the data infrastructure supports their requirements
- Ensure data quality and integrity through data validation and testing
- Implement and maintain security protocols to protect sensitive data
- Stay up-to-date with emerging trends and technologies in data engineering and analytics
- Closely partner with the Enterprise Data and Analytics Platform team, other functional data teams and Data Community lead to enable adoption of data and technology strategy.
- Knowledgeable in evolving trends in Data platforms and Product based implementation
- Comfortable working in a fast-paced environment with minimal oversight
- Prior experience working in an Agile/Product based environment.
Qualifications & Experience
- 2-3 years of hands-on experience working on implementing and operating data capabilities and cutting-edge data solutions, preferably in AWS cloud environment. Breadth of experience in technology capabilities that span the full life cycle of data management including data lakehouses, master/reference data management, data quality and analytics/AI ML is needed.
- In-depth experience with AWS Glue service and data engineering ecosystem on AWS.
- Hands-on experience developing and delivering data, ETL solutions with some of the technologies like AWS data services ( Redshift, Athena, lakeformation, etc.), Cloudera Data Platform, Tableau labs is a plus
- Create and maintain optimal data pipeline architecture, assemble large, complex data sets that meet functional / non-functional business requirements.
- Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
- Strong programming skills in languages such as Python, PySpark, Pandas, Scala etc.
- Experience with SQL and database technologies such as MySQL, PostgreSQL, Presto, etc.
- Experience with cloud-based data technologies such as AWS, Azure, or Google Cloud Platform
- Strong analytical and problem-solving skills
- Excellent communication and collaboration skills Functional knowledge or prior experience in Lifesciences Research and Development domain is a plus
