Senior Data Engineer | AWS
Guarde esta oferta e mantenha a sua pesquisa organizada
Crie uma conta gratuita para guardar ofertas, criar alertas e voltar a este anúncio a partir do seu painel.
Ao continuar, aceita os nossos Termos & Política de Privacidade.
Our nearshore expertise sets us apart. We specialize in connecting experienced technology professionals with complex international engineering environments, supporting organizations in building scalable, reliable, and high-performance data platforms.
We are looking for a Senior Data Engineer | AWS & GCP
To join a Life Sciences Data Engineering project , supporting a pharmaceutical organization specialized in research, development, and manufacturing of innovative therapies .
The project focuses on the evolution of an enterprise cloud data platform , integrating research, laboratory, manufacturing, and operational datasets to support analytics, data quality, and scientific decision-making.
Skills
- AWS: S3, Glue, EMR and Step Functions for ingestion, processing and orchestration;
- GCP: BigQuery, Cloud Storage, Dataflow and Cloud Composer for analytical/DWH;
- Python, SQL and PySpark for data transformations and distributed processing;
- ETL pipelines for enterprise data integration and processing;
- Data Lake & Data Warehouse architectures for scalable storage and analytical consumption;
- Apache Airflow for pipeline orchestration and dependency management;
- Data Quality & Governance for validation, lineage and traceability;
- Terraform, Git & GitHub Actions for infrastructure automation and controlled deployments;
- Performance Optimization for distributed workloads and analytical queries.
Responsibilities
- Develop AWS-based ingestion and processing pipelines , using S3 to organize source datasets, Glue for extraction and transformation, and EMR for large-scale PySpark processing;
- Implement GCP analytical data pipelines , using Cloud Storage for staging, Dataflow for transformation workflows, and BigQuery for curated analytical datasets and reporting consumption;
- Build Python, SQL, and PySpark transformations to standardize, cleanse, validate, and combine data from research, laboratory, manufacturing, and enterprise systems;
- Design and maintain ETL workflows connecting data processing and analytical layers , managing schema evolution, incremental loads, data dependencies, and processing efficiency;
- Configure AWS Step Functions and Cloud Composer / Apache Airflow to coordinate pipeline execution, scheduling, monitoring, retries, and recovery mechanisms;
- Optimize Spark processing jobs and BigQuery workloads , improving partitioning strategies, query execution, storage efficiency, and cloud processing costs;
- Implement data quality checks and validation rules using AWS Glue Data Quality , PySpark and SQL , leveraging Google Cloud Dataplex for data quality monitoring , metadata management and lineage;
- Automate cloud infrastructure and data pipeline deployments through Terraform and GitHubActions;
- Collaborate with Data Architects, Analytics Engineers, and pharmaceutical domain teams to deliver reusable data models, reliable transformation frameworks, and scalable data products.