Databricks Engineer - Bogotá, Capital District
hace 1 día
Descripción del trabajo
We're seeking a hands on Data Engineer to help plan and execute the migration of analytics and ETL/ELT workloads from their current solution to an Azure + Databricks lakehouse stack. You'll help design ingestion and transformation pipelines, optimize Spark jobs, and ensure governance and compliance suitable for a pharmaceutical environment
Key Responsibilities
- Analyze current pipelines and re‑platform to Databricks (PySpark/SQL) following bronze, silver, gold patterns.
- Rebuild schedules/orchestrations (e.g., Databricks Workflows, ADF/Synapse/Fabric) and replace current instance specific operators with native Spark/Delta patterns.
- Map data models/virtualized objects to Delta Lake tables with partitioning, Z Ordering, and optimized file layouts.
- Develop scalable ETL/ELT in PySpark and SQL
- Implement unit/integration tests and data quality checks (freshness, completeness, schema)
- Tune Spark (shuffle partitions, broadcast joins, AQE), implement caching, and cost‑optimize clusters (auto‑scaling, auto‑pause, spot instances as appropriate)
- Contribute to validation documentation (requirements, risk, test, evidence)
- Ensure lineage, cataloging, and metadata standards
- Build monitoring/alerting for pipelines and SLAs; track cost and performance budgets
- Support incident response, defect triage, and root‑cause analysis; contribute to runbooks and knowledge base
- Work with product owners and analysts to translate requirements into technical specs
- Partner with platform/SRE teams on VNETs, private endpoints, networking, and environment hardening
Required Qualifications
- 4–5 years of professional experience as a Data Engineer working with Spark (Databricks preferred) and cloud data platforms (Azure strongly preferred)
- Strong PySpark and SQL skills; experience with Delta Lake
- Proven experience delivering production ETL/ELT pipelines, CI/CD (Git, DevOps/GitHub Actions), and testing frameworks for data
- Familiarity with orchestration (Databricks Workflows, Azure Data Factory/Synapse/Fabric pipelines)
- Understanding of data modeling (dimensional, lakehouse medallion), performance tuning, and cost optimization
- Experience in regulated industries (healthcare/med‑tech, pharma, financial services)
- English proficiency (written and spoken) ability to collaborate with global teams
Trabajos similares
We bring together a global team of engineers, scientists and architects to help the world's most innovative companies unleash their potential where you can make a difference. · Design build and maintain data pipelines and ETL processes using Databricks and Apache Spark. · Optimiz ...
hace 2 semanas
We are looking for a Databricks Engineer to help us design and build data pipelines and ETL processes using Databricks and Apache Spark. · ...
hace 2 semanas
We're seeking a hands on Data Engineer to help plan and execute the migration of analytics and ETL/ELT workloads from their current solution to an Azure + Databricks lakehouse stack. You'll help design ingestion and transformation pipelines, optimize Spark jobs, and ensure govern ...
hace 2 días
Join us for a career full of opportunities where you can make a difference and no two days are the same. · ...
hace 1 semana
Senior Engineer, MLOps Expert in Databricks + Spark
Solo para miembros registrados
Licenciatura o Máster en Ciencias de la Computación con más de 47 años de experiencia en ingeniería MLOps. · ...
hace 5 días
Se requiere Profesional en Ingeniera de Sistemas, telemática y/o afines · Debe contar con certificación vigente en Databricks en alguna de las siguientes modalidades: · Databricks Certified Data Engineer (Associate o Professional) o · Microsoft Certified: Azure Data Engineer Asso ...
hace 21 horas
Somos una empresa líder en tecnología y estamos buscando un Data Engineer con experiencia en Azure. · ...
hace 4 días
Experiencia en arquitectura de datos o ingeniería de datos con maestría homologable relacionada. Certificaciones en Microsoft Azure Data Engineer Associate y Databricks Certified Data Engineer Associate. · ...
hace 2 semanas
+Estamos buscando un/a Ingeniero AI con nivel de inglés B2. · +Python intermedio/avanzado. · Experiencia en data / ML / AI (mín. 1.5 años). · Experiencia con LLMs y frameworks GenAI (LangChain, LlamaIndex, AutoGen, CrewAI). · +,+<p style= ...
hace 1 semana
Buscamos un Data Engineer con experiencia en Databricks y Apache Spark para diseñar, desarrollar y optimizar pipelines de datos. Experiencia técnica requerida en desarrollo y mantenimiento de notebooks en Databricks, creación y gestión de Jobs y Workflows. · ...
hace 1 semana
Estamos buscando un Ingeniero AI para trabajar en Inteligencia Artificial Generativa. · ...
hace 1 semana
Buscamos un/a Ingeniero/a Data con sólida experiencia en Databricks y Apache Spark. · ...
hace 1 semana
Vacante para un AI Engineer - Python. · Experiencia total en el mundo de datos.Conocimiento y experiencia en ciencia de datos e ingeniería.Inglés conversacional B2. ...
hace 1 semana
Bilingual Azure Data Engineer
Solo para miembros registrados
We are looking for a Bilingual Azure Data Engineer to join our team. · Develop and maintain efficient ETL pipelines using Azure Data Factory and Databricks. · Analyze large datasets and build machine learning models using Python and Spark. · Utilize Apache Spark to process and an ...
hace 3 semanas
At Capgemini Engineering, the world leader in engineering services, we bring together a global team of engineers, scientists and architects to help the most innovative companies unleash their potential. From autonomous cars to life-saving robots our digital and software technolog ...
hace 3 semanas
We are looking for an experienced Data Engineer with strong foundation in Python, SQL, and Azure, · and hands-on expertise in Databricks. · Migrate legacy data infrastructure while ensuring performance, security, and scalability. · Build and optimize data systems,pipelines,and an ...
hace 2 días
We are looking for a Senior Data Architect to design and implement scalable, secure and high-performance data architectures. · Design and implement scalable data architectures Build ETL/ELT pipelines using cloud-native technologies Lead the development of data platforms on Azure ...
hace 1 mes
FACTECH es una empresa multinacional especializada en proporcionar a las grandes compañías servicios de IT en las principales tecnologías. Contamos con más de 2.500 profesionales al servicio de nuestros principales clientes y con presencia en España, Colombia y México. · Actualme ...
hace 1 día
Fusemachines is a leading AI strategy, talent, and education services provider. · ...
hace 4 días
We are seeking a motivated Data Analyst to join our team. · The ideal candidate will have a strong foundation in supporting business intelligence tools and financial analyses, · data modeling and management,and maintaining data engineering workflows within Databricks.Proficiency ...
hace 1 mes