Databricks Data Specialist - R01569707

Data Specialist

Primary Skills Databricks Engineer

Role Overview

We are seeking a highly skilled Databricks Engineer to design, develop, and optimize scalable data engineering and analytics solutions on the Databricks Lakehouse Platform . The ideal candidate will possess strong expertise in Databricks, PySpark, and SQL , with hands-on experience building batch and real-time data pipelines, implementing Lakehouse architectures, and ensuring data governance and performance optimization.

Key Responsibilities

• Design, develop, and maintain end-to-end data pipelines using Databricks and PySpark .

• Build and implement Lakehouse architectures utilizing Bronze, Silver, and Gold data layers.

• Develop and manage Delta Lake solutions with ACID transactions, schema enforcement, and data reliability features.

• Create, monitor, and optimize Delta Live Tables (DLT) pipelines.

• Implement scalable and efficient data ingestion processes using Auto Loader .

• Develop and manage real-time data processing solutions using Structured Streaming .

• Orchestrate, schedule, and monitor data workflows using Databricks Workflows .

• Design and implement Lakehouse data models to support reporting, analytics, and business intelligence requirements.

• Establish and enforce data governance, security, and access controls using Unity Catalog .

• Optimize Spark jobs, SQL queries, and overall platform performance to ensure efficiency and scalability.

• Collaborate with cross-functional teams, including data analysts, architects, and business stakeholders, to deliver high-quality data solutions.

Required Skills (Must Have)

• Databricks Platform

• Delta Lake

• Delta Live Tables (DLT)

• Unity Catalog

• Databricks Workflows

• PySpark and Apache Spark

• Structured Streaming

• Auto Loader

• SQL

• Lakehouse Data Modeling

• Strong understanding of data engineering best practices and scalable data architectures

Preferred Skills (Good to Have)

Azure Ecosystem

• Azure Data Factory (ADF)

• Azure Synapse Analytics

• Microsoft Purview

• Microsoft Fabric

AWS Ecosystem

• AWS Glue

• AWS Lambda

• AWS Step Functions

Data Engineering & Integration

• Apache Airflow

• DBT

• Fivetran

• Informatica

Streaming & Analytics

• Apache Kafka

• Power BI

Data Governance

• Collibra

• Alation

GCP

• BigQuery

Qualifications

• Bachelor's or Master's degree in Computer Science, Data Engineering, Information Technology, or a related discipline.

• Proven experience in designing and implementing cloud-based data engineering solutions and scalable data pipelines.

• Strong analytical, troubleshooting, and problem-solving capabilities.

• Experience working in agile and collaborative environments.

• Excellent communication and stakeholder management skills.

Preferred Candidate Profile

• Hands-on experience with modern Lakehouse architectures and enterprise-scale data platforms.

• Strong understanding of data governance, security, and compliance frameworks.

• Experience delivering both batch and real-time data processing solutions.

• Ability to work independently while collaborating effectively across global teams.

Key Technologies

Databricks | PySpark | Apache Spark | Delta Lake | Delta Live Tables (DLT) | Unity Catalog | Structured Streaming | Auto Loader | SQL | Lakehouse Architecture | Azure | AWS | Airflow | Kafka | Power BI

Specialization • Databricks Engineering: Lead Data Engineer

Job requirements Databricks Engineer

Role Overview

We are seeking a highly skilled Databricks Engineer to design, develop, and optimize scalable data engineering and analytics solutions on the Databricks Lakehouse Platform . The ideal candidate will possess strong expertise in Databricks, PySpark, and SQL , with hands-on experience building batch and real-time data pipelines, implementing Lakehouse architectures, and ensuring data governance and performance optimization.

Key Responsibilities

• Design, develop, and maintain end-to-end data pipelines using Databricks and PySpark .

• Build and implement Lakehouse architectures utilizing Bronze, Silver, and Gold data layers.

• Develop and manage Delta Lake solutions with ACID transactions, schema enforcement, and data reliability features.

• Create, monitor, and optimize Delta Live Tables (DLT) pipelines.

• Implement scalable and efficient data ingestion processes using Auto Loader .

• Develop and manage real-time data processing solutions using Structured Streaming .

• Orchestrate, schedule, and monitor data workflows using Databricks Workflows .

• Design and implement Lakehouse data models to support reporting, analytics, and business intelligence requirements.

• Establish and enforce data governance, security, and access controls using Unity Catalog .

• Optimize Spark jobs, SQL queries, and overall platform performance to ensure efficiency and scalability.

• Collaborate with cross-functional teams, including data analysts, architects, and business stakeholders, to deliver high-quality data solutions.

Required Skills (Must Have)

• Databricks Platform

• Delta Lake

• Delta Live Tables (DLT)

• Unity Catalog

• Databricks Workflows

• PySpark and Apache Spark

• Structured Streaming

• Auto Loader

• SQL

• Lakehouse Data Modeling

• Strong understanding of data engineering best practices and scalable data architectures

Preferred Skills (Good to Have)

Azure Ecosystem

• Azure Data Factory (ADF)

• Azure Synapse Analytics

• Microsoft Purview

• Microsoft Fabric

AWS Ecosystem

• AWS Glue

• AWS Lambda

• AWS Step Functions

Data Engineering & Integration

• Apache Airflow

• DBT

• Fivetran

• Informatica

Streaming & Analytics

• Apache Kafka

• Power BI

Data Governance

• Collibra

• Alation

GCP

• BigQuery

Qualifications

• Bachelor's or Master's degree in Computer Science, Data Engineering, Information Technology, or a related discipline.

• Proven experience in designing and implementing cloud-based data engineering solutions and scalable data pipelines.

• Strong analytical, troubleshooting, and problem-solving capabilities.

• Experience working in agile and collaborative environments.

• Excellent communication and stakeholder management skills.

Preferred Candidate Profile

• Hands-on experience with modern Lakehouse architectures and enterprise-scale data platforms.

• Strong understanding of data governance, security, and compliance frameworks.

• Experience delivering both batch and real-time data processing solutions.

• Ability to work independently while collaborating effectively across global teams.

Key Technologies

Databricks | PySpark | Apache Spark | Delta Lake | Delta Live Tables (DLT) | Unity Catalog | Structured Streaming | Auto Loader | SQL | Lakehouse Architecture | Azure | AWS | Airflow | Kafka | Power BI