Consulting Lead - Databricks

India Trivandrum/Chennai Full Time We’re seeking a Consulting Lead-Databricks with strong experience in Databricks, Python, and PySpark, along with hands-on knowledge of Unity Catalog (UC) and Lakehouse architecture. In this role, you’ll design and optimize large-scale data pipelines, implement robust governance and security through UC, and build scalable solutions in a modern cloud data environment. The ideal candidate must have 5+ years in data engineering, expertise with any one of the cloud platforms (Azure/AWS/GCP), and proven ability to deliver high-performing ETL/ELT pipelines that support business-critical analytics. Qualification Hands-on expertise in Databricks, Python, PySpark, and Delta Lake/Lakehouse architecture. Experience implementing and managing Unity Catalog in a production environment. Strong SQL skills and experience with cloud platforms (Azure, AWS, or GCP). Proven ability to design scalable data pipelines and optimize performance. Strong communication skills and ability to work closely with stakeholders. Key Responsibilities: Design, develop, and maintain large-scale ETL pipelines using Databricks and PySpark. Optimize data workflows for performance, scalability, and reliability. Collaborate with data scientists, analysts, and business stakeholders to deliver high-quality data solutions. Implement best practices for data modeling, governance, and security in a cloud data environment. Migrate existing AWS Glu/Redshift based data pipelines to Databricks