Software Engineer II
- Posted
- Employment
- full time
- Work mode
- hybrid
Skills explicitly mentioned: SQL, PySpark, Apache Spark, Databricks, Azure Data Factory, Azure Synapse, Data Engineering, CI/CD
Summary We are seeking a Senior Data Engineer to architect and build modern Azure-based data solutions, lead large-scale data platform modernization initiatives, and mentor junior engineers. The ideal candidate should have deep expertise in Azure Synapse Analytics, Databricks, Azure Data Factory (ADF), and modern data engineering practices . Experience with Informatica PowerCenter is a plus but not mandatory . Key Responsibilities • Design and implement end-to-end Azure data architectures using Azure Synapse Analytics, Databricks, and ADLS Gen2 . • Lead and support data modernization and migration initiatives to Azure. • Perform solution assessment, complexity analysis, and migration planning. • Develop scalable data pipelines using PySpark, Spark SQL, and Azure Data Factory (ADF) . • Implement data engineering best practices including SCD, CDC, UPSERT/MERGE , and data quality frameworks. • Optimize data processing performance, storage design, and cloud costs. • Design partitioning, data distribution, and performance tuning strategies. • Mentor team members and establish coding standards and engineering best practices. • Drive adoption of AI-assisted development tools such as GitHub Copilot to improve productivity and code quality. Required Skills • Strong hands-on experience with: • Azure Synapse Analytics • Azure Databricks • PySpark • Spark SQL • Azure Data Factory (ADF) • Advanced SQL expertise across platforms such as: • Oracle • SQL Server • PostgreSQL • Experience with Git/GitHub, CI/CD, and Azure DevOps. • Strong understanding of: • Data Warehouse Concepts • Medallion Architecture • Data Governance • Data Security (RBAC, Key Vault) • Experience designing scalable and enterprise-grade data platforms. Good to Have • Exposure to Informatica PowerCenter (IPC) . • Experience with Kafka, Azure Event Hubs, or Streaming Architectures. • Terraform or Infrastructure as Code (IaC). • Power BI and Analytics solutions. • AI/ML integration with data platforms. • Domain experience in Insurance or Financial Services.