Job Description
Lead Data Engineer
Primary Skills
- ETL Fundamentals, SQL, SQL (Basic + Advanced), Python, Data Warehousing, Modern Data Platform Fundamentals, Data Modelling Fundamentals, PLSQL, T-SQL, Stored Procedures
Specialization
- Databricks Engineering: Data Engineer
Job requirements
Job Description – Data Engineer (Databricks)
Company: Brillio Technologies
Role: Data Engineer – Databricks
Experience: 5–10 Years
Location: India (Hybrid/3 days)
Employment Type: Full-time
About Brillio
Brillio is a leading digital transformation partner delivering innovative solutions across Data, Cloud, AI, and Digital Engineering. We help global enterprises unlock value through cutting-edge technology and data-driven insights.
Role Overview
We are looking for a hands-on Data Engineer with strong Databricks expertise to design, build, and scale modern data platforms. The ideal candidate will have experience in PySpark, Delta Lake, and cloud-native data engineering and will work closely with analytics, product, and business teams to deliver high-performance data pipelines.
Key Responsibilities
- Design, develop, and maintain scalable ETL/ELT data pipelines using Databricks and PySpark
- Build and optimize data lakehouse architectures (Bronze–Silver–Gold layers)
- Develop data ingestion frameworks from multiple sources (batch & real-time)
- Implement Delta Lake features including schema evolution, ACID transactions, and optimization
- Create and manage Databricks notebooks, workflows, and jobs for production pipelines
- Ensure data quality, governance, and lineage across pipelines
- Optimize Spark jobs for performance (partitioning, caching, tuning)
- Collaborate with cross-functional teams (Data Scientists, Analysts, Architects)
- Support production deployments, monitoring, and troubleshooting of data pipelines
- Contribute to migration of legacy data platforms to modern cloud-based architecture
Technical Skills (Must-Have)
Strong experience in:
- Databricks (Workflows, Notebooks, Jobs)
- PySpark / Spark SQL
- SQL & Data Warehousing concepts
Hands-on experience with:
- Delta Lake / Lakehouse architecture
- Data pipeline development & orchestration
Good understanding of:
- Data modeling, ETL/ELT processes, SCD concepts
Cloud & Tools
Experience in at least one:
- AWS (S3, Glue, EMR) / Azure (ADF, ADLS, Synapse)
Workflow tools:
- Airflow / Databricks Workflows
Version control:
- Git / CI-CD pipelines
Good to Have
Experience with:
- Streaming frameworks (Kafka / Spark Streaming)
- DBT / Snowflake / Redshift / Fabric
Exposure to:
- Unity Catalog, Autoloader, Delta Live Tables
Experience Required
- 5–10 years of experience in Data Engineering
- Minimum 2–4 years of hands-on Databricks experience
- Proven experience building enterprise-scale data platforms and pipelines
Soft Skills
- Strong problem-solving and analytical skills
- Ability to work in a fast-paced, agile environment
- Good communication and stakeholder management
Why Join Brillio
- Work on cutting-edge data & AI programs
- Opportunity to build large-scale data platforms
- Collaborative and innovation-driven culture
We may use artificial intelligence (AI) tools to support parts of the hiring process, such as reviewing applications, analyzing resumes, or assessing responses and identifying potential inconsistencies or verification signals in application materials based on available information. These tools assist our recruitment team but do not replace human judgment. Final hiring decisions are ultimately made by humans. If you would like more information about how your data is processed, please contact us.
