Job Description
Job Description
Lead Data Engineer
Experience
10+ years of relevant data engineering experience, with a strong track record of designing and leading large-scale data platforms and a working knowledge of Large Language Models (LLMs) and GenAI-enablement.
Location
12:00 Noon – 10:00 PM IST
Bangalore (Hybrid – 2–3 days in-office per week)
Profile Summary
About Company
Trantor is a pioneer in creating enterprise technology solutions and state-of-the-art CaptiveCoE™ assisting enterprises across the globe with their digital transformations and business needs. Our commitment to excellence and authenticity has led to long-term working relationships with our clients and solution partners.
Website: https://www.trantorinc.com
We are seeking an experienced Lead Data Engineer to drive the design and delivery of scalable, reliable, and cost-efficient data platforms. This role requires deep, hands-on expertise across the modern data engineering stack — distributed data processing, data modeling, ETL/ELT pipeline development, and workflow orchestration — along with a solid understanding of Large Language Models (LLMs) and their data requirements. This role requires strong, hands-on expertise with the Databricks Lakehouse Platform, including the medallion (bronze/silver/gold) architecture, Apache Spark, and Delta Lake. The ideal candidate will combine strong engineering fundamentals with technical leadership, mentoring engineers, setting best practices, and partnering with architects, data scientists, and business stakeholders to enable advanced analytics and GenAI use cases.
Key Responsibilities
- Lead the design and development of scalable, high-performance data pipelines and platforms across batch and streaming workloads
- Architect and maintain ETL/ELT workflows for data ingestion, transformation, and curation across the medallion (bronze/silver/gold) architecture
- Define and govern data models, schemas, and storage strategies for data lakes and data warehouses to support analytics, reporting, and ML/LLM workloads
- Build and curate high-quality datasets, feature pipelines, and retrieval-ready data stores (including embeddings/vector data) to support LLM and GenAI applications
- Establish and enforce engineering best practices, coding standards, code reviews, and CI/CD for the data engineering team
- Implement workflow orchestration and scheduling using tools such as Apache Airflow (or equivalent), and automate end-to-end data workflows
- Lead migration of data warehouses and pipelines from legacy on-premise and cloud platforms to modern cloud-native and lakehouse architectures
- Apply performance tuning and cost-optimization techniques across distributed processing, partitioning, caching, and compute configuration
- Establish data governance, security, lineage, and access control aligned with compliance and privacy requirements
- Build monitoring, logging, alerting, and data-quality frameworks to ensure pipeline reliability and proactive issue resolution
- Mentor and guide data engineers, and collaborate with architects, data scientists, and business stakeholders to translate requirements into robust technical solutions
- Participate in and lead Agile/Scrum ceremonies including sprint planning, daily stand-ups, and retrospectives
Required Skills (Mandatory)
- 10+ years of hands-on data engineering experience, including technical leadership of data platform initiatives
- Strong, hands-on expertise with the Databricks Lakehouse Platform, including Delta Lake, Delta Live Tables, Databricks Workflows, and Unity Catalog
- Proven experience designing and implementing the medallion (bronze/silver/gold) architecture for data ingestion, curation, and consumption
- Strong expertise in distributed data processing using Apache Spark (PySpark and Spark SQL) or equivalent big-data frameworks
- Proven experience designing and building ETL/ELT pipelines for both batch and streaming data at scale
- Expert-level proficiency in Python and SQL for data transformation, validation, and pipeline development
- Strong experience building and managing data lakes and data warehouses, including dimensional and lakehouse data modeling
- Working knowledge of Large Language Models (LLMs) and GenAI concepts — prompts, embeddings, vector databases, and Retrieval-Augmented Generation (RAG) — and the data pipelines required to support them
- Hands-on experience with at least one major cloud platform (AWS, Azure, or GCP) and its core data services
- Proven experience leading large-scale data migrations from on-premise (e.g. Hadoop) and cloud platforms to modern data architectures
- Strong understanding of distributed data processing, partitioning, and performance optimization techniques
- Experience implementing data security, governance, lineage, and access control (e.g. IAM, encryption, cataloging)
- Strong understanding of object-oriented programming, software design patterns, and CI/CD practices
- Familiarity with Agile/Scrum delivery methodologies and experience mentoring engineers
- Excellent problem-solving, analytical, and stakeholder-management skills
- Strong verbal and written communication skills
Required Skills (Good to have)
- Experience building LLM-powered applications or data pipelines using frameworks such as LangChain, LlamaIndex, or similar
- Experience with vector databases such as Pinecone, Weaviate, FAISS, or pgvector
- Relevant cloud or data engineering certifications (e.g. Databricks, AWS, Azure, or GCP)
- Knowledge of streaming frameworks such as Spark Structured Streaming or Kafka
- Experience with infrastructure-as-code and DevOps tooling (Terraform, Git, Jenkins, Azure DevOps)
- Exposure to MLOps / LLMOps practices and model lifecycle management
- Exposure to BI/visualization tools such as Power BI, Tableau, or QuickSight