Job description
Job Summary
You will be responsible for designing, building, and maintaining scalable machine learning pipelines, deploying models to production environments, and ensuring the reliability and scalability of ML operations. The role involves managing infrastructure, implementing CI/CD pipelines, containerization, API management, monitoring, security, collaboration with data scientists, and performance optimization.
Reporting Structure
· This job reports to the Manager – AI.
Job Objectives
· Design, build, and maintain scalable ML pipelines and deploy models to test and production environments.
· Set up and manage cloud and on-premises infrastructure to support ML operations.
· Develop and maintain CI/CD pipelines for ML models and automate build, test, and deployment processes.
· Utilize Docker and Kubernetes for deploying ML models and manage containers for smooth operation and scalability.
· Develop and manage APIs to support ML models, monitor and secure API calls, and ensure seamless integration with external applications.
Job Responsibilities
Pipeline & APIs Deployment and Management
· Design, build, and maintain scalable machine learning pipelines to ensure efficient data processing and model deployment.
· Develop and manage APIs to support machine learning models and services.
· Ensure seamless integration between machine learning models and external applications.
· Utilize API management tools to monitor and secure API calls, enforcing access control and data protection measures.
· Deploy machine learning models to various environments, including testing and production, ensuring seamless integration and functionality.
· Ensure the reliability, availability, and scalability of ML pipelines by implementing robust monitoring and alerting systems.
· Provision pipeline operations effectively, managing resources such as compute, storage, and networking to optimize performance and cost-efficiency.
(CI/CD) Implementation & Containerization
· Develop and maintain CI/CD pipelines tailored for ML models and applications.
· Automate the build, test, and deployment processes.
· Utilize containerization technologies such as Docker and Kubernetes for deploying ML models, ensuring consistency and portability across environments.
· Manage and orchestrate containers effectively to optimize resource utilization and maintain scalability.
Performance Monitoring and Optimization
· Implement comprehensive monitoring and logging solutions to track the performance of ML models and pipelines, enabling proactive issue detection and resolution.
· Set up robust alerting systems to detect and respond to issues and anomalies promptly, minimizing downtime and performance degradation.
· Ensure compliance with security standards and regulations, implementing measures to protect data privacy and model security.
· Continuously monitor and optimize the performance of ML models and infrastructure, identifying and resolving bottlenecks to improve system efficiency.
· Respond to and resolve incidents related to ML operations promptly.
Scalability and Resource Optimization
· Set up and manage both cloud and on-premises infrastructure to support ML operations.
· Optimize models and infrastructure for performance and scalability in production environments, ensuring efficient and reliable operations.
· Manage resource allocation to ensure cost-effective operations.
· Develop scripts and automation tools to streamline ML operations, automating repetitive tasks to improve operational efficiency.
Disaster Recovery and Incident Repo rt
· Implement backup and disaster recovery plans for ML models and data.
· Ensure data and model availability in case of failures.
· Conduct root cause analysis and implement preventive measures to mitigate future occurrences.
Collaboration and Best Practices
· Collaborate closely with data scientists and engineers throughout the ML lifecycle, from model development, and testing to deployment and maintenance.
· Collaborate with data scientists and AI researchers to develop and test machine learning models.
· Provide support and guidance on best practices for ML operations, facilitating effective teamwork and knowledge sharing.
· Implement best practices for model versioning, testing, and validation.
Job Requirements
Educational Qualification
· Bachelor’s or master’s degree in computer science, Engineering, Data Science, or a related field.
Previous Work Experience
· 4 years of proven experience as an ML Ops Engineer or similar role in a production environment.
· Experience with Azure cloud platform. AWS experience is a plus.
· Experience with containerization technologies (Docker, Kubernetes).
· Experience with API management tools (Kong)
Skills and Abilities
· Strong programming skills in Python
· Proficiency in CI/CD tools
· Familiarity with machine learning frameworks (TensorFlow, PyTorch).
· Strong understanding of DevOps practices and principles.
· Excellent problem-solving skills and attention to detail.
· Strong communication and collaboration skills.
