AWS Dev and Pro Management // Sahil Khanna // MLOps Podcast #145 clip

MLOps Coffee Sessions #145 with Sahil Khanna, Griffin, ML platform at Instacart co-hosted by Mike Del Balso. Sahil discusses their process for managing development and production environments within Instacart. They use AWS heavily and have separate Dev and production accounts, which have different resources and network configurations. To ensure consistency, they mount the same configurations when developers run things locally. They have a standard interface for integrating custom models into their system, and they deploy them first to a Dev environment, run integration tests, and then to production with a canary deployment to allow for a rollback in case of issues. They also manage the transition of feature pipelines from Dev to production, making the pipeline immutable once it is transitioned. Here are some of their best practices for data pipelines: 1. Separate development and production accounts: Ensure that development activities are done only in the development account and not in the production environment to avoid any unintended changes. 2. Consistent environment configuration: Ensure that the same environment configuration is used locally and remotely so that users have the same access to AWS credentials, environment configurations, and resources. 3. Use of feature store: When the development pipeline is successful, use it as a signal to transition the pipeline from dev to production. This ensures that the pipeline is reliable and has been tested before being deployed in the production environment. 4. Standardizing interfaces: Standardizing the interfaces between different products, such as using RSTK to package models, helps with reliably deploying models and reduces the number of issues that arise during integration. 5. Canary deployment: Use canary deployment to test the deployment of models in the production environment and to have the option of rolling back in case of issues. 6. Immutable pipelines: Once the feature pipeline is transitioned from d