Simplifying Application Deployments on HPC Clusters

Key Takeaways

researchers around the world are making scientific breakthroughs from discovering drugs to accurately predicting the weather to simulating safe landing at high speeds on distant planets to achieve these breakthroughs researchers need not only computational power but also easy access to robust software in this demo you'll see two tools the verification container and ngc container environment modules that help you deploy applications faster and easier on your hpc cluster as well as in the cloud one of the major challenges that hpc users face today is easy access to optimized application software modern application software is increasingly complex with multiple version dependencies and system admins have limited resources to maintain these software environments containers help address this challenge by packaging application and all its dependencies this enables rapid deployment on any system with just a single command regardless of the underlying software stack this approach also provides reproducible results because the software environment is consistent on every system nvidia's ngc catalog offers gpu optimized containers for hpc deep learning and visualization applications these containers can be deployed on x86 or arm systems running either docker or singularity in a data center in the cloud or at the edge the verification container is like a preflight check that confirms if your system is ready to run containers from ngc it verifies the container runtime and hosting gpu infiniband drivers displaying the most relevant information here in red you see that no infiniband devices were found by docker if the system doesn't have infiniband then you could ignore this warning but in this case the system does have infiniband so this is an issue that requires follow-up additional options are needed on the docker command line in order to enable infiniband inside containers re-running the verification container shows that the docker runtime on this system is now ready to run ngc containers now that you verify that your system is container ready it's time to deploy an application you can pull a container and run it using your container runtime or you can use the new ngc container environment modules which are lightweight wrappers that use environment modules to further simplify configuration and deployment let's see this in action using chroma quantum chromo dynamics application the chroma container environment module is already loaded and a benchmark data set is available on the host the mpi run command to run the benchmark is transparently mapped into the chrome ngc container one of the key benefits of the ngc container environment modules is that they're transparent ensuring a small learning curve and minimal changes to your workflows now you can take advantage of all the benefits of containers such as portability and reproducibility without the learning curve and by using the latest hpc and deep learning containers from ngc they ensure that you're getting the maximum performance from your hardware the ngc container environment modules are open source references that you can customize for your specific site requirements for instance you may want to enforce use of a library of pre-downloaded images or bind mount a high performance scratch file system into the containers see what the ngc catalog verification container and ngc container environment modules can do for you visit the link in the description for this video