Create a RAG Chatbot Using NVIDIA AI Workbench

Key Takeaways

welcome to Nvidia AI workbench demo for hybrid retrieval augmented generation project on GitHub this project allows you to run a rag application that can use different sources for inference from GPU on a local machine to a cloud endpoint or a self-hosted microservice this demo is done on a workstation powered by Nvidia RTX 6000 Ada gpus it shows how to do inference in two different modes local and self host microservice for this demo we assume that you have the AI workbench installed on a system with gpus and you have a GitHub account we start on GitHub with hybrid rack project this is a git repository with the code and environment configuration required to build and use the project we then clone it in AI workbench AI workbench pulls the repository to the local machine and builds the containerized environment which includes jupyter lab for development Environ and gradio app for using the rag application once the container builds the Open chat button turns green and you can click it to start the gradio app the chat app has three inference modes for rag local cloud and self-hosted microservice the workstation used in the demo has Nvidia gpus so I start with the local inference option I start the rack server and then select Lama 38b at 4bit Precision once the model downloads I can start the inference server which runs directly on the GPU on the workstation note that this video is significantly sped up for the purpose of the demo it takes about 3 minutes to complete the initial setup once the server is ready I get a message saying that service reachable and happy chatting I say hello to make sure model is up and running and I ask a question about Nvidia Blackwell super chip launch since Nvidia Blackwell details are not included in the models training data there is some hallucination this isn't necessarily a flaw it's expected for factual queries on topics that are not covered in model training data rag is a way of providing a model with appropriate context so that it can better answer this kind of query to show improvements rack and offer I add a PDF for nvidia's Blackwell architecture the rag application extracts the text and embeds it into the vector database where the data is embedded stored and ready for query when Vector database is disabled question comparing blackw and Hopper architectures yield inaccurate answers however enabling the vector database results in accurate responses additionally using the show context feature allows to see the direct text llm uses for generating responses providing a clear retrieval context to perform remote inference Begin by switching to a self-hosted microservice on the chat app ensure that you have an NGC account for accessing the Nim and generate the API key for authentication use the latest image from NGC catalog to take advantage of most recent updates and improvements after that log into Docker and run the Nim container in the chat app enter the required details for containerized models such as IP address and the model itself follow the same steps as local inference note that the vector database is agnostic of the inference mode and provides responses for all modes enabling Vector database results in more accurate responses thank you for watching our demo on setting up hybrid rag using Nvidia AI workbench on a Workstation