Nvidia’s Chat with RTX is a promising AI chatbot that runs locally on your PC

Nvidia today is releasing an early version of Chat with RTX, a demo app that lets you run a personal AI chatbot on your PC. You can add YouTube videos and your own documents to it to create summaries and get relevant answers based on your own data. It all runs locally on a PC and all you need is an RTX 30 or 40 series GPU with at least 8GB of VRAM.

I briefly tested Chat with RTX last day, and while the app is a bit rough, I can already see it being a valuable part of data research for journalists or anyone who needs to analyze a collection of data. documents.

Chat with RTX can handle YouTube videos, so all you have to do is enter a URL, and you can search transcripts for specific mentions or summarize an entire video. I found this ideal for searching through video podcasts, especially for finding specific mentions in podcasts from the past week amid rumors about Microsoft’s new Xbox strategy change.

However, it wasn’t perfect for searching YouTube videos. I tried the transcription of one Edge YouTube Video and Chat with RTX downloaded the transcript for a completely different video. It wasn’t even a question I’d asked about before, so there are clearly bugs in this early demo.

a:hover]:text-gray-63 [&>a:hover]:shadow-underline-black dark:[&>a:hover]:text-gray-bd dark:[&>a:hover]:shadow-underline-gray [&>a]:shadow-underline-gray-63 dark:[&>a]:text-gray-bd dark:[&>a]:shadow-underline-gray”>Screenshot by Tom Warren / The Verge

When it was working properly, I was able to find references in videos in seconds. I also created a dataset from FTC v. Microsoft documents for Chat with RTX to analyze. When I was handling the lawsuit last year, it was often overwhelming to quickly search through documents, but Chat with RTX allowed me to search them almost instantly on my PC.

For example, the chatbot did a good job of summarizing Microsoft’s entire Xbox Game Pass strategy based on internal documents revealed during the lawsuit:

Based on the context information provided, Xbox Game Pass is a gaming content subscription service that provides access to a library of games for a monthly fee. It aims to be a platform that allows players to play their games anywhere and publishers to reach players everywhere. The service is supported by Project xCloud and Xbox Series X and aims to create a fertile ground for publishers to monetize their games. The Gaming CSA’s main strategic goal is to scale Xbox Game Pass, and it is believed there is significant subscriber opportunity globally, with a potential of 750 million subscribers. The service faces three primary content dynamics, including the need for differentiated content, expansion beyond the console, and limited content offerings.

I also found this useful for scanning PDFs and checking data. Microsoft’s own Copilot system can’t properly handle PDFs in Word, but Nvidia’s Chat with RTX had no problem extracting all the important information. Responses are also virtually instantaneous, without the lag you normally see when using cloud-based ChatGPT or Copilot chatbots.

The major disadvantage of chatting with RTX is that this is the case Real feels like an early developer demo. Chatting with RTX essentially installs a web server and a Python instance on your PC, which then uses Mistral or Llama 2 models to query the data you add to it. It then uses Nvidia’s Tensor cores on an RTX GPU to speed up your queries.

a:hover]:text-gray-63 [&>a:hover]:shadow-underline-black dark:[&>a:hover]:text-gray-bd dark:[&>a:hover]:shadow-underline-gray [&>a]:shadow-underline-gray-63 dark:[&>a]:text-gray-bd dark:[&>a]:shadow-underline-gray”>Screenshot by Tom Warren / The Verge

It took about 30 minutes for Chat with RTX to install on my PC, which is powered by an Intel Core i9-14900K processor with an RTX 4090 GPU. The app is almost 40 GB in size and the Python instance takes up about 3 GB of RAM out of the 64 GB available on my system. Once it’s up and running, you can access Chat with RTX from a browser, while a command prompt runs in the background showing what’s being processed and any error codes.

Nvidia isn’t offering this as a polished app that all RTX owners should immediately download and install. There are a number of known issues and limitations, including that citation is not always accurate. I also initially tried to get Chat with RTX to index 25,000 documents, but this seemed to crash the app and I had to clear preferences to get back up and running.

Chat with RTX also doesn’t remember context, so follow-up questions can’t be based on the context of a previous question. It also creates JSON files in the folders you ask it to index, so I wouldn’t recommend using this across your entire Documents folder in Windows.

I love a good tech demo, though, and Nvidia certainly delivered that here. It shows the promise of what an AI chatbot can do locally on your PC in the future, especially if you don’t want to subscribe to something like Copilot Pro or ChatGPT Plus just to analyze your personal files.

Leave a Reply

Your email address will not be published. Required fields are marked *