CoreWeave infrastructure observability in W&B Models

Key Takeaways

Hi, I'm Russ from Weights and Biases and today I'll be discussing an exciting new integration between Coreweave and Weights and Biases that allows AI and ML engineers to automatically see Coreweave events in their W&B models workspace. When training and fine-tuning AI and ML models at scale, where runs can last days, weeks, or even months, it's critically important to constantly track progress, or lack thereof. WMB models provides AI and ML engineers with a real-time window into every run, offering visibility into key metrics and trends using intuitive, interactive charts and tables. WNB models has always shown the what during the run. When we see emerging problems, W&B models provides us with the answer to the question, what's going on? What are the issues? But now with Corewave observability, users will see the why. Detailed information and alerts from corewave mission control about issues such as degraded Infiniband links or thermal throttling are surfaced directly in the W&B workspace and visually correlated with line charts showing run progress. So it's possible to not only understand when problems begin to occur, but whether these problems are tied to underlying infrastructure problems. Even when running on the best possible infrastructure, components do sometimes fail. And single component failure can sometimes compound into multiple component failure and degrade or halt training runs. But now AI and ML engineers no longer need to fly blind when such issues occur. Understanding exactly what performance metrics are failing during a run and why they are failing allow AI and ML engineers to better collaborate with platform engineers to address problems together and resume or restart training or fine-tuning runs as soon as possible. The value of coreweave observability in W&B models is clear. Now let me show you what it looks like in the workspace interface. This project includes our runs training a large language model over multiple days. From keeping a close eye on the results, I know that many of them have had performance issues and even crashed during the run. Digging deeper to investigate our crashed runs, I'm going to add a filter. So, I'm only seeing the runs where state equals crashed. And as I scroll down my list of runs on the left here, I recognize this sequence of runs called 68 smoke test overlap. I want to drill down into these. So I'm going to add this reax filter in the search box and investigate these runs further. As I scroll down the workspace to these performance metrics, I can see that after some period of steps, each of these runs seems to be experiencing performance issues indicated by a drop in throughput and MFU. Using WMBB models without the core weave and WMBB models observability integration. I can now click on one of these runs and see clearly when significant problems started to occur and cause the run to crash. I can also start pouring through run logs in W&B models and wherever else I might have them and also ask a platform engineer to help though I can only provide relatively limited information. Now, let's see the new information available when using the coreweave and W&B models integration and how it can help me understand the problem, communicate the issue better to my teammates, and resume or restart my training runs as quickly as possible. Let's go back to our list of runs. I'm going to expand this list to see the data for each run in a tabular view. And I'm going to add this column, issues. And now when I go back to my list of runs on the left, I can see the issues column next to each run name. And if I click on this issue button next to our first 68 smoke test overlap run, this drawer on the right pops out and provides information directly from Coreavee mission control about why our run has failed. Node CPU herz throttle long. The node CPU herz throttle long alert indicates that a node CPU frequency has been throttled below 201 mhertz for at least 30 minutes. We now have some insight into our run failure, but we can drill down even further. Clicking on the run, I can now focus again on the performance charts for this specific run, but this time I can see these annotations on each line chart exactly where the problems are occurring. Corewave observability inside of W&B model surfaces these issues and presents them directly on the workspace dashboard so you know immediately how hardware problems may be impacting the performance of your training runs. It's basically like having the answer key overlaid right on top of your run metrics charts giving you critical insight into the health of your infrastructure. Here we see a drop in performance metrics. So I can zoom in and see when the performance metrics drop right when the throttling issue occurred. Clicking on the issue itself and the chart opens up our drawing. Expanding the overview section provides the information we also saw a moment ago, the issue summary, a detailed description of the problem and also a description of the efforts in place by coreweave to resolve the problem. I can also click on this graphana link here and go directly to the job metrics using corewave observability and we can see right here that there are indeed alerts for node CPU herz throttle and also looking at this chart we can see where our performance drops from over 500 pops down to 100 plops. Going back to our issue summary in W&B models, I can also click on a graphana link for each individual issue and see the node details with a superdetailed breakdown of the current status of this specific node. All this information is invaluable in helping an AI or ML engineer and a platform engineer decide how to best proceed and whether and when to resume or restart the run. So for a quick recap on the benefits of the increased visibility delivered by this integration between CoreWave's mission control and W&B models, alerts and remediations are available to W&B models users directly in the workspace interface without having to pour over log files searching for information. Annotations overlaid on experiment tracking line charts allow users to immediately understand when infrastructure events occurred and the effect each event may have had on any given metric. Detailed infrastructure status information allows AI and ML engineers and platform engineers to work together to debug issues and make critical decisions about how to proceed. WMBB models allows AI and ML engineers to quickly rewind and resume runs from the last checkpoint, meaning no need to start over and lose precious, expensive time and resources. To sum up, Corewave observability and W&B models means increased productivity when building models. Better models faster with Coreeave and Weights and Biases. [Music] [Music]