Newsfilter article unlocker

All right. Well, thank you everyone for joining. My name is Sean Kinney with RCR Tech. We've got a great discussion lined up for you today on AI-era network fabrics. Some quick housekeeping notes. We are recording the session today. We will make that available on demand if you'd like to revisit the content. I've got a companion report in the works that will expand on the conversations you'll hear today. That's going to be emailed to registrants in the coming weeks. And we've got some time set aside at the end of the hour for Q&A with the audience. So please submit questions throughout, and we'll get to as many as time permits. To give you a brief bit of context, we're all familiar with the historic ongoing AI infrastructure investment cycle that we're in. Goldman Sachs estimates cumulative global AI infrastructure capex at $7.6 trillion between 2026 and 2031. Right now, that investment is largely focused on hyperscale data centers that are used to train frontier models. But at the same time, we're tracking a shift away from training towards inference, where the thinking is that in the coming years, 80% of AI infrastructure will be used for inference. So as AI-enabled use cases continue to develop, particularly things like physical AI, the idea is that the infrastructure needs to be distributed closer to the endpoint. In addition to performance benefits, particularly latency that comes with compute living close to the application, you can also see benefits around data locality, data sovereignty, control, and cost. So in practice, this means that hyperscalers, neoclouds, network operators, and enterprises need to think about how to place and orchestrate workloads across a distributed heterogeneous infrastructure in a way that delivers the desired outcome while also optimizing for cost and resource allocation. And to talk through the topic, we've got a great panel for you today. We're going to hear from Clayton Wager, who leads AI and high-performance networking at Nakia, Atul Deshpande, Principal Chief Architect in the Field CTO Office for the Global Telco Vertical with Red Hat, and Urvashi Chowdhury, Vice President of Product Management for AI Platforms and Services with CoreWeave. So thank you all for joining the conversation today. Reminder to the audience that we are able to take Q&A, so please submit those questions. And now we're going to kick things off with a presentation and a conversation with Clayton. Thanks, Sean. I appreciate it. Great setup. So we've known for a while in the AI industry that we would have a transition from training the models, building these, you know, wonderful frontier models and open source models that we have towards actually consuming them. And if you look at a lot of the data that's out there, it supports the idea that we are still at the very, very beginning of AI adoption. While it might seem to some of your audience members that AI has become a part of their day and they use ChatGPT or Claude or another tool to assist them during the course of their day. The reality is that the consumption of AI is still in extremely early days. And as part of that, we take a look at, you know, how do we then deploy that inference? And you're exactly right to say that it's tempting to take a Look at the big super data centers that we're building today and to think that that's where all the inference will happen. But it's not. In fact, the distribution of it is happening as we sit here today. And we expect it to be a very, very long-lived investment cycle in order to do that. So if we advance one slide, we take a look at what that distribution means. Today, a lot of focus, when you talk about CapEx numbers, when we talk about analysts that are tracking the industry, a Lot of that is focused on the right-hand side of this diagram here in the AI factory, and certainly incredible amounts of CapEx being spent relative to what we're used to in the technology industry. We know that there is still today a gap in the demand and supply of compute. And so that build looks as though it will continue for many years to come. But a lot of that still, of course, is in the development of the products. We see a lot of inference serving these days, certainly. But it still is sort of limited to those products that we have now, ones that are coming down to an individual user. Perhaps they help you code. perhaps they help you as a knowledge worker organize your thoughts or develop a product specification whatever it happens to be that lives in the cloud today in a very centralized location well with the advent of ai now sort of moving into a bunch of different parts of our lives physical ai robotics is the easiest example and so physical ai is is likely going to have a lot more video spatial data it's going to be more time sensitive when you're talking about robots operating at high speed on a factory floor. We want that inference, the AI that's controlling them to live closer to them. So too, enterprises are starting to have a huge adoption cycle as they start to learn from the first cloud products on how they might use AI, train their own models, take their important proprietary data and make something useful out of an AI model. Enterprises will start to bring that back on premises to have the better unit cost, First of all, perhaps there's regulatory concerns like, you know, healthcare or financial. And so all of these trends are pushing the distribution of AI. Sovereign AI is another one, right? So that may not be down to an edge, but it's countries are starting to build capacity within their borders in order to have the sovereignty of AI for their citizens. And so that distribution of Ai as we go into the field causes us to take a look at how we do that. What's the underlying data movement and network that supplies that? If we fast forward one slide, we can see that that ends up being a very complex many-to-many type of function. And guess what? This looks like the internet that we have built. Many of us who have been involved in this industry for a number of decades, we have been building public internet, private internets, you know, clouds that are connected together across the globe. Now, really, for more than five decades,we've been doing this. And so the lessons that we learned there about bringing large domain networks together, federation, security, all of these pieces that we use to build a robust, resilient internet globally, Nokia had an insight that said, look, this is actually AI is going to be a brand new type of workload on these networks. and so we're going to go through a cycle of discovery and investment on how we can carry those AI workloads in the best possible way. New features, new solutions, new platforms, all of the things we've been doing are going to have to be rethought with AI. You may hear this called the AI grid, right? The AI grid is this idea that workloads will be distributed in many, many different forms and fashions, either based on technology drivers or based on economic drivers And so being able to distribute AI in a very programmatic way to have structural pieces in there like settlement and billing and interchange, interworking, all of these things will come to AI. And we're, again, at the very beginning of that investment cycle. If we go one more slide, please, what we'll see is that some of the viewers today may be familiar with the very famous scale networks and AI right scale up, which allows us to build a very big rack or double wide rack or set of computers that are connected together to make a larger compute complex scale out being then the network that connects all of those individual scale up domains together and allows us to move data around both in a training and inference and an extremely you know critical part function of scaling and then of course as we sort of escape the walls of a single data center and need to bring two buildings on a campus or two buildings in a region together to provide a very large compute complex that's called scale across and so that extends that scale out domain across again buildings on campus or within a region allows us to build a bigger computer but But all of those networks, scale up, scale out, scale across, are really within a single operator domain. You see them used to, again, create a bigger compute complex. As we start to bring enterprises together with clouds and we bring sovereign AI together with edge AI, as we startto distribute things like GPUs to the bottom of base stations in order to both manage the wireless network in the form of AI RAN, but also provide compute capability for jobs which might run on them as part of a large pool of assets of resources. That scale beyond is what we're calling it, right? The idea that scale beyond going beyond the data center walls. So scale beyond as this discovery space now that as an industry, we will have to take a look at and figure out again, what are the features and the solutions, capabilities, new platforms that we'll need to invest in as we connect all of these pools of AI compute together in capability. And really, we're at the very beginning of understanding this. We know that with the explosion of agentic AI, that we have more demands on CPU, on storage, disaggregated memory. All of these things are causing the size and shape of AI to grow. And so Scale Beyond will allow us to, again, make the right technical and economic decisions to distribute that capability, whether it's moving inference to one location and potentially KD cache being replicated in order to create some efficiency from that, whether it's tiers of memory that allow us to store our conversations that are happening right now and those that we had six months ago and three years ago and synthesize all those together. Those might have to live in different physical locations or even logical locations within a campus. And so Scale Beyond allows us to capture that. Again, built on many of the same fantastic service provider technologies, which have been deployed over the past many decades, but with a view towards the unique requirements that AI will drive over that infrastructure. If we fast forward one more, we'll see what might those capabilities be. Well, certainly a large part of today's internet was built for human consumption. If we look at typical traffic in the internet, it's very asymmetrical, headed outbound towards consumers. Much of that is because of things like streaming video, the media that we're consuming. We don't generally upload as much media as we do consume it downward. And so the symmetry of our connections in the ScaleBeyond network will have to be evaluated. We're going to see machines talking to machines 24-7, 365. And so, the ScaleBeyond network will no longer be limited by human consumption, right? This is now a machine network where machines are talking to one another, and they'll be passing data around quite symmetrically from place to place. You know, the performance of the network has always been important. We have focused on that as network engineers for many, many decades, but taking these new workloads and the type of traffic that they're generating, we can't assume that they will always be the same internet-style traffic, IP, TCP that we've had before. We might have high-performance requirements using DMA technologies to transfer data from one place to another in order to gain efficiencies, and so the service level agreements around that will have to change. This is a really interesting part of Scale Beyond. As I talked to customers about the concept over the past several months, this one was one that spoke to them as sort of an immediate action because we have carriers and telcos around the world which want to take advantage of this AI super cycle and offer products to clouds and enterprises. But the language to do that doesn't exist yet. We don't have standardized products around the globe that allow us to be effective purchasers and suppliers into the market. So this idea that we can structure a new set of SLAs around that is a key part of Scale Beyond today. The workload placement, again, using concepts like the AI grid, which was really orchestration layers to put different pools of capacity and different types of capability where they make the most sense. And being able to program the network and do that is also of critical importance. Today, many of those ads, moves, and changes are done with human intervention or at least human triggering. And certainly, we expect in the future that the networks will have to be AI-controlled so that AI can make a determination on exactly what bandwidth it needs where and instantiate that in the network on demand, even at 4 o'clock in the morning when there's not a human around, and do it with high precision. So the programmability of the network is key. And then, of course, like anything, security is always a key part of what we do. We have sovereignty concerns, privacy concerns with AI. And so having deep security throughout all layers of the stack, every single one of the network transport layers is a key part of that. And we have invested quite a bit, at least in Rokia, we've invested quite a bit on security within the network. And we expect those demands to increase over time with AI as a workload. So scale beyond is this, again, this new discovery space. It's for the industry. It's not something that at Nokia that is a product of ours. This is just a concept that we want our peers in the industry to start living in and understanding how we can, as a collective group of suppliers and consumers, make the right decisions around product roadmap, around investments, and optimize AI as a workload across the entire planet. No matter what shape and size it might be, all the way from a pair of glasses you have on, smart glasses, alltheway through the very largest gigafactories. on the planet. So that concludes the prepared material, Sean. All right, excellent presentation, Clayton. I appreciate you taking us through that. Some follow-up questions. You know, you mentioned this on your second slide, I believe, but if you look at scale up, scale out, scale across networks, and then look towards this idea of scale beyond, what is the principle constraint today? And then where does that shift as this scale beyond topic is sort of realized in the real world? Yeah, so some of your clever viewers might recognize that Scale Beyond looks a lot like what today we call data center interconnect, right? And so in some ways it is, in one of its forms, a basic uplift of data center, interconnect. And this is an area that we focused on, the industry has been focusing on, we at Nokia have for quite a while. We have a strong heritage in service provider routing and data center interconnected. And so today we lean on those technologies as a solution to connecting these AI domains together. With Scale Beyond, we're thinking beyond that, ironically, right? That's why the term exists, is it's not just simply about connecting big buildings, which is the job of something like DCI. It's about how do we programmatically offer that smart connection down to any size and shape? Again, something that might be as small as a device I'm wearing, or something as large as a large data center. The primary constraint is that this is a new space for everybody. We're all learning as we go. Much of the learnings are within large hyperscalers and neoclouds, and as forthcoming as they generally are in industry events, and they do a wonderful job documenting what they've done, we as an industry and suppliers need to jump in and come up with some more clever solutions. So just like anything, one of the key limiting factors is just the time to spend innovating around this and coming up with better solutions. We're going to have to think outside the boxes, they say on this, and not make the assumptions that traffic is the same as it has been with the internet for a long period of time. The example that I have used before, I've been around the industry for a while, there was two real big events that had the same sort of effect on the internet. The first is when we had IP voice. So if you think about the days when Skype came out and we were replacing landlines and we had to create an internet that could support these IP voice connections with robustness across the planet, because that's what consumers expected. The other one was streaming media, right? With the rise of Netflix and Hulu and all of these others, we had to re-architect, especially our broadband access networks, in order to support those applications that consumers were taking us to. I think the same exact thing is going to happen with AI. We're going to see those requirements develop. And as suppliers and consumers in the industry, the biggest thing that stands in our way is simply working together to document that so that we can accelerate road maps and grow this new scale beyond network to be suitable for the AI workloads of tomorrow. You referenced this a little bit, but I wanted to explore this idea of the KPIs that we should be tracking. I think you had a slide where you talked about, you know, right now we have humans that are clicking and that's where our AI traffic comes from. But as we start to stand up agents at scale, you Know, that's moving at machine speed. It's always on. It'S more symmetrical. So as that progresses, what should we be measuring beyond just throughput and latency across a network? What's important? Yeah, it's a great question. So if you look at what value AI workloads provide, of course, it is very common to use tokens, right? Tokens per minute, tokens per second, the amount of tokens that you pay for when using an API. But when using AI in a more robust fashion, again, I mentioned the fact that we have more CPU capacity, we have more memory capacity and storage, not every AI workload is token generation. Some of it might be retrieving things from storage, like off of a cloud service, OneDrive, Dropbox, these type of products. Someof it mightbe a disaggregated memory service. An agent running within an enterprise or running on my laptop, wherever, and running in a base station might consume different bits of those things. And not all of those are equal. Some have higher weight. They're more time sensitive. Some are more valuable. They have more value to deploy. So what we're looking at with everything on the table, again, returning to first principles about how we measure across a network, it might be that the different types of AI workloads or the different kinds of sub workloads within those might be metered and built in a different way. And so having the instrumentation to be able to do that at very high precision is of keen interest to us. Certainly things like the amount of bandwidth and the guarantees we make about performance of that bandwidth are the bedrock of interconnection of networking. We'll still have those, but then starting to understand on top of that, especially for those service riders and carriers and clouds that might have blended economic models, profit formulas. Some might be very happy to provide a pipe between two places and they want to specialize in that. Some might decide to move up the stack and offer more robust services, more user-facing services. And so being able to measure those different types of workloads, I think is going to be one of the key pieces of attention as we move forward with Scale Beyond. Then one last one, Clayton, and if we could keep it relatively brief, but you mentioned this concept of AI grid. And last week, Nokia made a significant portfolio announcement around AI RAN. So I was hoping you could maybe put that into context of our conversation today. Yeah, so AI RAND is an exciting technology. It helps in both the optimization of the radio network itself, which relates directly to the ability of carriers to deliver the right service at the right cost per bit. And then also a future view of AI RAN takes that capacity, which is within a base station, and might allow us to deploy workloads on top of that, which are not related to the RAN, right? So it's the very beginning, if you will, of this idea of AI grid. There will be many hundreds of thousands or millions of locations where this will take place. And AI grid allows us to orchestrate that and move those workloads where they might make sense for physical AI, they might makes sense based on cost of energy, right? Follow the electron instead of follow the sun. They might make sense because we need more capacity for an event that's happening at a stadium or something like that. And so AI grid, if you really can imagine it as that smart orchestrator of knowing all those business rules and being able to move things around, the connectivity fabric of the AI grid is scale beyond. That's what will allow us to move those workloads around and optimize them with relative freedom, right? That's what we want from a scale beyond network is that it doesn't get in the way of optimizing the AI workload. Excellent, Clayton. Well, I really appreciate you sharing your perspective with us and you're going to rejoin us for Q&A. Audience, please submit questions for Clayton and now Atul, I would invite you to join us and I will get your presentation up. Atul, if you can join us on audio and video, please. Yeah. Hi. I'm just starting my video. Okay. So it's a very interesting topic. I think what my eminent panelist Clayton has talked about, the AI grid. And right now, what we will do in my talk about how we can actually build some building blocks of this AI grid altogether. So the topic I'm going to talk about is how you can schedule or orchestrate the inference, which is a kind of taking a central stage across the open hybrid cloud. So it's moving across from edge to core kind of architectures. And the theme of this talk is all about any model, any accelerator, and any cloud. That's the real theme which we're going to focus on. So what we're gonna do is that we will start with the problem statement, which is what you see on your screen right now, is what makes a Gen-AI model different from the cloud-native apps, right? And as you could see, we had a decade-long experience on running the cloud native apps. Mostly they are stateless. They run on CPUs, and the requests going to those applications are mainly uniform, and they're very predictable in nature. So any load balancer around roaming technology can actually push those requests at the back end in a symmetric way. But then when you look at the Gen-AI, right, the generative AI which is taking the storm, it is completely different, okay? So it runs on GPU. First of all, that's a fundamental difference, right? The hardware itself is very, very expensive. And the request, right, when we talk about the request that typically the prompt request are very non-uniform. Some are very lengthy, some are very short, some are reasoning requests. So there are a lot of types of requests. And you see the responses are going to vary, right. So that's another difference. and the Gen-AI is typically a stateful, I would say, right? So when I say stateful so it simply means that it utilizes the KV cache from the previous request, right. So if you look at on the right-hand side of the architecture, right if you put a standard Kubernetes load balancer you're going to hit a latency spike and cache crashing, right so even though the other nodes had a GPU available or the VRAM available, right? So that's completely blind. It's not going to see those state, right. But if you look at it to solve this problem, right, you need to have a state-aware router and that state- aware router is actually going to address these challenges, right So it is going to say the warm KV cache where the VRM is full, where the DRAM is available and then it's going to basically send that inferencing request to the right set of GPU port optimally, keeping those SLA intact. So that's what we are heading to. And that's where the Gen-AM model or deploying the Gen AM model for inferencing makes it unique. Now, the question is, how do we build a platform to do exactly the same thing? How we can actually take the request and send it to the right backend port where there's warm KV cache, where the GP available, there's VRAM available. And that's one of the next slide. We will see, this is the AI platform which Red Hat is building with the ecosystem as well as with the upstream community. And as you could see, right, that's a stack altogether. So I'll start from top. But you have an AI gateway. That's an interface. That's a typical gateway. It will do your API management. It will doing your token tracking, your billing. It will also do the authentication. and many other functionalities. It's a single box. And then what comes next is a distributed inference stack, which is based on LLMD and KSERV, okay? So that's the stack which we are actually going to address the challenge what we just saw in the previous slide, the stateful nature of the Gen AI models, right? So with this stack, LLD, which is running by KSRV, right, you're going to take those requests and send those requests to right set of pods at backend. Now, another interesting fact is that you will also have your validated optimized models, right? These are open source, open-weight models available from Red Hat Catalog as a part of this deployment architecture. And everything is going to run on VLNM as an inference server, right, that's a standard deployment model. And one last part of that at the bottom, right. This all, the deployment is going to run across different accelerator families, be it NVIDIA, AMD, Intel, IBM, right? So the interesting part of that, now, when you look at the inferencing requests going to send back in, you need to have this stack in place, which call it a distributed Gen-AI inference platform altogether. Now, with this platform, which is running across different accelerators, what kind of architecture you can build? You may ask that question, right, with this stack right in place. So on the next slide, that's an example of an enterprise model as a service architecture where you would see, and this is a very popular architecture pattern coming in recent times where many enterprises, they want to basically have their own model catalog, and they wantto serve those model catalogs to different consumers, consumers, so be it a developer, be it business user, be other enterprises. So the use case is of similar nature, but the consumption model is completely different. And what we are doing is we have actually built this architectural pattern as part of our AI stack, which is again on the previous slide, what you saw at the distributed journey app platform. And here, what you see a central IT team, I'll go from left to right. So typically a central IT team for an enterprise, they will have a GPU host, and they want to basically use those GPU hosts to offer as a model and service. The second thing what they will do, they'll take a set of more validated models, which they are looking to offer it to their consumers, whether it's a developer, again, or a business user. So they can actually take it from Red Hat's validated repo. So it's all governed now, right? So you know your hardware, you know what models you want to serve, And then you want to take those models in a compressed way, so LLM compressor, which will offer you an unprecedented amount of accuracy, efficiency, and GPU utilization. So I'll not get into the details of smooth cons or GPTQ, but that's an LLOM compression technique, which Will offer a tremendous amount of GPU capacity to wider consumption. And in the end, once you have all this figured out, you are going to expose this all through an AI gateway, which if you remember from the previous slide, that's a top layer, right? And it's going to securely expose through APIs, and then this operational economics could be monitored. And then on the right hand side, the developers are going to consume those APIs and build a different type of AI applications, different agentic applications, or you have business users that are going play around with a different set of models. On top, what you see is that the operational economics, you can actually track the real time telemetry. So you can track a lot of KPIs and this set of KPIS are evolving. are evolving so so you can see the time to first token time to output token kv cache utilization and a lot of other uh model related matrices we are building so you can actually track those real-time telemetry so the it team can actually see who's consuming what uh and how as it is it really meeting their select criteria right so so that's how uh the enterprises are adopting this kind of platforms to offer uh the model health service but now but But now real questions come, has this been proven somewhere, right? Is this something, someone has adopted this, right, is it real? So if you see on next slide, this is something which we have been doing with our customer. And as you can see, this is a public case study and joint engineering blog from Renad with Tesla. It talks about how this state aware routing, right. Or state aware orchestration is driving the efficiency at the next level, right? And on the left-hand side, what you see here, just to give a sample of the architecture, and I've not gone into the details of this architecture, is the pre-fill and the decode phases. That's a transformer architecture. So now what you do is that you run these two different transformer stages on different nodes, rather than putting them on a single node, right. And pre-filled phase, as some of you may be aware of it, pre-filled phase is very compute intensive, right, So probably you want to run it on a separate node and the decode phase is more on the memory bandwidth intensive, right? So rather than putting these two architectural stacks together on a single node and content for the same GPU, what you're doing is that you're putting in a different nodes and let the intelligent orchestrator LLMD, right, orchestrate those inference requests to the different phases or different nodes. So with this architecture, now what we see and with the blog results, right? There are three X more capacity, same hardware. No, you're not adding any hardware here, okay? And you're actually getting a two X faster responses, right, like TTFD, okay. And there are many other details. So this type of architecture, what you saw, right is solving the problems of statefulness of the journey and models. The stack is getting deployed. you're building those architectures and you're putting in the production and you running it already on scale, right? That's what we have been doing with some of our customers. On our next slide, and that's the last slide, we have done some performance benchmarking of this distributed inference stack. And as you saw right from previous slide, right, it's offers 2x TTFT reduction, 3x capacity, but then we have, we have done a third-party industry benchmark. That's an ML for inference, right. Virgin 6.0, and what do you see, right? That's a clear model, a vision model. And you see that 50% faster vision on B200 versus the standard baseline of B300 setup, right, and then Whisper 30% faster. So the third-party benchmark itself also quoting that the VLLM stack, what you're seeing, is outperform the global throughput benchmark here. And on the right-hand side, the other card, right?, which talks about, that's what from LLMD launch block with IBM, what we have done. We have been testing it for quite some time. And what we see is that under strict SLA maintenance, right? P95, time to first token less than two second latency, you're able to basically still sustain the two X queries per second. That's phenomenal now, because now you're going to sell two times more consumers, two times More Inferency Request under the same hardware, maintaining that SLA. That's phenomenal. So in summary, what we see as a transforming, right? And when we look at the AI grid, just want to connect that idea. The AI grid is all about how you can intelligently transfer, orchestrate, and schedule those requests on a different set of hardwares across multiple data centers from H to core. And that intelligence requires a complete stack which is a distributed journey app platform and and with that platform you can actually drive the performance utilize the capacity drive the optimization and it meets the inference standards and and it also meets the baseline qps and restricts sla yeah so that's it from my side uh happy to take any questions uh later thank you yeah atul thank you for that and if you're able to join me on uh camera i did have a couple follow-ups um you know you drove but you uh you made the point in your presentation that red hat's approaching this whole space with a openness and portability front and center uh you had one slide that took us through the diversity of accelerators that we see in the field gpu cpus tpus etc i'm just kind of curious as those accelerators become increasingly differentiated how portable can an ai workload really be without sacrificing performance oh that's a great question and uh okay let me start my video i think there's some uh yeah so oh that'S a that'S A very interesting question and and i think i think a lot of our customers right and a lot OF our partners right where we work with ecosystem uh the architecture is heterogeneous right no one no one is uh want to just stick around one accelerator, right? Because of a lot of challenges, right. But if you look at my slide on the Janiya platform, right, the inference platform, that's exactly the answer is all about, right You can actually take that stack all together with the AI Gateway, the LLMD, the model repos and the inference framework and deploy across your all your heterogeneous architecture. What I'm saying is that with that single operational layer, you can actually run it across different sort of architectures, whether it's an NVIDIA, AMD, or Intel, right? So that's a phenomenal thing. What we see that there are certain nitty-gritties on an accelerator side, which still needs to be figured out, right, and still to be optimized, certain networking parameters. But apart from that, the operational layer remains the same, right. So the portability is given, right ? And that portability allows you to run this stack all together optimally and schedule those inference requests optimally to a different port. So that's where we see the trend is growing and the deployment pattern is getting more and more portable across different set of accelerator architectures. One more question for you, Atul. To this idea of distributed inference AI grid scale beyond, You know, the orchestration layer, it needs to balance the workloads, latency requirements, GPU availability, interstate consumption, data locality, all these different factors. What happens when those compete with one another? How does the platform orchestrate that? How does it balance kind of competing priorities? Okay, that's a really interesting and challenging question. Okay, so simple answer, there's no standard formula. Okay, industry, right, correct, okay. And I would say there's no static formula also. So you can think of latency or data locality or resiliency as probably a fixed SLS, right? Non-negotiable, probably non-governable, because that's how we see. And then you can actually play around with certain other parameters like cost, energy, GPU choices, and all of those things. Right, so this weighing of this different parameters is not static, right? Probably you can think of like a chat bot or a lot of our customers, Telco customers are now building a lot of assistant, right, we call it an assistant chat bot, right, you can maybe look at the speed, right. How you can prioritize the batch processing with VLLM. But then the other applications, they want to maybe look at the other parameters, right? Correct. Like Azure Deep Workflows, right. So what we see that the same hardware with tweaking around different parameters of SLAs, you can still optimally route and meet those SLA requirements across. So, but there's no static formula. It needs to be played around with a certain characteristics. Yeah. Excellent. Well, thank you so much for the presentation and the discussion for the audience. Atul is going to be around for Q&A. So if you have any questions for him, please put those into the chat and we will get to as many as we can. Now, Urvashi, I'd like to invite you on for a brief fireside here. So to start, hey, how are you today? Just to start things off, maybe you can give our audience just a brief overview of your role and responsibilities at CoreWeave, and then we can discuss some of the ideas that we're exploring today. Sure. And I don't have slides, so hopefully we'll just have a great conversation. I'm Urvashi, lead product for our AI services and platforms at Coreweave. I've been at the company about a year and a half, and I spent over a decade at AWS prior to this in building AI cloud platforms and services. And the focus for our discussion today that I want to have is what do AI workloads mean for customers? What's their lens as they are trying to deploy how we work with them? And then I think Clayton and Atul have shared some great insights and I'll try and connect the dots wherever I can. Well, let's look at this macro trend of the shift from training to inference. I'm curious, from your perspective, what changes in the infrastructure and services? What becomes important for inference that maybe mattered less for large-scale training? Sure. And before I jump into answering the question, I just also want to set the stage on what CoreWeave is. Some of you may know us for our GPU capacity, but we are much more than that today. CoreWeave has grown into a full AI stack cloud designed for the most demanding AI training and inference workloads. So CoreWeav supports every stage of the AI development lifecycle from cluster management and cross-cloud networking to observability and reinforcement learning. And we haven't taken our eye off infrastructure while we've built that out. We're still pushing the cutting edge of compute and recently we're the first AI cloud provider to bring up and validate NVIDIA's Vera Rubin NBL 72, the most advanced AI system NVIDA has ever built. So hopefully that gives the audience a good overview of CoreWeave. And Sean, coming back to your question about the difference between training and inference. A good way to think about this is with training, you control the workload and it's one enormous tightly coupled job where the whole goal is to keep thousands of GPUs that are interconnected, saturated for days or week at a time as they train through the model and crunch through the data while minimizing failures or interruptions that could require jobs to be paused or restarted. So with training, throughput is the primary focus. With inference, the patterns shift a little bit, and I think Clayton and Atul have talked a little bit about this as well. You really don't control the demand, and you can have millions of requests that might be bursty or unpredictable, and all of these requests might show up with their own required latency and cost budgets that you need to meet. So many of the things that weren't as important as training start to become much more important for inference, things around workload placement, latencies, the request level SLAs that we're trying to meet, and then scaling down between spikes so you're not paying for idle GPUs. There's two other things that may be different. So inference fleets can also be multi-tenant or where you are serving many different models across customers. And so you're now working through shared capacity and scheduling of workloads. Isolation also becomes harder. And then tying back to network fabric, The network problem also changes shape a little bit between training and inference. So with training, typically you're stressing scale up and scale out. That is the fabric within the rack or across racks within a cluster. Whereas with inference, in addition to that, you're also adding scale across, which is the fabric between regions and towards users. So different stresses across both these different workload types. To expand on your comment about workload placement, what are some of the characteristics of a workload that would genuinely benefit from being distributed closer to a user or closer to the source of data? I think keeping the lens on workloads is critical, and I would work back from where it's required to have a strict latency flow where you can't run in a decentralized fashion. And so, you know, on-device personalization, real-time voice video where the latency budget is very tight. And then also when you have requirements like data residency or enterprises may have regulatory compliance, sovereignty constraints that they're trying to meet. So all of those are really good workloads. You know, we also talked about physical AI earlier, you know, where you need those edge or remote deployment patterns for strict latency use cases. What we're also seeing is for many of the large language models, the models are so large and the inference time, particularly as models reason through their responses are higher that the latency requirements get a little bit more relaxed. And what customers also care about is utilization, being able to batch requests so that they can have more efficient economics, being able to take advantage of regional or global capacity pools, leveraging cost-effective options like Spot or Flex that we also support on CoreWeave. So consolidating the workloads and running them helps drive higher utilization and streamlined operations, but for latency-sensitive workloads, we may need to consider a different path. I also think there's a hybrid pattern here, which is something that we've started to see customers explore where small, fast models that are closer to the users take care of the first level of turns and responses and then escalate or offload to central models for deeper or more involved answers. So that helps you also balance latency and cost trade-offs. You gave us a great overview of CoreWeave at the beginning. I wanted to focus in on the platform and services piece. You know, from a developer's perspective, I guess the platform is meant to sort of simplify infrastructure. But at the same time, the performance of the AI workload is sensitive to that infrastructure. So how much complexity should the platform abstract away and where do developers still need that sort of infrastructure awareness and insight? Yeah, that's a great question and something that we work with customers on and help them understand the right abstraction for their workload. So my rule of thumb is that the platform can own things that don't really change your model's behavior or cost profile. So areas like provisioning, networking, infrastructure, cluster management, failure recovery, managing the software and driver stacks, And being on point for the 3 a.m. page when your node may invariably die, I think those sort of undifferentiated heavy lifting and operational toil is better handled by platforms because they can operate at a larger scale and have much deeper coverage and automation around these areas. and teams can focus on building their products and solving for their use cases. At the same time, you know, while abstracted interfaces make it easy to get started, but if they don't offer enough knobs and controls so customers can't influence or improve the quality or unit economics, that can become limiting as well. So we want to make sure that customers have the ownership and ability to tune their evaluations and price performance profiles with being able to set things like batching behavior, quantization strategies, ability to select from different hardware accelerators and make topology-aware decisions. So just stepping back, you know, there isn't one abstraction level that works. Our goal is to create a easy path with simple defaults that help customers get started quickly, that have all of the optimizations baked in, but then also provide the knobs that help them tune price performance. So for our inference capabilities that CoreWeave offers, we have the serverless paper token service where you can just use models out of the box without having to think about the infrastructure underneath. Or you can tune certain parameters and get more controls and configurability of your deployment with the dedicated inference option. Or, you can run on Kubernetes and have full control without actually having to manage the infrastructure underneath, so different abstractions depending on the level of expertise across teams, the amount of control needed and the goals that customers are working back from. So you mentioned unit economics there. I'm sure all of CoreWeave's customers measure different things, whether that's cost per token, cost per request or time to completion. But from their perspective, how should the platform expose and optimize those KPIs without forcing customers to really get in the weeds with managing the infrastructure that they're consuming? So different players in the stack will default to different metrics depending on the role that they are playing. So you may see infrastructure providers optimizing for tokens per watt. Influence providers might be optimizing for tokens per dollars, and these can all look very good in isolation and in benchmarks, but when we step back and think about workloads that customers are running, sometimes they may miss context on whether these tokens have actually landed as usable output within the service-level objectives that the customers may be trying to meet and within the right context. So we think customers could approach this from a different lens and really think about useful work per dollar for their specific workload. This means that the metric that customers could consider measuring is cost per useful token, where you're actually paying to generate the output that you are using versus the sticker price of what the model or inference provider might be listing. So this metric really correlates with the workload and the amount of useful work that is being delivered versus theoretical numbers. Excellent. I really appreciate you taking the time to share your perspective and tell us about the work CoreWeave is doing. Now, let's get into some Q&A. Clayton, Atul, I'd invite you to rejoin us on camera here. You know, if we think about where we're going with scale beyond, we've got distributed infrastructure, we have an orchestration layer, all of this is networked together. There's intelligence that resides in each of those pieces that's valuable to the others. So I'm kind of curious how you think about truly achieving a unified fabric as opposed to different domains that are coupled together. Clayton, maybe we can start with you and then go around the table. Sure. And it's a very insightful question because the fact is that to date, unless it's a private infrastructure like a global infrastructure that a hyperscaler might have, but if you think about the public internet and the connectivity that most enterprises and most everybody who's not a hyperscaler leans on, you're right. That has been developed oftentimes to the specification of the consumer. But because the entire profit formula, being a service provider, relies on you building products which have enough commonality that you can amortize that investment across a bunch of people, we've developed these layers relatively separate. There is some communication you can signal from one to the other in what I would say some rudimentary ways today. But in the future, when you think about the type of signaling that will need to happen, the first example that comes to mind, I alluded to it earlier, with power being the number one concern with the growth of AI. We see it a lot with where we place data centers. But in the future, we want to have the ability to move workloads depending upon where power is more available. Again, follow the electron. And so the workload management system, in concert with the tenants of that management system—the actual clouds themselves will need to signal down into the network there and say i'm going to need more capacity facing in this direction because i anticipate in 30 minutes time i'm gonna be moving a bunch of my live uh you know hot kb cash or i need to move some data i needto move some stuff off of warm hot storage excuse me warm storage to hot storage and so being able to signal from the workload management system at the request of the tenant that's there is kind of an easy example it's a pattern we can follow. It might have to do with bandwidth, it might have to do resiliency, you know, I'm about to move this data and so I need redundant paths and I need to know that those paths are geographically redundant. Or, you know in today's world of geopolitics, you know, we want to avoid certain routes because they transit a certain country and I have concerns about that. Or it's it's not allowed by regulation. Certainly in Europe that's very strong with the privacy regulations that they have that they need to avoid landing certain data within the United States or other countries because of just the regulatory framework that they have. So that's signaling as it moves from layer to layer. Some early examples exist. I know that there was one hyperscaler who does use sort of follow the electron, you know, interlocks the energy, available energy capacity with their workload needs. Again, that's kind of a pattern that we can follow going forward into how we might have top to bottom signaling. And again, remember, these are going to be different people. They're going to have a supplier in the form of a service provider. They are going be a consumer in the form of a cloud and the end consumer of AI is on the other side of that, all three of those constituents will need to have an input on exactly what the optimum connectivity layer looks like. So again, I think that's the most practical example we can think of today about how we might interlock those elements together and have open ways to do so. Yeah, Atul, you touched on this in one of your slides, particularly pulling network telemetry up into the orchestration layer, but how do you think about this over time as the infrastructure continues to get more complex more heterogeneous, more distributed? Yeah, so I can actually give two examples, right, which I can think of, right? See, the network itself is very disaggregated and distributed in nature these days, right. So if you look at one of the key use cases, and a lot of the Stelco customers right now are exploring the possibilities that if they're building a different AI factories, how they can actually intelligently place the request to the right set of data centers or each data centers, okay? So if you have a use case where you can only have certain SLAs for latency, right? So that's a primary criteria. And as I was mentioning, there's no static formula depending on the use case, right, if you haven't used it. So you have latency as one of the first requirement. So based on that age latency requirement, you are going to send those requests or inferencing request to the each of the data centers, right? If you have a different criteria, you can actually send those, orchestrate those requests to different data centers. So that's the first case. Now that sovereign AI, right, that's another use case. That's very interesting because what we see is that AI grid is basically nothing but a higher form of sovereign AI where you are actually now building up a governance layer around those AI factories and then operating that within that boundary of geography, right. So in a sovereign way, let's flip that example and you see the data residency has become a primary SLA, right? You want your inferencing. So first of all, you take that model from the model catalog deployed in one of your stacks. And then you actually automate the routing within that governing parameter. And then, you start monitoring with those parameters, the time to first token, the SLAs, right, the consumption, the cost, all of those aspects, right. So, with this kind of governing option and the portability layer, you can actually manage this different set of use cases which requires a different set of SLH. So, that's how we see. And this architecture itself is evolving. And more monitoring, more elements are coming in, which allows the telcos or the customers to operate it efficiently. Great. And Aravishni, I was curious to get your thoughts on this movement in the AI infrastructure that's happening kind of simultaneously towards open and portable, but also towards deep integration that are optimized for performance. Where is openness essential? Where do you see specialization producing enough performance or economic value to justify itself? that's a great question and you know a rule of thumb is open at the interfaces and integrated underneath so we can deliver value to customers and if i can give some examples you know openness means areas where customers are interfacing with the stack whether it's the kubernetes layer the observability, the API interfaces, or model portability. Our goal is to be agnostic as well as support customers that may be running multi-cloud hybrid deployments because it delivers the most flexibility for customers. But below those interfaces, we optimize each layer of the stack for accelerating AI workloads. And this vertical integration gives significant opportunities for us to improve the customer experience and deliver significant price performance gains. So as an example, with the CoreWeave mission control agent and the autonomous AI research agent, ARIA that we've built, we can integrate all of the observability and metrics from metal all the way to the job because the whole stack is well integrated. So we can not only detect hardware failures, correlate them to a job that was running, but we can recover automatically and then also provide full visibility to customers. So this level of automation as well as advantage for customers is hard to achieve in separate environments. So the approach that we take is enabling specialization where the coupling and vertical integration delivers value for customers without creating lock-ins and still delivering the flexibility that they would need. Excellent. I'm sensitive of time, but we're going to run slightly over here. I got a couple more questions. So this is the lightning round where I ask you something really complicated and get you to respond to it very succinctly. When we look at this from a service provider perspective, a telecom network operator perspective, what has to change for them to participate more significantly in the AI economy? Clayton? Yeah, I mean, service providers have a strategic asset that's unmatched, which is their territory that they cover and their customer relationships. And so what needs to change, quite frankly, is, I would say, quite easy and quite difficult. You know, there's a lot of times it's a culture change and just an approach. I know of one European longstanding telco who, you know, a new management team came in recently and they have a completely different thesis. I was just talking with someone last week. They have a complete different thesis about how they approach the market and they develop products. You know the cadence of the AI industry is as fast as it's ever been. And so service providers will need to adopt that cadence. And I think they'll have tremendous success because of the, again, that strategic asset they have, which is geography. From telco to techco, Atul, is there anything you'd add to that? No, that's exactly right. You're thinking of the exact one that I am. Yeah, no, it's paying dividends, right? So I think that we're on the cusp of a transformation of service providers in that way. Excellent. Atul anything you add there? Yeah, so we always believe that, right? In order for CSPs to join this AI bandwagon, right, the three things are the core pillars, the technology adoption, right. We start with small pilots, scale it, right., govern it properly. The processes, align the processes with the new adoption. Okay, so you might have to have some, modify certain processes, adopt some new processes. And the people, right?, the people and the culture, right.? So you have to actually have people, right set of skills, people from top to bottom who are going to adopt and drive that process. So the people, process and the technology is the answer. Align that and the CSPs are basically ready to go. Excellent. Now, this will be my last question. I'd like to hear from each of you on this, starting with you, Urvashi. But for our audience, what is the one or two architectural or operational decisions that they can make now to prepare for what we know will ultimately happen with scale beyond and distributed inference? I think if I can leave the audience with one decision, it's more operational is to know how they will measure the workload ROI and also evaluate the quality of the workload because placement decisions can be technology decisions, but at the end of the day, they have to meet whatever internal goals that have been set. So having a clear way to measure the success of the workload will be critical. And then working back from what data might be needed across these different placement options so that you have the same view to decide whether the placement decisions are indeed accelerating your goals or they are hindering you in terms of adding operational toil and balancing the two is important. same question for you uh at all yeah two things uh that i i would reiterate that message the portability of the stack right across different hardware architectures that's probably one of the key decisions and the governance how you basically govern monitor uh operate right seamlessly so so that's there's a key right whatever architecture you choose the portability and governance should be the probably oneof the key decision so architectures give the final word to you Clayton what are those decisions that organizations need to make now to prepare for scale beyond yeah phenomenal quite phenomenal answers to your question both just in front of me there I would say you know from my standpoint understanding the workload intimately is key you can't wait on your consumers especially if you're a carrier or service provider someone offering services into the market don't wait for your customers to tell you what their requirements are because, quite frankly, they're changing every 60 or 90 days. New model architecture comes out. It changes the shape of compute, especially on the advanced side of things, on the frontier lab side of thing. And so using AI within your enterprise and being familiar with the workload is of paramount importance. That's going to allow you to anticipate what changes will take place in the future and be on the leading edge of that product creation, servicing those needs right away. That cycle is as short as it's ever been. I've been doing this for over 30 years. That cycle is incredibly short. I mentioned streaming media and IP voice and other broadband access 25 years ago. All of those took a relatively long time to unfold so we could organize suppliers into the market. But AI is different. As I said, every 6, 8, 12 weeks, we've got a different view of things. So that flexibility, adaptability starts with the measurement of the network, being able to have the portability on the network. And then finally, just understanding that workload before you're even involved with an architecture decision is really the key. Right. Well, we're going to leave it there. I really appreciate everyone on the panel joining. It's a very insightful, interesting conversation. So thank you for your time. And for everyone in the audience, we appreciate your time as well. So please follow us along at RCR Wireless News and RCR Tech and continue the discussion. So thank You, everyone. Thank you. Thank you, thank you.