“I'm Not Worried About an AI Bubble” — The $13B Company Proving It

In 2022 when I joined Baytown, I spent my first year mostly trying to not get fired. I really didn't know what I was doing. I got lucky just like joining the right company at the right time. I didn't have some strong thesis about the market.

So plain English, what does B 10 do? So B 10 is an inference provider. We do the work required to run AI models in production. Get the GPUs, put the models on the GPUs, and make them run fast.

Which customer is your favorite story to tell? One of them is this company called Open Evidence AI companion tool that's deployed in pretty much every hospital you can think of across America. If a doctor is in the middle of an operation, turns to their assistant to get some information and it's slow or it's down or the information is wrong, like they're literally lives on the line. And going back a little bit, why is there a GPU shortage?

And I am not worried about an AI bubble all the way down. like everyone is providing genuine sort of differentiated value and everyone is is Today's guest is Phillip Kylie. He joined base 10 over four years ago back when the words AI infrastructure company got blank stairs and he spent that time both doing inter inference engineering and explaining to the rest of us. Fun fact, this year he turned four years of notes, customer conversations, and team interviews into a 256 page book called Inference Engineering.

Thank you for the coffees. If you've ever wondered where tokens come from, why GPUs cost so much, or what the people behind the scenes actually do all day, this is the guy. Phillip, welcome to the show. Hey, thanks for having me, Julia.

I'm excited to talk. So plain English, we're at Thanksgiving dinner. What does B 10 do and what does your job there look like? So B 10 is an influence provider.

What that means is we do the work required to run AI models in production. If you have an application, it needs to be able to take inputs and get outputs from AI models. That process is called inference. From a technical perspective, it basically looks like three problems.

get the GPUs, put the models on the GPUs and make them run fast. Uh my role at base 10 is similarly broad broad and uh difficult to difficult to define but I work across content across engineering across go to market to just try to not only kind of bring base 10 to the market that already exists for inference but help develop the next market because everyone's going to need an inference strategy in a couple of years. Why do you say that? Well, today I think that a lot of companies are relying on closed model labs for everything they do with AI.

AI equals open AI anthropic Gemini tokens coming into whatever application you're building. What I'm seeing increasingly is companies want to begin to own their intelligence. They want their road map to be under their own control and not controlled by a single lab. They don't want to be renting their inference a token at a time.

And so we we help them, you know, either adopt open source models, fine-tune their own models, or even train models from scratch that are amazing at their domain specific task. And because of that, they're able to save a bunch of money on compute. They're able to, you know, get better latency, better reliability, all that kind of stuff. But most importantly, have an AI roadmap that's actually in their hands and not just wrapping someone else's.

And what's been on your plate this week? This week, this has been this has been a fun week. We did a couple of of customer events which were really nice. I love to, you know, speak on on panels and in events with our customers and sort of share what they're working on.

I've also been working on uh shipping a bunch of books. We just launched the book in India, which has been super cool. And um I'm working with a distributor over there. So far, it's the bestselling technical book on that distributor's page in India.

um since launch, which is amazing. Um and yeah, I've got a couple of other projects cooking that I can't talk about yet. Um but exciting things to come in this world of influence for sure. That's awesome.

Congrats on that. It's very good news. Which customer is your favorite story to tell? I mean, you're not supposed to have favorite customers.

Okay. But you know, some some customers that I love to talk about, one of them is this company called Open Evidence. Open Evidence does search and information retrieval for doctors and physicians. They're a sort of AI companion tool that's deployed in, you know, pretty much every hospital you can think of across America.

Like if you go to a doctor's office, there's a good chance that someone behind the scenes is using open evidence. And the reason I love working with them in particular is like they really get it. they understand how missionritical inference is because if a doctor is in the middle of an operation, turns to their assistant to get some information and it's slow or it's down or the information is wrong, like they're literally lives on the line. So that level of intense engagement with something as sort of boring as infrastructure because they understand the stakes is I think the the sort of company that I really really enjoy working with.

Are you more of an engineer who writes or a writer who engineers? That's a great question. I I would turn that back to you. What What What do you think?

I think of yourself. Of yourself. Of me? Yes.

I would say I'm a writer who engineers attention in the startup world. Engineers attention. I need to get better at the attention piece. That's uh you know like like it says in the paper, attention is all you need, right?

Yeah. distribution too and like being able to really stand out. Exactly. I think you know I definitely have that engineering background.

I studied computer science in school. I worked as a software engineer before base 10. But what's cool about today is that being technical is becoming much more of a spectrum. And compared to the folks on the model performance team or some of our Ford deployed engineers who are in there with the customers every day, I'm like, "Wow.

those people are are the real technical experts. I'm just the, you know, guy waving a book around. But at the same time, because this space is so new, because there are so few people who really understand AI technologies in general and inference in particular, you actually can become an expert in the space pretty quickly and become sort of a technical resource for other people coming into the space just like I did. So I've seen people come into inference with both engineering backgrounds and not with engineering backgrounds knowing almost nothing about the space and within months making actual sort of frontier contributions to anything across you know runtime infrastructure the way the market understands it.

There's just so much to do here and so few people working on it relative to the size of the opportunity that anyone can become a technical expert pretty quickly. Do you think being in SF2 really helps? Yeah. Yeah.

I I started out remote for the first three years of my career and just as base 10 and the market sort of hit an inflection point around January of last year. That's when I was actually able to move out to the Bay Area full-time. Um, and now I actually live pretty much at the base 10 office, which is awesome. Getting to, you know, see it 247 what's going on.

Yeah, I like it here. Um, and it's it's nice to be in a room where everyone gets it, everyone knows what's going on. Speaking of B 10, you joined B 10 in January 2020. Chav GBT didn't ship until November of that year.

what were you doing for those first 11 months and what did you think the company was going to be? So in 2022 when I joined B 10 I spent my first year mostly trying to not get fired. Um I really didn't know what I was doing. I I joined B 10 as a technical writer and at the time base 10 was a very ML focused platform.

We still had influence as one of our core pillars but we were running much smaller models. So, you know, back then when when whisper came out, that was the big one for us more than chat GBT was whisper this audio transcription model where we'd been using like wave 2 and other stuff to try to transcribe audio wasn't working. Whisper comes out, it works, but it's a billion parameters, a billion. Like, what am I supposed to do with that?

I can't put that on a CPU. I can't, you know, I've got to like get a T4, like an A10G or, you know, at the time what some of these hot commodity GPUs were and and run inference on that. And and that was, you know, one of our one of our early sort of pivot points as a company is is focusing more on the the GPU inferencing piece back then. Um, even even before a lot of these sort of language models came onto the market, being more focused on audio in, audio out, search, embeddings, that kind of stuff.

So yeah, it was it was a very different time, a very different market. In many ways, I got lucky just like joining the right company at the right time. I didn't have some strong thesis about the market or or the product. I mostly just thought it was a really cool group of people who I wanted to spend time with and learn from and I still think that.

So I've just been very fortunate that the the market has developed a very strong tailwind for us. That's amazing. And what's the thing you believed about ML infrastructure in early 2022 that turned out to be completely wrong by mid 2023? Well, a lot of people back then kind of thought that inference was the easy part.

you know, inference was a solved problem and that you either had to build at the layer above that in sort of MLOps tooling or you had to build the layer below that in terms of more of the the runtime or maybe that it just wasn't a worthwhile problem to work on at all. But fortunately, like what we've seen is that influence in fact is the most important workload out there. It's the biggest, it's the stickiest, it's the most sensitive. And so, you know, where I I was not, you know, banging the drum of influence in 2022 necessarily.

Um, so I would say I was I was not so far ahead of the curve back then. Um, but I've been fortunate to have kind of hopped on the train early in in the last couple years and and begun to to understand what what influence means in this market. And did the team see the LLM wave coming or did you like have to scramble like everyone else? We saw it coming to a degree.

Like I said, we were more focused in the early days on other modalities. Still generative models, still transformers based, you know, like boat models, that kind of stuff. But the the issue was that like early LLMs were just not that good. You know, we had the first language model I used was GPTJ.

uh which yeah GPTJ which was a a sort of chat tuned variant of of GP22 I believe. Um then there was Llama and Llama 2 and those weren't very good. I mean amazing research projects for the time but you couldn't really do anything with them. And then like the uh what was it called?

There was the the the Zepho model that came out. um there was some some sort of fine tunes of of Llama that made them a little bit more useful. And then it really wasn't until some of the you know Llama 3 era models and then finally like Deepseek 3, Deepseek R1 that LLM, Open LMS really started hitting the the sort of use cases that they're now powering today. And so up until that time again, we were still a lot more focused on audio embedding, search, image, video, like all these other modalities that we still do a bunch on today, but those form the foundation of our business for a long time.

And when did you personally realize this isn't about deploying models anymore? This is about one specific kind of model at a scale nobody planned for. Yeah, I mean I still do think that it's about more than one model because we're still very heterogeneous uh heterogeneous across both model modality runtime developer like everything. Um we're trying to be multimodel, multi- cloud, multi- everything that you could possibly think of eventually.

That said, like the the LLMs are centralizing as the core piece of this workload. Um, and I think that piece has really emerged in the last say 16 months. Okay. Okay.

And what does this mean for like founders building in the space now? Well, it means that you actually have a huge opportunity, right? Because I think that that you and I have both seen a million GPT rappers, right? That was kind of the whole thing.

And in 2023, it was super cool to build a GPT rapper. And then by 2024, it was like not so cool anymore, but they were still scaling. And then last year, it was like, well, if you're a GPT rapper, you're pretty much dead. And this year, they just are all dead.

But along the way, it's like you actually now have the opportunity to build something that's way more exciting than a GPT rapper, right? You have the opportunity to build both incredible depth at the harness and context layer to build actually differentiated agent systems, but you also have the ability to take control over your own inputs. Like I was talking about earlier, you have the ability to own your models, own your intelligence, own your tokens, and build something that you control end to end and thus develop a better user experience and a more sustainable business. Awesome.

Thanks for sharing that. Let's start with the book. Inference Engineering, 256 pages, came out February 23. For a lot of people in this field, it's the closest thing they've got to a textbook, but things move fast.

What's the part you'd already rewrite if you could go back? And this was something I was thinking about every day as I was writing this book. It's like, I got to get this done. I got to get this published because as soon as it comes out, I'm gonna start feeling like pieces of of it are out of date.

But at the same time, I tried to write in such a way where look, I'm I'm not saying this thing's going to last a decade, but I think 12 or 18 months is a reasonable timeline for it to feel fresh and sort of at the frontier. There's been a couple things. 1. Like, there's a couple model specific things in there that have updated.

I would definitely want to include more about a couple of new quantization methods that have become popular. Talk a little bit more about context parallelism which I'm seeing more and more of. But the thing is that the fundamentals are all there. Everything in the book is still true today a couple months after it's been published.

And I think that, you know, one thing that I've been able to do being in the space for so long is kind of layer my knowledge generation over generation of technology. And, you know, with with a with a book like this, like you can kind of catch up to what the frontier was and then lay stuff on top of that foundation um until such time as I have a minute to sit down and do a second edition. Wow. Okay.

you know, pro probably probably a year or so until that again. Like I really wrote this with a mind for being as timeless as possible within a space that moves this fast. That's really good. You had that mindset like making as timeless as you can so people can continue looking through it.

When you first started writing the book, did you have was it would you say you treated almost like a a journal entry? I mean, you compiled a bunch of different notes and then when you first started when you ended the book, did you have more? Did you know more when you finished the book? Like it was almost like a stage like it was from stage.

Absolutely. Yeah. So, writing this book really sharpened my thinking and knowledge on a bunch of pieces of the problem and exposed places where I thought I understood what was going on until I had to explain it in writing and then I thought, wait, let me go back to the research. Let me go back and talk to the team and and really get a deeper understanding of what what exactly is a hidden state again and you know how does how does a a CUDA graph relate to a CUDA kernel and like some of some of these some of these pieces uh that as you're sort of working in the dayto-day you just sort of use the information without thinking about it and then once you explain it you have to think through your fundamental assumptions again.

Um, I will say with with this writing project, like more than anything, my skill is is writing books and long form writing. I sat down, I put together an outline over the course of a couple weeks and just sat down every day after that and plugged away three, four sections out of the outline until the thing was complete. So, there wasn't a lot of previously like written material. There was all my blogs and whatnot, but that didn't really fit in the context of a book.

So I had all these ideas swirling in my head of of what this field meant and I had to find a single structure that would kind of carry all that information in a linear narrative which was pretty tricky. It's like reducing something from N squared to an O N log N space complexity in my head. Wow. If I started reading the book today, which chapter should I read with skepticism in in what way?

Are there parts where maybe some things are there like there controversial parts maybe in this book that maybe some people wouldn't agree on or how do you make sure to like kind of balance those all those opinions? Well, one thing is I mean look the the book's green for a reason. This is a very Nvidia first view of the world. And part of that is due to just the technologies that I use in my daily work.

Like we are a very Nvidia focused company when it comes to the hardware and software that we use. Um so you know maybe you could imagine a book like this that has more about other types of hardware or other other software ecosystems. The other thing maybe is the chapter five on the different model performance techniques which is my favorite chapter of the book. But whether or not quantization really kills your intelligence, whether or not you know one method or another of speculative decoding is really the way to go.

I think that there's a lot of active research in in all of these areas. And so if you imagine a if you imagine a spirited debate around some of these topics that that might be the the chapter that causes it the most. Awesome. What's the one thing you nailed in this book that you're still proud of?

Influence is a really hard challenge because it's not just one thing. It's not just get good at BLM. It's not just understand distributed systems. It's not just learn how to write a CUDA kernel.

It's everything together at the same time. And I think what I'm most proud of is developing and delivering a structured introduction to the space and showing a topic that is as wide and varied as inference and delivering that in a way that really anyone even I'm finding a lot of people without computer science degrees or formal engineering backgrounds picking up this book and and getting at least something from it. So that's what I'm most proud of. What else do you think you want to share about your book to everyone watching?

Maybe, yeah, some like sneak peeks or maybe if a non-technical someone like me would pick up this book, what should they keep in mind? Um, what what could they take away from this? Yeah, I think the the most important thing is to realize how early we are in inference. influence has been what it is today for only a couple of years.

And you know, while the stack has finally solidified just enough to feel good about putting it all on paper, that there is still so much left to do in this space and that everyone today has an opportunity to come into the space You're often the person at Bay 10 who writes about a new model drop the day after it lands. Would you agree with that? Yeah, that that has been me for a long time. Fortunately, we now have a a team who are a little bit bigger company and so uh we now have many experts working on the problem instead of one guy running around with his hair on fire.

Um and and the results have been much better because of that. But you had the chance to get a front row seat to the war room. So when Deep Seek drops something, what's happening inside the company in those first 24 hours? So we've gone from a new model of substance every quarter to every month to every week to at this point almost every day it feels like there's some new model that comes out that that requires some kind of response to it.

So we've had to get a lot better at that process and and also you know develop partnerships with so many of these labs where now for example with with Neotron and Gemma and many of these frontier open source families were getting approved for early access and working directly with the developers to bring these models to market. But whether you find out about the model on Twitter, get the weights off of a a download link or get a partnership ahead of time, the process is pretty much the same. Step one is understanding who's going to want the model. So whether that's some some eval to figure out what it's good at, what it's not good at, um whether that's understanding how it fits into the existing market map.

Number two is figuring out like how are we going to run this thing? Where's the software support come from? There's a lot of dependence in these sort of releases on overnight builds and other kind of quick and dirty ways of getting it up, making sure that the quality is high, making sure that the output matches what the developer intended. So you got to you got to get it up.

You got to get it in the hands of the people who want it. And then you have to make it go fast. So some models share architectural features with each other in which case you can take the optimizations that you've already built and deliver them onto the the new platform. In other cases they introduce new architectures like when DeepSeek introduced Deepseek sparks attention or with um Kimmy K2 and they had the whole like int for native weight thing.

These are some modelsp specific sort of quirks that you have to figure out how to work around and figure out how to deliver a new paradigm for faster inference under this architecture. What's the reason for like all the urgency? I mean everyone everyone wants to try the new model. I will say something I've noticed is that a year ago when a new model dropped, the entire industry wanted to try it that second and today there's a little bit more deliberateness in the way that new models are adopted.

I think that's for a few reasons. Part of it is that there are so many new models that you can't have a fire every day when a when a new one drops. But these models are still delivering big steps in intelligence. So it still is a very urgent problem to get them online.

However, now so much more is done by the system around the model, the the harness and and the context, the memory. Like every piece of an agentic system besides the model itself is much more loadbearing now than it was a year ago. And new models introduce behavioral quirks that this entire endto-end system has to account for. So now we're seeing people take more time, days, not hours, to integrate new models into their production workflows because they have more to lose and they have to make sure that this model is not introducing any regressions in the massive amount of product that they've already built, which is which is a change from say like a year or two ago where it was just like oh just it's better ship it.

How much of that race is engineering versus just having the GPUs sitting there already? A lot of it is is the engineering piece. Like obviously you need capacity but just doing a little bit of R&D takes one or two nodes. Um, obviously deploying at scale takes a whole lot more, but the the main thing is a deep understanding of the existing model architectures out there, the runtimes and techniques that support those and then how you can port over stuff that's the same and adapt stuff that's different.

What's the worst surprise you've personally seen between, you know, we got the weights and we have a working deployment? Yeah, I think there's there's a few things that can be challenging. Um, one is definitely figuring out the correct quantization scheme, especially if the model has a different native quantization. I mentioned Kimmy coming out in int 4 where we prefer more of an NVFP4, which you can't translate directly between these two formats.

So you have to take a round trip through a different format and make sure you do that without introducing any kind of of loss on quality. Um and then there's the facing the fact that like you have to repeat all of the work that you did to get a model to frontier performance. So if you have for example a speculative decoding algorithm that guesses ahead multiple tokens every forward pass through the model, you need to retrain that speculator on the new model on its new hidden states on the way it thinks about how to tokenize input and and create output. So there's there's a lot that has to happen every time and you're also working on cutting edge software.

Everything is day overnight builds and day zero releases. So there's just always going to be bugs to overcome and always a lot of pressure to get it done quickly and get it done well. It's it's a it's a hard project, but it's what moves the industry forward. You mentioned this is the second time you mentioned overnight builds.

Yeah. Why are why are overnight builds like important? Yeah. What do you mean by that?

So usually when a lab releases a new open model, they're going to work with the open- source community, folks like VLM, SG Lang, Nvidia with Tensor Rotlm, and Dynamo to make sure that the existing tooling in the world supports their new model. And that support often rolls out at the same time or even a couple hours after the weights do. And so if you're using yesterday's version of VLM, it doesn't even know that this new model exists, much less have support for it. But at the same time, this means you're installing a brand new version of the software.

And there's always a bunch of breaking changes. And so there is that sort of balancing act of of figuring out how to adopt the new stuff both within the within the influence ecosystem as well as like I mentioned within the agents themselves without breaking what what you already have looking. Okay, correct me if I'm wrong, but RA R1 needed 16 H 100s just to hold the weights. If Deepseek 4 is bigger, are we hitting a wall where just run the open model stops being realistic for most companies?

Yeah. So, we for Deep Seek, we actually were first using like H200s. They have a little bit more VRAM per chip and and you could do it with with eight of them. The reason that eight matters is because Nvidia GPUs are delivered in nodes of eight.

And these eight GPUs have high bandwidth NVLink interconnects with them which allows them to you know share weights share information while the inference is running and if you don't have this NVLink connection you have to go over Infiniband or Ethernet or some multi-node connection between sets of nodes which is way slower and makes your inference much slower. So the the big limiting factor is like how much memory do you have? you need about at least twice as much twice as many gigabytes of VRAM as you have gigabytes of model weights so that you have headroom for the KV cache which stores uh intermediate representations of the information that you're computing during inference. So yeah, DeepSeek was was really big at its first time.

Um the the R1 model was 671 billion parameters which in FP8 was 670 gigabytes. Today we have trillion parameter openweight models which in FP4 is like 500 GB and you're running that generally on B200 systems which have a little over a terabyte of VRAM available. And for anyone who's not an inference engineer, why is knowing all about this important? Yeah, it's important because it gives you a sense of scale.

If you have a model like GPTOSS and you can run it on one a couple of GPUs versus a model like Kimmy K2, you you understand you start to understand like why is this model faster? Why is this model less expensive? But also maybe why is this model like less capable? It's it's not as big.

you understand the sort of physics behind the trade-offs of of the models that you're using. Nice. So in the future a few inference providers like B 10 owning all the big model deployment and everyone else just calls okay is the future that like is the future a few inference providers like B owning all the big model deployment and everyone else just calls APIs. I don't think so.

No, the the majority of our business is not a sort of payer token API model. It's folks building dedicated deployments where they get the GPUs for themselves. They are able to put the models like put whatever model weights they want on there. They're able to introduce fine-tune models.

They're able to set their own inference parameters to maybe index more towards latency or more towards throughput or whatever it is that they care about. So I think about it a lot less as us owning the deployments and a lot more as us allowing every vertical AI company in the world to own their own deployments. Okay. And what's the break even at what scale does it make sense to own GPUs versus just rent?

Well owning GPUs that that's an interesting question. Like we don't even own GPUs. We sit on top of a doz a dozen different clouds and across a hundred different regions where we're able to sort of aggregate capacity from all of these partners and deliver it to our customers. Um, if you're thinking about owning in terms of having dedicated influence on GPUs that you're renting yourself versus paying per token, it depends a lot on the modality.

In voice where models are very small and run on less expensive GPUs, it can be lower in the order of like five figures a month. In language models, you know, usually you're looking at a high five or or six figure a month range where it starts to make sense to to cut over to dedicated. Um and of course if you are a Neol or if you are running something like video generation or something very differentiated and demanding like obviously you have no choice but to to cut over at the beginning. Um but again like these sort of large frontier models can be very very expensive to operate at scale.

Is there a model size that even base 10 won't host? It's not so much about like what we won't host and more about what won't get developed right now. And it comes back to that that VRAMm headroom. So with every generation, Nvidia GPUs get bigger.

They're able to hold larger models on them. And the size of of models that open source labs create gets pushed higher and higher. Right now, the frontier is approximately a trillion parameters where most people aren't releasing openweight models that are larger than that. But as we get the B300 systems rolling out into the world and then as of course Ruben comes along that next generation Nvidia GPU, we're going to see a big increase I think in model size.

I could see models doubling in size again to run on these new architectures and these more powerful pieces of hardware. When will in how many years will we see this increase? When is the shift? Yeah.

So, it's interesting because the hardware is going to start shipping at the end of this year, but it takes a long time for hardware to make it into the market. Today, Blackwell's been out for 18 months, and we're still working on porting every kernel and every workload to a sort of Blackwell native world. I think that it'll it'll be faster this time around. It's faster every cycle.

I've been through it three times. I've been through it with Ampio, with Hopper, and now with Blackwell. But every time it gets faster because the stakes are higher, it's more urgent. We need to absorb the massive demand in this market with the newest and most powerful capacity available.

Awesome. And is there a version of Deepseek that scares you like architecturally where the inference just like won't be able to keep up? 2 two was a tough one. That that model had a lot of experimental architectural features and was, you know, difficult to run with a high degree of stability, especially around things like super long context.

But that was explicitly a sort of experimental release. Um, I think that every time you come up with a new architecture, it takes the market a little while to adjust. But then everyone shares the best ideas and and today you'll see new open models come out with ideas and architectures that were developed across multiple different labs kind of all making their way upstream into one. If it's in terms of more like capabilities, no, I'm I'm very I'm a very optimistic guy.

I think things are going to go great and I think the way to make sure that things go great is for everyone to have access to frontier intelligence on an equal playing field. And when you talk about context like what is what is a visually like like digestible example of of some of this like context or what would people be using for? Yeah, I work with a ton of of vertical AI companies, which is cool because if intelligence alone was enough to get things done, we could all just like sit around thinking all day and live in a perfect world. But it turns out that there's a lot of messiness in between a sort of theoretically awesome solution and something that actually works.

So we work with companies who are building in healthcare, in finance, in technology, in you know all kinds of sectors of the real economy to apply AI solutions to common and repeated problems. And I'm just really excited for all of these builders to have access to even smarter models, even faster and cheaper tokens because their penetration in their markets is so low. They have still 10 to 100 to in some cases like a thousandx room to expand their their usage base and we need to deliver them infrastructure that will support that expansion. What's the gap between a model works in research and a model is production deployable?

It's a great question. I think that a lot of the gap just comes down to software support because, you know, even something as simple as the matrix size when you do your your general matrix multiplication can impact whether or not something is fast enough to be usable or or not, right? Whether or not you have a specific kernel for that specific matrix size. So there's just a ton of implementation details that come out in production.

There's a ton of testing. You know, these these language models are not deterministic. And so if something goes wrong, one out of every 100 or thousand or 10,000 runs, you need that production scale in order to start to see those issues frequently enough to diagnose and fix them. Yeah, there's there's a there's a lot of work that goes into it.

Um, and unfortunately sometimes that what kind of happens in the field, but I mean ultimately we're able to to catch the bugs and and patch them and upstream the patches and just keep the industry moving forward. I'm curious, what do you mean by deterministic? They're not deterministic. Yeah.

So, if you give it the same input, you're going to get different outputs on on different runs. Um, and because of that, maybe one out of every so often runs trips some race condition or triggers some looped output or some issue like that. And and those can always just be really tricky to debug. You talk to dozens of customers a month.

What are smart teams doing right now in this shortage that not so not so smart teams aren't? I think there's a big focus right now on fine-tuning small models for task specific quality because if you're able to take a 20 billion parameter model and replace either a frontier open model or a closed model from a lab with something that you can run on a single H100 and get pretty great output from then you're just going to be able to take the capacity you have and move it a lot further. So it takes a lot of work. You have to have the data, you have to have the infrastructure, you have to have the knowledge to be able to do a successful fine-tuning run.

But that's where the frontier is right now is is in post training. And going back a little bit, why is I think that there's a lot of worry in the world about an AI bubble and I am not worried about an AI bubble because where I sit, you know, we have our customers who are selling to their customers in the sort of real economy and they're providing a ton of value and they're getting paid which they then turn around and and and pay us a portion of that to run their infrastructure in a way that you know is is like good unit economics for for the end customer and then we run that infrastructure in a way that's like good unit economics for us. We buy our compute from folks who have good unit economics who buy from Nvidia who has fantastic unit economics. So all the way down like everyone is providing genuine sort of differentiated value and everyone is is making money.

And then when you look at the explosion in demand at the top that's really what's driving it is so many people want to add AI to what they're doing every day. the amount of AI that is consumed out there in the world like here in San Francisco might feel very high but it's actually pretty low in in the real world and so you know as I see the potential for 10x 100x a thousandx more demand from the customers of our customers I see how that's going to spill through the entire rest of the ecosystem everyone's getting ready for that everyone understands that you know once you're you're growing at these historic rates month over month for long enough you start seeing unprecedented numbers and so we just have to prepare with as many uh bits of silicon as we can get our hands on. What what I like to say is every token that can be generated is going to be sold. Every GPU that can be racked is going to be rented and every wafer of silicon that can be manufactured is going to have a chip pressed into it.

So that's the where we're at in terms of the the demand in this industry and the entire supply chain is racing to meet it. That's a really good quote, too. Who's going to suffer the most and who's are there people who are not going to suffer from this? Like Yeah.

But I mean, unfortunately, it's pretty much a situation of the better capitalized you are, the better you're able to take advantage of of capacity. You know, it's it's turning into a little bit of a money game, which is unfortunate. But there's also so much engineering work you can do around this problem. Whether it's like I was talking about earlier with fine-tuning, whether it's figuring out new inference optimizations to increase your throughput so you don't need as many GPUs to serve the same traffic.

There is a lot of engineering work that we can do here to make the capacity crunch less What's a bad habit teams deployed in 2023 when GPUs were easier to get? You know, one thing I'm actually still seeing is when you have a company with a certain fixed allocation of GPUs, the teams within that company are are really struggling about who gets which GPUs when um and and maybe like holding on to more capacity than they need or running at a a higher rate than they need just to to make sure their stuff doesn't scale down and someone else's scales up. I think that it's going to take a lot of collaboration within every single company and and even between companies to figure out how to get every GPU into a place where it's doing valuable work. Is there a team you secretly admire for how they're handling it?

You don't have to name them, but yeah, I mean, I think I admire a lot of I admire everyone who's building in this space. It's a hard space. It's very competitive. Um, and if you're if you're doing something if you're doing something and it's working, then like you've gone through a lot to get there.

I think, you know, one one team that I compete directly against is like, you know, I'm I'm fans of of what a lot of our competitors are doing, like I think that that modal does a great job with developer experience and that, you know, Charles over there, who's my sort of counterpart, is always making really great stuff. I think that you know when I look at companies who are scaling you know big sort of cloud when I look at companies that are scaling like big big cloud platforms like that's really cool when I look at the way that even you know the hyperscalers are are integrating new AI capabilities in it's like it's hard to sort of pivot that quickly. So like everybody in this space is doing something really well and the trick for me is to just like figure out what we're uniquely good at in terms of of performance and scale and then you know deliver that in a way that is is differentiated and deliver that in a way that our customers love. Give me a a quick rundown of what are you guys uniquely good at.

Yeah. So what we really focus on at base 10 is a few things. Number one is performance. Like you want the lowest latencies, you want the highest throughputs.

All that comes from having the best possible runtime for your models. And then the other piece is scale. As as we've talked about, like it's hard to get your hands on compute. Demand is exploding.

You need a infrastructure provider who's actually going to be able to scale with you as you, you know, 10x and 10x again and 10x again. And both of those things alone are important, but what we really like to think about is how do we deliver them together as closely as possible? And then how do we make sure that we're sitting really closely with the customer, we're working hands-on with their engineering team, and we're trying to, you know, make them successful because if they succeed, we're going to succeed. So ultimately, kind of the base 10 way is is that last bit.

It's the customer obsession and it's the hands-on engineering support to get people to success faster. Cool. You mentioned customer obsession because yesterday at the Stripe conference, um, Sam Alman was talking about how now he would fund someone who doesn't know how to code but is so deeply like in with the customers and with their users, deeply understands their users. Like that is so important.

Yeah. Yeah. And we have the fortunate position of being very in with our users and more or less remembering how to code. Yes.

Yeah. There you see a balance of both. What's the most underrated I don't think it's underrated anymore. I think people who are in the know are picking up on this, but KV cache reuse is still the thing because the the KV cache, it's what you create during prefill when you process the prompt and you update it during decode when you create the subsequent tokens.

If you're able to remember this information and reuse it on the next request that has the same context, you get to skip a ton of expensive work and deliver your answer much faster. So, it's it's faster and cheaper. And when you look at what Nvidia is focused on right now with Dynamo and Nixel and all the technologies they're building there, like it's very clear that there's a big focus on KV cash management and I think that's going to continue to be a focus for a couple years and and we've only really scratched the surface on on what that's going to deliver for us. Is the H00 still the right thing to want or are people quietly moving to B200, MI300X or TPUs and not talking about it?

It's a great question. That's two two questions. Um, one is like Blackwell adoption. Blackwell adoption is here definitely.

There still is a lot of stuff written for Hopper, especially because of export controls. So many of the open source labs only have access to Hopper, at least officially, and thus release stuff that is optimized for Hopper. So there still is a ton of demand for Hopper H100, H200 on this market. But Blackwell is here, B200's are here, B300's, and they're definitely the sort of frontier chips for the most demanding workloads.

um you know AMD and and TPUs, TPUs in particular, like I said earlier, every piece of silicon that gets manufactured is is is going to get a GPU put on it and it's going to get put in a data center somewhere and it's going to get used. Um but the the software stacks are still less mature than Nvidia's ecosystem. That's always been their whole thing is is having the best software stack. And so, you know, we are still like all all on Nvidia chips.

Um, but I'm definitely seeing more and more market adoption of AMD and and especially TPU. I think that Google's done a great job there. The the new um V8s of course look incredible on paper. They're getting a lot further on the software side and and making them more available.

Um, so I'm really excited for a future where we have a lot of different options for what shape our silicon is and and where we get it from. And I don't think that any part of that future is going to be bad for Nvidia because they're going to just continue to to crush it at at the things they build. They they sort of acquired a lot of Gro technology which is allowing them to diversify a little bit and I think they're incredibly well positioned for this future as well. How real is that?

We're testing my 300X announcement versus we're running production on it. Yeah, a lot of it is testing again because of the the software stack. Like we test all kinds of things. Um but you want to have a great degree of confidence in the thing you're using before you put some of these mission critical workloads on it.

I mean, you probably mentioned this, but will Nvidia's moat ever crack? I I'm not I'm not betting against team green. I think they know what they're doing. Um when you look at the time it takes to develop and tape out and manufacture and install a chip, it's years even for the best companies on earth like Nvidia.

So if you look at something like Blackwell which was released last year or maybe even technically the year before means it was designed in the sort of 2020 to 2022 era when transformer models were only just becoming a thing. That makes it kind of the the first generation of silicon that's even aware of generative AI. And then if you look at at Ruben coming out like this is one that actually is built for ideas like disagregation with CPX and with now the Glock LPUs. It's built for the scale of some of these models.

So because of those the lag in those generations you know the the things that are going to be released in three or four or five years are getting their design finished today. That is where I think there's a huge opportunity for them to just continue to forecast the future. But uh you know like they say like the best way to predict the future is to invent it. And they've had a lot of success in you know arriving at the market at just the right time with just the right solution in an industry where you can't just put up a PR overnight and and ship.

You have to have years of research, development, and supply chain in place to to meet the market at the right moment. Walk me through the spot market for GPUs in 2026. Is there actually a black market? Who's the buyer?

Who's the seller? There is not very much liquidity in the market today. Um, there was maybe some a few months ago, but everyone understands that the demand for compute isn't going anywhere and everyone's just like locking down what they can get. I think that, you know, there is going to be a lot more supply coming online.

But yeah, I mean, you're just you're just not looking at short-term GPU reservations if you're looking at any kind of volume. If you want a node here or there for a research project or a benchmark, like that's still possible, but large allocations are just just uh you know, becoming harder and harder to come by. Why is there liquidity? Because everyone understands that they they they got to buy this stuff up.

Um, you know, obviously we've purchased a a a great deal of capacity. Um, many other folks in the space have as well. And like there's always a way to find a way, but um the the compute market is becoming much much tighter in a way that again like signals the need for us to build additional sort of technical abilities that that help us navigate compute. um whether that's something like multi cloud capacity management where we can take bits of compute from here and there and mix them together and and use those to serve requests whether that's you know really excellent granular autoscaling so that when certain customers aren't using GPUs they're they're available in the pool whether that's inference runtime optimization so that we need fewer GPUs for the same workload there's a ton to build on the technical side for a world with scarce capacity as we continue to navigate it from the business side and make sure that we have the capacity that we need to serve our customers and that our customers have the capacity that they need to operate their businesses.

Have you ever seen a GPU deal that made you say that's definitely stolen? I mean, you know, if you ever see a guy running out of a data center late at night with some B200s on his back, uh, check that it's not me before you call the police. Um, I think that I think that, you know, there's a a uh a lot of incentives in this market to take a quick buck. Um, and that the way that we're going to build a durable business is by not doing that.

Um, so yeah, I I think I think that there's certainly some like sketchy or questionable deals that happen um in this market and you can still be a very successful player while staying away from that. What do you think about the export control workarounds? You know, I'm not like particularly wellqualified to comment on that. Um, I think that I view things from a open source lens and I want everyone developing open source to have access to the best tools no matter where they're developing it because open source is good for everyone in the world.

It's good because everyone can almost like access it and collaborate. Exactly. The ideas that get released into open source are then adopted by every team off of the merits of the ideas themselves and move the industry as a whole forward. So you're for open source.

Yeah. Do you know people who are like against it? Is there a lot of people who are against it? Yeah.

I mean of course the big labs are not so thrilled about it in in some cases although you know Google has a fantastic open source effort with Gemma. OpenAI has actually contributed quite a bit to the open source ecosystem with Whisper and GPTOSS and they just released a new open source model for PII reduction. So I do see an understanding of of the balance that's needed in the world. Um I'm maybe a little bit more open source build than average but at the same time that's because I work at a company that benefits from open source.

How much of the shortage is real versus allocation games by hyperscalers? I think it's real like if you have GPUs that are just sitting there and not doing anything like you have a very strong in economic incentive to put those GPUs to work and so while of course people are going to be getting capacity in anticipation of of need that's going to come into the market um I think that right now a lot of it is just like people are are using what they're buying. Let's go into inference engineering as a career. Three years ago, inference engineer was a job title.

Now it pays a fortune. Is this a five-year window or a permanent role? Where do you see this going? I mean, I'm not going to speculate so far in the future right now, just because who knows what things are going to look like.

I had no idea five years ago that I'd be doing any of this stuff today. That said, I do firmly believe that inference is something that developers should learn and it's going to be great for their career whether their title is inference engineer or not. Within inference engineering specifically, there's a massive expansion. A couple years ago, there were maybe only a few hundred people who did that.

Now there's there's thousands. I think even with the gains in AI assisted development, there's going to be hundreds of thousands or maybe even a million people when you look at the entire world and there being 20 million developers whose job involves understanding and managing inference technologies to some degree. So I think when you when you look at everything the trends with open source and owned intelligence when you look at you know the the GPU supply and and what that means in terms of market demand translating to supply that there's just so much work to do in this space and we need more people to get into this space learn inference and start doing whatever they want whether that's open source whether that's startups whether that's big labs everyone's going to need Closing question. What's a take you have that most of your peers in inference would disagree with?

That's a good question. I think I'm I'm pretty consensus on a lot of things. Um because it's been it's been proven it's been proven right to me. Um, I do think that one take I have that is perhaps insufficiently nuanced but still pretty useful is that there's really only one type of model or really only two types of models.

There's like auto reggressive transformer model where if you squint enough it's an LLM like embedding models and and TTS and ST like if you stand really far away and look at it it looks kind of LLM shaped and then you can do LLM shaped optimizations on it and then if you look at any kind of diffusion model whether that's video generation or image generation or or any of these you know more advanced modalities on top of it like if you stand far enough where it's like oh we're just like iterating through a unit it's like kind of the same thing we can do kind the same optimizations over here. Again, this is a little bit too general for someone who's actually like doing the the last mile optimizations to feel good about as as a mental model, but I find it helpful to kind of categorize the space that way and understand what the first tools I'm reaching for are when I look at one modality versus another. Awesome. Thank you, Philip, so much for doing this with me.

Yes. Thanks, Julia, for having me. Awesome. and everyone get get the book.

Yep. Yeah. Where is it? Where can you buy it?

com/influence engineering. The PDF is free. You can also order a physical copy. I don't really like make any money on these.

It's it's just, you know, to cover the cost of printing and shipping. But for you, Julia, I have one here today.