He’s Building the Document Layer for AI Agents

I think the bottleneck is in the context layer. The reality is all these models um especially with the latest releases from you know anthropic and open AI are like absolutely insane. They're like smarter than humans. I think in the end context is everything cuz without context the models will just like sit still.

In October 2023 you published something called rag is a hack and it went everywhere. When I say rag is a hack it's because there's really no first principles to how like why you should like chunk retrieve that type of thing. It was just one of the easiest things that like kind of worked so that people could build some like basic PCs. How would you explain Llama index to someone who isn't an ML engineer today?

Basically, we are building the document infrastructure for AI agents. So, you published five ML papers. I mean, you mentioned how you have a really good research background. You were doing self-driving car research at Uber.

You fundamentally have to ignore all customer feedback in order to build something that's like more research related. What's that highest leverage challenge you're facing? It basically just comes down to execution and scaling, but I think the core product is like pretty sound. I mean, I actually don't think that's like necessarily a existential challenge.

Jerry, thank you so much for coming on to the podcast. Thanks for having me. We're at your office right now, which is really cool. What when did you move into this office?

Yeah, we moved into this, you know, beautiful space at 118 Second Street, uh, about 6 months ago. Wow. And what made you choose this? Um, it was just really nice.

I mean, you could probably tell from the lighting. We're kind of at the heart of uh Fi. You know, I could be a real estate agent at this point. Uh but it's it's great.

You know, there's kind of a lot of startups down the street, too. And it's great to be in the heart of kind of where a lot of AI startups are building. Yeah. Let's get into it.

So, most people in the developer world know Llama Index, 25 million monthly downloads, Fortune50 customers, 27 million raised. But the whole thing started with one commit and one tweet in November 2022. Tell me more about that. Yeah, and you know, the company's evolved quite a bit since then, but I'm happy to trace the origin story.

So, I think this is back when, uh, GBT3 was just coming out. Uh, and people were starting to play around with some of the capabilities of the GBT3 API. Um, you know, Chad GPD had not even come out yet, but a lot of builders, especially in San Francisco, were starting to test some of the capabilities there. And one of the things I've realized was I wanted to feed GPT3 some of the private context that I had available within my systems instead of just having it regurgitate what it remembered.

Um, this sounds pretty obvious right now, but at the time it was quite a novel idea. And so basically what I built was this like open- source set of tooling to enable these LLMs to ingest and almost like index some of this data in some sort of tree. So it actually could fit within a 4,000 context window that GPT3 had. Um, and so this predates rag.

This predates retrieval. This predates a bunch of like the more advanced techniques that have come out since then. Um, but basically I had this kind of like hierarchical structure that could surface some basic context to uh LLM um at the exact point that it needed it. Um, and be able to index an entire collection of data um to be able to deal with the 4,000 token context window.

So at the time I called it a GPT tree index. Um again this predates retrieval embeddings rag uh not embeddings like in general but just like rag as a concept. Um but it took off on Twitter because clearly like everyone was trying to figure out this problem of how do I you know inform the LLM with like my own private sources of data so I could actually build various types of applications with it. Um and it was totally by a stroke of luck.

Um you know I think I was working on some other ideas at the time that candidly I was like more interested in. Um but you know because of this uh I just I felt the urge to keep working on it because it took off on Twitter. Um and then you know investors came inbound. Um a lot of builders became interested and at a certain point I thought you know this probably could just be a company.

Um if I actually just became a little bit more serious about this because it was pretty clear that uh we're kind of at the cusp of this like new wave of like emerging technology. Um tragically came out uh you know we kind of started this company. Um, I got Simon on board as a co-founder and CTO. Um, raised from Greylock in April of 2023.

Um, and the rest is history. Wow. And when was the moment where you decided to make this company like no actually start building this like seriously? Yeah, it was probably a month in um when I it's one of those things where you know I've always been interested in AI.

I spent my entire career um in some combination of machine learning research, machine learning practical applications of serving at a large scale in production um to actually building kind of ML related products at a startup. And so I I knew that was going to kind of embark on this journey at someday. Um it just took a little bit of time about a month to realize like you know this is probably one of those points in time that is relatively rare to come by and I should really just take advantage of it even if the form factor of the products and the mission uh would evolve since then and of course it has. Yeah.

Nice. Let's go back to November 2022. You're working full-time at robust robust intelligence. What were you trying to solve for yourself when you built GPT index?

Yeah. So, it was basically what I described where I had like a bunch of calls stored in notion um through my call notes um because basically I was an engineer but I was on like probably four or five like customer accounts just like making sure I kind of understood their needs and could help resolve some of the pain points. Um but the issue is because I was working on a bunch of different things. Um every time prior to the upcoming call for a customer, I would kind of forget what was going on.

All right. And this again sounds very obvious now. these days there's Fireflies, there's Granola, there you can use like quad to pull transcripts, but at the time um there really wasn't a good tool for this. Um and so I was thinking about what if I had uh GPT3 ingest, you know, just like all the context of all the previous call transcripts about this customer.

Um so I could actually feed this in and have it synthesize some context about an upcoming call, right? Um, and of course that exceeded the 4,000 token context window um because the uh volume of calls was just so much higher um than kind of like 4,000 tokens is not that much. Um so you need to figure out some clever retrieval strategies to actually inject kind of relevant data to actually help answer you know specific tasks or questions at hand. Um that's exactly what like data structure helped to help to serve.

Oh and then you pushed the first commit and the first tweet the same week. what happened and the days after. Yeah. And then basically, you know, I think uh I pushed the tweet, got 300 likes on the tweet.

Um I think some people retweeted it. Um I had like 20 followers or something at the time. Um so 300 likes was like a lot. Um you know, I was very excited about it.

Woke up in the morning. " And then um afterwards, I just kept pushing like more PRs. Um started getting a little bit more distracted. You know, I don't feel great about this, but it was just like a little bit more distracted by like kind of having it as a side project.

um more and more people started using the tool. So I opened like a Discord to have like people in the community join um started to create like a really basic website or like Twitter account for like the tool itself. Um and it kind of just took off and again like a month in I was like all right I should probably actually get a little bit serious about this and actually start to kind of build something real. And so you study at Princeton is that right?

How was that just experience there? And what kind of moments do you feel like shaped you um to to get you to where you are now? So I mean Princeton was cool but you know I think basically I got into machine learning pretty late um in in kind of like my years in college. I got in probably like third or fourth year of machine learning.

Um at the time you know I think um it was like 2016 2017 um there were these like new advances across deep learning that were about to come out or had just come out. Um, transformers would come out like a few months after I kind of started getting into it. Um, but at the time there were these like generative models called like GANs, like generative adversarial networks that like for anyone like who kind of like was following deep learning at the time, for me it was awesome. Like basically it was kind of the first set of models I saw that could actually start generating pretty realistic like images of like people of uh kind of like uh scenery and and some other things.

And like you know before that like all machine learning was just for me at least like you know regressions or decision trees or kind of like something pretty basic. Um but this is one of the first instances where I actually saw it start to do like really cool stuff. And so I think that's where I really saw a lot of opportunity to just like kind of want to dive in more deeply understand how this like worked uh like the science behind it and then start to think about some of the applications. Um that kind of like fueled my entire journey um post college into into ML.

And then I mean you were at Kora and then Uber um in the ATG like uh research sector and then robust intelligence and now Llama Index. Was there did it all feel like separate chapters or do you feel like there was kind of a through line everything like connected to Yeah. I mean I mean I think in in general everything kind of tied back to building up my experience around machine learning. Um, you know, Kora was really like serving large scale recommener systems um to kind of like the, you know, millions of users that were using Kora to kind of look up questions.

Um, and I think I got a taste of like what it takes to actually like build really cool ML at scale. Um, also I had a lot of really smart people. I don't know if people would know about like the Kora like mafia, but like you know Alexander Wang came from there. Like a bunch of Iy people were there or like IMO people.

Um, so I just felt like I was in a group with like pretty smart people. Um and then afterwards I joined Uber where I met my co-founder Simon. Um because I wanted to get a little bit deeper into like just uh ML research like deep learning research um where you know you were actually submitting papers like kind of actually uncovering various discoveries. Um so I joined the self-driving research lab at Uber ATG.

Um and we were publishing papers to CDPR uh Nurups and a few others on um you know I worked on everything from like sensor compression to perception to planning um a lot of just like concepts around like both uh deep learning neural net architectures to like RL to kind of like a lot of other concepts. Um afterwards I joined robust intelligence which really much was um a little bit different in that sense because uh going from research back into like startups actually building product gave me a sense of like what you actually need to do at a small fast growing company to you know try to achieve like PMF. Um so I kind of like have a well-rounded experience of like research and product from a variety of different uh perspectives right from both kind of like the research side to the startup side. Um, so I think it was a nice like blend of different things.

Um, I think of course when you start a company, um, like you learn a lot of things that there's no way you could actually learn before starting a company. So there's just like in general a net new set of like new learnings and experiences that you pick up. Um, but I do think, you know, a lot of the existing career stuff was like formative. Yeah.

And I want to talk a little bit more about how you met your co-founder at Uber. Was this like kind of coincidence how you guys crossed paths or were you guys working together there or closely? Yeah. Yeah.

So I met um you know my co-founder Simon back at Uber when we were doing uh machine learning research and um yeah I mean it was it was great uh Uber had a lot of just extremely smart uh hardworking people that I deeply respect um where people were one just like uh extremely interested in advancing the frontiers of machine learning um and then two they were just like grinding all the time like it was um it was awesome you know I think kind of coming from a world I would say in like the mid2010s where like I I would say tech was like relatively chill. Um, and you know, kind of that like Silicon Valley was still relatively chill, but getting into a place where people were so passionate and just like were kind of like doing anything it takes to actually advance the frontiers of science was was uh great. And you know, I think Simon is probably one of the smartest people I've met. Um, and is also just a great person kind of like culturally.

Um, so I think when I was looking at the set of potential co-founders, um, I think he was the top top of the list. Yep. That's awesome. And then you mentioned how San Francisco was very chill.

So in what ways uh was it ch more chill? I think back in like 2010s and you know 2015 especially and again like people that have actually more directly worked on like different startups at the time um probably have a deeper sense. I do think um and take this with a grain of salt. uh people probably just like um like uh cared a little bit more um at least in my circle about just like going to some like big tech company uh kind of like chilling with all the benefits that it awarded.

But I felt a little bit less of the like hunger and energy of just like shipping as fast as possible because I think you know ever since the advent of like AI um I think the landscape has transformed so much and also the people that are able to take advantage of it are able to also reap the highest rewards such that like you have a lot of smart people in this like race to try to kind of like build some really cool stuff. I feel like some of that urgency maybe was like not there as much right in that space. definitely not in the AI space because I think um you know 10 years ago the space was still maturing. Yeah.

And then how could you explain how would you explain Llama Index to someone who isn't an ML engineer? So today our core mission is probably a little bit different than the origin story. Um and so today basically we are building the document infrastructure for AI agents. um you know fundamentally if your organization has kind of like PDFs, powerpoints, word documents and you know like 90% of the world's unstructured information is within some of these document containers.

Um every large enterprise from financial services to insurance to legal to manufacturing has volumes of paperwork and just like company knowledge and context stored within these documents. For us, we're building the best-in-class technology to actually unlock information from these documents, parse them into clean representations, and also enable agents to kind of like operate over these documents um at the semantic layer. So I think for us you know we care a lot about building the best-in-class technology to uh really like you know OCR digitalize these documents extract them into the right representations and enable agents to kind of like be able to like operate over them uh clean of hallucinations um and be able to you know like operate downstream accurately at scale. Um it's a little bit different than kind of a broader mission when we first started where we uh were a little bit more at the orchestration layer, right?

We were a rag framework, but like since then we've really gone all in on like document processing um because it's just such a core need across both like kind of AI startups to some of the largest enterprise out there. And um especially as agents get better and better, the thing that we've realized is they need access to kind of like clean best-in-class data, right? And so I do think us and you know like Composio you guys um and a few other players are kind of playing within this space but in like complimentary patterns um whether it's kind of like document based context or like web context or like SASbased tools. Okay.

So the bottleneck isn't model intelligence it's documents. Explain what that means in practice for a real enterprise. I think the bottleneck is in the context layer. Um so basically uh first of all I mean I you know that's great marketing for us.

bottlenecks in documents, but just to you know put it a little bit more broadly and including you guys too. The the the bottleneck is in like the ability to actually uh inject context into these like ever intelligent agents. And so I think the reality is all these models um especially with the latest releases from you know anthropic and open AI are like absolutely insane like they're they're like smarter than humans. Um they can do like a crap ton of different things.

Um the issue is people are still figuring even the smartest people in SF like right now are still trying to figure out how to make the most of these uh models to actually get it to do work. And if you think about like how much the SF like leading AI engineers are struggling with this like the average user within an enterprise that is not like following Twitter every day um has absolutely no idea how to actually make use of it to kind of get leverage on their time. And a lot of that is because the AI needs um actual uh specifications of the workflows plus like deep ingrained context within how an organization operates to actually be able to do real work. Um especially real work that's up to the human standards are defined by you know like kind of um by by the company itself and not necessarily some some like objective grading.

Um context is the thing that actually enables agents to actually do real work. Without context, agents are just like blank slates that like, you know, can't really do much. Um, if you give it like a simple task, it will do the simple task. Um, but in order for it to actually solve tasks, it needs to be deeply embedded and deeply understand kind of all the data that you have.

Um if you think about the massive volumes of data that exist within any organization, it exists in like conversations, it exists in call transcripts, it exists in like you know structured systems of record and software uh companies from like CRM to ERPs to web data. Um and of course for us it also exists in just like massive piles of like unstructured document based paperwork. Um, so I think for us, you know, especially across all these industries, we see a lot of opportunity in being able to again like digitalize that information and make sure that's actually stored in some accessible knowledge base that agents can tap into. So when they're solving a really hard task, they actually know what's going on within the organization to actually be able to do the task.

So I think in the end, context is everything cuz without context, like the models will just like sit still. They're not going to do anything unless you actually like prompt it to. Um and of course you know like even if uh kind of the future of these interfaces evolves beyond like chat uh into something a little bit more proactive um these agents by definition need to kind of understand how you know your world works and they need context to do so. And so you have an open source framework that 25 million people like download every month for free and a cloud platform they pay for.

So how do you think how do you think about what stays free and what gets monetized? Yeah. Well, um, it's actually pretty simple now. So, I think we have this like core platform.

It's called Llama Parse. Um, we kind of changed the name from Llama Cloud to Llama Parse just to make it a little bit more direct, but fundamentally it helps like parse your documents. Um, and then helps like, you know, parse and extract your documents. Um, the free tier just gives users, you have like some set of free credits to get started.

We also have a startup program with like more free credits. Um, and then afterwards, the paid tiers kind of like just scale with consumption. Um and so basically, you know, we offer different self-s serve levels at different tiers with different volumes of credits and volume discounts. And how did you get the name Llama Index?

Yeah, so we um it was kind of an amalgamation of llama plus like kind of what we were doing at the time, which is, you know, like a rag like indexing a bunch of data. Um the animal llama has the letters lm in the name. Um so that's how we came up with it. Um we came up with this name before like Meta came out with the llama models.

Um, so they like, you know, like technically stole the name from us. Oh my gosh, interesting story. Wow. And then in October 2023, you published something called Rag is a hack and it went everywhere.

What were you trying to say there? Yeah, I mean I think you know 2023. So, you know, I kind of came out with this project, right? This thing called like llama index.

By the way, before it was called Llama Index, it was called like GPT index. Um, and I changed the name to Llama Index to avoid like, you know, trademark issues or whatever with GPT. Um, but you know, it didn't really work, right? It was like a tree structure.

Um, this predates rag. Of course, rag came along. Um, and what is rag? Rag is basically you, uh, take a bunch of text.

Um, you instead of like indexing it into a tree, you put it into a vector database with embeddings. Um, so you can actually, you know, retrieve information from it. It's very much like people everyone was building rag back in like 2023 because it was the the simple thing was like a five-step process. you just like chunk put it into a vector database and then give it to an LM to generate an answer.

Um it was pretty clear to most people that it just like didn't really work super well um for like the majority of use cases. It was fine for like answering basic questions. Um but you know just for any like answering any complicated question or even starting to like do something like summarization even like slightly more complex tasks uh it would totally fail. Um, and so when I say rag is a hack, it's because there's really no first principles to how like you know the why you should like chunk retrieve that type of thing.

It was just one of the easiest things that like kind of worked so that people could build some like basic PC's. Um, so I think ever since then, especially with like evolving agent loops to kind of like better abilities to actually structure your context in a variety of different databases to more standardized interfaces like MCP and tools where agents can actually interact with external sources. Um, a lot of this has hope like you know like luckily improved a little bit over time. Um, and so I think rag was like a good concept at the time but uh to get started but it was like woefully incomplete.

Um, and luckily you know I think the space has advanced a lot since then. And what's the new thing that people are doing today that you think is the new hack? Yeah. Well, I think there's um luckily, you know, I think the um definition of the word hack kind of implies that you like things like are still not working super well.

And I do think the capability frontier has improved quite a bit. Um but I also do think there's still a lot more things to do. Um these days, uh there's a few general patterns. One, um you know, I think agents have gotten much better at reasoning and they standardize interfaces around like tool use and MCPs.

Um, and so you're actually able to have agents be uh clever enough to be able to loop on themselves to be able to uh you know launch multiple questions uh at a time, break down a complex task into smaller ones and kind of be able to like pull apart different pieces of information to synthesize a response. Um and then the second is I think people start realizing I mean like the the search stack is still generally like BM25 like keyboard search or uh vector search. um you know just the importance of kind of like interle this with kind of more flexible tools for the agent to actually read and operate over you know unstructured documents or like other data. I think the thing that is still unsolved though is um I don't think people generally know how to build a production scale like search system on their own.

I think rag is like kind of a extremely simple version of a production scale search system. But if you actually wanted agents like your AI agent to tap into just like a massive pile of like unstructured context, um your engineer would need to like build a lot of that stack towards actually ingesting data from your uh systems and be able to index it into the right way for search and retrieval. And I I don't think most people really know how to do that. And why what do you need to know like to be able to build it then?

you kind of need to have like um basically there's a few things um one you know with MCP these days like you know anyone can kind of like connect their data sources and like you can kind of like use interfaces like composio for instance you like compose your SAS like and and like other tools and you have your agent just like interact with this and I think that's a good enough start because um you know again agents themselves are getting pretty smart even if the tool interface itself is like not standardized they'll be able to get like decent results but if you look at you kind of how Google works or any sort of like web index or if you actually want to like improve the capabilities in terms of accuracy and cost you need to actually start looking into like uh indexing instead of just like um directly letting the agent query the service you probably need to actually pre-index this data that's why like this idea of a web index exists right when you launch a query on Google it's not like it actively like calls like every single domain that it has out there and like crawls the links like on the fly it has a pre-index of like kind of web links um and it's able to search that index. That allows for both like faster search and also more relevant search and retrieval. Um building that is non-trivial because you need to maintain data pipelines that actually connect your data sources. You know, kind of actually update the index with like fresh data.

uh make sure that you're able to kind of like tune your uh like uh search and storage stack so that you actually rank all items um with like the right weights and have like rigorous benchmarks and evaluation metrics so that you can actually like surface the right information at the right time. Um I think that's actually honestly um not very trivial to build. I think plugging in MCPS into an agent is pretty easy and it'll get search like 80% there to the point where honestly for a lot of tasks like that's totally fine but if you really want fast relevant search um and you want to just be able to do it at scale um I think there's like a long tail complexity in actually making sure that works. Okay.

And then you so you published five ML papers and then you mentioned how you have a really good research background. you were doing self-driving car research at Uber and so let's talk more about how research background changes the way you build products. Yeah, I think so. Um I was actually kind of thinking about this uh the other day because fundamentally especially with what we do as a company today.

Um we actually have to integrate a a mix of both like research and product. Um because we're building uh something at the paro frontier of like document intelligence or understanding to in order to actually be able to unlock data from like you know PDFs, powerpoints um into well structured context to some extent. Um, we need to make sure there's a significant gap versus what like the frontier models are able to come out with versus there's always like new open weight models coming out all the time and we need to improve that both at the kind of like harness layer at the orchestration layer and then also we need to kind of like do applied research to actually distill fine-tune and RL like smaller models for like specialized document understanding. So for us to actually build something really good for document parsing, we do need to do uh research and combine that with product.

I think one thing that I've learned um especially kind of having worked in research and also having worked at startups which is the opposite end is how fundamentally different the two are. Um so that you kind of have to reconcile them in some way. Like I think um working in research gave me an appreciation for just like the fundamental uh effort you need to do like general discovery um general just like deep understanding of the space and then being able to kind of like um actually commit to kind of like some sort of bet and like uh iterate on that until you're successful. uh from a technical point of view um it is quite different than building a scrappy product that has product market fit because basically um again I kind of like mentioned this um uh I think I was talking to uh you know Simon about this you uh fundamentally have to ignore all customer feedback in order to build something that's like uh like kind of core research related like basically customer feedback still kind of matters but it doesn't really matter like the idea of like finding product market fit is you build some like scrappy product and you'll like rapidly hill climb on it and the implementation details don't really matter.

Um you just need to find something that like satisfies the user. Um research requires you to make a bet that like fundamentally you need to make a piece of tech really really good um and that that on its own is like a worthwhile like uh achievement um in order to actually kind of like you know push both the frontiers of science and also like worth spending your time on over a longer period of time. Oh and then what what do you mean about hill climb? So hill climbing is just like improving the capabilities um for us just to give you a sense of the tension right like you know we need to kind of like generally improve the capabilities of uh like document parsing and also extraction which means there's a lot of these parsing benchmarks where we need to make sure we are able to parse even the most complex documents across like tables charts um you know like uh kind of like everything from research reports to contracts and all that stuff with like uh close to 100% accuracy we are not there um in order to out there um and and make that like super cheap as well.

You need to commit a bunch of resources towards like training the best models and towards deeply understanding the latest ML techniques towards kind of um you know like optimizing whatever you can on the agent to harness. So that let's just say like takes up the bulk of resources on the research team. Let's say then you have like kind of a few different customer requests come in and oftentimes the way customer like kind of feedback comes in is it can be very custom right like hey I need like a specific help or a specific like annotation on this like table or I need like kind of the UIUX to actually be able to review you know this specific uh type of document. Um like basically you know if we were only building a product we would just like drop everything else and just like do what the customer told us and try to generalize that into something that we could sell.

Um, and you know, like to some extent, it's not like the tech needs to be super good. It just needs to, you know, fit their needs. But for us, like if we were actually committed towards improving the core capabilities to some extent, you do need to like almost like balance that feedback, not necessarily ignore it, but put it on like some sort of like priority queue with the rest of research. So otherwise, um, you can actually continue to focus on like capability improvements over time.

Okay, awesome. Thanks for sharing that. Can we talk about your paying customers? Is that something we can disclose?

Um, depends what um the the bigger ones like data bricks and KPMG. Data bricks isn't paying customer but I can I can talk about some of the public ones but I can't I can't sure the cost role is yeah what does it mean when your enterprise customers are also putting like money in? Yeah. Um in terms of just uh our customer distribution you know we serve some of the uh like AI native companies to some of the largest enterprise.

Um, so this includes companies like, you know, like Lovable to, uh, KPMG, Carile, uh, Sex, um, and other companies across, you know, kind of insurance, uh, finance, energy, and and others. Um, and so we're pretty fortunate to be in a position where we work with kind of pretty large enterprise customers. And the way that we serve them is we basically provide the core document infrastructure that's able to help ingest a lot of their paperwork within the enterprise and help serve both kind of like AI native use cases around like chat and aantic search to kind of like aent tech workflows of being able to automate various types of like kind of like human based like document workflows. Okay.

So you use Composio connect. Let's talk a little bit more about Composio for your day-to-day workflow. What does your day look like when AI is handling that coordination layer? Yeah, so I mean just as we are to kind of like documents, um I would say composio is kind of like the connector layer to you know the broad set of like integrations that you typically use.

Um oftentimes like non-document based right but it's like the broader set of data that you typically want to connect to. Um and so you know if you think about it you have all these like general agents like claude codeex like openclaw co-work and everything else and um everybody especially those in SF is kind of using a set of like services right um this includes CRM like you know Salesforce HubSpot to ERP systems to you know like ticket management like linear Jira to Slack of course to notion um to Gmail um to a bunch of other things and so for me you know um a lot of this comes down to like internal usage of AI um you know our entire engineering team is using a lot of these like coding tools as with like you know a bunch of other companies in the space. Um for me personally um especially since oftentimes I'm not doing as much like technical work. Um I oftentimes really want to use these agents to help me like do comprehensive like uh research on like customers um to help me again do the call prep use case I kind of mentioned in the beginning but now you know it's way easier.

um to doing like uh kind of like almost product strategy synthesis based on like customer feedback. Um to I guess like being able to start to automate like various workflows like uh create like email replies um you know be able to kind of like prep for like uh or analyze our sales pipeline, be able to like prep for various types of like one-on-one meetings and so on and so forth. So I think I mean obviously to do that you need data. Um, and you know, I think when I was looking at just like tools, um, first off, like I really, um, I just think quad code, like the Cly, um, even though CLI's are kind of a terrible user interface, um, is just fundamentally more fully featured than like co-work um, and so I continue to use the Clyde and like it was really annoying to try to try to figure out and configure like every single MCP um, through kind of like oftentimes the integrations were not supported or had to manage like access in some weird way.

Um, and then I found, you know, like Composeio, obviously, I mean, I kind of know knew about it, uh, like a while ago, but I think, um, what I enjoyed about Compose for you is instead of like me as a developer trying to integrate this into my application, it was just like something I could use out of the box. Um, so I just used it, clicked a bunch of buttons to connect like all my integrations and it becomes like a single source where uh, you know, my cloud code um, and codework and codeex to some extent can just like interface with uh, all the like data sources I have. Nice. I want to go back and talk a little bit more about your experience at Uber.

So, you were there during the most ambitious period of their self-driving program. Would you say that? Yeah, that's correct. Yes.

And what was it like to be inside that? What did you learn about building AI systems at scale that you carried into Llama Index? I would say the thing I learned at Uber was more um because a lot of the work was around like at least within the group uh research as opposed to kind of like necessarily like the platforms or engineering um because I think we were in like the research lab so it was like core R&D um is fundamentally the amount of like uh appreciation you need to have for some sort of like medium to longerterm horizon bets um and the fact that if you actually want to build something extremely technically deep you need to build both like the research culture and talent and also like organize the product roadmap such that you're able to generally improve like core technical capabilities over time um and you know as I mentioned it is a little bit different than like if you were only focused on building products because um when you're betting on this uh especially when you want to like um kind of build a real piece of deep tech you need to make sure that your timelines are able to reflect that and it's a little bit different than kind of like the you know like it ship it whatever like mentality of just like building a core piece of like a SAS offer. Okay.

You probably mentioned this a little bit already, but what do you think is the most important unsolved problem in the space right now? Yeah, I think there's a lot of different things. I kind of hinted at this. I do think there's some open questions around just like massively solving agentic like context retrieval and injection.

Um I think like MCP is a start towards connecting to all the different data sources but actually how you organize all this information. Um and like how you can actually get agents to actually you know follow this like entire decision trace of like context across your data sources and not just kind of like make that data accessible somehow but make it like super efficient to search over. Uh make it very relevant so they're not like corrupted by like stale context. I think it's still an open problem.

You know, I think MCPs kind of get you some of the way there. Um, but again, search is slow. It doesn't necessarily return to you relevant context without you aggressively prompting what to search for. Um, and basically figuring out how to manage like context across different sources and feed it to an agent.

I think it's still it's actually still an open problem in my opinion. Um, we're kind of trying to tackle a subset of that. I'm sure you guys are also trying to tackle a subset of that too. Yeah.

Um, but I just think uh there's still like more to be done and I guess just time time will tell. I mean, yeah. I mean, I think it's like some kind like it's almost like people need to like put their heads together, right? Like we're we're kind of like figuring out everything on like the document side.

Yeah. Because you kind of need to solve problems on both like the data ingestion and also like merging stuff together and also retrieval. Um so you need like the MCP connectors, you need authentication obviously. Um but you also need like legitimate like retrieval like um kind of uh like expertise.

Um and then you need uh like to be able to combine all these different data sources together. you need some sort of like self-improvement loop as like users actually, you know, kind of like uh search and analyze over this data like kind of improves the scale over time. Yeah. What's the thing agents still can reliably do today that you expect them to be able to do in let's say 18 months or so?

So I think what's happening right now is that I think agents are actually pretty good at like doing um specified tasks um in an end to end manner like yes you know maybe I there's a bunch of benchmarks like coming out on like coding and whatever like I think on verifiable tasks like like coding I think as long as you have the specification laid out um it's able to do it like extremely well. Um I think what maybe is starting to become like the next territory. Um and I think people are thinking about this like you know you see all these like threads on Twitter is for all these like specified tasks the human has to be deeply in the loop. Um right which means for you to actually get leverage on AI you have to actually be in the loop of defining the specifications because if your specifications are off the agent is not actually going to be able to you know build the thing that you like precisely want.

Um and so you're obviously not going to trust it. The net result though is that it doesn't actually save the human that much time because you have to basically be extremely obsessive about following every single one of your cloud code sessions or codec sessions. And you know if you look at all the discussion threads on Twitter, people are like more tired instead of less tired than before because they need to manage all their agents that they're trying to spin up um and make sure they don't fail. I think um over time um especially as like agents are able to kind of get a little bit more senior and mature in thinking um there needs to be like some sort of like outer loop where um that it's almost like you have the agent doing this the brainstorming of like what specifications to describe so then it can loop over these like agents that are actually taking the specifications and like solving the task.

Um and if you think about it that's basically like how humans self-organize right you have like tech leads you have managers and you have like IC's um like you know top level defines like OKR is the tasks like product defines you know the product specs uh engineers like write code and implement it I think there's still a lot of open opportunities and how uh kind of like agents are able to take more underspecified tasks break it down um and be able to kind of like solve things um but also encode all the like cultural context and norms within organizations So it solves things according to human preferences. And just um elaborating a little bit, what are some of those specifications you're talking about? Oh, I mean like I'm kind of using the term like generally, but if you just take coding as an example, right? If you give like the coding agent a detailed specification um and this is where context comes in applied to um of like exactly kind of a here's like the coding like you know not just the task itself but also like the coding style like here's the general code base that you should explore and kind of understand.

um and then like give it some like more complex long long horizon tasks um and have it be able to do it end to end in a way that like a production engineer could just like review um and approve without like any further reviews. Um I think like it's it's like kind of getting there. I know like the benchmarks are getting harder, but like as long as you specify the thing really well, it should be able to do it. The issue is then like how do you then kind of have this like outer agent be able to almost like it's like the um I think claude code and codeex have like the slash like goal command um and then everyone's like building their own agent loops to do this.

How do you give it like a less specified task? Um, but like a a high level like goal or something um just like and then give it some access to like cultural norms and like context um and how do you basically enable it to do things without giving it as much information and can it like delegate an entire like team of like agents to start to do this? You're kind of starting to see a little bit of this flavor with like Fable and whatever. But um I don't think it's I I think it's almost like less a technical problem and more like a like a model problem and more like a harness problem and how you like delegate this and human behavior patterns towards like actually using it.

Yeah. And if you're starting Llama index today, what would you build differently? Would you do something differently? Well, I mean that's kind of a I mean like I think the um I think fundamentally what we're working on today like we wouldn't be doing this thing like right now if it wasn't like durable in the market.

So I think fundamentally I think the context layer like there's still a bunch of like opportunities within the context layer to actually solve. Um and and I I do think like the document market's like huge like I mean I honestly probably would be doing something similar in terms of just like how do agents like manage and operate over documents. Um and because like just there's still open questions on how agents are able to kind of like access and structure uh like unstructured data. Um but of course like you know especially at this current moment in time um there's a lot of other opportunities too, right?

There's stuff at the con connector layer. There's stuff at the kind of like um like context graph decision-making layer. There's stuff at the web search layer. Um and then of course there's still opportunities for various verticals um to get like you know oper like automated um like or sorry like made more efficient with a lot of like AI automation.

Yeah. Can you tell me something about the product which you haven't talked about publicly yet? Um I think uh it's a good question. So I think well I I we don't have like kind of a like this like major splashy thing coming up but just to give give users a sense you know we have like just incredible capabilities around like document parsing and extraction um we've invested very deeply on both like benchmarking for document understanding we evaluate every single frontier model provider and open weight model and OCR solutions on document understanding with our recently published benchmark called parsbench we are working on expanding the ben benchmark um it contains right now 2,000 pages of enterprise documents across like financial services, insurance, legal um and the goal is to make sure that um you know these solutions are able to structure these documents um in the the right clean context.

Um and it turns out you know Frontier model providers are not like amazing at actually reading like a good chunk of real world documents. Um and that's that's actually why we think we still have um just like offer a ton of value because we're both able to offer higher accuracy with like much lower cost. Um in in addition to that though um I think we're also looking at like document extraction like given a schema how do you actually large scale like analyze a document to then like fill in kind of like a a set of fields um and there's a lot of research we're doing at the agent layer to make sure that we're able to do that very efficiently. Um and the model layer to distill models that can do that um and we're also work working on like benchmarking efforts there.

And what are you most looking forward to this year? Yeah, I mean I think there's there's a lot of things. I mean we are um we have this like incredible team um that we're looking to grow. You know I think we're like 45 uh you knowish people or so and we're looking to kind of like scale that up.

Um we are looking to kind of expand some of the surface area around like documents. Um so uh in addition to kind of like the core intelligence like parsing extraction engine extending into like editing capabilities to like general management capabilities you know besides retrieval and search getting into versioning permissions and a few others. Um I just think there's like a lot of value to be had in building like software where like agents are able to collaborate and kind of like manage uh this like unstructured data and also kind of figuring out how the format of like unstructured data evolves over time. M what's the like highest leverage challenge you're facing right now?

Um I think there's I mean there's a few things. One I think the main it basically just comes down to execution and scaling. Um you know we're trying to uh you know hire folks in San Francisco and and uh just hire really really talented people. We're trying to kind of like generally make sure that we have like a very tight ship that's able to kind of like scale based on um kind of like customer demands and feedback.

Um, but I think the the core product is is like pretty sound. I mean, I actually don't think that's like necessarily existential challenge. Yeah. Well, thank you, Jerry.

This is a really great conversation. Thanks for sharing.