A Microsoft AI Futurist on What Everyone Gets Wrong About Agents

an agent with its own identity. And this is kind of emerging. You don't really see this in the wild yet. Well, I have built a number of these agents myself.

This thing will have its own email address. It'll have its own like Teams box or Slack box or whatever it might be. How many agents have you built? I have been building agents since 2023.

You know, I mean, my my job is AI futurist at Microsoft. You've argued that giving an agent your credentials isn't just a technical challenge, it's a risk. Take us to the moment you realize delegating your identity was the wrong way to build this. " For one thing, from a security perspective, it does not just automatically inherit all of your permissions.

What is a white nightmare scenario you'd be terrified of? Later on today, I'll be meeting with a guy from a bank and banks are very conservative about this. I think a couple of years ago, the Air Canada incident. I think you're Canadian.

This guy went to the Air Canada chatbot and he uh somebody in his family had died and he asked whether they had a grief discount. So, the chatbot completely hallucinated this discount Marco, hi. Hello. Thank you for being here.

Thank you for having me. Where are we right now? We are in Salesforce Park. So, we are five stories above the city of San Francisco uh in Soma uh next to Salesforce Tower, which is the tallest building in the city.

What does this place mean to you? Well, I've been here for a while. So, I have lived in the San Francisco Bay area my entire adult life. 27 years now.

Uh, in fact, the building behind us, I used to live in that building over there for 11 of those years. Yeah. Uh, so I have been in this area. I've worked in this area uh almost my entire adult life.

That's awesome. Okay, so it's Saturday morning and you're building something in GitHub pilot co-pilot. Walk me through what does a VP of products build for fun? Uh well okay so this weekend it was Saturday morning I guess actually and I was in the GitHub copilot app in this case and I decided that I wanted to build something to test a contact center scenario.

I mean I mean it doesn't sound like something you might build for fun but I thought it was fun. Uh I come from a contact center background and so back in the day over 20 years ago I wrote a lot of contact center applications and things like that and uh people are trying to do that now with with models with with AI. Um it does have some challenges and I have experienced some of these challenges firsthand in particular these things have a tendency to just halt. you know a model when you talk to a model if you're talking to a model that is real time that can talk or whatever when it makes a tool call you know when it when it makes an integration to something by default it goes silent it just does not keep talking and it will not make any noise at all and this is very disconcerting uh so I wanted to see this weekend if I could solve this problem and I did succeed that's wow that's amazing and so what does this mean mean to people building in the space?

Well, for me personally, what it means is that I am much better prepared for a keynote that I'm giving in October. So, I'm actually working with a company called Parlo, which is in the contact center space. And one of the things that I wanted to talk about is exactly this problem of like these things go silent, right? I don't think Parlo does that.

I think they have also worked around it probably in the same way that I have. Uh but uh you know I I basically I wanted to talk about the what I call the four cardinal sins of the AI context center. There are these mistakes that people keep making and it really annoys me especially when I call one of these things and I experience it firsthand. I'm like oh come on guys like you could do better than this you know.

So anyway uh so that's something that I was looking to illustrate. So, in a way, this is fun for me, I guess. And I guess it's fun for me to feel like I'm well prepared. Yeah.

Uh so that even though it's all the way in October, that I have, you know, some time to kind of bake this in and so when I do this on a stage in front of hundreds or however many thousands of people are going to be there, uh that I'm not going to look like an idiot. Okay. Expand on the four cardinal sins. I'm curious.

Okay. So, the four cardinal sins. Well, cardinal sin number one and the most common one is uh the tool call silence thing. And so the you know you have this AI context center thing and nowadays they sound realistic, right?

They will not sound like a robot. They'll sound like a real person. Uh so cardinal sin number one, the most common one is they will go silent. And this certainly will tip you off immediately that oh this is AI.

Like you're like hey are you still there? And it literally will not respond to you at all. It'll be like it's just quiet, right? uh cardinal sin number two and three uh are related to each other.

So cardinal sin number two is that you don't explicitly identify the thing as being AI. I mean you know honestly you should just tell people upfront look this is this is artificial intelligence you know moderate your expectations. Number three is uh a lot of times people will not have an obvious escalation path to a human. So, if the AI just cannot do what you wanted to do, don't make me jump through hoops to get to somebody who can do what I'm trying to do.

And the fact is, if you are calling a a call center, if you're calling somebody, uh, chances are you tried to solve it yourself in some other way with an app or whatever. And so, if you're calling somebody, there is a problem. Right now, increasingly the AI will be able to solve it. But if it can't, then get me to somebody who can.

And uh cardinal sin number four is bad grounding. And that is actually kind of related in a sense of that is to say that if the AI can't answer the question or do the thing that you need it to do, well that means it's not grounded well. It's you have not given it the tools, the integrations, the knowledge that it needs to do its job. And that's going to cause people to fail over to human escalation, which is exactly what you don't want.

I mean the reason that people are doing this stuff is what they call deflection. They don't want you to talk to a human if they can avoid it. That stuff is expensive. So those are the four cardinal sins.

Silent tool calls, representing itself as a human, not enabling a human escalation and bad grounding or in integration. A thank you for sharing that. This is very useful. And is this useful for what kind of people should know about these things?

I mean, for one thing, I think I I show this a lot to executives who are optimistic about uh quote unquote modernizing their contact center. I I've been in the contact center business in some fashion for almost 25 years now. Uh so I've been doing this for a while and this has been a theme all along. I mean all along people have been looking for deflection and certainly when I got into the business it was the IVR.

So, like here in San Francisco, if you call United Airlines, you're hit first with this IVR. And what do you say to the IVR? Customer service. Customer service.

Because the IVR is useless, and it always has been. Uh, but now we're moving past that. Now we're getting to a place where you really can do something interesting and fun and useful with an AI in the contact center if you don't do stupid stuff. And so you've argued that giving an agent your credentials isn't just a technical challenge, it's a risk.

Take us to the moment you realized delegating your identity was the wrong way to build this. Okay, wait a minute. So let let's talk about we got we got to roll this back a little bit. Okay, let's do it.

All right, so if we're talking about agents, there are kind of three ways that you can use an agent today. And that agent could be anything. It could be chat GBT, it could be, you know, Microsoft Copilot, could be something you build yourself. The first way and kind of the default mode of using like chat GPT is anonymous mode.

It's not logged in really as anything. I mean you might be logged into chat GPT or something like that but it's not really doing anything on your behalf and it's mostly kind of answering questions based on public knowledge. So it's not going and looking in your emails or your calendar or any of that stuff. Mode number two is on behalf of mode.

And so here for example I am using chat GBT at least in my personal life in an on behalf of mode. In my business life I use uh primarily Microsoft copilot and co-work for this but uh on behalf of means it's logged in as me. Uh and that's super useful. Uh so that means that it can send emails on my behalf.

It can make calendar entries on my behalf. And so, uh, this weekend I did actually, uh, make some calendar entries in my, you know, we have a shared like family calendar. It's a Google calendar that we use, uh, me and my wife and my daughter to like coordinate between us. And, uh, I added some stuff to the calendar with this thing.

Specifically, my daughter was going to the Oakland Pride Parade and I didn't really know when or where it was. And I was just I went to Chat GBT. like add the pride parade to my calendar and you know invite them so we all know that she was going to be there at that time and it worked. It looked it up.

It found where it was and it you know put it in the calendar at the right time. Great. On the other hand uh if it does something stupid it's doing something stupid as you. So now I bought a new car uh about a month and a half ago.

So, I bought a new car. Uh, and uh, the thing is I came to find out that well, I haven't got my license plates for this car yet. So, the dealer uh, I called the DMV. The dealer apparently did not like properly submit the application.

So, that still in some kind of pending status. So, I had chat GBT draft an email to the dealer and uh and I was like, "Okay, copy everybody I know at this dealer like the because you know the salesperson is this person going to want to talk to me anymore. " Probably not, right? So, you know, CC the contacts person, CC these whatever.

CC everybody. Tell them what the problem is. And it sent the email. So, it sent the email.

Then I went and looked at the email after the fact. " like it sent it just to the sales guy who like I said is probably not gonna want to talk to me and it in the body of the email it says CC contracts at whatever and you know whatever instead of literally CCing them like oh you like when I say CC I mean actually copy them on the email you know uh so when it does something stupid it's doing something stupid as you just like in my business life something similar happened last week. I was using uh in this case I was using co-pilot to book my book me a room at our 555 California office here in San Francisco. It booked me two rooms.

Why? I don't even know. I got two confirmations. I I I don't know what happened there.

It booked me two rooms. So, you know, okay, easy enough to reverse, right? Neither neither of these things are fatal. I can, you know, just I can CC the people manually.

I can unbook one of the rooms. still it does a stupid thing. It did a stupid thing as me and that required my intervention to fix it, right? Um the third mode is an agent with its own identity.

And this is kind of emerging. You don't really see this in the wild yet. Uh well, I have built a number of these agents myself because I can. Uh and you're going to start to see it over the next year.

You're going to start to see more and more agents with their own identity. Uh so, you know, yours might be Julia's agent Julia's assistant or something like that. And you will probably have one, right? You will have one.

And these things, this thing will have its own email address. It'll have its own like teams box or slack box or whatever it might be. Uh and so you'll be dealing with it as if it were a separate user and it will like for example email people as a separate user. So, if it's scheduling this podcast with me, for example, like I'll be dealing with not you, but you know, Julia's assistant, uh, and that has some benefits, right?

For one thing, from a security perspective, uh, it does not just automatically inherit all of your permissions, right? Now, this thing, when I use an agent on my behalf, okay, it booked a room twice, but it inherits all my permissions. So it could have gone and deleted a bunch of files if it really went sideways because I can delete a bunch of files. But I can set it so that my agent that has its own identity in its own user space, it can delete files.

It just cannot do it. And so as it can go as sideways as it wants and it can try as hard as it wants to delete files, it can't do it because its user just is disallowed from doing such a thing. Uh so from a security perspective that gives me a certain kind of a firewall uh that's kind of useful. Uh but also from a social perspective you know if the email let's say stupid email right your your assistant sends me stupid email and you can be like ah it's my stupid agent right and it's obviously you know it's obviously not coming from you.

Yeah and you know this too. Yeah. So it gives you a little bit of stupidity isolation which can be useful. Uh so that that is a nice separation to have.

On the other hand because it has a separate identity that means that you the user you may have to jump through hoops because it can't naturally do everything that you can do or see everything that you can see. And so you may have to if you wanted to share a document or a video or something on your behalf. Well, you have to share it to it first because otherwise it can't see it. It's a separate user.

Yeah. Do you think that's a bit do do some people would they not like if you have an AI assistant? Would they see you as lazy or like what's what do you think people's opinions about or thoughts about this are? I'm not sure yet how this is going to go.

Uh you know I think it's interesting that when I like yesterday I was composing an email using co-work and co-work is so in this case I'm using M365 co-work which is kind of a a variant of anthropic co-work but it can also use other non-enthropic models uh and it's pretty useful I mean it can see my calendar can send emails and stuff like that. So yesterday, uh, I was kind of on behalf of one of the customers I've been working with, I was emailing some other executives at Microsoft to obtain their assistance, which I did succeed in, but co-work does this thing where at the bottom of the email, it says sent by co-pilot coowork. Uh, and I don't like that. I don't know because Yeah.

You know, I don't like that, too. So, uh, I make it, save it to my address, and then I delete that part. Cool. Okay.

Uh, before I send it. Okay. And that also gives me an opportunity to just check for stupidity, right? Is it kind of like did it?

And sometimes that stupidity is in the text itself. But sometimes the stupidity is like it mal formats the thing like it doesn't put any line breaks or something that every now and again that happens. For the most part I've kind of conditioned it past that. Now you can give these agents memories and things like that to kind of condition them past this stuff.

And most of the time that works. Not always. Um, so you know, it gives me a chance to give it a quick glance before I go send it to a bunch of executives. Uh, so yeah, I mean I'm a little sensitive about it.

Uh, I don't know if other people would care or even notice it. It is pretty small down there underneath my signature and stuff like that. As long as it's going to drop. Uh, but maybe as it becomes more common, you know, it'll it'll just be a thing that you do, right?

It'll be like, well, of course I have a scheduling assistant. Of course you have a schedule. You know, I have my agent talk to your agent. We'll do lunch.

You know, that kind of thing. Yeah. Yeah. So, it's very I think now because it's very uncommon, people find it so weird, but every people get used to things and everything's evolving anyways, right?

I mean, in the same way that like people send you like a calendarly link or whatever, right? It's like one of those things where you like use the website to book yourself. And I think you sent me one of those. Did you send me one of those?

Now, what you may not know is that I had my agent do that for me. So, actually, when we booked our first meeting Yeah. Uh so in this case uh I was using something new that most people don't have which is called Microsoft Scout. It's a kind of a variant of copilot.

It runs locally and it has a computer use agent. Computer use agent uses a computer. So you sent me this calendarly link if I remember correctly. And so I said okay go you know look up that email from Julia and go you know to her calendarly link find some time in my calendar that works and go book it on her thing.

And it did. So cool. And that's how we ended up even sitting here today. That's really helpful cuz it's like less thinking on your end too, less deciding.

I mean, I could have gone to my calendar, find the time, go to your thing, go find a could have done that stuff. It would have taken me, I don't know, five minutes, right? But I during that five minutes, I was doing something else. I don't know what, but whatever it was, it was probably useful.

Yeah. Uh, and then it was done. How many agents have you built? Uh, a lot.

A lot. I have been building agents since 2023. Uh so you know I mean my my job is AI futurist at Microsoft and so I am supposed to be in the future a little bit and so okay yeah the first agent that I built was actually a calendar management agent and I used to show it on stage to people and they would almost fall out of their chairs. They'd be like oh my god you know what I used to show the canonical demo that I would always show.

I had my daughter's school calendar and I would say to the agent go look at a school calendar and book spring break in my calendar. and it did it uh pretty reliably and you know that to people they were like what that's super useful and it is super useful. I mean as I said I do it all the time. m.

Riyad time, not San Francisco time. Do it right. And then I figured it out. So time zone issues are still a thing.

Back in 2003, it was a really really a thing. Like used to be that large language models and time zones did not get along at all. Now it's gotten quite a bit better. Okay.

Um so yeah, I have built I don't know hundreds of agents maybe. Wow. So you're you're the world world class at it now. I'm pretty good at it.

Uh yeah, I'm pretty good at it. Uh and you know the thing is though nowadays when you when you build an agent when you're thinking about building an agent to do a certain task the first thing you need to think is do I need to build this thing? So for example um I built this agent that is an unsubscriber agent. You know I do a lot of like marketing stuff and whatever and I get a ton of like crappy marketing emails from all over the place every day including this morning.

Right. So back in 2023, I think it was 2023 or 2024, I built an agent that was an unsubscriber agent that used at the time it used this thing called browser use, which was also a computer use agent. It would open up a browser. So basically the trick is I would put the emails in this unsubscribe folder in my Outlook and this thing would connect to that folder, pull the emails out.

Now the thing about unsubscribe links is that you can't really unsubscribe programmatically. It doesn't really work that way. So from Constant Contact, Mailchimp, and all these other things, these pages are what we call post render. So if you go just to the link itself, the page itself is kind of blank unless you render it in a browser.

And then after the fact, it kind of renders that stuff. And all of these are different, right? They're all sometimes you got to put your email address in there, sometimes you got to check a box, sometimes you got to check multiple boxes. It requires a little bit of reasoning to unsubscribe from this thing.

So I built this agent that did this quite effectively. It would just go through my unsubscribe folder and for each thing it would open up a browser, do the thing, whatever, figure it out what it needs to do, unsubscribe. Great. But now, if you fast forward to today, how I do that now, well, I don't need that agent anymore.

That agent is on my YouTube channel, so you can find it still, right? And in fact, it's in a public repo. You can use it still if you want, but I don't use it anymore. I don't need it anymore because now there's a couple of different ways that I can do this.

But um there is a a preview feature also in M365 copilot called Opel. And Opel is a computer use agent that provisions a virtual machine. So it basically provisions a little computer in the cloud for you. And in Opel I have made an unsubscribe skill and in this unsubscribed skill it basically describes the process.

It says go connect to my Outlook which it can do. connect to my Outlook, go grab these emails for each one, open up a browser, go to the thing, figure out how to unsubscribe and do it, right? And it does it. And I just run this every morning.

So I like as I was sitting here waiting for you to show up, I actually put one of these emails in my unsubscribe bin. Later on, it's going to run. It's going to fire itself up. It's going to follow the skill.

It's going to unsubscribe from the email. I won't even look at it. And it works. Uh, and I didn't really need to build anything for that.

Now I don't need to have any infrastructure up. I mean this is just a general purpose agent. So the question is now you can make these skills in co-work in M365 copilot. I think you can make it even in chat GPT certainly in anthropic co-work there's lots of places now that you can make skills and a lot of times now you could do with a skill what you would have had to build an agent to do even two years ago even one year ago.

Uh skills also, by the way, came from coding agents. And so I have skills up the wazoo in GitHub copilot app and in claude code and stuff like that. I got skills all over the place. Uh but now they're kind of leaking into the real world, you might say.

Do you know about composio? I am familiar with it. Yeah. Yeah.

What do you know about Well, I mean Composio is a mechanism by which you can integrate agents to other things. And do you do you use I don't presently use Composio? No. I mean generally do you build out your own like integrations?

Well, I mean a lot of times the things that I'm using already have an integration. I mean a lot of the things that I'm integrating to well in the Microsoft ecosystem for example uh you know Outlook, Word, uh SharePoint and Teams and we have an integration layer for that which is called work IQ. Oh. So Work IQ is kind of the headless version of all the Microsoft applications that agents can use.

Uh, and so, you know, at least in my business life, that's what I'm primarily using to integrate. So, most of the things I'm trying to integrate, too. Yeah. Um, yeah.

So, I haven't really had a need to use Composio yet, uh, for that reason. Okay. And so, let's push on this like identity advantage. So, if you're a startup founder without the benefit of owning the whole stack, like no M365 or Workspace identity suite, what is the one nightmare scenario you'd be terrified of?

Well, if you're a startup in the agentic space, you do you need to think about your agents going sideways. And there's a couple of different ways that they can go sideways. Uh, and you also need to think about your customers environment. So, how are your customers going to be running these things?

And in what environment are they going to be running them? Uh, you know, later on today, I'll be meeting with a guy from a bank. And banks are very conservative about this. I mean, in a couple of different ways.

First of all, from a security perspective, as you might expect, they are uh very diligent about ensuring that these agents can't go sideways and do crazy stuff. But there's also the compliance angle. I mean, you have to think about these agents going sideways in a different way where they're not doing something that's insecure or they're not being exploited per se, which is also a concern, but that they are doing something that is non-compliant. And some people may remember, I think a couple of years ago, the Air Canada incident.

I think you're Canadian, you might remember this where uh this guy went to the Air Canada chatbot and he uh somebody in his family had died and he asked whether they had a grief discount. Sometimes airlines do that. " And then when he went to go book the ticket, he found no such thing exists in Air Canada. That's not a thing.

Actually, the chatbot completely hallucinated this discount and he sued and won. mostly I think on principle. I mean I don't think it was a huge amount of money but I mean you know and and he kind of had a point right now if this were a bank now this would be a problem right this you know you cannot as a bank promise you know for example I cannot promise you if I was a banker or a wealth manager that your portfolio is going to rise 10 times that's not a thing that I can say and you know if you have a wealth manager or you're dealing with a bank they will record all of these calls and they have these compliance checker things to make sure that people aren't saying stuff that they are not legally allowed to say, making promises that they're not allowed to make. So that's a major concern uh for these folks.

So as a startup founder, you need to think about who is your audience, who are your users, and what are they going to be concerned about, and how can you mitigate those concerns? These agents with their own identity thing are part of the mitigation now, or at least that's one of the options. Uh so at least if you're if you're deploying this in a in a Microsoft environment I mean Microsoft has this kind of stack through and through right we have Entra as the identity provider there are the applications themselves uh when you make an agent it gets an email box and things like that because we have outlook and stuff like that uh Google has that also if you're doing that with a different system it might be a mishmash right you might have octa as the identity provider you might have salesforce over here as the agent host or whatever and then you're going to have to think about how to put that all together and how that identity works. perks and what all the tendrils of it are.

That's part of it. And then the other part is uh you need to think about again how is the agent behaving. Now what you find today is that we and pretty much everybody in the industry are moving in the direction of what we call rubric based evaluation. Rubric is kind of a rule that the agent needs to follow.

So it used to be for the last 3 years or so we have all been using these relatively generic metrics and they started with things like groundedness. Groundedness is the opposite of hallucination. So basically you would feed to this metric you would say here's what the agent responded here's all the documents it retrieved and it would use another large language model an adversarial LLM to check to check the response on the documents and say did everything in this response come from these documents. So if it had checked that Air Canada thing, it might have, you know, the Air Canada chatbot probably retrieved some documents from Air Canada.

It did not find anything about a grief fair in there and it said it anyway, right? So groundedness, it would be like you get a zero, you get a thumbs down uh for groundedness on that. Uh and there's a bunch of other metrics that uh that we use now which are like agent style metrics which are like task completion. Did it complete the task you asked it to do?

task adherence, did it do only that thing and not some other random thing? Uh tool selection, like did it pick the right tools and stuff like that. But the thing is these metrics, these metrics that we have, I'm going to say traditionally have been using even though it's only been a couple of years. They're pretty generic.

I mean, they don't know what a given agent is supposed to be doing. So, you know, you say task completion, for example, like let's say that I ask my agent to let's say I have an apartment pricing agent. This is a real thing that I was talking to with a real customer one day. So, let's say that they have a brand new apartment complex, right?

Avalon or something like that. They got all these new apartments and they they have an apartment pricing agent. They give it a spreadsheet of here's all the apartments and their bedrooms and their layouts and the things. Now, I want you to spit me back a spreadsheet with prices in it.

Okay. So, now consider task completion. Task completion as a metric. You asked for a spreadsheet.

You gave it a spreadsheet. You asked for a spreadsheet back. " Yeah. I mean, you ask for a spreadsheet, you got a spreadsheet, right?

But now the apartment pricing manager, the human opens the spreadsheet up, looks at the prices, and let's say that for the one in the two bedrooms looks pretty good. Okay, but the three bedrooms, oh no, like something has gone very wrong here. This is way off, you know. But task completion as a metric is going to have no idea about this.

I mean, it all it knows is you ask for a spreadsheet, you got a spreadsheet, but it doesn't know what that spreadsheet's about, why it's doing this, any of that stuff. This is a generic metric. A rubricbased metric or what Anthropic calls an outcomebased metric, they call it outcomes, everybody else calls it rubrics, uh, is different because it's tailored to the specific use case. And so, a rubric based metric would be like, okay, so for three bedrooms, you need to consider X, Y, and Z when you're doing this price.

for one and two bedrooms, you're going to consider A, B, and C, whatever it might be. Or for a bank, the rubric might look like you need to be compliant with regulations D and E. Literally, I actually have a rubric that says that. Uh, or, you know, you need to be not just polite, but you need to not make promises to the user about X, Y, and Z.

Uh, and so you'll have this big long list of rules that this agent must follow. And you can run these rules, these rubrics in bulk. And so you can give it a whole bunch of like simulated conversations and make sure that it doesn't run a foul of the various rubrics because now each one of these rubrics, each one of these rules becomes a metric in its own right, something that you are now measuring in bulk and you want a certain threshold of them like you know say for compliance you might never ever wanted to make a non-compliant utterance. Uh which is challenging because it is a statistical measure.

These things are nondeterministic. Yeah. But there are ways that you can ensure that it doesn't run a foul of this. Um anyway, so uh this is another thing you need to think about as a startup founder is like if you're doing something in AI and agentic AI and stuff like that, how do I make sure that it's not going sideways from a security perspective and from a what you might say an alignment perspective?

In banks that's compliance, but in other contexts it's alignment. uh and uh these are the kind of evolving mechanisms to do so and so with over 20 million users now on Microsoft 365 co-pilot we've moved past the novelty phase so in your view what separates organizations where co-pilot genuinely stinks from the ones that stall after the pilot yeah so there's a couple of different ways that uh any AI can stall I mean this is Microsoft copilot or really anything and one of them is this kind of overpromising I guess. Uh you know for example uh in 2025 Excel co-pilot kind of sucked you know Excel co-pilot at that time it really did nothing useful nothing that I could discern and so the problem at that time was that if you open that little C-pilot sidebar in Excel and you tried to make it make a pivot table or something which is what something I tried to do it just it just couldn't do it. In fact, it couldn't even like format cells.

" And it was like, "I'm sorry, Dave. I'm afraid I can't do that. My name is not Dave, but I'm I'm quoting the you know what what what movie is that? " Uh anyway, um but then in January of 2026, it changed completely.

So, Excel Copilot got a complete retrofit and now it's super massively useful. In fact, I was using it yesterday for something uh where it could totally can make pivot tables and format things and I used it to fill out whole giant questionnaires. I used it to fill out a 1,185 questionnaire last week uh in like an hour. It's ridiculous.

I mean, the stuff you could do with this now uh is is absurd, right? The problem is if somebody had tried this in like December of 2025, they'd be like this is useless and they will never open it up again. They have no idea that it went through this kind of sea change in the intervening time. So that is I think one of the challenges with any AI system is this idea of overpromising and a lot of people you know have tried like Microsoft Copilot prior to also 2026 and thought okay well you know it could kind of answer questions but for example like until relatively recently it couldn't manage uh your calendar it just couldn't do it.

Uh but then suddenly it could and now it could do it in spades. I mean like I said I mean I made these meetings with it. I use it all the time. So if you're not keeping a breast of what's happening with these things, a lot of times you're you're missing the point.

You're missing the you're missing the thread. Um and so sometimes it's said that the definition of insanity is doing the same thing twice and expecting a different result. However, in AI that doesn't really apply. In AI, you actually have to kind of try it and try it and try it again because eventually it does work.

And this was certainly true also of Chat DPT. You know, for years I had wanted Chat DPT to like manage our family calendar in in this Google calendar thing, which honestly Google Gemini should be able to do and it's super bad at it. I mean, sorry Google people, but you got to fix that. I mean, it's just terrible.

Anyway, but a couple of months ago, Chat GPT showed up with a Google Calendar and a Gmail plugin that really worked well. And now, as I said, I I do use it to manage my personal like family calendar every day. It's super handy for that stuff. So, you know, I tried it and I tried it and I tried it again with Chat GPT, with Gemini, which still frustrates me.

Uh, and finally, I got ChatG would work. So, am I insane? Maybe. But still, yeah, you kept trying.

That's it. It works. And looking at how people use it day-to-day versus how you imagined that they would, what's the most surprising thing you've seen? Uh I don't think that we have seen the surprising thing yet actually.

Uh I think that the voice capabilities if you really use the voice capabilities and like for example voice mode in chat GBT it's really quite good. Uh it still has problems. I mean sometimes I use it for example in the morning when I'm shaving and sometimes it actually gets distracted by the noise of the shaving. I mean my facial hair is quite thick actually.

you wouldn't know what to look at me. But, you know, uh, so yeah, sometimes I'll like talk to it about, you know, investigate some company I'm talking to that day or whatever. I'll talk to it and it's a little bit overly interruptible, but for the most part it works. And so, I was uh I was actually in an Uber up in uh in Redmond.

I was coming, you know, going to the airport in this Uber and uh uh my came to find out my taxi driver was Kenyan. Well, I was in Nairobi uh a little while ago. " Like, "There's no way. " So, I busted out my chat to you, put it in voice mode.

Now, it turns out, no, most people in Kenya speak Swahili. Swahili is kind of the the the language that you would expect a lot of people to speak, but there are lots of smaller languages. They're kind of tribal languages. And he said that he was from the Cuyu tribe.

" So I was like, "All right, Chad GBT, you know, I I can't remember what I I told it to tell me about something. " 100% just bust it out in Cuyo. Absolutely can do that, right? And I have done that.

I mean, I've done it in Cuyo and Luo in uh Tita and like those are all Kenyan languages. I've done it in Ebo and Ha and Euroba, which are Nigerian languages. And especially in Africa, people really fall out of their chairs when they hear this because you might expect Swahili. There are like hundreds of millions of people that speak Swahili, maybe tens of millions that speak Kikuyu, uh, you know, things like that.

And so, even for like lower resource languages, now you're starting to see the AI perform pretty well. And that opens up some really interesting new possibilities for I mean, there are like 6,000 languages in the world at least. Uh, and so for people that maybe primarily speak Kikuyu, what does that mean? You know, I mean, what does that mean for their medical care?

What does that mean for how they work with governments? This is true in India also. I mean, in India, you know, there are 22 official languages in India and many more sub languages and dialects. You know, we say a language is a dialect with an army.

Uh but you know if you if you speak like chhattisgari or something like that and there's not that many people that speak chhattisgari I guess mostly in chhattiscar in India uh I think that's the the least spoken Indian language of the 22 official languages but now the Indian government has created these interfaces that can speak all 22 some of them using our technology and uh and they can approach these folks that speak chhattisgari that speak like udu or speak you know well delegu lots of people speak Dell. So anyway, I think that really opens up some new possibilities. So that's I think one of the more surprising capabilities of AI today is its ability to speak and speak well a huge variety of human languages, but we're not really using it yet. Not like we will soon.

Okay, let's do some rapid fire. Um, okay. Would you build it or would you buy it? Co-pilot or Claude?

Well, I mean I I don't know. Yeah, co-pilot. These are these are these are well I use I use GitHub copilot app actually. So there are different ways that you can use GitHub copilot.

So cloud code and I like cloud code and I have cloud code too and that's that's great and they did a great job with that but it is a CLI. It's a command line interface and I am not a huge fan of CLIs when I can avoid it. Uh because they're limiting I mean that form factor is really limiting. Now the GitHub copilot app came out just recently actually it just went G in June and it's an application uh which is not an IDE so not like Visual Studio VS code things like that things that I have traditionally used copilot you can use it as a CLI there is a co-pilot CLI there is a co-pilot in Visual Studio and VS Code if you want to use it that way and then there's this standalone application which is the C-pilot app and I think the C-pilot app absolutely nailed it and part of the reason for that is uh Well, a lot of times when I'm building things now, I will spawn or it will spawn a whole bunch of sub agents.

So, it it's not just one agent building anymore. Cloud code does this, too. Um, but to see what those agents are doing and stuff like that, this is where the textbased interface of a CLI really kind of falls down. You know, it's not great.

I mean, you you can you can say the magic keywords in there and be like, okay, what are the sub agents doing and stuff like that? You know, that's not wonderful. But in the app, I mean, in the co-pilot app, there is a little sidebar mode called insights. For whatever reason, they called it insights.

And you can actually see all the agents enumerated and what they're doing. And you know, you can click into them and stuff like that, what they're thinking, so to speak. And it's freaking great, right? Because I mean, sometimes it happened to me last week.

I was doing something and it spawned 12 different agents at the same time. Uh that was kind of useful to see because and I and I kind of was looking through them because you know I kind of want to see is anybody doing something stupid you know and intervene and you can do that. You can actually go into one of the agents and and intervene with that specific agent or give it an instruction or you can give the instruction to the master agent to pass it along over there if you want and that usually works too. Yeah.

Uh but anyway, yeah, the app it just gives a much richer experience. Plus, you know, a lot of times in these CLIs and uh I think both cloud code and copod CLI are guilty of this. Your normal hotkeys don't work normally. So like to paste something into the CLI uh you can't just or to copy something you can't just use control C uh in Windows uh because that means break in a CLI context.

So that will stop the app altogether. You have to use a different hotkey. Uh so the CLI kind of breaks your normal patterns. And so you know a lot of times I've accidentally quit the thing thinking I was just copying some text like ah damn it you know.

Uh whereas the app just works normally you know normal copy paste normal hotkeys just like you're used to. Uh more pleasant rendering and everything. And so yeah, I just I prefer it that way and I always have one agent with your identity or seven with their own uh one agent with my Well, it depends on what I'm doing and what I'm using it for. Uh you know, so there are things that I do want to do on my behalf.

Sometimes I do want to send it like the email that I sent to these Microsoft executives yesterday to help me with this customer I was talking about earlier. That needed to come from me. that was not going to come from my assistant agent or my scout or whatever. That needed to come from me.

Uh but if it's like an agent that is managing my calendar, that does not need to come from me. Especially if it's booking two rooms, then I can, you know, two rooms by accident when I just wanted to book one. Then I could tell the guy who runs all the rooms at our office like, "Ah, sorry. " Yeah.

Just give me one. I'm good. Uh so then I can blame it on the thing. So it depends on what I'm using it for.

Okay. And if you blame it on the thing, then it's not really your fault. I mean, it's still my fault. It's still my fault, right?

I mean, you know, you still got to own it. Yes. Uh, and it's up to you to, as I say, condition your agents. Okay.

You got to teach it. You have to teach it a little. You're growing it. I mean, bear in mind, agents have skills and agents have memories.

And so, you can give it skills. And I have given my agents skills on how to write an email like me. And you know, part of that, the beginning of that is don't be AIish. That is, you know, no M dashes, which is ironic because I actually used to use M dashes a lot.

Sure. Right. And now I can't use an M dash anymore. Freaking AI killed it.

Right. Thanks, Chat DVD. Uh anyway, so uh no M dashes. Don't bold print random stuff.

All that stuff that this or that, you know. Oh yeah. Or from this to that. I hate that crap.

Uh that's just the beginning of it, though. I mean, you know, the way that I write my emails, I mean, I have a a little bit of an idiosyncratic style as most of us do, right? So, for example, I don't write a signature on my email like, you know, sincerely, you know, or whatever. I don't do that best, right?

I don't do that stuff, right? Um, because I have a an email signature at the bottom there. And so, I'll usually say no more than like thanks, but not like a sign off kind of a thing. And I don't I just don't do that.

uh which my wife sometimes takes me to task for. She thinks it's impolite, but whatever. So, my emails can be a little bit tur and that's just how I've always been. Yeah.

But the the agents, the AI naturally wants to sign off in this way that I don't do. So, I have conditioned my AI not to do that. Uh so that it comes off sounding like me. And so, uh yeah.

So you you it's still your fault if it goes sideways, but you can minimize the possibility of that by correcting it when it goes wrong and telling it remember this. Say make a memory not to do this thing in the future. So you know, like I said earlier, uh when I sent this email to uh my car dealer that hasn't properly filed for my license plate and it did the CC wrong, this is chatb's fault in this case that it did the CC wrong. didn't actually make a memory that when I tell you to copy somebody on an email, you are literally putting it on the CC line.

This should have gone without saying, but okay. Yeah. Uh and so hopefully in the future it will not do that stupid thing again. Awesome.

Almost there. Okay. What's the most overhyped AI term right now? What is the most overhyped?

Autonomous. Autonomous is the most overhyped AI term right now because uh you know for the most part we're still in this phase where we can't really trust what the agent is doing. And so even for me where I run agents up the wazoo for all the things. Uh the idea of just allowing chat GBT to autonomously send an email to my car dealer which I did do turns out to have been a mistake.

" Like, you didn't put the CC line where you're supposed to. So, uh, yeah. So, we still have to be really careful about autonomous agents, uh, or autopilot agents, which ironically at the moment is Microsoft's own branding for certain types of agents, which I'm not a fan of. Uh, we have to be very careful about that.

We will get there. We'll get there eventually, but we're not there yet. Also, yeah, automating everything like automating your work, like finding ways to automate your work. I mean, it's okay to automate your work, right?

Just like I automated the invite, you know, that that we made together the calendar for this uh event itself. Cool to automate, but to do it autonomously. Okay, that's okay, that's different. Okay, like literally like on its own fire and forget.

Fire and forget. Okay, there it's relatively rare the occasion when I can fire and forget even still. Well, thank you, Marco. This is incredible.

All right. Well, thank you