Knowitall Doctors, Mac Keyboards, More Love for CoWork, and Maybe it was the Models After All

Announcer (00:04): Welcome to Raw Data with Rob Collie. Real talk about AI and data for business impact. And now CEO and founder of P3 Adaptive, your host, Rob Collie.

Rob Collie (00:20): All right, Justin. Well, welcome back to day like 15 of the Rob is still sick podcast.

Justin Mannhardt (00:25): It's like a sinus infection now. There's no way it's been that the whole time. You're on this wandering progression of some kind.

Rob Collie (00:32): I don't know what's going on. I would definitely have a sinus infection. Do I still have something else at the same time? I don't know. But I didn't notice the transition, the changeover to sinus infection. And I had this virtual doctor appointment yesterday with ... He had to just now get out of med school and he was an insufferable no-it-all. It was like I had two different experiences of this appointment with him. One experience was me talking to a doctor and not getting what I wanted and that wasn't good. But the other part of me was experiencing this like, "Oh my God, what great sport this is kicking the crap out of this know it-all." I kept him there and tortured him for more than 30 minutes. I wasn't mean to him, but I just kept saying, "Nah, I've been to this rodeo. I know what happens when I get a sinus infection. I get a nasal rinse prescription that dissolves the antibiotic in it. And you just irrigate your nose with antibiotic. You don't take antibiotics in your whole system. If you can get the antibiotic to the right place and not your whole body, go ahead." And he's like, "I've never heard of this." And I'm like, "Well, that's not my problem, man."

Justin Mannhardt (01:41): I just graduated med school and they gave me this laptop and a webcam.

Rob Collie (01:44): He said, "You've ever gotten that here?" I'm like, "No, you know what? I only got it in Indiana. I know it's a high-tech, cutting edge Indiana and back here in backwater, Seattle, you might not have heard of this yet." These are literally word for word what I'm saying to the guy. He's just asking for it. And he's laughing and I'm laughing and we're just constantly just busting each other. Neither one of us is really giving in. No one's getting really upset. But he's like, "Well, what is it? What is the prescription antibiotic?" I'm like, "You're the doctor. You tell me what it is. Not me tell you what it is. I don't remember what it was. Look it up, man." He's like, "Mm." And I'm like, "You want me to look it up?" And he goes, "Yeah." So I started looking up and he goes, "Wait a second. Are you just Googling this? " And I'm like, "No, I'm not Googling this. I'm Clauding it." He thought I was looking it up in my records. I'm like, "If I had my records dude, I would've told you. No, I'm going to go look it up." He still didn't want to give me something at all.

(02:40): And in the end, I'm like, "I don't know. There's blood coming out of my nose." And he goes, "Oh, blood. Oh yeah, we'll get you some antibiotics."

Justin Mannhardt (02:47): Oh, real.

Rob Collie (02:49): I was giving up and just getting the hell out of here after 30 minutes of jousting with this guy. Part of me would pay for the fun of jousting and beating up on someone like that who is so full of himself that he needs it. I feel like I'm doing something for the greater good, but I want to separate that from my doctor experience.

Justin Mannhardt (03:12): Yeah. I can see that.

Rob Collie (03:14): We're going to be doing some antibiotics.

Justin Mannhardt (03:17): How long is the regimen?

Rob Collie (03:18): In the end, he wouldn't even give me the compounding thing that I wanted. He's like, "You know what you're going to do? I'm going to give you the ointment and you're going to dissolve the ointment in your saline rinse." And I'm like, "Whatever, just give me the prescription and let me get out of here." I don't know. I'm on my own.

Justin Mannhardt (03:33): God speed.

Rob Collie (03:34): I've got two rants today. I just experienced a Windows update. I joined the podcast and then I had the mother of all Windows updates. It was awful. It doesn't even understand what percentage means because I got to 21% and then it went to 7%. Every time it got to the next phase, it's like, "Oh, now we're applying another update." Anyway, the reason that my laptop was in that state is because I haven't been using it very much. This is my Windows PC laptop because I've been using my MacBook, basically for the solid past week. And it's been great. However, there's one thing that I am shocked about, just absolutely shocked about, and I can't believe it. And there's an old Chappelle Show episode that's relevant here. So for a while there, there was a TV show called Wife Swap.

Justin Mannhardt (04:24): Okay. Yep.

Rob Collie (04:25): I am proud to say that I have lifetime watched 0.0 minutes of Wife Swap.

Justin Mannhardt (04:30): Which rounds to Zero.

Rob Collie (04:31): I was aware of the existence of the show and Chappelle Show in the early 2000s made fun of this by having their own version of it where he's like, "I'm tired of the Wife swap show." It's always people of the same race swapping. I want to see the version where a white family and a black family swap. The skit is that. And of course it's over the top. Everything you'd expect to happen in this skit happens. At the end, it's almost like an outtake at the end of it with Chappelle. He's the black guy that went and lived with the white family for a little while. And it's like this B-roll, blooper roll or whatever of him going, "And you know what? White people don't use washcloths. What is wrong with you? You all are filthy. You're just rubbing bars of soap all over your body." He's just going off. He couldn't believe that there's this whole segment of society that's quietly not been using washcloths this whole time. And of course, my reaction to this was like, "Yeah, what is wrong? I use washcloths. I've always had washcloths. Do most white people not have washcloths? I don't know." To me, it was the funniest part of the whole segment was his discovery that all along under the radar, something absolutely mind-boggling has been happening, right?

Justin Mannhardt (05:48): Right.

Rob Collie (05:49): Okay. So here's my observation about the Mac world, and I'm going to try to do it in the Chappelle voice. "Wait, Mac users, you don't have home and end keys?"

Justin Mannhardt (06:00): I knew it was something about the key.

Rob Collie (06:02): "You don't have page up and page down? What is wrong with ... Are you even doing real work over here? Are you serious? Are you doing serious work? I use those keys, not really page up and page down, but home end and delete?"

Justin Mannhardt (06:19): All the time.

Rob Collie (06:20): I have to use a key combo, like a control key combo to delete the letter that's in front of my insertion point rather than the one that's behind it.

Justin Mannhardt (06:29): Yep.

Rob Collie (06:29): I don't get it. I hate this about the MacBook. I hate it. Oh, everything else is beautiful. I'm loving it. Cowork runs great.

Justin Mannhardt (06:40): Yeah, nice.

Rob Collie (06:41): It is a dream, but I'm still getting used to command left arrow, command right arrow to replace home and end. I'm really not used to function backspace is my delete. Have you gotten this far yet?

Justin Mannhardt (06:58): No. Before we started recording, I was mentioning I'm in the middle of a renovation of our bedroom/office area. My Mac Mini arrived yesterday and I declared, "Ooh, I'm so excited to set this up," which my wife responded, "Yeah, right after you're done with the project." So I've got an incentive that I need to accomplish because I have a nice mechanical keyboard, but it's got a Windows layout. And I was a heavy Mac user in college. Went to music college and that was the thing. This muscle memory, there's going to be some issues. I'm anticipating this and I don't know if there's a good solve for it or not. Everything's going to be desktop with that and I've got a KVM switch so I can go between both machines. So we'll see how it is.

Rob Collie (07:45): Well, I'm even wondering, does the Mac receive home and end from your other keyboard and do the right thing with it? Does it receive delete? I would sure hope it does.

Justin Mannhardt (07:56): I'm at least hoping there'll be a way for me to make assignments somehow.

Rob Collie (08:01): I'm DOA on the MacBook. It has the built-in keyboard. There isn't another key to map.

Justin Mannhardt (08:07): Well, you just got to get really good at dictation, it sounds like.

Rob Collie (08:10): Oh my God. The number of times I'm having to do home and delete and being reminded of how awful it is just ... There's always trade-offs.

Justin Mannhardt (08:19): This is worse than learning a new language.

Rob Collie (08:22): As a bit of a segue, on last week's episode, I was saying, and I think you were agreeing, Copilot Cowork is going to be a significant wake-up call to a lot of people about what AI is capable of. I believe that last week, I come back to you this week twice as committed to this. Here's a story you will appreciate. Of course, there's a block diagram in the book, showing pieces, parts of an AI system. And I've just hand sketched this for the moment and taken a cell phone picture, pasted into the Word doc, moved on. I don't want to get hung up on the diagrams. I want to keep rolling. The design and graphics team can follow behind me filling in these things. So this is really funny. Christie's been working with our designer and she gave that napkin sketch to the designer and he said, "I need a little bit more detailed version of this." He didn't want to work with this sketch. So Christie had to go sit down with an AI image generator and describe to it to make exactly the same damn block diagram, but just in a more polished form to give to him. She was even laughing with me. She's like, "Yeah, he needed a napkin sketch of a napkin sketch. He couldn't ..."

(09:36): And I saw all the iterations she got of the block diagram from working with the AIM generator and it was better than you would expect. It was doing well, but it was still off in so many ways. And when you start giving it extra instructions, it'll fix the thing you're talking about, but then it messes something else up. It's the usual thing. And I thought, "You know what? No, this isn't how it's going to be." So I sat down with Cowork, I gave it my sketch and I said, "This sketch appears in chapter two. Go read chapter two. You have access to it. Go read chapter two for context, for background, all that you need, generate me a brand new PowerPoint file in the folder with a block diagram of this." And it did. It went and did it, came back, and it wasn't good. It was better than a lot of the AI versions. But then I said, "Okay, good first start, make the following adjustments," and it did, and then make another following adjustments, and it did. And I could kick it off and let it run, and I could go back and do something else. And in the end, I finally ended up with something where I had to manually reach the point where the diminishing returns were there, and I got into PowerPoint and I just manually moved some things around and spaced them better.

(10:47): But it was producing a PowerPoint deck for me with objects that I can manipulate. And if I had given it an existing PowerPoint deck that already had a style of blocks in it, of the components that have been predefined, it would've used those. And this is just so much better than any other way of making this kind of graphic. And I had the foresight as I was doing this to have it every time, instead of just changing the existing slide, I had it make the next slide so that I had the version history. And then I essentially copy pasted the prompts I was feeding into the notes field of each slide. And I shared that with Adam and Christie today. I was talking to them about it in Slack. You tell these are believers. They believe in AI. They're like, "Sounds like it was really useful, blah, blah, blah." But you could tell it wasn't really landing. And then I gave them the slide deck with the iterations and the prompts in it, and they came back like, "Okay, wow, I could feel the electricity."

(11:50): This experience is about to come to every human being that's on the highest end skews. We might be approaching an event that's on the order of magnitude of ChatGPT release in terms of how much it stands people's hair on end. It doesn't have to be that big. I'm saying it's in the range of outcomes that it might be that big, and that in itself is really, really, really significant.

Justin Mannhardt (12:18): I saw someone that's in one of my feeds, one of their bold predictions was Copilot Cowork will have the same Teams versus Slack moment relative to ChatGPT specifically. They're like, "This might actually take off, and this is the way Microsoft really picks up steam in this race."

Rob Collie (12:43): First of all, whoever that was, I feel like, no, you are plagiarist. You are stealing. I'm just kidding. This is what happens when you happen to occasionally be right about something. There are other people who are also right. But I think it's going to be a seismic event for the world.

Justin Mannhardt (13:03): I think it's important because ... And this isn't about any particular tech vendor. I talked to so many people whose experience working with AI is void of this concept of something that has access to your stuff, tools to help you, and they're still boxed in of like, "Oh, I have a question answering machine." So for them to have this experience where they ... Once you come across that event horizon, it's like you can't remember not understanding it anymore and your aperture just gets so much wider about what's possible. And it's something maybe a bit nuanced for some about what you were describing with going through your PowerPoint deck is this really important difference between AI that just gives you output versus AI that works with something that has components and parts that can be refined in a very specific way. I've been enjoying this benefit of picking technology specifically for how well it works with AI in that regard.

(14:09): Image generation is actually a great example of this because most image generators, even the best ones, you just get the image. It's not like you're getting a layered design file or anything like that. This is funny. So I was doing some stuff and I asked, "This is perfect, but change the background." It can't do something like that. Just can't.

Rob Collie (14:30): If it's Cowork and it has access to the Photoshop API-

Justin Mannhardt (14:35): Yahtzee.

Rob Collie (14:38): Or Claude Code, right, whatever.

Justin Mannhardt (14:39): Cowork stands to really deliver on this experience, certainly within the Microsoft ecosystem, but most certainly beyond that as well. But I find myself just constantly underwhelmed with the AI that exists inside of a product. What I'm really looking for is where's the product that is open, has a rich API layer or a CLI or something like that because I want to work with the components inside of the thing.

Rob Collie (15:07): At the suggestion to Brian Jones, I went and tried Excel Copilot out again. In Excel Copilot now, there's a dropdown. You can choose Opus 4.6. Bring it on. Opus 4.6, that's my buddy. So I said, "Hey, build me a Monte Carlo simulation that tells me how unlikely Mike Tomlin's coaching career was. The 16, 17 seasons and never had a losing season." I can't put my finger on it, but it did not feel like Opus 4.6. It had its iteration depth or something set to slower. I have no idea what.

Justin Mannhardt (15:43): Yeah.

Rob Collie (15:44): It didn't go well, but it's going to go well. I think Copilot Cowork is going to be the first thing that helps them start to reclaim the positive associations with Copilot. The word Copilot is one of the best brand names you could ever come up with for an AI. And the fact that it's been so thoroughly trashed by so many failed half-step offerings is from a marketing and branding perspective, is just like a million branding people cried out in agony and then we're suddenly silent, right? It's-

Justin Mannhardt (16:21): Pain.

Rob Collie (16:22): Yeah, exactly. But anyway, so Copilot Cowork, of course, most of that goodwill branding wise, word-wise, is going to accrue to Cowork. It's going to be a wonderfully successful Microsoft offering. The Microsoft and Anthropic partnership here, that version of it is going to be better than what you would get cobbling it together yourself, but again, if they're throttling it in some way, and I think that's one of the things that we see in the built-in products is that for cost of goods control, they're throttling, limiting, using lower end models. All the people who apply for jobs with us that are using these AI powered services to just auto apply to all the jobs, the how many hipsters does it take to change a light bulb question that's multiple choice, and there's only one obviously correct answer to this question if you're a human being reading it. And if you take this question and you paste it into off the shelf ChatGPT, off-the-shelf Claude that has full power, it nails it. But still, the AI job appliers, somewhere behind the scenes, someone's decided, yeah, whenever we hit a multiple choice question, we just set it for the lowest LLM. And so these things are just constantly saying one or we don't know or whatever to the number of hipsters. I just short circuit those people into the reject pile, right?

Justin Mannhardt (17:51): Right.

Rob Collie (17:54): If you don't get that question right, you're not getting a job here. The built-in products are going to have to just open the throttle, the built-in AI integrations, and that might not be the only problem that they have, but you can feel it.

Justin Mannhardt (18:06): I don't know enough about the inner workings of all the details either, but it just seems that there's something with ... I think you made a really good point about how they're maybe trying to optimize for cost of goods, controlling tokens, compressing the semantic meeting of a question, or applying their own markdown instruction, if you will, of like, "Here's how to do this." That as models have gotten better, one of the things that if you've been close to, you've witnessed is they need less help in certain places as the models have gotten smarter. So as an example, there's a product that I use every day that has a well-known MCP server. I don't tend to use that MCP server all that often anymore because Opus 4.6 is just so freaking capable of just understanding the API and writing scripts for what it needs when it needs it, which wasn't necessarily possible with the context windows and the limitations and the way models were at the time. And so I just think how much of these solutions that are in software are built on a set of assumptions that are getting challenged on a frequency and a pace that we're not used to? You've sat in this seat and I haven't. If you're a product owner for something like a Canva or even Excel or PowerPoint, this is a different world entirely to think about how users interact with what you have.

Rob Collie (19:37): Yeah. Along those lines, you're talking about MCP servers and APIs and things like that, something mind blowing that I learned last week. And it's one of those things like in hindsight, I should have known this. All the information to come to this conclusion was available to me, but I didn't. In hindsight, I feel silly. So I was talking to Brian about Excel and how Excel makes itself available to the LLM in this Copilot experience, because it really can do anything and it will make live modifications to your existing document now, which is not where it was.

Justin Mannhardt (20:12): I remember when it would be like, "I can walk you through this one.

Rob Collie (20:15): I can tell you how to do this." And some of the Office apps are still in that state, but Excel has reached the point where they're like, "No, bring it on." I was like, "Okay ..." And again, it's going to be so funny how obvious this was. I go, "All right, well, it's not messing with your file format, right? The LLM isn't tearing open your file format like it would on my computer." When Cowork generates a PowerPoint file for me, it is going after the file format. That's what it does. It's assembling a file from atomic units. It's not messing with any APIs. So I'm like, "It can't be messing with your file format because that would be changing things underneath Excel and Excel won't know that it's changed and you'll get all these out of sync problems between the application and the file format." He goes, "Nope, we're definitely not doing that." And I go, "So what are you doing?" He says, "Well, it has access to our entire API." And I said, "Okay, the documentation of your API weighs as much as an encyclopedia. That is way too much to put into context. Where's the magic here? Where's the free lunch?"

(21:21): Okay, so here's the answer. No, man, the knowledge of our API is in its pre-training. It knows everything it needs to know about our API. We have home field advantage because the Excel API is so well known and so well documented and so ubiquitous all over the internet that it knows it already. It does not need an MCP. It does not need anything. It already knows everything it needs to know and it can write whatever code it wants to write against our API. And it's like, when you're a big enough product like that, you essentially get this crazy home field advantage. Isn't that wild? And at the same time, obvious. Why wouldn't it be in there?

Justin Mannhardt (21:58): Yeah. Yeah. It is one of those, of course, because I'm like, "Well, it can't be this and it can't be that because same problem." Wow.

Rob Collie (22:08): Seriously, the Excel API to know it would drown the context window right off the bat. If not full, you'd already be distracted and diluted to the point where you'd be getting poor results. This is an impossible thing, unless it's already in the pre-training. Claude almost opens up this idea of product placement, right? So if I've got a product with an API, can I go to Anthropic and say, "Look, we'll give you a billion dollars if you include this in your training corpus for your next model."

Justin Mannhardt (22:44): That's an interesting thought, Rob, because I imagine there's a variation of the block diagram you were describing here we could play with a bit. So there's the knowledge of the model itself. What's been happening in the past, let's just say the 12 months that have gotten people like you and I really excited, it's been things like skills, MCP servers, Cowork, Claude Code, this extension of the LLM into tools and other parts of software and then OpenClaw and things like that. And then Jensen was out at his keynote saying, "In the future, every computer will have something like this on it." And so this sort of era of tooling around the AI, I wouldn't put it past there to be a commercial opportunity there for these shops. I've been working with Canva recently, but I don't want to work with Canva. So they go, "Here, learn everything about our product," because the AI in there is just garbage, right?

Rob Collie (23:44): It's almost like a protection racket, right? Oh, you want your API to be in our ... It'd be a shame if your API weren't in our pre-training. Nice product you got there. It'd be a shame if our LLM didn't ... It's up to them. It is. It's up to them what it includes.

Justin Mannhardt (24:05): That's crazy. You probably already have some good nouns for this stuff, but when you move from AI as this question answering machine to this thing that gets work done for me thing, it's just night and day different understanding of what's possible. And I think for me, and obviously everyone I've seen go through, it's on net positive. It's unlocked all kinds of creativity for me, all kinds of productivity for me. I'm getting things done that I would've never even touched before. That's out of bounds for me. I can't do that type of work and I can't afford someone who can. I just wouldn't do it.

Rob Collie (24:44): One of the things that we've been saying, or at least I've been saying, and I think you as well, but I don't want to rope you in with me unfairly, because I'm about to say that this thing I've been saying I've been kind of wrong about. So it'd be unfair to say, "Hey, jump in this boat with me. Let's go somewhere," and then say, "Aha, we both suck."

Justin Mannhardt (25:02): We both suck.

Rob Collie (25:06): So I've been saying that a lot of the breakthroughs in AI lately haven't been about the AI. They've been about connecting it to non-AI stuff. So to use our magic Lego brick analogy for the LLM and then the regular Lego bricks, I've been saying that a lot of the advancements we've been seeing have been more like creative use of the regular Lego bricks connected to the magic Lego brick. And funny thing happens when you're writing a book. You start to think, "Okay, I'm going to put my thoughts down on paper," but then you realize, "Okay, now I'm really putting it out there and so it's going to be a permanence to it and I really actually need to double click. I need to ..." So I started doing some research on ... I had an AI buddy helping me with this research, but I was verifying the sources and all that stuff because I do not want to be called out for being factually inaccurate in some tiny little detail that doesn't make any difference, but it still torpedoes the credibility of the book.

(26:09): So I was researching the timeline of AI assisted development. It starts with Copilot, GitHub Copilot, and GitHub Copilot was essentially an auto complete machine. You could think of it as code to code. It was reading your code because code's easier to read and write than English and natural languages, So code to code is the easiest thing possible. And it was reading comments a little bit and using those as additional context, but this was happening. People were using the hell out of it before ChatGPT launched. By like 18 months, ChatGPT launched, and that was the next era. For the first time, it was English to code. You could write something and say, "Hey, I need a power query script that does blah, blah, blah." And it would spit it out, and then you'd go copy paste it and find out it didn't work. But then it's basically two full years before the next step change.

(27:05): It's two full years before cursor launches agent mode, and then it's like two to three months after that that Claude Code launches. So this two-year absence of a step change is really conspicuous. And again, so now I'm like, "Okay, I got to keep digging deeper." It turns out it was the magic Lego brick. So a number of things happened in those two years that were necessary conditions for the next step. It wasn't like humanity was sitting around going, "Well, we don't know what to do with this." And suddenly one day decided to put the Lego bricks together and they got something different. It literally wouldn't have worked with the LLMs. The LLMs had to get better to enable this. And so there were actually multiple factors here. One of them is that token windows went from 4,000 tokens to 200,000.

Justin Mannhardt (27:59): And now a million. Now we're at a million.

Rob Collie (28:03): We're now at a million. But from 4,000 to 200,000, 50X, you could maybe barely write a function in a 4K token window. By the time the back and forth of the conversation, you're eating up all kinds of overhead. It's not just the function it's writing that's going to be eating those 4,000 tokens is like 3,000 words. So that needed to change. Cost needed to drop. The inference cost, like cost per token of inference use, like actually running the LLMs had to drop. Accuracy had to go up. So the analogy I ended up using in the book is agent mode and clog code that we now take for granted will go and execute many steps on its own before it comes back to you. Up until then, it was one step. So if you have an 85% accurate LLM, that's actually kind of okay.

Justin Mannhardt (28:53): Yeah, you need that checkpoint.

Rob Collie (28:54): Back and forth with the human, right? So the human has this light finger on the steering wheel that's compensating for the 15% inaccuracy. But if you got a 12-step process, what happens is the LLM does something and then it feeds its answer into a brand new chat with itself. So you get this magnifying error rate. Like 85% error, I don't know if math off the top of my head, but you take it to the power of 12, you get like a 3% chance of a correct answer coming out the other end. So you have a not viable product. They could have chained this stuff together and built these long-running chains years and years ago.

Justin Mannhardt (29:30): Would have been devastating.

Rob Collie (29:32): But they would have sucked. All these factors, all these things that were changing in LLM research had to reach a certain critical mass before Cursor was able to go and do what they did and before CloudCo was able to go do what they did. And so I found that journey educational, even though it didn't suit my predisposed opinions about it, but I'm smarter for it.

Justin Mannhardt (29:53): When you're so in it's fun when you have moments of reflection like that and you stop As you were describing this, I was remembering, yeah, the first couple applications that I used AI to develop, I was very involved. Sort of this back and forth human in the loop thing. I reviewed every plan meticulously and we're going to compartmentalize this like we would a software team with epics and issues and be very methodical. I'm building a web application right now and I was like, "Hey, I think we need this feature and it needs to work like this. Can you just figure out what makes the most sense and when it's ready, commit it and push it up and I'll check it out." But that's just because I have this experience now where I just believe I'm probably going to get something pretty close to what I want.

Rob Collie (30:46): I was messing around with the math using Excel last night when I was writing this part of the book. If you want a 12 or 15 step chain, which is what these things are doing a lot of the times.

Justin Mannhardt (30:56): It's probably common.

Rob Collie (30:57): The accuracy rate needs to be in excess of 99% at each step for you to get a good result, a reasonable percentage of the time. They have to be really good at each step. I kept hoping that the math would be a little bit more dramatic. The difference between 85 and 95, I was hoping that I would show the reader, well, the difference between 85 and 95 is dramatic, but they both suck.

Justin Mannhardt (31:18): Yeah. It's like the concept in Six Sigma where they explain parts per million, a failure rate. You think like, "Oh yeah, we're on time 95% of the time." And then it's like, "Oh, that means this many million households are disappointed every day with your product. This is a brand killer."

Rob Collie (31:34): Yep, yep, yep, yep, yep, yep, yep, yep. Yeah. Totally. It also reminds me of audio feedback. When you get the microphone too close to the speaker, nothing makes that ooh sound nothing. It's just error being compounded and fed back in. And that's what it eventually ends up in. And it was getting the equivalent of audio feedback. You needed to get the clarity higher so that it could survive 12 round trips. It's just bananas.

Justin Mannhardt (32:09): If you've been doing the same stuff with AI for a while, now's a good time to get something like Cowork, whether it's in the Microsoft Cowork or the anthropic Cowork. And take these things for a test drive to see what's going on. It'll really open your eyes to where we're going.

Rob Collie (32:29): If you have to get yourself a MacBook, do it. Just be ready.

Justin Mannhardt (32:34): Just drop a couple grand real quick so I can get a $20 subscription.

Rob Collie (32:39): Oh no, it's worth it. It's totally worth it. And I didn't need to get as good of a MacBook as I did.

Justin Mannhardt (32:44): I don't even know how I would begin to formulate the ROI on some of these things where I spend, gosh, I probably spend close to $1,000 a month in subscriptions. Of course.

Rob Collie (32:55): Yeah. And that's cheap relative to what you're getting. We talked about this. Don't tell them. Please don't tell them, but I would pay 5,000 a month for this. Please don't tell them.

Justin Mannhardt (33:06): And it's always been this way. There's things that I legit just couldn't even consider doing at the cost it would've been prior to being able to use AI to move things forward.

Rob Collie (33:18): You know that's what they're doing, right? The reason why there's a bubble here is because they're all trying to outlast each other to be the one that charges $5,000 a month.

Justin Mannhardt (33:27): I forget how you characterized it the last time we brought this up, but there's a big difference between the financial bubble in the markets and the technology itself having any sort of bubble. And you can study the same fundamental concepts, even those very different in prior revolutions like this. AI will survive whatever popping of bubble occurs to the business institutions involved.

Rob Collie (33:54): It don't care. Technology, it's like ... Even the LLM itself, right? OpenAI's LLMs don't care about the open AI company.

Justin Mannhardt (34:06): And I think the races ... It feels a little different going back to your point about the advancements in the LLMs for that two-year span. It certainly feels now like, okay, who's actually starting to tap into the value equation of how businesses work and how people are able to use these things in ways that is transformative and productive and on net good for us all? But I don't know how you sleep on it. I just don't.

Rob Collie (34:31): If you're listening to this podcast, you're not.

Justin Mannhardt (34:32): Yeah. You're not. But we've both had conversations like this. Some people, for a variety of reasons, whether it's the software their company has chosen at the moment or what is approved or what is not approved or their personal situation, how people are encountering using AI takes on all kinds of formats that aren't you and I's experience. But yeah, if you have a friend, nudge them in the right direction because it's going to show up one day and it's going to be different.

Rob Collie (35:02): Another one of the things that writing something that's going to feel like forever forces a number of things that don't happen normally. One of them is double checking your prior hypotheses, but another one is it drives a thoughtfulness about things that wouldn't have occurred. I think that that two-year lull in the qualitative step change in programming powered by AI, I think that lulled us to sleep. GitHub Copilot comes out, and of course the hype was breathless. Then ChatGPT comes out and the hype is even more breathless. Now it's taking English instruction and writing code, holy hell, sky is falling, no developers are going to have jobs. And then for two years, nothing else happened. And I think that interruption in the trajectory, the breadth trajectory, confirmed in a lot of people's minds the thing that they wanted to believe, which was it was all bullshit.

Justin Mannhardt (36:03): It's nice and convenient to believe that.

Rob Collie (36:06): And so at the end of those two years, these new things started appearing, it's actually amazing how fast these things have gotten to multiple billions in annual revenue. Cursor and Claude Code each are north of two billion in annual revenue, like in an eye blink. These things are one year old.

Justin Mannhardt (36:24): They made the old charts of that pace obsolete.

Rob Collie (36:27): And at the same time, I'm pretty sure that that two-year lull has damaged the adoption curves of these things.

Justin Mannhardt (36:34): It would've been way better.

Rob Collie (36:35): Yeah. If it had been 12 months earlier, it would've felt like the continuation of that trajectory. But I think enough time passed that people just flipped the switch in their heads that this plateaued, that we'd peaked. They're still missing this other thing. I don't know. Not for long.

Justin Mannhardt (36:54): I think we're right on the cusp of it. Even when Cursor and Cloud Code and those types of products came on, it's like the way Power BI was for the Excel crowd and we tapped into this audience of people. Cowork did that in a way, and I think we'll do that in a similar way because cloud code is ... Even like GitHub Copilot. Okay, you're writing software all day every day. This is who this is for. Cowork is for basically anybody that sits at a computer for a living.

Rob Collie (37:23): That's a lot of people.

Justin Mannhardt (37:24): That's a lot of people.

Rob Collie (37:26): Oh, I also did some math last night. You'll like this, I think.

Justin Mannhardt (37:29): You've mentioned doing math several times.

Rob Collie (37:30): You're right. I did multiple maths last night, it turns out. So I sized the professional developer audience versus the knowledge worker audience versus the, what we've called the data gene audience.

Justin Mannhardt (37:45): Tell me more.

Rob Collie (37:47): We've talked about this before, that there's going to be more developers joining the fray as a result of things like Claude Code.

Justin Mannhardt (37:55): I agree.

Rob Collie (37:57): We've hemmed and hawed about what the ratio would be. I was saying, yeah, it's going to be an expansion, but I don't think it's going to be an expansion of the same size as what we got with the BI thing. The BI world was tiny. The number of people who could build analysis services models was like maybe double digits worldwide that were actually good at it. And then we ended up with a million. You don't get 80,000 X multipliers very often. So I went and researched what the ratios are. I backed into this because the things that are factually available are the number of developers versus the number of knowledge workers. That ratio is like 40 to one. 40 knowledge worker for every professional developer, like someone who actually deserves the capital D developer. So many different sources, the ratio can wander higher or lower. 40 to one is a good ratio.

(38:53): But then I have to use my empirical knowledge that's unique to my career path to the backend from the number of knowledge workers to the number of data geners, the tweeners. And we've talked about that on this show for years now, that that's about 16 to one. So what you end up with is a two developer, two professional developers, five data geners, 80 knowledge workers, two to five to 80 is the first place where the whole numbers work out. You can go one to 2.5 to 40.

Justin Mannhardt (39:31): Yeah, 2.5, 80.

Rob Collie (39:32): 2.5, 80. In terms of thinking about the expansion of the people in the world who can produce applications and platforms, the two people, the developers are going to become much more effective than they used to be. They're going to essentially become entire teams, and then the five are going to be about as capable as the developers used to be, but still faster. You wouldn't give them an entire massive sprawling architecture to build. Yeah. So two to five to 80.

Justin Mannhardt (40:01): Two to five to 80. I like that it's the math because there's this argument between is vibe coding a thing to not a thing? And there's this middle ground of the people that just understand how to work that way and get through, whether you want to call it the last 20% of the final minal of like, you can get something going really quick, but if you actually have the persistence to understand how things work and the curiosity to get something that's usable, that's a data gene crowd that's going to be able to figure that out. Everybody else is going to be like, "Oh, I one shot at a thing that's like this app, but now I don't know what to do with it.

Rob Collie (40:40): " I'm being forced in this current chapter that I'm writing to formulate in a written down crisp way for the first time my best current advice for business leaders As to how to think about this new era of software abundance that's coming and how to think about staffing. And so the two to five to 80 is part of that, is having them understand what that ratio looks like and then explain to them, honestly, what it is almost certain if this book makes it to a second edition, I will revise this section. I can't skip this. I have to give advice on how do you relate the two to the five? If you're a small company, you're going to have to just probably make do with one of the five.

Justin Mannhardt (41:23): Which is going to be great, by the way.

Rob Collie (41:25): Freaking amazing. But for more ambitious projects, how do the two and the five relate? And we have dynamic at our company, at P3. We now have both of those demographics at our company, and we are working out how they relate, How do they support one another? It feels good to be on the bleeding edge of figuring out a social dynamic that honestly, I don't think many people are working on right now. You're either in one camp or the other, there's a whole professional developer ecosystem, they're calling it agentic engineering now, right? Which is a great phrase, way better than vibe coding. Even the inventor of vibe coding, the phrase a month ago, the co-founder of OpenAI said, "It was just a throwaway tweet and really things have changed since then." It's gotten better since then as well. So not many people that I think are wrestling yet with this two to five to 80 thing. And so I'm like stepping into that void right now and it's exciting.

Justin Mannhardt (42:26): Well, I'd be curious to hear more about that or give a read when it's out on the shelves because I think you're right. And my conversations with leaders back this up. They're more or less thinking about their organizations as being mostly the same, same types of people, same types of roles, same types of processes, but like AI is now involved. So two to five to 80. I'm going to do some math and maybe I'll come back next week and we'll compare notes.

Rob Collie (42:54): All right. Yeah. Sounds good.