AI and the Art of Building Backwards

Announcer (00:04): Welcome to Raw Data with Rob Collie, real talk about AI and data for business impact. And now, CEO and founder of P3 Adaptive, your host, Rob Collie.

Rob Collie (00:20): Hello again, Justin. How are you today?

Justin Mannhardt (00:22): I'm doing very well, Rob. And how are you?

Rob Collie (00:25): I'm doing well too. Thanks for asking. I have a topic for us today.

Justin Mannhardt (00:29): Okay.

Rob Collie (00:30): In a couple of weeks, I've been invited to go over to Microsoft and speak with some members of the SharePoint team who are very much part of the forward deployed engineer type effort at Microsoft with their customers in the AI space. We deliberately don't have an agenda. I'm over there for an hour. I'm not being asked to prepare anything. If I was asked to prepare something, I wouldn't be quite sure what I would. It's going to be a relatively small room, I think seven or eight people. Some of them have flown in for this, not to see me. They've flown in for a general team building week long summit.

Justin Mannhardt (01:13): You're the agendaless part of a larger agenda.

Rob Collie (01:16): I'm one part of it. Yes. And two of the people in the room, the manager of the team as well as one of the people on the team are longtime friends of mine. He might even know one of them. The setup for it is, "Hey, Rob and his company are out there doing AI. Let's have a bit of a blast of reality from outside the bubble." They're also doing AI. They've got all kinds of things in flight with their customers. It's very interesting and I think completely unsurprising that they are doing the forward deployed engineer thing, and they certainly have plenty of engineers.

Justin Mannhardt (01:56): Just deploy them more forwardly.

Rob Collie (01:58): That's right. 100% fits our observations of the world that successful AI is almost always going to involve custom software as opposed to out of the box. It's one of the most unknown, unknown type of conversations. What common ground are we going to have? What is it that I know and have seen that will be novel and interesting to them versus things that they completely already know and vice versa? They're also buying copies of my book for everyone on the team, and part of the event is me signing them. I haven't taken a marker to an event in 10 years.

Justin Mannhardt (02:47): Is there a special book signing marker?

Rob Collie (02:49): There was a special book signing marker that was in my backpack. I think it's still in my backpack. I have no idea if it still has ink in it, this thing has just been sitting there. If the thing gets pulled out, it's going to be like, "Whoa, what's going on? I wasn't expected to be used again." One of my questions is how much will the people in that room but also the people that they work with every day at Microsoft, how much will they learn from my book?

(03:23): The things that I know about AI happen because of two things. Number one, immense necessity. We're in a business at P3 that is simultaneously expected to know what to do about AI, also in a place where if we stuck with our existing business model, AI was a threat to it. And also a place where if we do it right, AI is a huge opportunity for us. And this is my company. I couldn't possibly have more emotional pressure to solve a problem, to come to an understanding than what I did. And it still wasn't easy, even getting off the starting line despite those conditions.

Justin Mannhardt (04:12): I recall, yeah.

Rob Collie (04:13): And then I had the time to go on that journey and come back and report on it, and that's when I ended up writing the book. I don't think any of those things would be true if I were working at Microsoft today. If I were doing my old job at Microsoft, I wouldn't have nearly the pressure to go and do the thing. I'd still be under pressure, right? There would be plenty of it but it wouldn't be quite as existential as being responsible for both my own financial future in the form of this company and also 50 other families. But I also probably wouldn't have the time. I would've really needed a lot of enthusiasm for it, which I didn't have. I didn't have in the beginning. I don't typically have enthusiasm for adopting things which is what has made, first of all, me being the person who was most enthusiastic about Power BI in the world for a long time was really unusual for me. And then to, at this company, be one of the people most initially excited about it and fired up about it, also unusual.

(05:22): Would I have done that at Microsoft? I don't know. In terms of AI, Microsoft is... They've got great field position. Everyone's business already runs on all of their systems, or at least crucial parts of everyone's business run on their systems. And LLMs are increasingly a commodity and they're so easy to swap, it's just a dropdown. It's the easiest piece of technology to swap out ever. It's wild that's true. You would say, given that, you look at the long-term timeline of this and go, "Game over, Microsoft wins."

(06:05): At the same time, let's just talk about Anthropic because they're the people who seem to have understood this game the best the soonest. They're tunneling in from the outside. They're doing everything they can to worm their way into the work environment that already exists, that is largely a Microsoft work environment but they're constantly offering us incentives to turn on the Slack extension and anything that lets us get deeper and deeper into your ecosystem, we want that. Tunneling in from the outside versus sitting inside the castle going, "Okay, why is this even a competition? Why is Anthropic even in the game?" It's not because they own the LLM. Their LLM is an API call. It's because they're building better software at the moment. And Microsoft is a software company. I think one of the hypotheses that I'm forming is that if you work in Anthropic, it's guaranteed that you understand everything that's in my book. Everything that's in my book as a placeholder for, do you get AI?

Justin Mannhardt (07:19): It might not all be as effective at communicating it but they would get it. Yes.

Rob Collie (07:26): But if they read it, they'd be leafing through it going, yep, yep, yep. And maybe even in certain cases going like saying, "Well, he's not quite right about that. He'd be-"

Justin Mannhardt (07:35): Yeah, exactly.

Rob Collie (07:36): "...gatekeeping a little bit."

Justin Mannhardt (07:37): Cute little diagram but it's true.

Rob Collie (07:38): Yeah. Right. That would be the reaction. And so I wish that they were handing these books out in advance of me being there. And this is weird but making it homework that they all read it before I show up because that's my biggest personal curiosity is how far along are they as a culture? Because they can build software. They can build the hell out of software.

Justin Mannhardt (08:08): How much of it do you think is the inertia of the machine? I think we would generally agree that AI fundamentally changes the way software gets made into the future. How much of this is just this is a massive machine to adapt the way it works inside the 15 walls of Microsoft? It's more than four.

Rob Collie (08:32): Their size, their politics, their structure, their incentives, everything are always a problem. Your past success and your past revenue streams oftentimes even prevent you from doing the new thing until it's too late. Of course, their size and their inertia, that goes hand in hand with field position. It's the flip side of field position. They're not really separable but... I haven't even seen this movie but I've certainly heard the line. It's Bane talking to Batman saying, "You merely adopted the dark. I was born in it." Or something like that. No matter how big of an organization Anthropic is, and they're already big enough to have their own big company problems, their own big company inertia, their own paralysis and all that kind of stuff, they're welcome to the party, but they were born in AI.

(09:26): In the Substack that I posted this morning, I went back in time and marveled in hindsight at how visionary Claude Desktop was. Did you even know that OpenAI had a desktop version of the OpenAI app for Windows and Mac before Claude Desktop existed?

Justin Mannhardt (09:48): I did not.

Rob Collie (09:49): Me either. I started to write that Claude Desktop was the only thing like that and then I'm like, "This is one of those claims that can be fact checked. Let me go fact check it before I write this." It turns out, no, OpenAI had a desktop app. Maybe they still do. They definitely do. We definitely know that they have now because they've been following in the footsteps of what Anthropic has been doing. But the point is OpenAI's version of the chat app was just a richer chat app. Claude Desktop was explicitly built to get close to your work environment with its ability to absorb MCP servers, extensions and things like that. Suddenly, the LLM could have access to your file system if you wanted to give it to it. When I wanted to connect it up to Notion, a cloud service, there was an MCP server built by Notion that I could use but it turned out there was one written by the community that worked better, at least at the time.

(10:49): I remember when Brian Julius was... He was always using Cloud Desktop when I was working with him and he was getting me started, giving me that first push.

Justin Mannhardt (10:58): Yeah.

Rob Collie (10:58): Which is wild. Okay, why is he using this as opposed to using claude.ai? Anthropic has known for a very long time that the game is all about getting close to the work surface. OpenAI didn't know that.

Justin Mannhardt (11:14): Not as quickly.

Rob Collie (11:15): No. No. And the reason they know now is because of Anthropic's success. Anthropic is the first one of these companies to profitability and it's because they've wormed their way into the business environment far better than their competitors. Microsoft is a software company. You don't need to be an LLM company to build the kinds of software that Anthropic has been building. It is their software that has gotten them these footholds. It's not their LLMs. It's not because we all just have to have Opus.

Justin Mannhardt (11:51): And Opus 5 was bad. I just like... Bad.

Rob Collie (11:55): You always have these opinions about these things and I'm just like, "Yeah, it's just another entry in the Pantheon to me. I don't even..." I use these things extensively but I don't really catalog the differences between them. Just like the LLM starts fresh every session, I feel like I do too.

Justin Mannhardt (12:09): Yeah.

Rob Collie (12:09): I don't remember how it performed yesterday. Yeah, it's just for tracing this alternate version of me through the system. How much would I be... Because again, I was a upper 25th percentile, 75th plus percentile, maybe even 90th percentile software engineer at Microsoft, not a developer but software engineering as part of the teams. But if I didn't deeply get AI, how effective would I be?

Justin Mannhardt (12:41): This is timely. I was talking about this yesterday. You yourself went on this journey where people in positions of leadership or influence, or use whatever term you want to use there, the question we were asking is how much of your opinion about AI is based on your own experience with it as opposed to things you're reading or hearing about from other people? You went on this journey of hearing things about AI, being concerned about things, being interested in things, and then going down the wormhole, coming out this other side. And so yeah, I'd be really curious what the feedback you get from your meeting of where the people on this team are at.

(13:26): It is interesting to me about the quality of the software and the field position Microsoft has because I'm a high user of AI and I actually want to and feel the most engaged with my work at this point when I am spending most of my time in the desktop app, whether that is ChatGPT Work or Claude Cowork or Codex or Claude Code, Claude Desktop, whichever one. And I know Microsoft, they had a big announcement today about a bunch of updates to the Copilot ecosystem. I haven't absorbed everything. But what we need in terms of the baseline I think is essentially a solved problem.

(14:12): It'll be interesting to see, okay, who can win the software game and what can the Frontier Labs do that say Microsoft couldn't? Is there anything there? I don't know. Aside from terms and conditions like you can't use our model and creating some sort of closed barriers there. One of my favorite things about... And I think both Claude and ChatGPT can do the same thing, is they have the in-app browser essentially. If I'm working on a document, it's just there in the app with me. I'm not going off to my other browser. Okay, that's done. Let's go on to the next thing. Oh, we're working on ads. Great. Let's open up the ads portal in that platform. Oh great, what's the next thing? Oh, we're working on the podcast? Just...

Rob Collie (15:01): I'm still stuck in the Chrome extension because as far as I can tell, I can't get my Apple password manager to cooperate with the in-Claude browser.

Justin Mannhardt (15:12): Really?

Rob Collie (15:13): Maybe I'm just the usual all thumbs. Having moved to a Mac and having the passwords that are stored on my phone also now available, having unified password management through all of that just so effortlessly, oh, so good. But my first attempt at getting the built-in browser integrated was asking me questions I didn't understand. I'm just like, "Nope, I'm out."

Justin Mannhardt (15:36): Yeah. I do use a third party password utility and maybe that's why. But my point there is like, okay, if the future of work is primarily me with my AI agent or agents... I don't really like the pluralistic idea anymore. It's just my agent can kind of do everything. Why do I need to think of it as five separate things? Me with my agent and then whatever I'm working on, the things I'm working on are the things I would work on with the software Microsoft provides me today, why can't they win that game?

Rob Collie (16:11): Yeah. Why can't I be in a Copilot desktop and loving it? There isn't any reason.

Justin Mannhardt (16:19): I've heard some Microsoft software is essentially just layers of layers of iteration over the years. I think at one point, you might have even told me this, that the rendering tooling in Outlook to render an email was based on Word.

Rob Collie (16:35): Originally, they called it Wordmail. It was-

Justin Mannhardt (16:38): Wordmail?

Rob Collie (16:38): I swear to God. It was a feature. But then we moved to HTML and I have no idea if it was Word's version of HTML editing or if it was... I lost track of the story at that point.

Justin Mannhardt (16:51): Aside from the AI agent software and the LLMs themselves, I think the software that wins in the future is always going to be the software that plays the best in this AI agent type ecosystem.

Rob Collie (17:06): There's a very Slack versus Teams type of thing going on here too. If you get to choose as an individual, Slack is more fun. But if you're a big organization, you're very likely to choose Teams. Was it a Super Bowl ad that... I think it was where it was for Zoom, where IT had forced them all to use some really difficult to use meeting software?

Justin Mannhardt (17:31): I don't recall.

Rob Collie (17:33): There's no two ways about what it was, they were talking about Teams. It was like the classic Apple break the tyranny where there's this rebellion, people standing up in their cubes saying, "I want to use Zoom." And someone else saying, "I do too." And it was intended to foment those kinds of rebellions.

(18:00): In this article that I just wrote, published on my Substack, when I started writing it, it was titled Anthropic the Software Company because that is the thing that's differentiated them more so than their LLMs. It eventually ended up being, "Is our AI bill going to multiply by 5 or 6X?" And there's still very much elements of the Anthropic as a software company in the post. Robcollie.substack.com, by the way.

(18:30): Yeah. If they're a software company, and Microsoft is a software company, and the software company Anthropic is trying to get into the Microsoft ecosystem from the outside and Microsoft owns it from the inside out, it's like how Microsoft owned the best BI engine on the planet in analysis services and they owned the place that everybody worked on data, which was Excel, and yet Tableau still came in and was a monster but not forever.

Justin Mannhardt (19:04): There's been plenty of examples. Power BI is a great one where Microsoft has been able to just be patient, realize what something is and what it needs to be, and then they stand up a gazillion dollar R&D team and boom.

Rob Collie (19:19): And Power BI is one of the rare examples where it's not like the Teams versus Slack thing. Power BI is just better than Tableau. Certainly, there are things about it that aren't. But end-to-end, it is a better product. Whereas Teams is really just more enterprise corporate friendly than Slack but I do not think that Teams is a better product than Slack.

Justin Mannhardt (19:44): You said this is the SharePoint team?

Rob Collie (19:47): Yes, it's parts of the SharePoint team. Yeah.

Justin Mannhardt (19:50): This is all speculation. You just think about people that are on EAs with Microsoft and they've got M365, E whatever, and they've got Azure footprints, and now everybody wants all that information to be discoverable and contextualized and available for all these different AI use cases. And just doing the work that we do and that P3 does, we wouldn't necessarily always describe this as hard to do but it's not trivial to figure out this kind of stuff. Can you just imagine SharePoint repositories with just conflicting documents galore and all... I saw something that I agree with strongly the other day which, listen, every company is hearing they need to fix their data problem before they can go do AI. Every company has got data problems. And 10 years from now, every company is going to have data problems. It's a call to the Fawcett's First type methodology, right?

Rob Collie (20:56): If you're a giant or traditional consulting firm, you want to be talking exactly like that. You want to be saying, "Yeah, you got to go get your estate in order." Because that's an endless problem. They know it's an endless problem. It is an excuse to work forever. And how could they possibly pass that up given that working forever is what they want to do more than anything? We were joking backstage last time, we'll say it this way for the actual production podcast, we're updating the VTO for our company, the vision traction organizer in the EOS system that we run here.

(21:34): And the Fawcett's First methodology, working backwards from the answer, but also just the way that we build, and I'm sure this is the way that you guys build as well, there's a proven process section of VTO which we 100% filled this in before. We're just updating it now. Our proven process is a hybrid of Fawcett's First. Fawcett's First is the philosophy. But then there's the way that we actually go about delivering it. What we're calling it is we build in plain sight. There's this constant react, correct, and improve dynamic with the client, high frequency touchpoints. Not only is it Fawcett's First but a fraction of the working product is there from the beginning, something tangible to react to. You can never react to plumbing. How does a business stakeholder look at plumbing and go anything other than like, "Mm-hmm."

Justin Mannhardt (22:32): Well, they look back at you and go, "Rob, this works. Right?" And you go, "Yeah, of course it works."

Rob Collie (22:36): Yeah, totally. And you have no choice but to say things are on track, right?

Justin Mannhardt (22:40): Right.

Rob Collie (22:41): They essentially build in the dark for a long period of time and then you finally get the chance to react and it's going to have drifted tremendously. Correct is now the order of the game and fighting over who pays for it is the next big challenge. Whose fault is it that it needs to be corrected? There's never a chance to improve because you're going to just fight over trying to have it deliver on what you vaguely expected. Whereas in the way that you worked and Kellan worked when you worked here, the way that I'm sure that you continue to work today, when you get to see intermediate versions of the working thing over and over and over again and watch it evolve in real time, first of all, you have the ability to react as a stakeholder, as the client in a meaningful way.

(23:32): You can correct things right then before layer upon layer upon layer is built upon the wrong assumption, which makes it expensive to correct, almost effortlessly like Neo in the Matrix stopping the bullets with his hands and just dropping them to the floor, just fix it. Fine, no big deal. Right? Chicka, chicka, chicka on the keyboard and...

Justin Mannhardt (23:56): "Hey, Claude, go fix this."

Rob Collie (24:00): But then also the opportunity for the stakeholder to say, "Hey, you know what? Even my original picture of what we should do here wasn't as good as it could be." Now that I've seen we're in it, it can be even better. Oftentimes, even that improvement, it would be considered scope creep in the old model, in the traditional model. But sometimes these are just opportunistic things that can be picked up, low-hanging fruit. Yeah, why not? Let's just grab it. It's just as easy as the correction. If it's truly like a scope expansion, of course, we have to have a conversation. But those incremental low-hanging fruit opportunities being picked up over and over and over again throughout the course of a project results in something completely wildly different. And believe it or not, it was that damn VTO template insisting that this thing be there and be named and I have a diagram and I actually had to go and do it.

Justin Mannhardt (24:57): Oh, you drew it even? Look at you.

Rob Collie (24:58): Oh yeah. Well, with AI assistance, of course. Right?

Justin Mannhardt (25:01): Of course.

Rob Collie (25:02): And it's pretty freaking cool. I'm really grateful having to be forced through this. When I first saw I was like, "Good Lord, no." But it's actually very, very powerful to have this artifact now and be able to talk about it much more clearly with prospects and put it on our website when we update it here shortly.

Justin Mannhardt (25:25): It's like some factor are more important, I think, with AI. Of course, you can describe, "Oh, we could take these documents and we could throw them through AI and it would do this to them, and then we'd have this output." But for a stakeholder to actually see how that would work in their workflow and be able to say, "No, that's not right or it got this wrong." Or, "That makes the next step more difficult for me." Or, "That does make it easier for me." Or whatever the case might be. And I don't know if you've shared this opinion. I'm learning this because I think there's stages of belief when I see someone demoing so I believe it's possible, and then I see someone build me maybe the thing we might bring to the first checkpoint in one of these cycles. Okay, I believe. But then with AI, what's the right volume of work to throw through this workflow so we can be like, "Yeah, this is good and it's done."

(26:25): Because usually my experience, Rob, the things that we're spending time on, is we're usually dealing with a messy process, heterogeneous documents or information. It's not something we can run clean unit tests on so that's why I think it's even more important to have these tight cycles and a good body of work where we feel like, okay, it didn't just do it a dozen times, it did it 100 times.

Rob Collie (26:50): Yeah. Closer to the business was important to BI but most people didn't do it that way anyway. It's a weird parallel in the same way that having a good semantic model was important to BI but most people didn't do it that way anyway. You could cheat. I don't think you can cheat in AI. If you're going to be dealing with structured data, you need a semantic model. And you really need to be close to the business, it's an extra order of magnitude important. But of course, the big consulting shops are going to want to say things like, "Well, you got to get your data estate and you got to get all the blah, blah, blah, blah, blah." And again, they want to sell you plumbing that has no truly measurable or evaluable outcome. That's their favorite thing.

Justin Mannhardt (27:39): It's like little check boxes. Well, we built this data widget and we built that data wadget and we built this pipe and that pipe and this porthole and...

Rob Collie (27:47): Oh, but it's all platforms. The way you're describing it makes it sound these are small things. No, these are giant monolithic...

Justin Mannhardt (27:55): And now we're working on Interop scalability. Let me see if I can glaze you over, bill more hours.

Rob Collie (28:07): I think we're going to call them Byzantine Consulting.

Justin Mannhardt (28:11): Byzantine.

Rob Collie (28:14): So good in so many ways. It paints exactly the right picture, both old and complicated. As a closing note, that was one of the things that when I first worked on the Windows installer MSI technology for Microsoft, InstallShield went and wrote this gigantic book. I still have a copy of it, wrote this gigantic book on the installer on MSI, how to use InstallShield to produce these MSIs and wrote a gigantic book on it. And I read one of the book reviews, one of the early book reviews. And it said, "You know what the first takeaway one gets from this book is the Byzantine complexity of Microsoft's new installer." And I just chuckled. I chuckled so hard. And again, I'm going to defend that product. It's Byzantine because it had to be. The installation of software on Windows platforms is we met it where it was. We didn't pretend that it was something else.

(29:12): All right. Well, I have to go. Good short conversation. I'll definitely want to circle back with you and tell you how the journey across the bridge to Redmond goes.

Justin Mannhardt (29:21): Very intrigued to hear the report.

Rob Collie (29:24): Oh, I'm going to be giddy going over there. Again, just the uncertainty of walking into that room and having no idea how this is about to go is going to be the opposite of anxiety inducing. Like going to a thriller of a movie, it's also funny at the same time.

Justin Mannhardt (29:37): Yeah. I look forward to the call on the way home.

Rob Collie (29:39): Take care for now, man.

Justin Mannhardt (29:40): See you.