Everyone’s Priority. Nobody’s Job

Announcer (00:04): Welcome to Raw Data with Rob Collie. Real talk about AI and data for business impact. And now CEO and founder of P3 Adaptive, your host, Rob Collie.

Rob Collie (00:20): Welcome back, Justin. We had a little bit of a week off there for the 4th of July holiday, but we are now back in the groove. Sounds like you had a lot of fun.

Justin Mannhardt (00:29): We did. We've got some friends that have a place up on Lake Mille Lacs, which is maybe an hour and a half north of Minneapolis. Huge lake. Pontooning, jet skiing, seeing who could knock the kids off the inner tube the most. I think my wife won. At one point, my oldest son referred to me as being too soft.

Rob Collie (00:50): Really?

Justin Mannhardt (00:51): Driving the jet ski and so he got what was coming to him, naturally. Yeah, we had a great time.

Rob Collie (00:58): Excellent. My daughter was still in town. She was here with us for six weeks. It was her last weekend with us so we just kind of laid low and-

Justin Mannhardt (01:08): Nice.

Rob Collie (01:09): We'd been gone for 10 days and we got back and she'd been here the whole time. So it was a great time. We avoided all the fireworks. Not really fireworks chasers anymore.

Justin Mannhardt (01:19): It's always a chaos.

Rob Collie (01:21): So yeah, that's what we did. So earlier today I was talking to a reporter about a news story. As part of this book launch, we have a PR firm and the PR firm is connecting me with all kinds of article opportunities and interview opportunities and all that kind of stuff. And it's kind of been an interesting forcing function for bringing my attention to certain things. There's just so much news.

Justin Mannhardt (01:45): Too much news.

Rob Collie (01:46): Too much news. And so the PR firm connecting me with these opportunities has become almost like my curated reading list in a way.

Justin Mannhardt (01:54): Nice.

Rob Collie (01:55): It's like, oh, you're going to go talk to this reporter about this article. It's so wild, right? Because they're really good at pitching, but I haven't read the article yet. They're pitching topics on my behalf and then I find out, okay, you're going to go talk about X. It's kind of neat, right? It's Schrodinger's reading list. So today's interview was largely triggered by something you might have seen. This is something Satya Nadella said relatively recently, which is that everyone needs to be making their own LLMs. Did you see this?

Justin Mannhardt (02:25): I think I've heard a bit about this.

Rob Collie (02:28): His signature soundbite that he gave to this media outlet was there should be as many LLMs in the world as there are companies.

Justin Mannhardt (02:38): Is this the one where he said something to the effect of, "We can't let one or two players control this whole dilemma."?

Rob Collie (02:47): Yes.

Justin Mannhardt (02:48): I'm familiar with the general context of this.

Rob Collie (02:51): Now, it's very clear that this is a strategically aligned opinion for Microsoft to have with its own interests.

Justin Mannhardt (03:00): Of course.

Rob Collie (03:02): The more value the world gets out of open weight LLMs that cost very little is a classic economic strategy. You want the compliment good to your service to be as cheap as possible. Microsoft is very much going to be on the integration side. It's the place where all the business stuff comes together. And if the LLM is the most expensive part of the AI solution, there's less money available for Microsoft and also more money available to the LLM companies to do strategic things that slowly undermine Microsoft's core business, et cetera. This is a classic example of talking one's own book. That doesn't mean that he's wrong.

Justin Mannhardt (03:50): No.

Rob Collie (03:52): And the danger ... And this is a big theme of what I talked to the reporter about. The danger in this message is that we have this gigantic capabilities overhang. The world still doesn't begin to know what to do with AI. Everybody's still stuck on the starting line, which is why I wrote this whole book is to try to help people understand what it means to get started and how it's not like a layup, but how doable it is. And one of the key things I keep saying throughout the whole book is you don't need to modify the LLM. You don't need to be an AI researcher in order to do this. And so everyone's sitting poolside and Satya's talking about the deep end.

(04:42): Even if longterm we all do find ourselves customizing, fine-tuning, whatever, building custom versions of LLMs, that's a very potentially valid future state. I'm not opposed to it. It's gatekeeping in a way. It's making the problem harder because the skill that it takes to train, customize, blah, blah, blah is another level higher than what it takes to do the types of work which is super, super, super valuable right now, which is, we're going to use the air quotes, just using already available LLMs and plugging them into custom systems that give them access to custom information and context and training and additional tools within your business. Integrating them into a custom system in your business without even thinking about modifying the LLM. This is the first place we all need to go. It was kind of an interesting multifaceted conversation. I kind of wanted to get your reaction to that.

Justin Mannhardt (05:53): Is this turning into a written piece or is it like a video interview?

Rob Collie (05:57): It will be a written piece. You never know with these things.

Justin Mannhardt (06:01): They'll do whatever they do with it.

Rob Collie (06:02): Yeah. The PR firm keeps track of what the status is, et cetera. But in theory, there's an attempt to try to have a lot of these things land on the launch week of the book, which is August 11th. I don't know if this is one of the things that we're trying to sequence for that. So the reporter told me it's going to be a written piece. If it appears, it's going to be in Business Insider and it will be kind of quote heavy, I think.

Justin Mannhardt (06:30): You got my head spinning now, Rob. Let's see where this conversation goes. I do think Satya is right about something important, which is there is a real dilemma when providers of the LLM command so much of the value in the ... I'll use the term supply chain, I guess. I think even through my own experience using their products ... And by they I mean ChatGPT, OpenAI, Anthropic. And I could say this about even some of the things that I've built for myself or that I've watched other people build. You say this a lot throughout your book, which I've read. It's not just about the LLM. It's about the entire experience around it. I think we talked about this a little bit last week with Fable 5. I liked the idea that LLMs provide this sort of goodness to the world, balancing of the scales, fair game to play on the title of your book.

(07:26): People have access to knowledge and information, but not everybody has access to the same knowledge and information. How that all plays out in the future, I'm not sure. There's even rumors that Microsoft is trying to figure out how to replace LLMs in their own products with their own stuff just because of how much they spend when someone's using Claude in Excel or whatever it's called or they're using Copilot and they want to use Opus or Fable or whatever. Okay. How can we not just be spewing money out when our users are doing this?

Rob Collie (08:01): It's very interesting. The economics of these companies that are trying to go public right now, right? Now we're really just putting on the grand speculation hat, but they have spent so much money researching and developing and proving out what's possible. You look at the difference in pricing between some of these LLMs and it's crazy how much more expensive some of these are than others. And how much of that crazy cost is just covering their cost to serve, just covering their cogs versus paying off past investments, like trying to make themselves whole in a sense. The really interesting thing is that there is nothing secret about LLM research. It all started from a paper written by researchers at Google. Every time these companies go and prove something is possible, it makes the follower's life a lot cheaper and easier because who knows how many different things behind the scenes the researchers at Anthropic or OpenAI were trying out and discovering didn't work, but as soon as they find something that does work, someone else who's doing LLM research knows, "Well, I know where to point."

(09:20): So we're all better off if these things cost less. If what suits Microsoft's purposes, which is cheaper, more commoditized LLMs, if that comes to pass, then yay. But in the last time we talked, we also talked about the difference between LLMs at build time versus LLMs while the agent is running. There's a really good argument for maybe in a way the most capable LLM you need is the one you use with co-work because it needs to know the most. It's like general purpose knowledge work and you're creating and thinking and all these sorts of things. And it's like when I'm using something like that, I probably want the best and smartest buddy that I can find. Whereas when you get something really dialed in in an agentic workflow automation type of thing, it's very possible ...

(10:31): Again, depending upon the usage case, it's very possible that if you use that same like Opus whatever that you use to build it, if you use that as the runtime LLM, it might be tremendous overkill. That thing's carrying around everything. Does your agent need to know about the lineage of Roman emperors?

Justin Mannhardt (10:49): No.

Rob Collie (10:50): No.

Justin Mannhardt (10:53): Nobody is going to build a better general purpose AI assistant than what you would get in like a Cowork experience or Codex, Claude Code. That's just not going to happen. We've talked about this for what seems like a long time where we've had examples of people building AI things and the AI was like a really teeny, teeny tiny part of a final solution with a very specific job. And we've built some stuff like that. And we actually went through the task of running lots of tests with different levels of models because there's a clear rubric. You can kind of unit test this stuff and they all perform this task equally well. You don't need the big baddie. But if you're sort of in your general workspace, you want something that can do lots of different things and think about a lot of different subjects.

(11:45): And I was talking to someone we both know that works for a big consulting company and he was explaining how they're really cracking down on using models from the frontier shops. One, sure there's a cost concern. They have the capital to build their own infrastructure with their own open weight systems, but also why are we just pumping all this context directly over the wire here? That conversation started to make me think about the internet was really interesting because it changed how commerce could happen. AI has changed how software can get made, it's changed how software can work, it's changed how people can work, but it hasn't really changed how commerce happens yet. See kind of where I'm getting at? It's like this weird thing where a lot of the narrative is around, well, let's maintain the status quo, but we'll just have AI agents do everything instead of people.

(12:48): And I think people can improve their businesses and they can operate differently and they can do all those things, but it's not like ... The internet was weird in this regard because we went from the transacting for goods and services all of a sudden goes from this physical plane into this digital plane. And AI doesn't really do that in the economic equation. I don't think. I don't know if you've thought about this or if you'd even push back on that.

Rob Collie (13:11): I have not thought about it. In another article that I wrote for a news outlet, I think it was Chief Executive Magazine. And this one will be timed I think to come out with the book. This is one of the most obvious things one could say, but it's still I think good for people to hear it, that the change represented by AI is by far the biggest challenge to ever face business leaders in history. And there's lots of reasons for that. Now you're talking about the internet and it doing something to the world that AI so far hasn't done. The internet certainly was a huge challenge for businesses. No one worries now about how to internetify their business. This is settled science. AI very much not so. What would changing the way commerce works, what would that look like?

Justin Mannhardt (14:08): I have no flipping clue. If I did, maybe I'd paint the picture, but there's some backtracking on this narrative of there's a spectrum of truth to this still. Certain elements of knowledge work won't be necessary or valued because AI will cover that. Entire job functions either go away or change dramatically. But when the world has access to technology that can make things move so much faster, are we going to do more as a result? Is there going to be more output, more creativity, more value created for people? Yeah. It's like it hasn't created net new marketplaces like the internet did, I guess is kind of the thought I'm having on that.

Rob Collie (14:54): And I wonder if that's just the way it is or we just haven't figured it out yet. I remember when I first got to Microsoft, it was 1996. The worldwide web was still relatively new. And I remember one of my colleagues laughing about picking up a bag of M&Ms and seeing http: on the bag of M&Ms. It was literally funny that a bag of M&Ms would have a URL on it. And at the time, http://-

Justin Mannhardt (15:38): We didn't even get the S yet.

Rob Collie (15:39): I know. We didn't have the S yet. Was so techie to see that prefix. It was just showing up on M&M bags and like why does my M&M bag need to be internetified? It was kind of the joke. But that's what you did at the time. One thing we absolutely need to do is we need to get our URLs on everything. Did it change the world? Did anyone go to the M&M website based on looking at that? We didn't have QR codes yet. We didn't have phones to scan them with. There's a little bit of a parallel here where the first move was everyone slapped AI all over their existing out of the box products.

Justin Mannhardt (16:31): 100%. Yeah.

Rob Collie (16:33): Another thing that I actually thought was good about what Satya said was that he's drawing attention to the fact that you have to customize.

Justin Mannhardt (16:39): Agree.

Rob Collie (16:40): Whatever off the shelf it is. Whether it's from Anthropic, whether it's from OpenAI, whether it's from Microsoft. I think the big players have now internalized that the thing that we've ... It's kind of neat that we understood this. I'm sure that leaders at these companies did too, but we understood this before they were making public moves that confirmed it. That you have to customize AI to your business. You have to. No off the shelf solution is going to deliver these returns, these promised magical transformations. All of the hype isn't going to be found in out of the box AI offerings, off the shelf AI offerings, whatever they are and whoever they're from. Anthropic and OpenAI have spun up these big, or at least funded these big consulting arms.

Justin Mannhardt (17:35): Yes.

Rob Collie (17:37): Microsoft has recently announced their own. There's billions of dollars being ... Not large numbers of billions of dollars, but billions of dollars being thrown into these ventures that are acknowledging that companies need help and it is engineering work. But again, this focus on the deep end of the pool, maybe not yet.

Justin Mannhardt (18:01): It's always a very welcome and humbling reminder when I leave the comfort of my own desk and laptop and I go out to somebody's company and I sit with them and I talk with them. And you realize in a way, it's kind of my job to be out on the bleeding edge. I've got max plans to all of the big stuff I've got. I'm literally running like 5.6 Soul and Fable on these ambitious tasks right now while we're recording and somehow the bandwidth is still doing fine. Some people have ChatGPT and they use it or we've got Copilot and IT's working on a rollout plan. And it's not that they're doing anything wrong at all, but the world they're in is like every day they show up and they're trying to take care of their customers and their issues and their problems. And so I think the slowness by which this change will actually happen, it's hard to predict relative to the pace at which things keep changing, like you mentioned, the capability overhang.

Rob Collie (19:04): It's almost like perfectly no one's job AI-ifying a company. I said it in one of the articles. It's like, well, the org chart exists for a reason. That's because no one can see everything. But where's the pressure being focused on AI adoption? Well, it's all being focused at the top of the company. Okay. All right, great. That's where you start. That's where pressure should start, I guess. But how is the C-suite supposed to develop some sort of monolithic AI strategy when it's really many, many thousands of workflows at the company that make up the company and they're all potentially improvable? That's the opportunity. How many of those workflows do you know? This whole, it has to be done close to the business because you need to know the workflows. You need to be like up close and personal, like nitty-gritty with them. And those people who are up close and nitty-gritty with those workflows, that's not where the AI pressure is landing and it wouldn't make sense for it to land there either really, but it needs to be there. It needs to happen in that vicinity.

Justin Mannhardt (20:21): This is a refrain we've been to, but I love it every time. I think what I hear about a lot and see in conversations that I get to be a part of is that pressure from the top moves the AI problem. They pass that ball into a elite team. Maybe that's a technical team, an I team, a software team. And they go, "We're going to figure out what AI means for us." And a lot of times they come back and they're like, "Look, we built our company's chatbot. Oh look, we built our own internal ChatGPT." Honestly, I think that's maybe like that chat-based experience is marginally better than actual ChatGPT. Yeah, it has some more context, but Joan in accounting could also just hook up the connector for M365 and it could give her probably a similar experience. But when you get like, you were saying like the thousands of workflows ... I was having a meeting this week and we're like in this one very specific workflow in one specific department, we're talking about like, let's just focus on like 30% of the traffic that comes through this team and really small steps.

(21:31): And I'm running into this with my own workflow is you take for granted how much inferred context humans can apply to situations that AIs just cannot do. And so one of the things that companies I'm talking to right now are struggling with is like institutional knowledge is leaving the building with retiring workforce. Those processes or decision trees aren't ... They're not written down in a nice, neat, orderly way that an AI could follow. I've had AI at times misunderstand pretty severe why you and I know each other, but why would it know? But it'll infer all these assumptions. So that happens in your business workflow. And so if you really want it to help you, like you say in fair game, it is possible and it is necessary, but there's work there. There's a lot of work to be done to really make this all work for us and how many dollars is Anthropic and OpenAI going to rake in from all of us in the process? A lot.

Rob Collie (22:29): I'm doing my personal best to limit and restrict my usage of Fable. I'm not falling for that.

Justin Mannhardt (22:36): It is very good.

Rob Collie (22:38): Is it?

Justin Mannhardt (22:38): It is very good. I'm not nearly smart enough to understand all of the details about how hardware needs to change and all that kind of gobbledygook. And honestly, I feel a bit slighted by it in a way. Yeah, it's a complex thought process for me even. So like, okay, in a few days, so yeah, on Sunday, unless they extend it again, Fable 5 will be exclusively usage credits or billed by the token essentially. So it's like, oh man, that's a bummer because I want to be able to use the best intelligence for the workflows that need that there, but I don't really have a good way to predict that cost yet. We're trying to figure out, well, what's the framework where we use it for certain tasks and how do we coordinate all that and how do we make it work? But I wrote about this a little bit too. Made me think, Rob, let's say you and I had a vision for like a software product 20 years ago and neither of us could code.

(23:43): We would either A, have to learn how to code through many years of study and experience or we would have to get funding to hire a development team or we'd have to find a partner willing. So there's always been sort of this natural barrier to entry into doing something like creating software. And so that's kind of happening here, but the only barrier is financial. To use Fable 5 to create software, I don't need to be really, really good at writing code. I just need to have enough money to use Fable 5. For me, there was a bit of a hit to the knowledge and empowerment for everybody narrative that Anthropic has been pushing forward.

Rob Collie (24:25): I see that now. I see what you're saying. It's very much pay to play and pay a lot to play. So Fable is really good. Let me ask you a very specific question about that. I've had some enormous successes building custom applications with AI. Some of the successes were just regular software, like the playoff fantasy football website that I made for the company. It wasn't using any AI at runtime. And I've built some things that do use AI at runtime and they've worked really well. And I don't know if it was kind of like the lack of it being my primary focus or if it's just like it's both that it was lack of my primary focus and that I'm not a seasoned developer that came up as a developer before AI. But I have one application that I built that I kind of just got myself into this chasing ghost buggy state with the same level of effort that I put into the other applications. I couldn't get this one to just ... As a side project, I couldn't land it.

Justin Mannhardt (25:34): Yeah. The final mile was more intense kind of thing.

Rob Collie (25:36): Yeah. Going in, I wouldn't have necessarily guessed that this one would have been too terribly different from the others. And I think that all happened with Sonnet, the Sonnet four series, not even Sonnet five. So I'm wondering if you're encountering essentially that Fable is so good that it helps you stay out of those sorts of loops and backwaters. Is it filling that kind of gap? Because that would be very, very significant.

Justin Mannhardt (26:11): And part of this too honestly might be like a legit complaint about what I perceived as hallucination regression between Opus 4.6 to seven to eight. When I say hallucination, sometimes people still think of what we were talking about maybe like two years ago where it's like ChatGPT just like making stuff up. This is more in the category of poor judgment given context. One example is I was using Claude to prepare for a meeting, process some previous transcripts and some ideas I was having and just help me get organized. And it came right off the top rope when it was first response is like, "Hey, I noticed this person is going to be in this meeting. They seem like they're going to be really influential. You don't know them at all. That's important. Here's some advice." I've known this person for 20 years. And so it's just like the confidence to make all that up was interesting.

(27:12): What I noticed going from Opus 4.8 over to Fable 5, specifically building applications is, and it's more of a feeling like me having to have less, "No, no, no. Hold on. You didn't understand what I meant quite right. Back it up." I felt like I was more in sync with the model on the ideas plane and the intent plane, the relevancy of certain context. Another example I've talked about is, so we built this app and last month or so I did a pretty significant rewrite of some of the core features. And so this changed database schema and API routes and it just changed all this stuff and Opus 4.8. And I was putting this in cloud MD files and subdirectories in the code bases, like trying to get this to stop. Every time I'd be like, "Well, hold on. What are we going to do about our current users? We can't mess this up."

(28:13): We don't have current users. And it was like really, really concerned about it. And Fable was just like, "Oh, I got it. I can just like wholesale delete this and replace it with the new thing and we don't have to do a data migration or nothing." I don't think Fable 5 necessarily writes ... I assume maybe it writes better code, but the end application works. Like you were saying, I've built stuff that works well and I've done that with older models and I'm now doing that with newer models. But the experience of getting there was more pleasant and more enjoyable. What's that worth I guess is the question.

Rob Collie (28:47): Enjoyable, but as always, we've talked about how there's a point at which a quantitative difference becomes qualitative. Like you end up being able to do things that you wouldn't have otherwise. It's not just that it was more fun or that it was a little faster or a little easier. It's that it falls on the other side of the tipping point of it got done and it worked versus this one app I've got that I'm just like, "Maybe I should circle back and have Fable take a look at it and go, hey, let's get this really dialed in." But that would be an interesting test case for me.

Justin Mannhardt (29:28): If you do this stuff long enough, I think I could promise almost anyone if you've really jumped in Satya's deep into the pool to go back to that analogy, you will go from, "Oh my gosh, this is the most amazing thing to ever happen ever. Look how much superpower I have.", to like, "Oh, thank God I'm done with that project." And then you'll reflect and be like, but it's still amazing.

Rob Collie (29:51): Thanks for mentioning the book and I'm so grateful that you've read it. I want to remind our listeners that it is available for pre-order today. Fairgamebook.ai, and for one low price, you get all three formats. You get hardcover, you get the audiobook read by me and the ebook. All those release when the book goes live, August 11th. But even in the meantime, if you pre-order from that site, you get to start reading it the first few chapters immediately.

Justin Mannhardt (30:19): Yeah. If we want to stay on the promotional train for just a second, I got an email from you today, which I though was very interesting because we don't tend to email anymore. We text when we want to communicate. It was a promotional email from the book because I pre-ordered it and everything. I got the fifth chapter today, little pre-release bonus and an invitation to the Fair Game Insiders.

Rob Collie (30:44): That's right. The Insiders Program. Yep. Really mostly just an opportunity to talk to people who've been reading the book. It's honestly, even just emotionally for me to have put so much time into this thing and then put it out into the world, very few people have read the whole thing at this point. I've sent you a full advanced copy. Most people haven't gotten one of those. People who've pre-ordered from the site have now seen chapters one through four and now they're receiving chapter five and it's like speaking into the void.

Justin Mannhardt (31:15): I hear you.

Rob Collie (31:16): So I'm opening up a little bit of a community where people who've been reading the book, we can talk about it. The insiders are going to get a full advanced copy of the book because if we want them to recommend it to other people, they kind of need to have read the whole thing. So it's kind of one of the perks is that you get access to the whole thing and not just the teaser, but there's some other rewards in there if

(31:37): You convert even just one other person into pre-ordering it. We wouldn't be able to do this without AI. This whole website was built with AI and the community platform that we're using is a platform called Circle. Guess what? We would've never gotten that thing configured and populated if it hadn't been for Claude. We just cut it loose and said it was really hard to figure out this website like customizing. It's like customizing SharePoint or something, right? No, no, no, no, no, no, no, no. You're just going to drive the browser and go read the help, figure out these inscrutable settings and we're going to describe the end state that we want. It's kind of interesting that even this thing, the Insiders program, without AI, we wouldn't be able to do it.

Justin Mannhardt (32:22): It's cool. August 11th. That'll be here in a Jiffy.

Rob Collie (32:26): And we will do it again soon.

Justin Mannhardt (32:27): Sounds good, man. See you then.