Why Bigger AI Isn’t Always Better

Speaker 1 (00:04): Welcome to Raw Data with Rob Collie, real talk about AI and data for business impact. And now, CEO and founder of P3 Adaptive, your host, Rob Collie.

Rob Collie (00:18): All right. Well, welcome back, Justin. It's a very rare, very rare condition of being an episode ahead in recording.

Justin Mannhardt (00:28): Well, let's build a little list here. We're an episode ahead. You and I have recorded two weeks in a row. You're still not sick, fingers crossed.

Rob Collie (00:39): Correct.

Justin Mannhardt (00:40): Don't call it a comeback.

Rob Collie (00:43): Look, still 100% done. We've retired a couple of KPIs.

Justin Mannhardt (00:47): The way we evaluate success on the show is changing.

Rob Collie (00:50): Yeah. We need to change our KPIs. We've hit those goals. Time to set our ambitions a little higher. So, how have things been with you?

Justin Mannhardt (00:58): Today is the last day of school for the boys. April and I got to go enjoy the promotion ceremony this morning. We all stood outside in each class. They introduced everybody by name, which I think is a little overkill for first and third grade, if I'm being honest.

Rob Collie (01:14): Just a touch.

Justin Mannhardt (01:15): Just a touch, but it's fine. It was touching. They're really excited because their after-school program is having a pizza party, and then we are spending all weekend at the youth state lacrosse tournament. Rob, it's like six to seven hours at the fields Saturday and Sunday.

Rob Collie (01:34): Oh, that sounds great.

Justin Mannhardt (01:35): It sounds awful.

Rob Collie (01:38): So good. So good.

Justin Mannhardt (01:39): We're excited to watch them play, of course, but it's like big gaps where it doesn't make sense to drive back home.

Rob Collie (01:44): No, the logistics of these things are just really non-great for the parents.

(01:49): So we were talking backstage a little bit about something interesting to talk about today. It starts from a place that was just kind of mind-blowingly hilarious, a big deal at the Microsoft Build Conference, Satya Nadella unveiling this just absolutely beastly piece of hardware: 128 gigabytes of RAM, some absolutely massive GPU. The marketing around it is free tokens. You don't pay for tokens. Instead, you just pay for all the things that make tokens expensive yourself. You don't get to rent someone's GPUs anymore. And then the power, do you think this thing requires just a regular 110-volt plug? I want computers that have to plug into a dryer outlet.

Justin Mannhardt (02:37): Everybody's going to get 240 volts installed in their houses because they need one.

Rob Collie (02:42): Yeah. If you've got an electric car and your garage is wired for that, you can have one of these computers as well. You can plug in your car, or you can plug in your computer.

Justin Mannhardt (02:51): It's insane. Now, I didn't catch in any of the announcements, because I think right now, they're in the introduction marketing, get people excited, if they've clarified how they're going to deliver local AI.

Rob Collie (03:04): Yeah, so the models have to be local to run on this thing.

Justin Mannhardt (03:09): Correct. So I wasn't sure if they struck a deal with anything, or it's going to be their own thing, or just open source.

Rob Collie (03:16): There's multiple different models that they will run. Microsoft has their own LLM for the first time. I don't think you're going to get anything from Anthropic or OpenAI.

Justin Mannhardt (03:25): I wouldn't think so.

Rob Collie (03:26): They're not going to be letting their precious LLM weights out of the data centers. This is really an interesting thing. There's all these open-weight LLMs, which are really kind of interesting when you think about it. But then there are a handful of them that are, I think, licensable in a way. Apparently, there's a way to DRM an LLM so that you can have it on your computer. You can have it locally, but at the same time, it's not open weight.

Justin Mannhardt (03:56): Okay.

Rob Collie (03:57): Either way, though, if you have this box, you're not able to use the frontier models. The best and brightest LLMs on the planet aren't going to be available to you on this box that you paid... I don't even know what's the pricing going to be on such a thing.

Justin Mannhardt (03:57): I have a guess.

Rob Collie (04:19): It's a lot. It's apparently only meant to be like a dev box. It's also the exact opposite of shareable. If you build some sort of AI solution that helps your company work better, you don't want to be running that on a piece of physical hardware in someone's office. That's what the cloud was invented for. I mean, you can almost imagine reliving the entire arc of what we've learned about computing, like, "Oh, this is running really great on my desktop. I need to share this, so I need to find some data center where I can park this machine," which was what we called co-location. And then we realized, "Oh man, it was much better if we just had the software companies set up all the hardware, all the data centers, and run on their stuff." So when that announcement first hit, I'm having this ongoing book paranoia. So the book is locked. It has to be locked.

Justin Mannhardt (05:08): Yeah, it's frozen in the timeline at this point.

Rob Collie (05:10): We made some last-minute edits to the proof last night, and it's literally the proof right before it goes... Yeah, you know about printing, right?

Justin Mannhardt (05:20): Yeah.

Rob Collie (05:21): So the proof... I'm about to explain printing to you, of all people.

Justin Mannhardt (05:25): I used to physically assemble books as a job.

Rob Collie (05:30): Yeah. So we send in some edits on the proofs, very, very careful not to ripple the page count. With the proofs, you've got to make sure that if you're making an edit, it needs to be about a one-for-one swap in terms of characters. Because if it starts to flow into the next page, it might ripple, ripple, ripple, ripple, and now you're endangering the whole thing.

(05:50): But anyway, so this announcement comes out, and my first reaction is, I'm having this paranoia, this jumpy paranoia all the time now, "Should I go and add something about this in the book?" And I came to the conclusion, well, for multiple reasons, no. I don't think it really changes all that much. I mean, it's a very, very, very specialized thing for them to make such a big deal about. It does hint at other things which are relevant, which are you don't always need the best and brightest LLM. There's such a tendency to just overkill everything, and so I think that's an interesting thing for us to talk about. I don't know if you have any particular thoughts on this crazy hardware thing.

Justin Mannhardt (06:27): I might have too many. The cost of using AI systems is a really interesting story, in and of itself. So you and I, we probably pay Anthropic the $200 a month or whatever. Even though I think you and I are the type of people that are doing a lot of AI, or like people in our profession are doing a lot of AI, we're still probably closer to hobbyists than the people building things like Claude Code or the people just using AI to code at Microsoft by comparison. And that's where you're hearing these kinds of hype stories that I think are mostly clickbait of somebody using a bazillion dollars in token costs or whatever.

(07:07): So I think the pursuit of being able to localize the compute for certain types of tasks is part of this story. I've even thought about, "Well, should I be doing certain things on a really souped-up Mac mini or an Nvidia box," and then this thing from Microsoft, the Surface RTX, is sort of a descendant from the Nvidia tech in a way. "Or should I just keep paying the tokens?" It's like a rent-versus-buy-a-home type of thing.

(07:35): Then on the other hand, different types of models are capable of different types of things. I've done some experimentation with some of the open-weight stuff. I have a shit-ton of unified memory on my laptop, and I can run a model that I would probably put on par with maybe some of the earlier versions of ChatGPT. It's pretty good at summarization or brainstorming through an idea, maybe some complete, but it's not Opus 4.8, not even in the same ballpark.

(08:05): So that's why I was wondering like, "Well, what types of tasks is this box going to be doing, and what types of AI are going to be there?" I would suspect Microsoft has something up their sleeve here.

Rob Collie (08:15): Oh yeah.

Justin Mannhardt (08:15): Because my third thought on this, aside from which model makes sense for the right task, is there's something seemingly really important about the coding harness that is built around the model. Claude Code is built to work with Anthropic models. Codex is built to work with OpenAI models. You can use Claud Code with an open-weight model, and it's not going to be the same thing. So part of me wonders whatever open-weight or new Microsoft mini model that's going to come out, maybe it's optimized for GitHub Copilot or something, but I do think this pursuit of hardware, LLM, and then software around the LLM being kind of important for this to work.

(09:08): We did some math, a tiny bottle of like, "What kind of Mac minis would we need to buy to be fully local and like be able to use a model?" It's like an $8,000 box. This RTX thing's got to be in that neighborhood.

Rob Collie (09:21): The local hardware thing, there's just something really, really wild about it. There's two dimensions of this problem, and it's really difficult to keep them separate. One is literally don't overkill your LLM usage. What do I do with my use of the productivity stuff? With Claude Cowork, Claude Code, I don't even think twice about it like, "Oh, there's a 4.8 out? Well, that's what I'm going to be using."

Justin Mannhardt (09:48): Yeah. That's the default? Perfect.

Rob Collie (09:50): 4.7 wasn't even like the latest for very long.

Justin Mannhardt (09:54): 4.7 was kind of dumb.

Rob Collie (09:56): Was it? Okay.

Justin Mannhardt (09:57): It needed to go.

Rob Collie (09:58): Did it? All right. Well, I'm not really sure that I noticed, but that's the thing, right? When you're already in overkill zone, you don't know how far down you could go while still getting what you need.

Justin Mannhardt (10:17): I agree with this.

Rob Collie (10:18): So I think this is going to be an art and a discipline that we all need to develop, and it's different for our personal productivity use than it is for company-wide agents. Company-wide agent has the opportunity to burn a lot more tokens because there's a lot more users of it, which is another thing that kind of puzzled me a bit about this box, the Surface RTX. Because it's being positioned for developers, are developers the place where most of the token costs are generated? For an organization, it would seem like the deployed runtime solutions are the place where most of the tokens are going to get chewed. Maybe not today, right? Because today, people are going bananas with Claude Cowork and bananas with Claude Code, and maybe that's what the future looks like more than in terms of token costs. I don't know.

(11:07): But I've been assuming for a long time, I think by default, that when an organization becomes AI-ified, which no one's even close to, a truly AI-ified enterprise or even an AI-ified mid-market company that's truly gone completely through the looking glass, we haven't seen that yet. What will most of their token costs look like? Will it be from the custom agents that are built for and that are running lots of operations and assisting people at that company, very, very, very bespoke things, or will most of the token cost be chewed up still by essentially personal productivity?

Justin Mannhardt (11:52): This would be a fun chart. There might be a similar chart on this topic from Anthropic from a while ago. And if I recall it correctly, coding was a dominant use of AI.

Rob Collie (12:03): Yes.

Justin Mannhardt (12:04): It makes sense, right? If I'm a software engineer and I'm building an app, I've got a code base, the context of that code base itself is going to require a lot of context, a lot of tokens. Versus me like, "Hey, what's on my calendar today?" is a fairly cheap ask. This is maybe a simple binary example. The average user, and I think this will be this way forever, the average everyday user of an AI tool like Claude.ai or just regular ChatGPT, they're never going to pause and think, "Which model should I use for my requests?" like you were just saying. "I'm asking about what's on my calendar, so I should just down this to Haiku because it's a basic." They're never going to think about that.

Rob Collie (12:47): And why would they, right?

Justin Mannhardt (12:48): Right. Why would they? Yeah. So I think that's on the labs to figure out, "Well, how do we figure out how to route base..." But I think there's versions of that feature set in Opus 4.8, and you can tell it gets it wrong where it underthinks or overthinks a problem based on my prompt, essentially.

(13:06): The other side of this is if you're building something like an internal company agent or AI or an app that uses AI in different ways, you absolutely have to think about what models get used for what types of things.

Rob Collie (13:21): Yeah. And even there, there's the human tendency to say, "Nah, just use the biggest, baddest. Send the full attack." But there is at least an opportunity in that situation to make deliberate decisions, and is an opportunity to test. That's another, I think, difference is that the personal productivity stuff is also one-off in many ways that would you even have the chance to baseline a lot of it to get the sense of? Even if you were being really, really responsible, you're like, "Okay, I'm going to run 10 instances of this task on 4.8. All right, let me go down to Sonnet 4.6." Then you could do an apples-to-apples comparison of its performance, and maybe you would notice that there isn't any difference.

(14:11): And it's so funny that it wasn't that long ago that Sonnet 4.6 was the apex predator, and so it's weird to be looking in the rearview mirror at these things and thinking of them as somewhat degraded or cheap or less capable when they had their moment of blowing our minds. And for what percentage of my day-to-day work would Sonnet 4.6 still be blowing my mind? Probably a very reasonable percentage, but I'm not going to find out.

Justin Mannhardt (14:44): We did a mini version of that on what we've been building, just talked about last week. We came to the conclusion Haiku and Sonnet actually cover most of this really well. And the cost to serve is then really tolerable, and we can see the economics surviving. There wasn't much value in inserting Opus into that equation.

(15:03): We, when we're building the damn thing, get a ton of value from something like Opus. But when that's the default in Claude AI, like, "Yeah, we're just going to use it." And then you get the people on LinkedIn, they're like, "Did you know when you ask Claude what's on your calendar, you waste an entire bottle of water?" and da, da, da." It's like, okay, all right. I don't think your crowd is even using Claude. They're just out.

Rob Collie (15:27): Just circle back to the wild nature of this Surface RTX. It's opposite what we expect, right?

Justin Mannhardt (15:34): That's why I think there's a missing piece in this story.

Rob Collie (15:38): Yeah, there has to be.

Justin Mannhardt (15:39): To fill this gap, for them to advertise it as local AI compute, all this fancy tech, target market is the hyper AI developer persona, there has to be some sort of Microsoft or a partnership with a new coding harness for a new local model or something. Because if their thing is like, "Here, go run whatever you want from Llama on this box, and this box is great," I just... Ah.

Rob Collie (16:12): So you immediately knew where I was going. Let's spell it out for the listeners though, just so that we're not doing the telepathy thing.

Justin Mannhardt (16:23): We just had an entire episode in our brains.

Rob Collie (16:27): Yeah, we did, in our brains. They say audio is a limited format, but when you don't even speak the words, it's really limited. So you had just said that at runtime, the solution y'all have been building runs great on a mixture of Haiku and Sonnet, which are down-level models compared to Opus.

Justin Mannhardt (16:47): Correct.

Rob Collie (16:48): But at development time, you're just like all-in on Opus, right? Because it's just so much better, and it's a leveraged investment. It's kind of like a one-time leverage investment, and it's going to write you better code, it's going to help you head off problems better, all kinds of things. It is worth its weight in gold.

Justin Mannhardt (17:04): Absolutely.

Rob Collie (17:05): So then that runs into like, "Hey, here's this developer-focused box." Maybe it runs the Sonnet class. I don't think so. I don't think it's even quite up to that.

Justin Mannhardt (17:16): We actually went through this whole evaluation of open-weight models, and there's only a couple that can just almost touch the frontier models, not the same, but they're close. It was something like, "Wow, we really recommend having 256 gigs of unified memory for this." It's like a Power BI dataset. You need to be able to fit the compressed thing into memory.

Rob Collie (17:42): Yeah. I mean, the reason we chose GPUs for AI is because they need to run so many damn calculations in parallel, and that means there's a lot of data. When they talk about the weights of the LLM, the on-disk footprint of the LLM is like a terabyte. It might not be that much, but I forget. It's in the book. I researched it for the book. Just loading the thing's brain is massive.

Justin Mannhardt (18:09): You don't want to compress them any further because you're going to lose performance, and I think it's even worse than with data processing systems. If that thing starts spilling over to disk at all or spilling over to CPU at all, the show's over.

Rob Collie (18:25): There's clearly something there. We're not going to be those guys that sit here and go, "Well, this is really dumb," because clearly, it's not dumb. There's this whole notion of fine-tuning. I believe that it's going to be an area of investment and focus. It'll be one of the things that's talked about more and more and done more and more going forward.

(18:46): And fine-tuning is really interesting, and it actually forced me to add an appendix C to my book. When you're done with the book, you can't put things in the middle of the book for the same problem. It starts to ripple, and now you got to revalidate everything, so late-breaking stuff goes in the appendices. So one of the principles I cover throughout the book is that you don't modify the LLM, you just give it access to information and data, and instructions, and all that kind of stuff. And that's a really, really cool principle that people can wrap their heads around.

(19:17): But then along comes fine-tuning and invalidates a little bit of that. You actually can modify part of the LLM by training it more specifically on your stuff. And I could start to see maybe if this local hardware machine from hell would be useful in that scenario, like lots and lots and lots of iteration on fine-tuning. Then maybe a fine-tuned down-level model would perform adequately, or even exceptionally, on tasks that today, that same down-level model wouldn't succeed on. The frontier model would succeed. The down-level model wouldn't. But if you fine-tune the down-level model, maybe it does. I don't know. And maybe that fine-tuning is super, super expensive if you're doing it in the cloud. I have no idea.

Justin Mannhardt (20:12): Yeah, I'm not sure either. This was getting some traction maybe a year or so ago, and I think it just got drowned out in the noise. The concept is you can have a very small, very fast LLM that is essentially hyper-focused on, let's say, coding in C++. It doesn't know about humanities. It doesn't know about the Cretaceous period. It just knows C++.

Rob Collie (20:36): Doesn't know Julius Caesar, right?

Justin Mannhardt (20:39): Right. I can see applications like that, but then this is where this goes back to my point about the coding harness being built for the LLM it typically runs on. We still have to do things like plan mode. If the LLM isn't a very good planner but it's an excellent coder, you're back in this situation, right?

Rob Collie (20:58): And the planner, again, it's overkill learning the entirety of human history as a means of helping me plan my application. But it does result in something that is an incredibly nuanced and capable conversationalist, that picks up on all kinds of stuff. I give instructions to these things all the time that are just super casual and flippant, but it's just because it knows the rest of the conversation that it's like, "Okay, I know what he means." Again, it doesn't need to know about Julius Caesar to do that, but where's the line? How much can you shrink it?

Justin Mannhardt (21:36): It's interesting that it was also kind of positioned for the developer. The run agents locally thing, I think the OpenClaw thing, that's lost some steam, at least in the hype train. I think people are still out there trying those sorts of things, but that's not how they positioned it, right? They didn't position it for your personal AI assistant or anything like that.

(21:57): I'm curious your thoughts on this. I do think there's certainly the opportunity that there's a new device category. I mean, OpenAI has this where different LLM-type things are running locally on the edge, and maybe there's some compute benefits to certain tasks like that. But to be honest, this one has me a bit puzzled. I've run various size of open-weight models locally. I don't need that, so I want to meet the person that is spending $100,000 a month on tokens that needs this, and I want to sign up for your class.

Rob Collie (22:31): Right, so the person who's spending $100,000 a month on tokens who needs this and that this thing will work for them. The Venn diagram is very interesting. You and I both queued this up ahead of time, almost hoping that the other one was going to solve the mystery for us. Hey, this is the transparency of this podcast. We've talked about this for a long time. We use this podcast as a means of figuring things out, as much as we use it as a means of sharing things we have figured out.

Justin Mannhardt (23:02): So take Microsoft out of the equation for a second. I do see what I'm doing and what our company is doing as needing a local box with a coding LLM that's not Claude Code being a potential need state that we would get to, but a few other things need to happen. Anthropic would need to realize this Mac subscriber thing doesn't work for us. It's all usage-based now. Basically, we get forced into a buy-or-rent economic decision, and say, "We're going to buy our house. We're not going to rent anymore." There's a bunch of other assumptions in there, also.

(23:40): But then why does Microsoft come out with this type of box if they're also not coming with something special for on the software end, something that is just spicy in GitHub Copilot or like one of their own mini LLMs or something like that?

Rob Collie (23:58): As part of this, it's not the only LLM that will run on this Surface RTX, but they did announce Microsoft's first own LLM.

Justin Mannhardt (24:07): Is it Phi, or is this different than Phi?

Rob Collie (24:09): I got the impression that it was different, but that doesn't mean that I'm right. It was something homegrown, I thought.

Justin Mannhardt (24:14): Well, then that kind of ticks all the boxes, because then they're like, "Here you go."

Rob Collie (24:18): It would give them the opportunity to start, as you said, the coding harness. If they own an LLM, and this was something you and I talked about years ago, just how strange it was for Microsoft to be completely sitting out the LLM development and research game. The fact that they had this partnership with OpenAI gave them this interesting first-mover status at one point in time, but might have also lulled them to sleep a little bit in the way that Google didn't do that. Google went and started developing their own very aggressively, in fact, even using their own custom hardware. And we'll never know behind the scenes whether Microsoft regrets and is reconsidering that. If they've gone from 0 to 1 Microsoft LLMs, that's a bigger difference than from 1 to 50.

Justin Mannhardt (25:10): Yeah. I've had a couple conversations about this recently, how you can look at Microsoft and their traditional peers. Just for the purpose of this, let's say that's Amazon, Google, Meta, you could throw Apple in there, I guess. Out of that cohort, arguably Google was the only one that was ahead on the LLM level game. They had DeepMind and all this stuff going back many, many, many, many, many years, and they have this interesting position of having a frontier LLM, mobile devices, browser, all the right ingredients. Meta is seemingly flopping on their own LLMs. AWS is just happy to host whatever you want. They've had some spurts. And then there's Microsoft. They're just like, "Yeah, but we got Office," so our field position is going to be just fine for whatever we want to do with that.

(26:07): And then you've got Anthropic and OpenAI coming. How you own someone's full domain, either as an individual in their personal life, the tools they use or as a business, you and I know this, right? Excel is still running the world. That hasn't changed.

Rob Collie (26:23): As a sign of the absurdity of all of this, I realized the other day that I now have three different options within fingertip reach of using LLMs to modify Office Docs. I have Cloud Cowork that just modifies the file. That's its interface. So you basically need to close the file and tell Cowork to go make some changes to it, and it's very happy to do so.

(26:52): But there's also now a Claude button on my ribbon in Office. It's its own add-on installed Copilot competitor, and that one doesn't use the file format because you have the file open. That one's using the API to modify the document, just like Copilot is. And, of course, I also have Office Copilot sitting there that I could also use. It's just too much. It's too much to choose between. I did a little bit of a head-to-head between the Claude add-on and Claude Cowork for the same task. It was a bad test because the thing that it was failing the most on was understanding the assignment.

Justin Mannhardt (27:34): Sure.

Rob Collie (27:35): It failed to understand the assignment equally poorly both places. I do think I got a better result, though, out of Cowork, even still, a marginally better result than out of the Claude add-on. The ability to just chew through and generate the file format, I think LLMs still really prefer that way of working if you can give it to them.

(27:57): Anyway, that's just a throwaway thought that I now have three different assistants that are willing to help me make a slide deck.

Justin Mannhardt (28:03): Yeah. No, I have very similar experiences doing things and using Claude Design quite a bit lately.

Rob Collie (28:10): My daughter has been using Claude Design.

Justin Mannhardt (28:11): Yeah. So I'd be like, "Oh, now I have this HTML artifact, but it's a presentation. So, do I care to get it into PowerPoint? No, I probably don't anymore." It would be a fun retro if we could get our future selves to come back from 10 years in the future and tell us what we're doing. Can you shortcut us here a bit?

Rob Collie (28:31): And explain this Surface RTX box to us.

Justin Mannhardt (28:34): Yeah, did it work out? Should I get one?

Rob Collie (28:36): What is up their sleeve? What is the endgame for this? Yeah.

Justin Mannhardt (28:42): I would say if you are really interested in building things that inherently have LLMs in them, you need to be paying a bit more attention to the differences between different models and why you would use them. I don't think it applies to the general consumer, and it's not hard to go explore the open-weight stuff. You might not be able to run a big model, but you can go understand what it's like to get one on your computer, and hook it up, and do some stuff with it. Don't Mac mini it up and go buy a hundred of them. OpenClaw boosted Apple's stock score a bit.

(29:19): I don't know what the over/under, Rob, on one of us getting an RTX in the next 12 months, but boy, I'm worried about one of us.

Rob Collie (29:27): It's not going to be me.

Justin Mannhardt (29:28): Yeah, I know.

Rob Collie (29:28): I think that's pretty clear. I mean, who knows? Never say never, but I'm not finding any itch.

(29:40): All right. Well, we will catch you next week then, won't we?

Justin Mannhardt (29:43): Sounds good, man. We'll see you then.