What Happens After the AI Works?

Speaker 1 (00:04): Welcome to Raw Data with Rob Collie, real talk about AI and data for business impact. And now CEO and founder of P3 Adaptive, your host Rob Collie.

Rob Collie (00:20): Welcome, Justin, to the Rob No Longer Sick After Three Months podcast.

Justin Mannhardt (00:24): Wouldn't that be a great cover story? Is like the reason I've not been on an episode recently is because I refuse to listen to the Rob is still sick counter and had some sort of civil protest.

Rob Collie (00:35): Well, you and me both. I think I've done maybe one without you, maybe. I don't know. I don't even remember.

Justin Mannhardt (00:42): My sense of time is very distorted today just because of the last several weeks and I know for you too.

Rob Collie (00:48): Yeah, I got better from being sick just in time for my wife Jocelyn to need to go into the hospital for emergency surgery. We woke up 7:00 AM one morning, she's like, "My tummy hurts." And by 2:00 PM she's in the OR. Oh, her here's an unintentional joke. That really put a kink in our plans. Her intestine was twisted like a balloon animal.

Justin Mannhardt (01:07): Oh, man, those are bad.

Rob Collie (01:09): Yeah.

Justin Mannhardt (01:09): She's doing okay though and on the up and up?

Rob Collie (01:12): Oh, yeah. It is amazing. Medicine is still a very, very, very primitive discipline in many locations. Many places on the map of medicine are like, "Yeah, we're still figuring out. We have no idea what we're doing." Surgery is not one of them. Surgery is impossible. The things that they do, after they explain what they did to fix this problem, I almost wanted to say, "Bullshit. You did not." It's insane. This condition, the thing that happened to her was before 1950 was like 100% fatal. And now it's like, we just cut the twist out. We literally cut the twist out and splice the ends. What?

Justin Mannhardt (01:54): All of a sudden, one day back many years ago, someone just said, "Well, what if we just open people up and fix it?"

Rob Collie (02:01): Do you ever watch the show The Knick?

Justin Mannhardt (02:03): No.

Rob Collie (02:04): With Clive Owen. It's not historically accurate, but it is somewhat historically grounded. It shows you when it was the frontier and they're just like, "Well, let's try it this other way. The last 17 patients have died when they had this problem, so let's try this new technique. Oh, that's 18 dead." A lot of people died so that new techniques could be tried and discovered and we benefit from them today.

(02:34): Back to the Dark Ages part, no one in the medical establishment ever figured out what was wrong with me, like why I had essentially the flu for three months.

Justin Mannhardt (02:41): And they're just like, "Well, you're better now, so nobody cares."

Rob Collie (02:45): Yeah, see? Exactly, right? They're all in the back room high-fiving.

Justin Mannhardt (02:51): [inaudible 00:02:51] some sort of homeopathic regimen and he's fine.

Rob Collie (02:56): I think it's just as likely that I started putting butter in my coffee again that got me better. I have no idea.

Justin Mannhardt (03:01): Human body's a weird thing, man. It just is.

Rob Collie (03:04): Very, very, very strange. But yeah. So on the day that we're recording, we have put the pre-sales website for the book, my book. It is live. And yours truly is in Fortune Magazine today with an article about the migration of developers that I think we're going to see away from big tech. They're not going to need as many, but the rest of us are going to have more of them. We call it the developer migration to the suburbs. And apparently that was worthy of being picked up by Fortune and Yahoo Finance. And we've also changed the subtitle within the last couple of-

Justin Mannhardt (03:41): Oh, no.

Rob Collie (03:41): Last 48 hours, at the last minute. So the book is Fair Game, subtitle Customizing AI to Your Business is Easier Than You Think. And the website is fairgamebook.ai. That's a real thing.

Justin Mannhardt (03:56): Looks great, dude.

Rob Collie (03:57): Thanks.

Justin Mannhardt (03:58): Looks great.

Rob Collie (03:59): That is a Jocelyn project and a Chris Riley project. Chris Riley is our second full stack dev. Two of them working together in a tight iterating loop. It's kind of neat. You have such a relationship with your publisher like I have with Bill Jelen. So we can do things like pre-order bundle that gets you all three formats, audio, ebook, and hardcover for one price that Amazon will extract quite a bit more for.

(04:28): And the data nerds out there will love this. When you publish through an independent publisher like we do, like all of my books have been, the top seller lists, they're wise to the fact that someone who's publishing through a publisher like that might game the system. So I could go to Amazon for instance and order 5,000 copies of my book in the first week. And you think about it, like $75,000 depending upon ... Maybe it's $100,000. It might be worth it to a company like ours to spend that money and fake it just for the publicity, right? I'm sure they've been burned by this multiple times.

(05:08): So any pre-orders of my book that go through Amazon won't count towards bestseller lists because of the fact that we're using an independent publisher. So we're incentivized to drive people to this other site, give them a better deal. And then all of the orders through this website will go through independent bookstores, not through Amazon. So independent bookstores will win and the independent bookstores thing will turn those legitimate sales into things that actually count towards the bestseller status because they know that if I was going to game it, I would just go to Amazon and buy them. It costs more at an independent bookstore.

Justin Mannhardt (05:53): Very exciting. I've seen several sections in the process from you that I don't know if they made or didn't make the final to print in all cases.

Rob Collie (06:03): They probably did.

Justin Mannhardt (06:04): And I'm going to go order mine now.

Rob Collie (06:07): Well, I appreciate it. So how's your life been?

Justin Mannhardt (06:09): It's summer in Minnesota. I have two young children and they decided to play lacrosse. Most of my life is spent around navigating the lacrosse practice and game schedule, which has been pretty doable for me because of the amount of training I've had during hockey season, which is basically the same thing, only cold and inside.

Rob Collie (06:30): And probably worse, right?

Justin Mannhardt (06:32): And worse.

Rob Collie (06:32): In terms of what times of day and things like that. I would think that the hockey thing was even more grueling on parents than lacrosse.

Justin Mannhardt (06:39): No, we're having a great summer. Honestly, I think things have just been absolutely moving at crazy warp speed on my end. It's like hard at this point to feel like, did I do that yesterday or last week or last month? I mean, just so many exciting things happening for us ahead of schedule, been building a lot.

Rob Collie (06:59): What kind of stuff? Are you willing to talk about it or what-

Justin Mannhardt (07:01): Yeah, absolutely. Oh, I'd love to. I was building stuff when I was still working with the P3 team. I'm always building right now. It's like a very different mental mode. Some fun projects that I sprinkled in with the primary project that we've been working on is a system that we're calling Pathfinder. And the fun parts about it is it involves a lot of AI in it. Something I think you and I agreed is like a really helpful use of AI is to have AI interview and ask people questions and tease out ideas.

(07:37): But our main goal with what we're doing is how can we create systems, tools, resources, services that help people figure out where should they focus in their business for AI? Based on that, what should they build? How should they build it? How do they get it rolled out? And then how do they make sure that's working? This is a very simple explanation, but we're trying to cover this whole gamut.

(07:57): So the Pathfinder product at its core today essentially serves the beginning of that process. Where should I focus? And then based on that, what should I consider doing about it? So the really fun part in there, Rob, has been the actual implementation of this designed fleet of AIs that all have really important jobs. And the stuff that's required to make it a reliable product experience is really, really interesting.

(08:25): So let's say, for example, I wanted to create an AI agent that would interview you about what you wanted for dinner. And as it's asking you questions, you give it maybe some ideas and you answer questions and then part of the instruction is to have the AI extract some metadata from what you said and go put it in a database somewhere. But let's say I just let one AI handle this whole thing. It is unlikely to want to stop talking with you. It will be happy to keep talking with you about what you want for dinner until you're ready to leave the chat.

Rob Collie (08:57): Yeah, it'll exhaust you.

Justin Mannhardt (08:59): It'll just go and go and go. So just sort of the interesting puzzle of having this process we want to go through, controlling for hallucination is really an interesting problem. So there's separate AIs that look at what the main AI thinks it's going to say back to you and say, "No, no, no, no, no. Rob already answered that question, so let's try again."

(09:22): So it's been really fun and we've had the opportunity to go run through this with a couple clients that were gracious enough to be guinea pigs with us. You know this so much better than I from being a product manager for so many years, like when you just have to tie your hands behind your back and just watch. And the terror that goes through, because we can test the hell out of things or build simulated tests, but until you put someone in front of it, that's a different deal.

(09:47): And so far, all the feedback's been really amazing and we talk a lot about how the first 80% of building anything with AI, it's like amazing and fun and fast and you're like, "Oh my gosh." But that final 20% or whatever percent is just the slog of, "Claude, what are you doing bro?" Finding those things.

(10:08): But that project has been really, really, really fun and we're going to ship it pretty soon. It's actually going to have the consumer level product that just anybody could buy and use, a team product, and then we have our kind of main B2B consulting package that this will ride alongside with too.

Rob Collie (10:27): Very cool.

Justin Mannhardt (10:29): Yeah.

Rob Collie (10:30): So that multi-agent thing, I talk about it a little bit in the last chapter of the book and I sort of build up to it along the way as well. The instructions that you give to an LLM, it's like the actor in the scene. You say, "Hey, okay, what's my motivation in this scene?" So you've got to give them that. And the more you try to give it multiple motivations, the less likely it's going to be good in any of them.

Justin Mannhardt (11:04): I agree, yeah.

Rob Collie (11:05): And it's a little bit of a context window problem. As the chat goes along, the dilution of the chat itself versus its different motivations, if you gave it multiple missions and it's like system prompt or what I call its handbook in the book, just to make it easier for people to absorb, it's kind of wild, right? Long before the context window starts to get even fractionally full, you see that giving these things multiple parallel missions or missions that are in tension with each other, it's a little risky.

(11:49): And the same way that it's a little risky to do that with a human being. But a really, really, really good human being, the 99th percentile, could handle these sorts of instructions, could live with that. And these LLMs by and large feel like most of the time they feel like 99th percentile human beings, right? So it's a little weird when you start to see them not behave that way.

(12:15): But so taking the same LLM, the exact same LLM. So now I've got two instances of it and this one I've instructed to do this and this one I'm instructed to do that like as a critic or as a watchdog or as a referee of sorts for the other conversation, suddenly it's very effective. This multi-agent system is very effective and you would originally think that you could probably have crammed it all into one.

Justin Mannhardt (12:41): I'll have to admit, I initially would've thought I could do so. So my very first prototype of this that I built, that's one of the things, like example I gave of just like this thing just asking you about what you want for dinner ad nauseum, it'll never stop. You get into those modes, it just wants to keep digging deeper and deeper and deeper and these LLMs can seem so amazingly capable and brilliant, but then just have the poorest forms of judgment at certain moments in terms of like what to do next. I don't know if it's always the context window or if it's just that main LLM thinking like, "Yeah, I'm going to maybe bend the rules a little." I mean, you see this in Claude Code sometimes where it's like, "Oh, I've got enough. I'm just going to go ahead and kind of keep going."

(13:31): But to hand-off what's happened in the main conversation to a different AI, a different context, different LLM and say, "Hey, here's a rubric and a test that's been completed. Does it pass the test?" And doing those sorts of things exceeded my expectations of the reliability you could have in that type of AI-guided experience with, because you can imagine the types of people we're trying to hit, leaders, ICs, all kinds of industries, all kinds of roles, to be able to have this work start to finish has been a very fun adventure and we're really excited about it.

(14:09): We've learned a lot about how we work in the process of building. It's one thing to learn how to work effectively with AI to build stuff. But then to work with AI and also work with other humans who are also working with AI on the same thing, that's an interesting adventure.

Rob Collie (14:34): All of the context that you develop in your own workspace. Now parts of it, you're like, "Oh, shit. Parts of this need to be shared." And there's this inheritance structure that you find yourself craving, what constitutes facts.

(14:57): So in the book I call handbooks or the instructions, what's my motivation? It's not an original word, but I use backgrounder to describe anything else that's sort of like certified as fact but is textual. Even just my editor project, like the editor for my book, there's so many different background or files in this project now that it doesn't need them all the time.

(15:29): But when it does, there's a background around there that's like, "Okay, here are all the different versions of my bio," certified versions of my bio. This one's for media, it's long-form media, this is short-form media. There's like many different versions of it. These versions of my bio, they need to be shared with other people because like I've got a PR firm that's out there representing me and they're sharing a bio that's not ... It's like the Good Morning Vietnam is not ... They're reading unofficial news. They need the official news and how to kind of federate these, you can almost call it like settled science, right? Okay, these are the bios. How to federate those so that the rest of our company, but also like across company boundaries.

Justin Mannhardt (16:16): It's a similar and different dirty data problem. This is the way I've described it. And sometimes it's like a technical issue. For example, we ended up with like five different implementations of how we send emails out of our app. And then it's like, "Whoa, whoa, whoa, okay." There's some basic stuff of like we want these primitives and so you fix that. And other times it's just contextual in text and Claude especially will get really excited about discrepancies. So there might be, let's say I'm prompting with a document that I wrote about some requirements, but then there's maybe an older document that's still in the repository that doesn't agree and it'll be like, "Oh my gosh, are we left or right?" And you're like, "Okay."

Rob Collie (17:05): It sounds an alarm like a klaxon.

Justin Mannhardt (17:08): Yeah.

Rob Collie (17:09): Sometimes you're happy for it and some other times you're like, "Oh, good Lord, that old file that's still lingering there, go delete that shit."

Justin Mannhardt (17:16): Yeah, exactly.

Rob Collie (17:17): We don't need that anymore.

Justin Mannhardt (17:18): This is happening in companies too. I talked to someone earlier this week where they're rolling out these quoting tools and sort of this self-service mindset. Well, that means, okay, Rob's writing his claud.md file and I'm writing mine. And then we start clashing over both material and not material details, creating some productivity loss. And this was like an analytics context that has nothing to do with are the metrics coming out right. Now there's just like arguing definitions of how you should interpret things, right?

Rob Collie (17:51): It's another one-version-of-the-truth problem, but with things like what I'm calling backgrounders. Your hard drive is a very convenient place to put this stuff. And that's like where Claude Code and Claude Cowork have defaulted. But it is a really, really, really create tomorrow's absolute morass type of situation, right?

Justin Mannhardt (18:17): Yeah.

Rob Collie (18:17): Fast forward six months and there's going to have to be versions of these tools, like even the off-the-shelf tools like Cowork and Claude Code that are like more than happy to plug into some other storage system. Again, it's the one-version-of-the-truth problem from BI, but now it's applied to policy and who are we? And it's everything. It's everything about a business now has a one-version-of-the-truth problem. Now, that's the negative way to look at it. The positive way to look at it is that we can get shit organized and actually on the same page in a way that you've never done before.

Justin Mannhardt (18:55): And I think there's a couple, call it a blueprint or a playbook of like, okay, if I could go back in time three months and give myself some lessons to help with these sort of things, because AI makes it really easy to make stuff, documents, apps, decks. If it's digital, it makes it really easy to make it. That means we have a surplus of shit that doesn't matter in the end of it because we have all these iterations. And we've run into even simple issues where there'll be some sort of decision document that we'll go through to be like, "This is what we want to do." But then there's another document that governed the build of the next step and it's like, "Wait, but we have this thing."

(19:35): So dialing the process by which we manage the documentation that multiple AI assisted workflows look at is really important. I think for the user maybe more so than the AI, because what I experience is the user is just frustration just trying to correct Claude on what it's looking at more so than Claude's ability to go actually build the thing I want.

Rob Collie (20:03): At the same time, you're like one or two really smart people on the frontier of this stuff. And at an organizational level, this is just a nightmare.

Justin Mannhardt (20:16): Yeah, it really is. I was sharing a story with someone I was talking to today. There was an article I saw, I think it was some sort of program manager at Amazon who their job and mission was to essentially keep track of what everybody's doing with AI.

Rob Collie (20:31): Okay.

Justin Mannhardt (20:32): Right?

Rob Collie (20:34): Just an inventory.

Justin Mannhardt (20:35): Yeah. There's detailed stories in there essentially of like, "Hey, this team is creating a solution that creates all these detailed outputs of their work, but then this other team is creating AI solution to summarize those detailed outputs into something else. And we have documents that are getting created, but we don't know based on what information and we don't know where they're being stored." And basically it's like, we don't know. Rob is building the same thing as Justin, complete maximum chaos. So yeah, it's like if two people can struggle, take that to 10, to 50, 100, 1,000. It's an interesting opportunity for how to fill that need.

Rob Collie (21:12): Yeah. There's some sort of new storage tech that I think we're going to see, I don't really know exactly what it looks like yet, that's going to be optimized for these sorts of problems. I'm not smart enough in my spare time to just go, "Oh, this is what it looks like." Someone's thinking about it. Everyone's trying to, right? So every content management system in the world is trying to be this now. So SharePoint has introduced support for markdown files and things like that because you know SharePoint document libraries are a fine place to start-

Justin Mannhardt (21:46): Right.

Rob Collie (21:46): ... for this kind of stuff. And every corporate Wiki provider is also saying, "Hey, now we're the place to store all of this kind of ..." No, you're not. So it's a fascinating problem and I do think if you could fast-forward, you will see, in sort of the same way many, many, many years ago, the only storage for data was SQL, structured SQL databases. And this whole notion of things that we now call data lakes didn't exist. And people who saw this problem coming, that we needed the ability to store what one guy at Microsoft called curly data. The answer that people came up with was to add XML storage to SQL server.

Justin Mannhardt (22:36): Right.

Rob Collie (22:37): And I mean, there were a lot of big brains at Microsoft, big genius brains behind this effort and it was completely a dead end because the real answer was something completely different, just regular file storage, storing these files. It turned out that XML wasn't even the thing, it was JSON that ended up winning that game.

(22:58): And I think we're in kind of a similar moment where the industries recognize that there's a need for some sort of new kind of structured storage that holds things like this, like these markdown files, in a way that's inheritable so there's a hierarchy to it, but also there's different security roles and blah, blah, blah, blah, blah and all that. And I think we haven't yet realized we need data lakes, Hadoop and all that stuff. We're in that, I don't know, 2007 zone.

Justin Mannhardt (23:28): I think you're nailing it here. Obviously it's easy enough, like let's say you and I, we wanted to work something together. It's easy enough for both of us to sign up for something like GitHub and we can push and check out our code and share that way, but that's just that project. It doesn't care about what's on your computer or my computer unless we're syncing it. We already have lots of situations where the hierarchy point is spot on because we have dependencies that apply to all of our projects, like our brand, all of these things need to inherit down. Those are some examples of things you find yourself constantly trying to get them all in sync. So a hierarchy of knowledge is needed.

(24:11): The other wrinkle to this now is, and I think this was maybe kind of an attempt to deal with this, is like the memory features in these tools that'll perceive that the user gave some sort of feedback so it'll write a memory. I've turned it off in some cases because it'll write memories that conflict with my main document set because of the way I said something. So managing the context is kind of the whole ballgame.

Rob Collie (24:38): Yeah. You said something earlier, which is actually the topic of an article that I was writing when this meeting came up and I had to go record the podcast. So you said that it's a new form of data quality. 100%. That's the thesis of this article I'm writing is that the way that we used to understand data quality was factually incorrect information in a database, a sloppily kept inventory process, or you have a product in your product catalog, it's assigned to the wrong category, but who cares? So for example, if you have like a toothpaste miscategorized as a haircare product, it doesn't matter. When the toothpaste gets delivered to the store, it still goes in the toothpaste aisle. It doesn't interrupt the flow of money, people buy it. But when you start doing BI, this is something we saw over and over again, right? BI became the place where all of the sins came home to roost.

Justin Mannhardt (25:42): Build a dashboard, find all the problems.

Rob Collie (25:44): Right.

Justin Mannhardt (25:45): That was the issue.

Rob Collie (25:47): All the sins that you were getting away with in your organization, as soon as you started trying to get accurate with measurement, suddenly those sins mattered. And now you're going in, you're talking to people who don't have any interest at all in your BI project and trying to get them to cooperate, to fix the data quality problem. And we know that the world is not done with that and that is still going to be a problem for AI. Agents consuming incorrect data are going to make bad decisions just as frequently as human beings. Maybe they'll catch some of them that humans wouldn't, but they'll also miss things that humans would have caught. Let's just call it a wash.

Justin Mannhardt (26:27): Here's the other, third thing maybe for your article. They also might go to unnecessary means to try and resolve the issue on their own.

Rob Collie (26:34): Right, yeah. Yeah. Let me help you burn a lot of tokens and really mess things up for you. So that's still very much here. I don't think we've had a chance to kind of like victory lap a little bit on this podcast. And it's not as, maybe not quite as sweet to you anymore because you're not necessarily so deep in the Power BI universe as you used to be. But I did share with you the chapter of the book that talks about this.

(27:04): For so many years, we tried to explain to people why semantic models were valuable. Why does a Power BI semantic model necessary? If you came from like the Tableau world, your response to this was always something along the lines of, "Oh, just shut up, nerds."

Justin Mannhardt (27:24): Right. Right.

Rob Collie (27:27): Until you've lived the difference, you didn't care. Oh, a slightly new question evolves. Someone's got a different question than they had yesterday. We just send our BI developers in the Tableau world without a semantic model, we send them to go write a bunch of SQL. They're re-implementing the same semantic formulas that they were last week, but in a slightly different flavor and like it's page after page after page. And maybe they make mistakes, maybe they don't, but there's all this extra work because a slightly different version of the same question requires all this extra work and the semantic model didn't do that. The semantic model was prepared to answer that slightly different version of the question. Even that simplistic explanation I just gave was often enough to make people's eyes glaze over.

Justin Mannhardt (28:12): Boring

Rob Collie (28:12): Nerds.

Justin Mannhardt (28:14): Nerds.

Rob Collie (28:15): Right? And as we've seen in many Power BI implementations, it's just a straight up like, "No, we don't care about semantic models either." We see people with just like one wide franken-table of a model or a bunch of tables that aren't connected to each other. They're not leveraging the number one thing that makes Power BI better.

(28:32): But AI has changed all of this. Suddenly, all of these companies that have been neglecting semantic models have looked at AI and gone, "Oh." When an AI agent is the one asking the slightly new question, there isn't time to go wait the two weeks for the dashboard developer to go write all the new SQL. And if you let the AI agent go and write all of the new SQL itself, it's going to get it wrong. Not only is it going to get it wrong, it's going to get it differently wrong each time. Today, it's going to come up with one answer, tomorrow's [inaudible 00:29:10]. None of the answers are going to be correct, but they're not even going to be consistent day to day.

(29:14): So guess what? Tableau and others have banded together on something called the Open Semantic Initiative. I forget what the I, it's OSI.

Justin Mannhardt (29:25): The Open Semantic Interchange.

Rob Collie (29:28): Interchange. There you go. That's what OSI stands for. Of all the semantic models in the world that are deployed, like greater than 99% of them are Power BI semantic models. Power BI is the only game in town in terms of like footprint of semantic models that exist in the world today. So when you're that far behind, you can't just come up with your own.

(29:48): The thing is, Tableau had their own semantic model, they just never invested in it or cared about it. Even their old semantic model, they can't make that their thing because they're so far behind in deployed footprint. So the only move to get customers to care is to say, "But ours is open." That's one way they can try to outflank Microsoft with its huge lead in this space. And it's a good move banding together. But oh, there's some joy in seeing blog posts on the Tableau website that say things like the agentic future demands an open semantic model.

(30:25): Now, they had to throw open in there because that's the competitive angle, but it demands the semantic model. The semantic model thing is now, what's the Zoolander thing, so hot right now? Semantic model's so hot right now. And this is a new flavor of data quality.

Justin Mannhardt (30:39): Yes.

Rob Collie (30:40): But it goes on into the things that we've been talking about. Data quality now applies to essentially textual definitions of our identity or our policies or things we've learned the hard way and don't want to have to repeat, institutional knowledge in all of its forms. So the thrust of the article is that all the data quality problems that used to plague BI, still relevant in the AI world, but there are many dimensions of data quality that AI cares about, that BI didn't.

(31:18): Even the semantic model, which really was a huge thing for BI, you could get away with not having one. And there's so many things you can get away with that in the same way that BI used to be the place where the buck stopped, AI is now going to be that times 100.

Justin Mannhardt (31:37): Yeah. I was talking with someone this morning actually about, they wanted to pick my brain about data architecture to serve AI and we got on the topic of models and things like that. And I said, "The issue is that if the AI sees a mess of tables," I use this analogy. Let's say I as a human who can write SQL saw a mess of tables, I'd be like, "Oh my God, I can't." We have to talk about the design of this data model so that I can live.

(32:07): The AI don't care. It's like, "Oh, I guess we got to write 12 CTEs and make up some new window functions and we just going to do it." To your point you just made, it's going to get it wrong and it's going to be happy to try.

Rob Collie (32:18): A different wrong every time.

Justin Mannhardt (32:20): Yeah.

Rob Collie (32:20): And in the process, very expensive because it's going to burn a lot of tokens doing it. It's not going to be fast either. It's going to be doing a coding project in the background in response to your question.

Justin Mannhardt (32:33): I caught Claude Code today. I could tell it was just like kept trying to do ... It was like doing a simple task for me. So I stopped it, I was like, "Claude, what you doing?" "I'm just trying to convert this docx file to this other file." I was like, "Why?" "Well, because I think it would be better if you had it in this format." So it's just like burning token. It's like, "No, we're good. You can stop."

Rob Collie (32:57): Yeah, in the movie Animal House where they bump off the drum major that's leading the marching band and replace him. And they just lead the marching band down a dead end.

Justin Mannhardt (33:11): They just keep going.

Rob Collie (33:11): And then they just drop the baton and run backwards and the band just stays there marching-

Justin Mannhardt (33:15): Stays in place.

Rob Collie (33:16): ... in place, trying to get through the wall for all eternity.

Justin Mannhardt (33:20): Yeah, don't they cut back to them later in the movie?

Rob Collie (33:22): Yeah, they totally do. They [inaudible 00:33:23]. They're still there. Anyways, yeah, so Claude does that sometimes. All of them do.

Justin Mannhardt (33:29): They all do.

Rob Collie (33:30): Sometimes it can dig itself out of those dead ends. Other times it's like, "Wow, dead end. That's my thing."

Justin Mannhardt (33:37): My favorite Opus 4.7 tendency was it constantly asking me if I was ready to go to sleep.

Rob Collie (33:45): Oh, I don't think it's asked me that. It didn't care about me apparently.

Justin Mannhardt (33:49): Especially in a long session, it'd be like, "Justin, you've done a lot today. Do you want to just call it a day?" I was like, "I'm getting close to my token limit, aren't I?"

Rob Collie (33:58): Oh, yeah. Yeah. Those prompts that started appearing recently that said, "This conversation's three hours old. Would you like us to compress it?" I'm like, "Well, no, it has nothing to do with it being three hours old, but nice try."

Justin Mannhardt (34:12): I'm still working on it.

Rob Collie (34:12): Nice try. I'm still on my subscription level. They're trying to get people to burn fewer tokens under their subscription cost, which good luck.

Justin Mannhardt (34:23): Yeah. I mean, it started, certain use cases they've moved out of the subscription plans and you got to use the pay-as-you-go models. I do wonder if both OpenAI and Anthropic, they'll hang on to get to IPO and then there's going to be some pricing reckoning that eventually happens.

Rob Collie (34:43): Or they do something completely illegal but difficult to prove, which is some sort of backroom agreement. We will both switch to token only in three, two, one.

Justin Mannhardt (34:55): Two, one.

Rob Collie (35:01): Anyway, it's been a while. Good to see you again.

Justin Mannhardt (35:03): Yeah, congrats on the book and when will it hit shelves, so to speak?

Rob Collie (35:08): Sadly, I think it's like August.

Justin Mannhardt (35:10): Okay.

Rob Collie (35:10): But people who go to the website today and pre-order, they get the first four chapters immediately. They'll get the ebook and the audiobook at the same time.

Justin Mannhardt (35:19): Cool.

Rob Collie (35:19): They'll get them immediately digitally on the release date, even while their hardcover book is in the mail. We're doing some other releases, little teases, a chapter here, a chapter there between now and then.

Justin Mannhardt (35:32): I'm looking forward to it, man. Congrats.

Rob Collie (35:33): Thank you so much.