Cat Negotiations, Dishonest Chatbots, AI vs AI in the Job Market, and More Real World AI Use Cases

Announcer (00:04): Welcome to Raw Data with Rob Collie, real talk about AI and data for business impact. And now, CEO and founder of P3 Adaptive, your host, Rob Collie.

Rob Collie (00:20): All right, Justin. Well, welcome back. It's been two weeks of me flying solo.

Justin Mannhardt (00:24): The old back-to-back solo pod play.

Rob Collie (00:27): Yeah, which is both fun but also a little bit exhausting because it requires me to really carefully think about every single word rather than just having a conversation with you, which is way more fun. I think we have a couple of things to talk about, I know you have one. I'm going to tell you one other that you didn't expect-

Justin Mannhardt (00:44): Ooh, okay.

Rob Collie (00:44): ... which is through strategic negotiation I have reached a peaceful resolution to the podcast lair cat problem.

Justin Mannhardt (00:52): Tell me more.

Rob Collie (00:53): I don't think this has ever really made it into the show, but backstage this is an ongoing saga of Rob versus cat.

Justin Mannhardt (01:00): It's true.

Rob Collie (01:01): The cat that lives down here. If I let him in the room with me, he paws me and digs at my clothes to get attention the whole time I'm trying to talk, which of course is just not tenable. You can't have a podcast while you're being gouged. But if I lock him out of the podcast room he just digs at the door, and whines, and you hear him. Strangely enough, I can put him in a pet carrier and have him sit next to me, and he is happy as he can be.

Justin Mannhardt (01:31): This is a negotiation directly with the category?

Rob Collie (01:33): Yeah, the cat and I, we sat down at the negotiation table and said, "Well, I really want to be in the room with you." And I'm like, "But you can't bother me." He's like, "Well, is there anything else we can do?"

Justin Mannhardt (01:45): Right, right. Well hey, that's great.

Rob Collie (01:49): And he feels safe in this cat carrier, safe and warm, and he knows that he's next to me and he hears my voice. So, there is peace in a saga that has been ongoing for years. As long as this podcast has been going on, we've had this dynamic backstage.

Justin Mannhardt (02:05): It's a long time.

Rob Collie (02:06): Yeah. Small victories.

Justin Mannhardt (02:07): It'd be an interesting stat to mine for is how many instances of this have been edited out of the show?

Rob Collie (02:15): Oh, many. Yeah, very many. And maybe a couple did leak through, right?

Justin Mannhardt (02:19): Yeah.

Rob Collie (02:20): You also hinted that you've had a real world run in with AI of some sort?

Justin Mannhardt (02:24): It's like an interesting report I bring, a bad experience with AI from the real world. You are familiar with the product known as the Oura Ring.

Rob Collie (02:38): Oh yes. Don't have one, but I'm aware of it.

Justin Mannhardt (02:41): So, my wife's had one for a couple of years. I think she got it through a program at work, and she's always really liked it. And she said, "Justin, I think you'd really like one too." So, finally they just released a new design and it looks really sleek, and looks better than the last one. So I said, "Oh yeah, I'll check it out." So I go on, I order, and I got my ring this week. And you'll notice I'm not wearing it. I put the ring on and I took it off, and the outer sleeve just immediately came right off. So there's an inner unit, which is the sensors and the chip and everything, and then there's the outer is the decorative, the color, the design. And it's supposed to be held by some industrial epoxy that apparently have a defect. So, then I go on their website and I go get some help.

(03:26): And what happens? This thing called Fin pops up, and it's an AI chatbot. And I'm like, "Oh, okay, cool." I actually had a really good experience talking to this thing. It was maybe one of the better AI chatbot customer service experiences I've experienced. It explained what the issue was, it even linked me out, "Here, this is a known issue. Here's the article that explains what's going on, how we fixed it and what we'll do about it." And so, then it asked me for all my details, about my order number, I had to do an MFA thing to validate who I was and log in. And so it said, "All right, so what'll happen next is you'll get a confirmation email from our member care team. I've logged a case for you, they'll reach out to process your warranty and you'll get a replacement ring." I'm like, "Sweet, that sounds great." And I'm like, "This is a good use, good bot." So far it sounds great-

Rob Collie (04:17): Where's the AI failure here? It's not responsible for the ring coming apart. Fin seems like a very cordial fellow.

Justin Mannhardt (04:24): Yeah, I couldn't jailbreak it either. I asked it for a recipe for blueberry muffins. It said, "Ha ha, very funny."

Rob Collie (04:30): You didn't get it to write you some Python code?

Justin Mannhardt (04:33): Nope, I didn't try that. So, a few days go by and I've not received any sort of communication. I'm going to follow up. So, this time I go and I say, "I want to talk to a real person." This is where I find out, they're like, "Oh yeah, there's no case recorded. There's no anything anywhere about what's going on." And I said, "Oh. Well, your AI assistant was very confident that it had done these things." And they're like, "Can you send us a screenshot of the conversation?" I was like, "I have no idea how to access that conversation." So, either this agent was just completely syncopatic with me, telling me, "Yeah, I've solved your problem," and it couldn't actually do those things, or it was supposed to be able to do those things and there was some sort of integration failure that was occurring that it wasn't wise to.

Rob Collie (05:23): Yeah, like it had access to some tools. It called those tools, those tools were down and it got an error code or something.

Justin Mannhardt (05:31): Some didn't work.

Rob Collie (05:32): And it didn't bother to check the error code.

Justin Mannhardt (05:35): Yeah.

Rob Collie (05:36): I mean, we're speculating from the outside, but it'd be perfectly possible that's one of those cases where the chatbot was just too worn out in terms of its own context window by the conversation with you, because you frankly are exhausting. And when it got around the time to call the tools, it didn't bother to check the error codes or validate. But it's probably also just a poorly constructed system, very likely to not be an LLM related failure. You got lied to, you've lost a number of days of progress on getting your replacement ring because an AI said, "Right you are, Justin. I've got you," and didn't got you.

Justin Mannhardt (06:20): Yeah. So, just be on alert out there folks, as you're realizing you're talking to these things. And it had me. I was like, "Great, this sounds exactly like what I needed."

Rob Collie (06:31): I mean, they don't lack for confidence when they mess up, right?

Justin Mannhardt (06:33): No.

Rob Collie (06:35): That's the best part of it.

Justin Mannhardt (06:36): Shout out to Mary. I was on the phone with Mary in member customer care, she got me all sorted. My replacement is on the way.

Rob Collie (06:45): Cool. Very cool. So along these lines, have you heard of Danielson Labs? We're making this a thing at this company here.

Justin Mannhardt (06:54): I love how you just so nonchalantly, just ... I'm like, "No." Like, "Oh, yeah."

Rob Collie (06:59): Yeah, Danielson Labs. So, Danielson Labs is a Skunk Works division of P3.

Justin Mannhardt (07:04): Newly minted.

Rob Collie (07:05): It was founded like three weeks ago in silence.

Justin Mannhardt (07:09): Real grassroots.

Rob Collie (07:10): Didn't even know that it was being founded. And if you stand outside of Danielson Labs, you will hear all kinds of clanging, and banging, and flashes of electricity light and stuff like that. You're like, "What is going on in there?" It's like Nikola Tesla and Thomas Edison, and all of those people are all in a lab together. And it's Kellan Danielson, our president and COO, who has really, really, really, really, really caught fire with AI in two overlapping ways. He's using AI to help him vibe code and build a bunch of solutions, but he's also, within these solutions, just like I was talking about in this last week's podcast, the solutions that he's building are mostly made of regular Lego bricks. They're mostly regular code, regular stuff, like stuff that has existed forever. AI is being used to quickly help him build these things, but there's nothing AI about the end result. It's just code running, doing its thing.

(08:06): But in certain places in the workflow, certain places in the code, it'll occasionally will call out to what I'm calling the magic Lego brick. It'll make a call to an LLM to do something that only an LLM can do, and then the answers and results of that come back into the regular Lego brick workflow. So, sometimes they even get fed to other LLMs to validate or take it to the next step, or whatever. So, it keeps the context windows clean and everything, and it really keeps it reliable. And I think your chatbot example is probably an example that isn't very well segmented. If it was better segmented, it would at least know when it was experiencing a failure unambiguously. But anyway, so I finally got a peek into Danielson Labs and I saw what he was up to, and I'm like, "holy cow, I need to go back to a couple of things. At least one thing that I've been working on before and not doing well with."

(09:00): So, one of the things we've been stressing a lot on the podcast lately is giving people use cases. The first use case I'm going to talk about is not a business use case, it is a hobby use case but it is very educational. So, I am commissioner of three different fantasy football leagues, actually four if you count the two leagues here at P3 Adaptive, and these are primarily social things first and foremost. Yes, they're competitive, but they're mostly about interacting with each other, and if there isn't some water cooler to huddle around you have to create one. Over the years, what I tend to do is I tend to try to send some sort of periodic updates or just a recap, or predictions, or anything for the members of the league to coalesce around and react to, and then start conversation.

(09:51): And so, you're actually getting the social value out of it, because otherwise you're just like 12 people sitting in your silos and you interact with each other via some scoreboard. You don't even talk to each other. But I've been doing a really bad job of that for a lot of reasons, just way oversubscribed in life. No way I'm sitting down and writing three, four long emails a week. Two months ago, month and a half ago, I tried using Claude desktop and an MCP powered web browser. It doesn't matter, the details of this don't matter so much, but I tried to write or build some sort of tool that would go out and look at the websites, the fantasy football websites for these leagues, inspect them, find out what's going on, and then help me write, or rough draft anyway, what these updates would look like.

Justin Mannhardt (10:39): Sure.

Rob Collie (10:40): And I just got nowhere, because the LLM spent so much of its energy, so much of its thinking, so much of its context window getting worn out just trying to navigate the ESPN website. That was all it could do, and it barely was able to do that. And then, it would just collapse. But after seeing what's going on in Danielson Labs, I'm like, "Oh my God, let's try something different." So, fast-forward to seemingly like three hours later, I had something that was using regular Lego bricks. That's the problem, the first solution was all magic Lego bricks.

Justin Mannhardt (11:18): All AI.

Rob Collie (11:19): All AI. It was doing everything, it was fetching the raw data from the website, it was building a picture in its head of what's going on in the league, and then it was trying to write the email and it just collapsed. In this case though, really, really, really leaning into regular old Lego bricks. Turns out, I didn't even know this, turns out that ESPN has an API that's open

Justin Mannhardt (11:45): Really?

Rob Collie (11:46): For public leagues, if you set your league to public so that people can view it without logging in-

Justin Mannhardt (11:54): Okay,

Rob Collie (11:54): .. this API works without even any authentication.

Justin Mannhardt (11:58): Brilliant.

Rob Collie (11:58): If you think about the amount of effort that the LLM was spending to chew through these web pages and literally find the links to click on them, and then wait for the page to load and-

Justin Mannhardt (12:11): And deconstruct.

Rob Collie (12:12): Yeah. If you just want to measure it in power, I was using kilowatt-hours of GPU just to do something relatively simple that regular code, regular Lego bricks can do with an API with zero effort, doesn't wear anything out, it's just super, super efficient, and it's also super precise. It doesn't get it wrong. So, it grabs all the information, all the statistics, all of the match-ups, everything, puts them in a database on my computer that it created for me, by the way, and then uses the database as the means for writing the emails.

(12:49): So, now the LLM does come into play, but it has access to the database and can query the database to get the information it needs, and even explore around a little bit. And it wrote some pretty damn good emails. I got to that in like three hours. This thing is still really in its early phases, if I had more time I would improve it infinitely. But at one point I thought I had a hallucination. In the email it said, "Yeah, so-and-so left 330 points on his bench this week," which-

Justin Mannhardt (13:21): Seems impossible.

Rob Collie (13:22): ... is not possible. So I went and looked, and it was more like 110. So, I was having a chat with the vibe coding AI that helped me and I said, "Hey, I think we've got hallucination here. What can we do to rein that in?" And it turned out it wasn't, it was a regular bug. Just a regular bug that the regular code had been fetching results from the website and not deduplicating. So, it had three copies of the statistics. But I'm so used to seeing incorrect numbers from LLMs when they do their own local calculations, which we really need to avoid, right?

Justin Mannhardt (13:58): Right.

Rob Collie (13:59): It's not doing that. It turns out it's doing all the things right. It's letting the regular CPU, regular Lego bricks code, like SQL and Python, and all those sorts of things that are super, super reliable and deterministic, it's letting them generate all of the hard information, including the numbers, and it was an easily fixable bug and off it went. But for a moment there I'm like, "Oh, here we go again." But quite the opposite. You want to use the magic Lego brick sparingly. You want as much as possible to be done by old-fashioned tech.

(14:32): And the results are just, I went from completely DOA, not getting anything, to getting something kind of incredible. I'm off and running now, next year this thing's going to be running headless.

Justin Mannhardt (14:48): That's awesome.

Rob Collie (14:48): And just sending updates to everybody every week. We're going to send a recap, we're going to send a predictions mail, there's going to be a water cooler and we're going to just create one. Really, the key educational point in there is again, use the magic Lego bricks sparingly, lean on traditional tech wrapped around it as much, as much, as much as you can, the quality of result is just amazing. Okay, so that was one use case. Not a business use case, but now I've got a business use case.

Justin Mannhardt (15:19): Okay, what's going on in the business?

Rob Collie (15:20): We should add this to our ongoing list of examples of where customized agentic, whatever, either way, not off the shelf.

Justin Mannhardt (15:31): Not off the shelf.

Rob Collie (15:33): Custom AI solutions, things they can do for your business. So we, as you well know, we are hiring our first ever pure developer.

Justin Mannhardt (15:43): Yeah, like a traditional full stack web type person.

Rob Collie (15:47): And it's ironic and fun that we are doing that. We're doing that to support our AI efforts. The irony of course is that well, wait, I thought we didn't need developers as much because now we've got vibe coding.

Justin Mannhardt (15:58): Right.

Rob Collie (16:00): AI, that was supposed to be the reason that we needed fewer developers, turns out to be the reason that we need an infinite percentage increase in developers at our company. That's kind of fun. But anyway, wow, from both sides of the table, a hiring team and the applicants, boy, the job market has become a numbers game and it has become an AI powered wasteland. We started using a really great online application and workflow system for managing our hires. I think we were early to that game. We started in 2017, but we never did any AI machine learning scanning of resumes or anything like that, but big enterprises jumped into that game with both feet. It wasn't long after that all kinds of tools sprung up so that applicants can play by the same rules and can automatically apply to 500 positions.

Justin Mannhardt (17:02): There's a type of data visualization you see it a lot with how Microsoft would show how their revenues decompose.

Rob Collie (17:09): Is it the flow, like the river flow?

Justin Mannhardt (17:11): Like the river flow.

Rob Collie (17:12): Yeah, I think it's called a Sankey.

Justin Mannhardt (17:14): A Sankey. So, I saw that type of visualization from an individual on social media with the amount of jobs they had applied for, and they were categorizing didn't get a response, got a response, had to do a test, didn't have to, got all the way out to getting an offer. And I was like, "This has to be fake." This person was like, "In the last year I've applied for," some ridiculous number, like a million jobs, and ended up getting a job but it highlighted even if this is hyperbolic and a bit fake, that's what's happening. If you're trying to get employed, your full-time job is applying to as many jobs as possible every single day, just by the boatload.

Rob Collie (17:59): Yep. And so, you lean on automation, and that includes AI-powered automatic answering of free-form questions in an application. You can't really fault people for this.

Justin Mannhardt (18:15): No.

Rob Collie (18:16): This is a market force. I'm not angry about it.

Justin Mannhardt (18:19): Yeah, we're obviously pro-AI here.

Rob Collie (18:22): Right. But we're not using the AI screening systems that necessitated the arms race of applicants. I suppose if companies weren't using anything automated, these things still would've come for the applicants, the automated application tools, but we're human beings reading applications.

Justin Mannhardt (18:46): Hundreds of them.

Rob Collie (18:47): The other night I reviewed about 100 manually myself. When you're getting hundreds of applicants for a job, you're not going to interview hundreds of applicants. You can't. You need to get down to a short list. Culture fit is super, super, super important to us. And so, those free-form questions that we ask people about themselves are really good, or had been anyway, a really, really valuable way to figure out if people have the right kind of vibe and pulse to work with us. Essentially just looking for real people with personality, and just interesting dynamic people. Well, that's all gone now because it seems like probably 90% of the applications I'm reading, no human being ever touched the answers. And it's not that I can spot an AI written answer, it's that when I see the same four or five answer themes repeating over, and over, and over, and over again, we don't have 400 people applying for our jobs, we have three or four different LLMs applying for jobs.

(19:59): And those three or four LLMs have their go-to answers, and they don't remember that they just applied for this job five minutes ago. They don't, it's being called on behalf of a different user. They have no idea. And so, I am just seeing the same answers over and over again, and tell us something about yourself as a human being. The word curious is going to appear by word six, and then continuous learning is going to be in there somewhere. One of my personal favorites is the vast number of applicants for our jobs, who are very active volunteering with underprivileged youth, giving them access to technology.

Justin Mannhardt (20:38): Yeah, specifically that.

Rob Collie (20:40): Can you imagine getting into an interview with this person and asking them, "Hey, well, so tell us about your experience with volunteering with underprivileged youth." And they go, "What?"

Justin Mannhardt (20:51): Yeah, we saw that about 100 times.

Rob Collie (20:53): "No, you said that in your application." And they go, "No, I didn't." That's how it'd play out. They wouldn't even remember. Anyway, so it tells us nothing and there's plenty of good candidates, I'm sure. There's so many good people applying for jobs with us where we can't tell anything about them, and so we are not going to waste the time on an informational. We have to have a reason to want to talk to you, and they're missing the chance to give us that. Now, I don't know if most companies are operating the way that we are, or care about this sort of thing. Maybe this is a wonderful strategy in general, but for us it is backfiring for all of these applicants.

(21:35): But it also makes a tremendous amount of work for us, and I'm wondering how many other non-enterprise companies out there are having exactly this problem right now? If you have any discernment at all, if you're trying to be conscientious at all about the applicants that you actually want to talk to, the radar is so jammed now. It's not just the quantity of applications, it's also the fact that there's no differentiation between them.

Justin Mannhardt (22:00): And you wouldn't pick up on this with a sample size of one, right?

Rob Collie (22:06): Mm-mm.

Justin Mannhardt (22:06): You never would. It's the fact that you've gone through so many and be like, "Why are there so many people doing the underprivileged youth thing?" We've even seen similar tells in, we do a technical screen and some of our other jobs, and we now know the clear tell that ChatGPT wrote this code. If you saw it one off you wouldn't know, but because you can see it again, and again, and again, and again, and again, then validate it yourself. And even between the LLMs, I've seen some articles where if you go ask the same question to different LLMs, like, "What's your favorite color? What's your favorite number? If you could do a favorite activity in the fall, what would it be?" And you'll get a very similar answer from all these providers. It's because well, I hate to oversimplify it so much sometimes, but it's like, well, what are these things trained to do is to predict the next token successfully. Right?

Rob Collie (22:59): Yeah. And as I watched the incoming pipeline, so I went through 100 candidates, I got the applied bucket down to eight people before I got tired. I woke up the next morning and it's 208. I'm like, "Nope, not doing that." But it's over 400 now, 400 to be reviewed, to even be looked at for the first time. Well, we are entering the arms race, and we're entering it in our way. So, we've done a couple of things. First of all, to the front of the questionnaire we've added a very honest bit of advice, if there was a human being reading it anyway, saying, "Look, we get it. Applying for jobs is brutal, but we need to know about you. And when you use AI, you just sound like everyone else. Trust us." And that gave us air cover that I can feel like okay with myself. Then developing an AI detection routine, it's AI powered. It uses the magic Lego brick.

(24:01): So, I've trained it on a whole bunch of these questionnaires. It's looking for things that sound like they were written by AI, but it's also looking for, "Oh, is this one of those four or five just over and over again patterns?" Are we seeing those? It takes me probably not five minutes, but it takes me a couple of minutes to review a single candidate just to discover that nope, there's nothing in here. It's empty. It's all kinds of text, but there's nothing in there that was ever written by a human being, and a poor human being is trying to read it. Why should I bother to read something you didn't bother to write? Then we also added my favorite interview question of all time, we had this in some older jobs at one point. We have a joke, we have a joke question.

(24:41): It's multiple choice, "How many hipsters does it take to change a light bulb?" And it's got answers like 1, 0, 100, unknown. And then the last answer is, "It's a really obscure number, you probably haven't heard of it." And our kind of people, if they read that question they know that they answer is that last one. So, we just auto DQ, automatically disqualify anyone that misses that question. It's shocking to me that the AI systems miss that question, because when you feed that question directly to an LLM it knows what's up. Whatever these AI systems are using when it encounters a multiple choice question, it's really economizing and using a really, really, really dumb LLM or something. I don't know. It's like they're cutting costs there.

(25:27): But anyway, I've written and I'm mostly done with let's just call it an agent, and it's made mostly of regular Lego bricks but also calls the magic Lego brick at certain points, and it goes through all of these applications and it whittles out all of the obvious nos, either obvious no for, I've got a number of heuristics that I have it doing, but also if they're not telling us anything about ourselves we don't even want to read it because they're not going to learn anything. We do have about 20 candidates out of the several hundred we reviewed so far that are real human beings giving real answers, and those people are worthy of our attention. That's the applicant pool as far as I'm concerned. The rest of this is noise.

(26:08): So having a customized, to your needs-

Justin Mannhardt (26:13): Correct.

Rob Collie (26:14): ... candidate screening agent, I think this is an amazingly applicable use case. When we're done with this one, we're going to set it up to help us with the other positions. And again, this isn't going to make things less humane, it's going to help us weed through so that we can be humans with the ones that want to be humans.

Justin Mannhardt (26:33): Exactly.

Rob Collie (26:34): Our time is valuable, just like theirs.

Justin Mannhardt (26:36): And just to bold a couple concepts that are going on here, we empathize with the job seeker. It is just massively brutal out there.

Rob Collie (26:46): It is.

Justin Mannhardt (26:47): You're looking for every easy apply, quick apply. You're just trying to get a job, and doing everything you can to do it. But so, then on the flip side this is very different than, "Hey, ChatGPT, review this candidate and tell me if you think their application was written by AI."

Rob Collie (27:06): Yeah.

Justin Mannhardt (27:07): Right? Because if you had that system, it would get you absolutely nowhere.

Rob Collie (27:11): Yeah, I agree. It's trained on these questions and these responses.

Justin Mannhardt (27:15): Yeah, I don't think you could have gotten there without the human experience of like, "Okay, I've identified the patterns I really want to focus on." Now you've got a very customized situation, like you said, you're using regular Lego bricks combined with the magic Lego brick, to do a very specific task that's customized to your needs and what you're looking for. That's so different than just, "Let's just slap AI on the top of this and let it make us more efficient." Because it would theoretically do that, just plug it into some LLM and be like, "Oh, great, I've whittled the candidate pool down to 20," and you'd be like, "But why these 20?"

Rob Collie (27:53): So my system, I've run this on a small test batch, it moves a candidate to either the next step for actual human review or it disqualifies, and if it disqualifies the candidate, it puts a note in there in the candidate explaining which criteria and why was triggered, which allows me to review them. And it turns out all the ones that disqualified were correct disqualified, it's not getting any of that wrong, but it's still letting a lot of slop through. And so, I'm doing things like, "Okay, this candidate here, see that candidate? There's nothing authentic in those answers." Now, from the job applicant's point of view I do agree with that. We are sympathetic to it, but it's interesting, isn't it?

(28:38): If there's companies like us out there who are not going to engage with you unless we have a sense of you, what this bulk application process is going to do for you is eventually find you a job at a body shop, the more corporate end of things. It's not going to give you equivalent results with all kinds of companies. It's going to drift your success rate, and I know that if you're just trying to get a job anywhere that's the game. It's a tough world, and our hand is now being forced because we can't afford to have your time or my time go into, for weeks at a time, reviewing things that aren't worth reading. Bring the ones to us that are worth reading and we will spend a lot of time reading the 5 or 10% that they get through that.

Justin Mannhardt (29:28): Totally. What do we got a name for this thing yet?

Rob Collie (29:31): I haven't named it. I hear Fin.

Justin Mannhardt (29:33): Fin's taken. Yeah.

Rob Collie (29:36): Well, it's not only is it taken but it's got negative connotations.

Justin Mannhardt (29:39): Bad street cred.

Rob Collie (29:40): Yeah. I don't know if this is a keeper or not, but could we call it The Sorting Hat?

Justin Mannhardt (29:49): Gryffindor.

Rob Collie (29:52): This sorting hat says Slytherin a lot.

Justin Mannhardt (29:55): Let's take this to committee.

Rob Collie (29:57): Yeah, just another one of those use cases. Again, I've been emphasizing lately developing your muscles for spotting places where AI can be helpful as your company is probably the single most important thing to be working on right now, and every example helps.

Justin Mannhardt (30:12): Yeah. Like you were on your episode last week, talking about how really having custom-built solutions is really crucial, but also not that hard. The experience is just infinitely better, whether you're building a chat agent or some type of headless automated workflow that's just running on the background, whatever the use case is there, if that isn't highly customized everything's going to skew towards the mean. That's just not what you're looking for. Project I'm working on right now, I'm working on a couple, but I'm working on something that'll help our team with opportunities with their accounts, and the example is so striking because you could go to ChatGPT or Claude, and say like, "Hey, I had a meeting with my client and we talked about doing this. Help me understand how to get through this process."

(31:08): And it'll be like, "Oh, well you should do this and you should think about this," and it's going to use things that are incompatible with our process, incompatible with what we actually sell, incompatible with all these things, right?

Rob Collie (31:19): Incompatible with our ethos.

Justin Mannhardt (31:20): I'm just investing the time into customizing this knowledge base and instruction set that understands its job clearly, it's not just a general purpose chat agent, it has a very specific duty. And it understands the process it's following, so when you talk to it, it understands where ... We have vernacular about where we are in this lifecycle internally. It understands that, and based on where you are what should or shouldn't be happening, or what type of help, and you just get a much better experience investing into that crucial customization, and it's not difficult. And by not difficult, there is sophistication involved and there's some smarts that are needed, but I'm not writing complex code or building some crazy-

Rob Collie (32:07): Yeah. We're saying these things aren't difficult, and at the same time as I'm working on these sorts of solutions, I know that again, it's this data gene thing. I remember I was talking to Amir Netz a long time ago about vibe coding. He's like, "Hey, you should sit down and try vibe coding," and I hadn't yet. And he's like, "You're going to discover pretty quickly, Rob, that it isn't for everyone." So, even though we think of it as sort of not that hard necessarily, in the course of doing this work yesterday with this job application thing, I could tell that the system that it was building wasn't working efficiently. It kept going back and making API calls to Breezy over, and over, and over again for the same information. I'm like, "Hey, let's stop and just grab all this once, and create a database locally so that we're not having to make all those API calls," because every like 10th API call you make to Breezy, it puts you in timeout for a minute.

(32:59): It says, "No, you've done too much." So, this runs forever just to do something simple, and it turned out just fetching everything was actually fewer API calls and then now I've got this local database. You wouldn't think to do that if you're not a DataGener. Our kind of people, the kind of people that work for us, that work at our company, it's a really bright future for the DataGene crowd. I think the DataGene crowd is going to have the same sort of experience with this stuff as they had originally with Excel, as they had with Power BI. We're just really early in that societal transformation, but it's not the same as when you're working with AI, the one thing you can't change is the AI. The LLM is the LLM, tough. That magic Lego brick cannot be changed, except by Anthropic, except by OpenAI, and they only do that every six months because it's so expensive. You don't have the power to do the rocket science stuff, but you don't need it. What does hard mean? It means in reach. It's not hard, means it's in reach.

Justin Mannhardt (34:03): Right. It's still intelligent work, it's still maybe there's careful planning involved or if refinement iteration, best practices that are emerging, but it's accessible to a larger number of people than what many might assume. Anytime I can reinforce the idea that when you're designing an AI solution, it's really valuable to identify all the parts of that workflow or process where you actually don't need the magic Lego brick. I don't need you to scrape the website, just call the API and store everything in a database. And now that magic Lego brick's happy because it's like, "Oh, cool, I don't have to go scrape through this HTML. Oh, this is good for me."

Rob Collie (34:50): And by the way, it also reduces the expense of the solution. Every time you call that magic Lego brick, the longer it thinks, you're chewing tokens and that translates to money. In my original fantasy football recap writer, I had an LLM just boiling the ocean for me and failing, but along the way, burning so many tokens that Claude was telling me that I was on timeout for the next three hours. If I'd been paying via the API, I would've run up a pretty enormous bill there. Well, enormous for that short period of time anyway, but in production use the CPU stuff like costs nothing. Costs nothing to run. It's free. The LLM stuff is not free.

Justin Mannhardt (35:35): Right, right. Absolutely not. But it's fun to be where we're at, having these types of points of view, and you see them validated across multiple use cases, then you get more confidence that you're onto something and then patterns are starting to emerge about when the next use case comes up here's the way we try to go about it.

Rob Collie (35:58): And even four months ago I didn't have anything resembling that, I didn't have those patterns yet, but I'm really, really, really confident in those principles. I've seen more than enough now to know better. What'd you think about the chasm between the out of the box, off the shelf consumer usage of chatgpt.com? And as you pointed out earlier, you start asking it questions about what to do to help your clients understand AI better and our capabilities, and you fail, it doesn't work. I think the existence of those consumer experiences in a way almost work against adoption. I mean, they work for adoption and the fact that you get to see what the magic looks like, but you get to see what the magic looks like in your personal life.

Justin Mannhardt (36:48): Yeah, or for you yourself.

Rob Collie (36:50): Yeah.

Justin Mannhardt (36:51): It's an interesting journey to have been on, because that's where I started is just signed up for ChatGPT. Holy shit, this thing's cool. and it is cool, right?

Rob Collie (37:05): It really is.

Justin Mannhardt (37:06): It's very cool, but then the deeper you go, I think this is true about the DataGener type is you start to understand a technology and then you say, "Oh, but could I also do this, or could I do that?" So you advance to things like, "Oh, I've got my own custom instructions in my ChatGPT account where I've got my own little library of structured engineered prompts that I need to use on a recurring basis." And then you graduate into an environment that is either a team environment or a organizational environment, and you go, "How do I do this for everyone the same way?" That's where it starts to break down on we need a way to have this customization that is shared by all, which is similar to the Power BI and the data problem. We've all been looking, how do we get to assets that guide us all together?

Rob Collie (38:05): It's both dimensions.

Justin Mannhardt (38:06): It's both.

Rob Collie (38:07): One dimension is the published and working on everyone's behalf in the same manner, and that's difficult to do with out-of-the-box, off-the-shelf solutions, but then there's also the dimension of merging it into a business process. And even if that was just your business process, really bringing it onside into that business process is really not doable with the off-the-shelf. But because the off-the-shelf product is so magical for so many things, you have this false impression that it then should be good at the business thing and you don't have any good explanation for why it's not, and you also don't have any idea of where to go next. It's such a huge leap to go from that consumer experience to saying, "Oh, I need to be not using the off-the-shelf. I need to not even be looking at chatgpt.com."

(39:02): The LLM needs to be called by other things in the background, even if I'm using some sort of chat experience, it needs to be a piece in a larger Lego model and not just a standalone thing. But you've never seen one of these larger Lego models built out of regular bricks plus one magic one, never seen one. Us describing over and over again, I hinted on last week's podcast that we're working on basically dozens of internal agents at this point, when I recorded that I didn't even know I was going to be doing the Breezy candidate screening one.

Justin Mannhardt (39:34): Put it on the list.

Rob Collie (39:37): So, really just need to help flood the world with examples, what those things look like.

Justin Mannhardt (39:45): Yeah. And how they're put together, which I think has been, that's been really fascinating to get grounded in some ideas there too. When you talk about adoption in enterprises, I forget what the stat you shared earlier was, like what, 60, 70% of ChatGPT is individual user subscriptions or something like that?

Rob Collie (40:05): It's estimated that 60 to 70% of that revenue is from individual user subscriptions, yes.

Justin Mannhardt (40:11): If you've seen some of their ads on TV, it's like, "Hey, I want to be able to do 10 pull-ups by summer. Help me make a recipe that says I want to hang out but I'm not being too romantic."

Rob Collie (40:21): Which I clustered together by the way in last week's podcast as help me impress girls. That's what ChatGPT is advertising on NFL football games is like the-

Justin Mannhardt (40:30): Yeah, help me impress girls.

Rob Collie (40:32): The use case of AI has help me impress girls, and not a business use case but it's also an admission that those sorts of use cases are really the only ones that really you really use, it's at least off-the-shelf.

Justin Mannhardt (40:48): It's just like a fresh idea I'm having now. If you think about, I'll use myself an example, if I think about my personal life, there's way more dimension to that aspect of my existence than there is in my professional life. What I mean by that is I am a father, and a husband, and a friend, and I like to play golf, I like to mountain bike, I like to cook, I like to grill, I like to work on my ... And so, ChatGPT is quite good at me just showing up and like, "Hey, this thing happened to my air conditioner. Help me fix it." "I want to make Southern fried chicken tonight, give me a recipe." It serves that really well. Not that my life unprofessionally is simple, but now it's like I'm in a very different type of environment where I need the AI to understand my role at P3 and I need it to understand what P3 does, and I need to understand all these processes to get a higher quality of a result.

(41:46): Let's say I'm writing a business proposal for a client. I just go to Claude, it's going to spit something out. It's like, "Yeah, this is not how I write. This is not what we talk about, this is not the product or service we actually sell." If you think about what we've done too is like, yeah, we've rolled out some of these subscriptions internally. I just wonder if the dimensionality of use isn't as high in an organizational context. Don't get me wrong, there's still value there but it's just different. The chasm is the right idea, Rob. It's just a different situation.

Rob Collie (42:17): Yeah, and there's this huge air gap you have to cross to get from the off-the-shelf you use to your first customized use. Again, it's not a rocket science chasm to cross, it's like imagining something you haven't seen yet. And this is in a way a little bit sad about humanity. You were saying there's more dimensionality to your personal life, but what it really comes down to is that our personal lives are far less custom than our professional lives.

Justin Mannhardt (42:47): Yeah, general advice is generally good.

Rob Collie (42:51): These things have been trained on, not every, but 95% of what we're going to encounter. Me grilling a chicken isn't terribly different from someone else grilling a chicken.

Justin Mannhardt (43:03): Right.

Rob Collie (43:04): Fun things to do with kids, to the extent that I need to customize that, I need to explain, I just need to tell it just a little bit about my kids and it goes, "Oh, I know that demographic." It's just seen so many people that we're just sadly not as unique as we want to think we are. In our professional lives, things are just a lot more nuanced. It's almost saddening in a way that we're more unique in the professional sense than we are in the personal.

Justin Mannhardt (43:32): Yeah. Even in a large organization, you're operating in a very small tribe, in a way. A smaller community from the rest of humanity, and you have adopted your own set of rules, and principles, and policies, and processes or whatever that is not broadly applicable outside.

Rob Collie (43:53): Companies are organisms, and honestly, no two companies are really that alike. I know there's a famous book about business, Crossing the Chasm. We have a very similar thing going on here, there's this empty void space between the off-the-shelf use and your first truly effective business use, because the two just don't look alike at all. And your first effective business use is going to be so unique to you.

(44:24): Well, listen, it's been great having you back in the co-host seat this week.

Justin Mannhardt (44:29): Yeah, likewise.