What is “Botsitting?” (plus: Data Agents Continue to Blow Our Minds)

announcer (00:04): Welcome to Raw Data with Rob Collie, real talk about AI and data for business impact, and now, CEO and founder of P3 Adaptive, your host, Rob Collie.

Rob Collie (00:20): Welcome back. It's been a few weeks.

Justin Mannhardt (00:22): Thanks.

Rob Collie (00:22): Took a week off from the podcast. We've had guests kind of filling in the calendar, and it's been a little while since we've got the JAW session in. This time, I bring topics.

Justin Mannhardt (00:32): I get to be the reactor.

Rob Collie (00:34): That's right.

Justin Mannhardt (00:35): To the topics. I love it.

Rob Collie (00:38): So far, no one I've asked has heard of this term, bot-sitting.

Justin Mannhardt (00:43): Bot-sitting.

Rob Collie (00:45): Bot-sitting. It's like babysitting, but for bots. I know about bot-sitting because it's the only topic on which, I have a PR firm. We have a PR firm that helps me promote the book, and they connect me with journalists who have questions. I don't get to choose the questions. The journalists have the thing that they want to write about that week, because it's the thing that's trending, and bot-sitting is the only topic on which I've had multiple journalists want to speak to me on the same topic. That's an indication that this is hot. This is the topic du jour. Bot-sitting, and then I go ask all the people I know who are really savvy about AI. They're like, "What's that?"

Justin Mannhardt (01:25): Is the answer, "It's what you do"?

Rob Collie (01:27): It's funny. In the early going with this, when these questions would come in and I'd be scheduled for 48 hours from now to talk about this random topic that just, really just feels like it comes out of a random number generator, as far as I'm concerned, the first few times they came in, my first response was imposter syndrome. "I'm not qualified to speak on that," and then I'd say I'm not qualified to speak on it. I'd tell the PR firm, "No, I decline this interview request," and then 30 minutes later, I'm like, "Wait a second. Actually, I do have things to say about that." Anyway, so now when something comes in and I've never heard of it, I just go, "All right, well, let's just go research what bot-sitting is." This is very much a disparaging, negative connotation term.

Justin Mannhardt (02:04): Correct.

Rob Collie (02:05): It describes employees' reactions to being forced to use AI. There's sort of two sides of the suck for the employee. One is, well, AI is now doing the part of the job that they liked doing, but also that the sitting part, the one that's more pervasive is that it's not doing it right. The AI isn't just going and doing the thing that's being asked to do and doing a good job of it. They have to constantly correct the damn thing. "No, no, not like that." You can sort of imagine what the average workflow looks like. "Go write me a proposal," let's say, "For this situation," and maybe the first time you send a request like that, you're expecting something optimistic, because again, you've had this great experience with off-the-shelf ChatGPT, whatever. In so many corners of your life, it's a world beater, and then you just expect that to carry over to business, but then it doesn't, and you pretty quickly reach the point where you're like, "Yeah, it's going to write an absolutely terrible proposal, but I have to ask it to do it and then wait. It's not going to be instantaneous that it writes this thing. I'm going to have to wait and wait, and then it comes out and it's awful, just awful."

(03:26): Then I start saying, "Here are the 12 ways in which this is awful and needs to improve," and you just iterate and iterate and iterate, and in the end, it would have been faster to just write the damn thing yourself. If you need to be doing many of these in parallel, you're more like TSA. You're quality assurance. Human beings are terrible at these sorts of vigilance tasks, right, so mistakes and errors are going to get through, or things that just rub people the wrong way, as we've talked about before, right? This is a document that's written to be sent from one person to another. Things are going to go sideways, because no human being has that degree of attention that they can maintain on something, thinking that it's broken, and so this is really wearing people down. All the journalists want to talk about essentially this backlash from employees. I'm like, "Oh, this is perfect. This is exactly where we should expect to be."

(04:31): Okay, so you're a business leader. You're under pressure to adopt AI. What are we going to do about AI? What do you do? You go, "Okay, the most obvious thing I can do, I'll go buy subscriptions." I buy them just like I bought Zoom.

Justin Mannhardt (04:44): Tell everybody to use them.

Rob Collie (04:46): Okay, so that's step one, right? We buy it and we expect everyone to just use it, and we're paying all this money and we find out people really aren't using it.

Justin Mannhardt (04:53): Right.

Rob Collie (04:54): Then we start monitoring usage and we mandate that they use it. That's, again, very obvious move, too. Has anyone at this point understood the importance of context engineering? Has anyone at this point understood that there is a knowledge cliff between the public knowledge space and the business knowledge space? No. The whole thesis of my book, the thesis of almost all the work that we're doing all the time, you, me, all of us, right, when it comes to working with AI, is providing it all the information it doesn't have, but these off-the-shelf solutions, these subscriptions that are purchased, don't have that information, and they're not systems being built yet. First of all, not even understood to be a need, widespread enough, right?

(05:43): We take for granted how much we understand about it now, but the world at large is not awake to this. Originally, I don't know what this term means, but in the course of research, again, I'm like, "Oh my God, this is awesome. This is a symptom of the world not knowing really how AI works yet," but it also makes sense that this is exactly where we're at. In a way, age appropriate behavior for the business world to be right here right now. I think of it as a great way to introduce the topic. You could be all nerdy about it, like me, and say, "Well, you know, the LLMs only know the public stuff, and the private stuff, it's your job to teach them about it." Okay, fine, but you could also just say, "Hey, if your employees are bot-sitting, not only are you paying the subscriptions, not only are your employees hating this, for good reason, but they're also probably less productive than they were before it."

Justin Mannhardt (06:42): Absolutely.

Rob Collie (06:44): You're forcing them to use AI to boost their productivity, but you might actually be getting the exact opposite, while also pissing them off, while also draining them, and while paying these subscriptions. It's a triple loss. Yay.

Justin Mannhardt (07:02): Triple. Triple L. When you were talking about this phenomena called bot-sitting, think about what all the research labs are saying right now. I had a talk last week where I was explaining this. There's a lot of research coming out that's basically showing AI activity is now table stakes. 98% of companies are reporting they have AI and they're doing something with it, and enterprise spend is going through the roof on AI tools, and you see this one story about, AI use is increasing at an alarming rate. Then there's this other narrative about sentiment towards AI getting worse. People are getting more afraid of it, more upset about it, and then in the middle, there's always been this narrative about, the original report was the MIT study about how many projects are failing. There's still this dilemma of lots of people saying they're doing things with AI, very few people understanding what that's meaning. I think these things happening, when someone finds themselves in a bot-sitting situation, they're like, "WTF?"

Rob Collie (08:21): Mm-hmm, and it's clear to them that they're less productive in these modes. You can't tell your managers this, because the managers have been given the edict, "Be doing AI," so the way to speak to this problem is to address the business leadership community and wake them up to the fact that they're actually getting negative productivity, or certainly, even if it's still positive, they're not getting anywhere near the ROI that they could be, if they had the right systems, the right custom systems wrapped around these things to feed the right context and all of that.

(08:55): I was all ready to talk about this. I've done a few interviews. I don't know if any of those pieces have gone live yet, and these will be things where, when they do, if they do, I'll be like one or two sentences. It's not going to be some sharp thoughtful angle so much as it'll be just sort of a reporting about, "Here are the things we're hearing from different corners." It'll be articles like that, but on one of the calls, I get on and the reporter says, "Yeah, so let's talk about software engineers complaining about bot-sitting." I was like, "Whoa, whoa, whoa, whoa. Wait a second. Everything I've been reading has been talking about sort of knowledge workers doing this." She'd come in with it as developers, and so again, for a split second, there's this imposter syndrome. "I'm not prepared for this," but then I go, "No, I can talk about this." This is also really fascinating. Developers are complaining about bot-sitting as well. Developers who are being forced to use AI to write code are apparently really hating it.

Justin Mannhardt (09:58): Do tell.

Rob Collie (10:01): Again, I'm reacting to this in real time on the interview. Kind of new information to me, but then it starts to dawn on me. "Yeah, of course," so I talk about this in the book, and then I recently went through this section of the book and I was recording the audiobook version of it, and I was starting to regret what I'd written a little bit, because I'd sort of described in the book a developer persona that is reluctant to adopt AI-based coding. I wrote that April, maybe.

Justin Mannhardt (10:32): Okay. A lot's happened since April.

Rob Collie (10:34): Here in August reading it, reading my own work, I'm like, "I think they're kind of coming around on this, and I probably wouldn't write it this way," but then I get on this reporter call and, no, yeah, 100% this persona is real and is still, if anything, reaching a crescendo now. I talked to Nathan, our all-in AI architect dev here at P3. Talked to him afterwards about this, because I ended up talking about him quite a bit with this reporter, and he confirmed what I was thinking, but I think he had the best lens on it, so I'm going to use his lens. His lens was, two kinds of developers in this world.

Justin Mannhardt (11:14): Oh, I love binary choices.

Rob Collie (11:15): Yeah, I know. I know. Of course, it's a spectrum between the two and all that, right, but hyper-logical and hyper-creative. Which thing are you more addicted to? Are you more addicted to the logical thinking and being in the zone of crafting code and all of that, or are you more addicted to the feeling of creating something cool? If you're the second kind, you love AI powered development.

Justin Mannhardt (11:44): Agree. Yeah, I can see this.

Rob Collie (11:46): If you're the first kind, it's the end of your identity. It's taking away the thing that you enjoyed doing the most. It's taking away the thing that made you special, that made you feel valuable, and turning you into a manager, which most people of that ... Not all of them. Honestly, some of the most successful executives at Microsoft also turned out to have originally been some of the most incredible software architects, developers. The archetype of, they're really good at code but not with people, turns out to not be true. The outliers are exceptional, and some of the best ones were some of the best developers. They're just super, super, super smart. They can apply that to everything, but anyway, almost ironically, the better you were at development, the more likely you were to be amongst the elite of the developers, the less likely you are to want to work this way.

Justin Mannhardt (12:48): I can see this.

Rob Collie (12:50): Because, the hyper-logical ones were really the ones in the end that really separated, within a large software organization, anyway. At Microsoft, the ones that were really, really, really good were the hyper-logical ones. You wanted the hyper-creative ones on your team because they could go and do something that no one would think to do. When it was time to architect the kernel of Windows, handle device drivers or whatever, oh my God, you do not want the hyper-creative one. You want the hyper-logical one, and Nathan's one of the rare examples who was really, really good at it, at development before AI, and then still embraced it. Anyway, so bot-sitting apparently can also mean just kind of being incompatible with the way things are going to go, because there's only so many jobs available for the hyper-logical.

Justin Mannhardt (13:42): This is a fascinating comparison between the creative and the logical minded. A very similar thread that's come up in some of the conversations I've been in recently is, the greater your expertise is in a given discipline, the more resentment you have towards AI in that discipline, in general. We're talking about software development. We had a great conversation with someone who's built a wonderful career in more creative, brand type spaces, but a similar kind of, "I would never want to use AI for this because it invalidates the thing I am elite at." AI, it's this exceptional digital phenomenon that's been creative, but if you are truly elite, it is hard for you to experience or accept that it is at your level, I think, for some people, and so you're giving me this menu, Rob, I think, of, there's flavors of bot-sitting.

(14:43): There's the employee who's just being told, "You've got to figure this out," and they're like, "What am I supposed to do with this thing?" Then there's the sort of, and not elite like elitist, in the negative way, but truly elite, master of craft, 30 years experienced level folks that, maybe you're right. Maybe this is just, it'll be hard for them to come around to it, and I can see why.

Rob Collie (15:08): It continues to subdivide, because a professional developer, there's a certain productivity output measurement. How long does it take you to write this chunk of software? With the creatives, it's more like, how long does it take you to come up with something, with the one thing that's really, really, really good? The one thing that's really, really good, the production of that one thing is almost instantaneous. I suppose, if it's a painting, it's not instantaneous, but if we're talking about brand taglines and positioning, and Starbucks, that first sip feeling, which is genius, right, is AI a good co-brainstormer for that kind of stuff? I think yes. It almost will never come up ... At least currently, it's not going to nail it. The human still nails it. I have this same experience when I'm writing and I have Eddie in the copilot seat, not capital C, Copilot seat. Again, it shows you how good a name that is, Copilot. What comes out is so much better with this sounding board that I've got.

(16:24): Again, it always wants to go draft something for you, and if you let it, you look at it and go, "No. Even if I wanted to mail it in, no, it's not good enough." This is someone who, by the way, I spent ... I'm the record-holder now. Last week, I spent $500 on Fable tokens at this company. Yeah, an annualized rate of $26,000 or something like that, right? Scale that across 50 people, we're north of a million in tokens a year, and that's after I'm on the premium plan, right? I used my usage. I think that was double usage, too, so it would've been more like 1500 if they hadn't been on the 2X usage. Having a coworker who is valuable is so different than having this new hire, in the knowledge worker case, which is, you're just constantly training this new hire, and by the way, because the longer the conversation goes, the less effective the LLM becomes, right about the time you finally got finished giving it all of the instructions that it needs to succeed at the task is about the time that it starts to forget the first things you've taught it, and so everyone wants to preserve and use those long chats, to reuse the long chats that they've been having, right? It has a built in failure in it.

(17:45): Most of the reporters didn't know that. Anyway, so bot-sitting.

Justin Mannhardt (17:50): A frame, a mental frame, I guess, that I think has helped some of the leaders I've been working with is, you can think about these AI tools in different layers of systems. At the lowest level of that is the personal use system. This is an example of how you built Eddie.

Rob Collie (18:10): Yeah.

Justin Mannhardt (18:11): Eddie's valuable to you. You and I don't use Eddie in a collaborative way.

Rob Collie (18:15): Even this is already personal business use.

Justin Mannhardt (18:18): Sure.

Rob Collie (18:19): As opposed to personal, like, "Help me fix this lamp."

Justin Mannhardt (18:22): Where I'm going with this, instead of me coming and saying, "Rob, you must use ChatGPT as your copywriter," you figured out the way to use and customize that system to work well for you. I think a lot of people, when they get enough time with these, they find the difference between, "This is something that works well for me," and, "This is something that slows me down." As leaders, there is a degree of enablement and freedom I think people need to figure that out at the most smallest level. I mean, they need some coaching as to, "Well, how do you bring business context? How do you connect the tools and all those sort of things?" I think when we come down, it's like, "You'd better use this AI thing for this task because we think it'll do a good job," you might be setting yourself up for a tough road, and then you go out to, "No, we're actually trying to activate a specific workflow," in a particular department or across departments or whatever it is, and then there, context is the entire ballgame.

(19:22): It is the entire ballgame, and I think there are a number of situations where you learn and you discover what is required to not end up in a bot-sitting situation. Then you can have some pretty rational discussions around, "Can we solve that problem of getting this context to the AI, and do we want to and do we need to, or do we think about this differently?" Once you get into the meat of these things, like you say in the book many times over, this stuff isn't necessarily hard to do. It's not hard to necessarily implement all of these things, but it can be a little tricky to figure out what you should do.

Rob Collie (20:04): In the same sense, once you understand it, that context is everything. By the way, context is everything even in the personal use case, right? Because, Eddie, my editor, he has a handbook that gets loaded every time, and then he also has access to a tremendous amount of background files now, that he doesn't always need all of them. This is a very, very comprehensive OneDrive folder with all kinds of sub-folders now, that is Eddie's brain, but even just understanding, the LLM, when you interact with it directly off the shelf, it just feels like such a world beater, like such a super being, that it's really hard to see its boundary of where it starts to suck. You don't step back and go, "Okay." People aren't this scientific, right, and I wasn't either. I had to go on this journey, this vision quest, right, to learn all these things for myself, to come back around to explain it simply.

(20:59): You don't notice when it goes from being a world beater to when it sucks. You just sort of keep grinding at it when it's sucking, and if you could take a step back and go, "Oh, why is this not working as well as it was when I was asking it how to fix this lamp?" Get really thoughtful about it, people haven't reached that level of awareness. The amount that people understand about regular software, even if they've never built any software, if you took everything that someone knew about what software can do, can't do, dos and don'ts, examples of how it works, examples of how it doesn't work, blah, blah, blah, blah, blah, there'd be a 300-page book in the average knowledge worker or business leader's head, and they don't even know how much they know about software.

(21:48): We don't have that yet. Most people don't have that yet for AI, and they don't understand that they need a similar amount of information, and again, all of it's simple. Really, it is. It's this light bulb moment when it starts to dawn on them. It's fun to see.

Justin Mannhardt (22:05): It is. It really is, because then, I think people's ability to decide, "This is how we apply it, this is how we tweak it, this is how we make it relevant to us." I achieved what's my current standard of virality on LinkedIn recently.

Rob Collie (22:21): Okay.

Justin Mannhardt (22:22): Which, I was like, "All right, this post went off. That doesn't happen to Justin." There was a phenomenon going around where if you asked an LLM to pick a number between zero and 100, it was very likely that you were going to get 42 back.

Rob Collie (22:37): Okay.

Justin Mannhardt (22:38): When I saw this, I went to Claude. I went to ChatGPT. I had Copilot at the time when this all happened. This all came out many months ago, this idea, and it just stuck with me, and you talk about similar things in the book, where you really fundamentally break down, the main job of the LLM, it is to match the correct output token to the given input token. The idea is, for whatever reason, LLMs have decided that the correct output to pick a number is 42, and so if you're in your company and someone's like, "Hey, help me with my sales plan, help me with my marketing plan, help me with this process," LLMs are like, "42, 42, 42." That's where it's not meeting the moment, because ... Your lamp example is perfect, because if you're like, "Hey, my lamp's not working," I kind of just want 42. "I have a lamp, fix it for me," or, I had issues with my internet in my house and I was like, "I don't need a Justin Mannhardt specific internet fix. Something's not working right. Tell me what the fix is." I think that idea is, you've got to think about all these contextual things. Otherwise, you're just going to skew to the middle somehow.

Rob Collie (23:58): Yeah. Now that you say the 42 thing, that's what I was seeing in all of our job applications for the developer position. "Tell me something about yourself as a human being." 42. Now, in that case, the LLMs had different favorite answers. For some of them it was 56, but they were only like, 42, 56, 67. There were only a few different answers.

Justin Mannhardt (24:20): Somebody had done a study where they essentially created a loop to ask LLMs over and over and found the most recurring numbers in this pattern. Then it kind of dawned on me, through someone else pointing this out. It's like, "Well, Justin, that's because, Hitchhiker's Guide to the Galaxy."

Rob Collie (24:34): Yeah. I was thinking that the whole time in the back of my head, right?

Justin Mannhardt (24:37): I was like, "Okay, fair enough, but still, either way."

Rob Collie (24:40): I don't care how it arrived at that number. It still had a gravity towards it. In different organizations, it might come up with a different number. Let's close with a topic switch.

Justin Mannhardt (24:52): Okay.

Rob Collie (24:53): Going back a long time now on this podcast, over and over again, you and I have been talking about how data chat agents are in so many ways superior to, go find the dashboard.

Justin Mannhardt (25:07): Agree.

Rob Collie (25:08): Speaking for myself, I would put myself in the 99th percentile of believing this, in the world, and I've recently discovered that I don't believe it enough.

Justin Mannhardt (25:17): Go on.

Rob Collie (25:18): I need to believe it more. For one of our clients whose identity I sadly have to protect, because it's a really awesome story, but I can't spill the tea, we've built a WhatsApp harness for data agents, so all they need is WhatsApp on their phone. They're using WhatsApp to talk to an LLM that has access to their semantic models, and it's a very high stakes business. A lot of money on the line, and the leaders of this company, the C-suite, are just hammering away on WhatsApp, all day, every day, and I've been watching this from a distance, and I said, "Hey, can we give this WhatsApp thing to the hockey league?" Because, we've got the semantic model. It's a Rob Collie over-engineered semantic model. I mean-

Justin Mannhardt (26:08): Let's go. Indie Inline, back in the spotlight.

Rob Collie (26:10): Indie Inline, back in the spotlight. I was so concerned about token costs and all that kind of stuff. I only gave it to Spar. I only gave it to the commissioner. He's busy, et cetera, but this past weekend, I'm like, "What am I doing? Send it to all of the league elders. Let them start messing around with it." Okay. I was stunned by the reaction. The dashboards have been a novelty that people are aware of. I think one or two people might look at them in a given week, whatever, right? I post a screenshot from one of the reports every week on the league page, sort of like stars of the week, one of the pages I made, but that's kind of it. There are people in this league whose reaction to the dashboards was essentially, sometimes polite, sometimes not super polite version of, "Shut up, nerd," who are now texting me screenshots saying, "Hey, it's missing some of our team's games for this season." Right?

(27:18): I mean, it's nuts. Within the first two hours, they had named it Clyde. Jay Clyde's is the name of their dive bar, pregame bar, every Wednesday. This is a sacred place for them. I mean, it is their third space. They name their teams after this place, so for it to be given the name Clyde, and I didn't give it that name. I would never dare, right? For them to do it in the first two hours is, I'm like, "Oh, this is really ... "

Justin Mannhardt (27:51): Okay.

Rob Collie (27:53): Then, the guy that texted me the screenshot of it not having all of Green's games, I mean, he is the peak of, "Shut up, nerd. I don't want to hear about your dashboards." Right? I wanted to reply to him and say, "Yeah, the reason why it's missing some of Team Green, your games this summer, is because you, you prima donna, you keep renaming your team in the middle of the season and it's messing with the ETL, and we need to go and add another mapping for your prima donna team naming," but I don't say that to him. I'm just like, "Oh, you're right. We're on it. We'll fix it." Right?

Justin Mannhardt (28:28): "I'm so sorry."

Rob Collie (28:31): Of course, the same guy, to really underline just how, "Shut up, nerd," he is, I'm going to tell you the next thing he screenshotted and sent me. Okay? We're going to edit out the question he sent. It's completely inappropriate to share on the podcast, but we're going to keep your reaction afterwards. Okay?

Justin Mannhardt (28:48): Okay.

Rob Collie (28:48): All right, so the very next thing he sent me was ...

Justin Mannhardt (28:52): This is a WhatsApp agent connected to a semantic model.

Rob Collie (28:56): Right.

Justin Mannhardt (28:57): This is what we are asking it.

Rob Collie (28:59): Some of the questions on Facebook have been things like, which we can leave in, like, "How many members of the league have gone to jail?"

Justin Mannhardt (29:10): I must know.

Rob Collie (29:11): Okay, but those are the silly examples. The things that made me think, "Okay, as much as I believe this, I need to believe it more," first of all, the dramatic embrace that this thing's gotten. I mean, they are asking it a lot of real questions, and the nature of the people who are engaging with it and how aggressively they're engaging with it relative to how much they engage with dashboards, I mean, it's just an infinite ratio of engagement. Then, the naming of it, right? It's just nuts, but even the use cases. First of all, you've got a business question in your head. You have to translate it into, "Where is the dashboard? Is there even a dashboard?" You don't know for sure that there is one, so it kind of disincentivizes look Looking for it, because it might be an ultimate failure and a waste of time.

(30:03): Then, if you happen to know where the dashboard is, do you still know how to use it? Do you know the right dashboard, but do you really know how to use it? You might not know how to use it, how to drill down in the right way and control click this and shift click that, whatever, right? It might not be so straightforward, but one thing that immediately emerged was these people asking it questions that even I, knowing the dashboards and the data model as intrinsically as I do, I wouldn't want to answer these questions. I would not go and answer these questions, because they'd be too much work. For example, this one guy, Shane, who was the first to reach 1000 points in the league, he asked it a very thoughtful question. I think he was hoping the answer was going to be no, but he said, "Okay, if Brad Denny had started in the league at the same time as Shane Wheatley, would Brad Denny have gotten to 1000 points first?" Even if you have mastery of the dashboards, this is not an easy thing to do.

Justin Mannhardt (31:05): Yeah, it's a tricky calculation.

Rob Collie (31:06): You've got to do a multi-step lookup. You've got to go find, first of all, what was their first game, and how many games per season? You can't even then just do weak math to determine it, because there are breaks between seasons, and the breaks between seasons are not always the same length, so you need to know when they each started, but translate that into, how many extra games of league nights has Shane had, as a lead, over Shane? By the way, then, they don't play the same number of games per season. Brad usually only makes 60% of the games, and Shane makes most of them, so he's accounting for all of this and comes to the conclusion that Brad would have gotten there 10 weeks sooner if they started at the same time. Again, even me with my full knowledge of everything, that is a really bad, ugly, nasty task.

(31:59): Many versions of this have shown up already. Less sophisticated versions, but still, like, "How many times has Doby gone without a point?" There isn't a measure that says number of games scoreless. I never wrote that measure, so it has to go sort of run the query and count by games, and count how many rows are at zero. Right? It's doing so much work on your behalf. I even asked today, and this is one of those where it's so sophisticated that I kind of want to make sure that its thinking is correct, but I asked about this one player, the player who asked the dumb question about jail time on Facebook. I wanted to show him an example of using it the right way. I asked, the guy's name is Sean. I said, "Hey, Sean has always struck me as a really spiky player in terms of his output. Is he more variable than average?" It went and calculated standard deviations. There's no standard deviation measure in the model, and then it went the extra step and said, I forget what they call this in statistics, which is the ratio of your standard deviation to your average. His standard deviation is higher than average for the league, but relative to his average, it's actually, that metric, he's more regular.

Justin Mannhardt (33:11): Oh, sure. Okay.

Rob Collie (33:12): Even though his absolute swings are higher. It's a very, very sophisticated answer, which I'm sure that the guy who asked about jail time is really going to love. I mean, it's going to light him up, intellectually, but these questions that are a degree of slog, even if you have mastery, it turns out they might be the most advanced questions you could ask, but they're not ... In the lifespan of wanting to know things, they happen quickly. They're some of the most valuable questions. They happen soon in your curiosity, and dashboards have never been good for them.

Justin Mannhardt (33:48): No. I don't know that this would surprise you, necessarily, but it surprises people when I tell them, because they know my history. I don't have any dashboards about my business at all. I have lots of data, and I can ask questions like you're describing, but the LLMs, I think maybe two years ago when you and I were talking about Copilot and finding maybe these spots, it was like, "Oh, no, not so sure about this." The LLMs have come so far in that timeframe, and I think this is the thing in analytics that's really interesting compared to maybe just general software development. You ask it that sophisticated question, and if I were to pose it to you, you'd be like, "God damn it, Justin. I'm going to have to go do this and this and this and this and this. How often are you going to really need this?" The AI just goes, "Oh, cool. Got it. Great. Here you go."

(34:48): You bring up a good point. It's like, okay, let's make sure we have some trust in this system and verify that it's working these things right, which is always a risk we've got to be mindful of, but it's pretty neat.

Rob Collie (34:58): That one with the standard deviation is sort of like, okay, now we're getting far enough afield of what the semantic model actually directly answers that you are doing math in your head, LLM. Do I trust that? Yeah, I don't know. You don't hate dashboards, but do you have semantic models?

Justin Mannhardt (35:17): We have the equivalent of them.

Rob Collie (35:18): Okay.

Justin Mannhardt (35:19): Well, actually, that's not true. We do have a Power BI tenant, but we've not invested much effort into it.

Rob Collie (35:25): You have some sort of other semantic layer.

Justin Mannhardt (35:27): Correct.

Rob Collie (35:28): Homegrown?

Justin Mannhardt (35:29): Homegrown. We're using a database product for all that, and we have a contextual explanation of how everything's related, and all that.

Rob Collie (35:38): Are you using any of the OSI folks, the Open Semantic Interchange?

Justin Mannhardt (35:43): We are not.

Rob Collie (35:44): Okay.

Justin Mannhardt (35:44): To be fair, this part makes it sound like ... I am half of a small two-person company. Our data challenges are not immense, but we consume lots and lots of data about what's happening, right?

Rob Collie (35:57): I think it's more a testament to just the nature of who you are. I mean, you've got plenty of data-driven lineage, and you're going to have all kinds of really sophisticated questions that you're asking. It's not that your organization is small that allows you to get away with this. It's that you guys are basically making your own semantic layer. Should you, should you not? I don't know, but you can.

Justin Mannhardt (36:25): Swimming your own risk, but yeah, I think the more interesting point is just this idea of, as someone who, we're cut from a similar cloth in this regard, made a living for a long time making charts and graphs.

Rob Collie (36:39): I worked that into my presentation at the AICPA conference in Vegas in June. I'm like, "I'm a CEO of a company that has built dashboards for a living. I'm telling you, it's not the way." There are still some examples of dashboards that we're going to keep. The idea that dashboards were the way to answer data questions was a lie we told ourselves because we didn't have any other alternative. Another similar question I asked was, "Shane Wheatley got to 1000 points. How much longer would it have taken him to get to 1000 points if Rob Collie hadn't been in the league?" Because, I'm so bad at hockey, and I've played more games against him than with him. The theory is that his scoring average was higher against me than normal, and even though it was probably lower when he played as my teammate, we didn't play enough games together. He made it up on volume, right? Absolutely, that's exactly the case. He got to 1000 points six games earlier, statistically speaking, by virtue of my participation in the league. I did my part, Justin.

Justin Mannhardt (37:40): Measuring contribution-ing is everything.

Rob Collie (37:43): Again, would I have run down that answer with the traditional dashboard? With any dashboards? No, I would not have done it.

Justin Mannhardt (37:51): That's really cool. WhatsApp.

Rob Collie (37:54): That's all it takes. If you're listening to this and you want to kick the tires on the hockey agent, just message me on LinkedIn. I'll give you Clyde's phone number.

Justin Mannhardt (38:03): Oh, man.

Rob Collie (38:04): Justin, you want Clyde's phone number? You don't call him. You text him and say, "What kind of player is Rob Collie?"

Justin Mannhardt (38:08): Yeah. "How many times has Rob Collie been to jail?"

Rob Collie (38:11): Someone did ask. The, "Shut up, nerd," guy did ask, "How cool a guy is Rob Collie?" He asked that of Clyde, and Clyde came back with all the places that I'd showed up in the superlatives database. That's the best it could do. "If you want to know how cool he is, we're going to give you the qualitative answer from the textual database, not from the stats, because we looked at the stats. He's pretty sub-average there."

Justin Mannhardt (38:36): He's pretty sub.

Rob Collie (38:37): Yeah. Well, it's been a good Friday. Good to see you again.

Justin Mannhardt (38:41): Always.