Is CoPilot the End of Filters? (Plus a Cautionary Tale)

Rob Collie (00:00): Justin, welcome back. You're still excelling at dodging podcasts with me, right? Getting sick, going on a cruise with your family.

Justin Mannhardt (00:10): Hard to pack the mic and all that on such a trip.

Rob Collie (00:15): Yeah.

Justin Mannhardt (00:15): I don't buy the internet package on this vacation.

Rob Collie (00:19): Yeah, and well, you shouldn't. Anyway, welcome back, glad you're refreshed.

Justin Mannhardt (00:24): Thank you. Yeah, it was awesome. We had a great time.

Rob Collie (00:26): And I thought we'd start today by talking about how angry I am about something good happening in Excel. Then of course we'll talk about Copilot and AI, right? And it's so ironic that I'm so grumpy about something so good happening. Pivot tables in Excel now auto refresh, meaning you change the data in your table of data and the numbers in the pivot table just update just like a chart would, and this is so good and I think it's even the default now that they auto refresh.

Justin Mannhardt (00:58): Indeed, it is.

Rob Collie (00:59): So why would I be so grumpy about something so deliciously positive? This is good for humanity that pivot tables auto refresh and I am grumpy about it. Do you know why I might be grumpy about this?

Justin Mannhardt (01:10): Well, having been the former product manager of Excel, I have a sneaky suspicion this idea is not new.

Rob Collie (01:22): Yeah, well, there you go, good instincts there. In the 2007 release of Excel, one of the missions I was given was make pivot tables more accessible to a broader audience. I had a great team. The good things that happened in that release, pivot tables did become more accessible, a lot better in a lot of ways. They also became more advanced and more capable, more powerful, and most of the credit to that goes to someone who worked for me at the time, Alan Fulting. I don't want to steal any of Alan's glory. I helped. I was part of the discussions. I was there for the whole thing. So I do claim some fair share of credit for how much better pivot tables got, but the lion's share of the credit should go to Alan.

(01:59): However, there were two things that I arrived at, two conclusions I arrived at relatively early in that process that if we've been able to do them, I think would've significantly expanded pivot table usage. Now, pivot tables were only used by 6% of the user base. We had the information on that. Really fundamentally, there's such a powerful and simple tool. They do something relatively simple and valuable that everybody needs. So what were the obstacles to people using them?

(02:29): And so the two conclusions I came to, number one was the name. Pivot table is a gatekeeping name. It couldn't be more designed to scare you off and I wanted to rename them to be summary tables. Holy hell, with summary because they actually explains to you what it does, what it does, and everyone understands what a summary is and it's inviting.

Justin Mannhardt (02:51): Can I make an admission right here?

Rob Collie (02:53): Sure.

Justin Mannhardt (02:54): I didn't understand why it was called pivot until I was introduced to the concept of unpivot.

Rob Collie (03:04): Yeah.

Justin Mannhardt (03:06): And then I was like, "Oh, I see."

Rob Collie (03:09): Yeah. I don't even know if that was even, I mean clearly that is a close cousin of why they named it that pivot table. It might've just been the ability to pivot rows and columns to flip them. That was what they were, which again is part of the pivot and unpivot operation in sequel, et cetera, all that kind of stuff. But epic fail of a name choice. We had OpenOffice coming up as a competitor at the time. We didn't know that. We didn't have Google Docs coming yet. And OpenOffice, they named their pivot tables Data Pilot, which still tells you nothing about it, but at least it sounds inviting.

Justin Mannhardt (03:44): Sounds cool.

Rob Collie (03:44): It's a thousand times better than pivot tables, so I'm like, "Look, the number one thing got to the number one thing we got to do is change the name. The number two thing we got to do is make the model refresh so there isn't this refresh button, they just need to feel like a chart in the grid." And I was blocked.

(04:00): I was told in no uncertain terms that we really couldn't pull either of those off, and I campaigned for months to change the name and at least weeks for auto refresh and was told no. So neither of those important changes got made in pivot tables, they're still called pivot tables. But to see auto refresh arrive and me not have anything to do with it, Justin, the pettiness, right? If I'm not allowed to have nice things, no one can, right?

Justin Mannhardt (04:31): Right.

Rob Collie (04:32): Anyway, I'm told, and this does make sense to me, that given that they're in continuous release mode now, when I was there, we were still releasing a product every two years, every two and a half years. And so if you were working on something, it either needed to be complete at the end of two and a half years or you have to completely rip it out because it's a risk. The half written code can't be in there. Now, their new model, the way that they operate now, they can work on a feature as long as they have to. If it takes them five years to get it done, that project can continue. They can put it on hold, they can come back to it, et cetera. And in other words, they now have all the time in the world to do something that annoys me.

Justin Mannhardt (05:14): Gift to humanity, annoyance to Rob, I think we'll split the difference and call it a win.

Rob Collie (05:20): I begrudgingly agree. I haven't even done it yet. I haven't even gone in and seen it work, I just know I'm going to be upset. But okay, the left brain knows this is a good thing. The emotional side of me, the lizard brain is we'll just try to ignore that part

Justin Mannhardt (05:37): Next time, or maybe you've already done this, but next time you talk to Brian, just tell him, "I told you so," and then move on.

Rob Collie (05:43): Oh yeah, I mean I have been texting with Brian and Dave about this, and you know what it got me? You know what it earned me? It earned me nothing. But the other thing it earned me was a week later, Dave, both these guys have been on the podcast, Brian Jones and Dave Gaynor. Dave Gaynor is an old mentor of mine. Brian Jones is an old colleague and peer of mine. Brian is back running Excel again. Dave is not running Excel. Dave is higher up in Office doing other things.

(06:13): But Dave texts me a picture of an error message in Excel that's something along the lines of tables don't allow this when you have multiple sheets selected, and he sends me this picture of this screenshot with the message like, "Epic fail, Rob."

Justin Mannhardt (06:32): Still your fault.

Rob Collie (06:33): And he's right that this is my fault. When you have multiple sheets selected, you can't do something to it. I think you can't insert a column into a table when you have multiple sheets selected. Many people after me in Excel have also failed to fix this. Anyway, by opening up the door and saying, "Goddammit, people, you did the thing that I was told I wasn't able to do, you bastards." That just opened the door for me to get attacked.

Justin Mannhardt (06:59): Right.

Rob Collie (07:00): By Dave for something that I failed to do properly apparently many years ago. Be careful waking up the past.

Justin Mannhardt (07:10): A cautionary tale.

Rob Collie (07:11): It does come back to haunt. All right, so let's transition into the thing that we're actually going to be talking about today, which is going back to Copilot as the front end to our data models as the front end to our reports and sharing a three-part journey of, one, an epiphany for me anyway, of yet another way in which this stuff is going to be better than what we've been used to. I'm rediscovering all of the implicit assumptions I had about dashboards that I wasn't aware of, but were wrong when I have a new lens to test it with. That's a positive message about AI front ends over our data, a way that it's going to be better. That's part one.

(07:55): Part two is a cautionary tale of how these things, there are risks, and there are standards that the software needs to reach that it hasn't yet reached.

(08:06): And then the third part is how we can already here at P3, we are already, and other people have been doing this as well, how you can preview where Copilot is going to get to by using other approaches. You can see the future In a way. The next version of Copilot isn't in any sort of preview that anyone's got their hands on yet, but we can roll our own preview. We've been learning a lot from that.

(08:32): Okay, so back to the hockey model, my hockey league. I keep using this example because it's one again where I have such direct access to the stakeholders. The end users of it are my friends. I know their personalities, I know the culture, I know everything about. It's just so rare to have such a direct connection to an audience of dozens of people and know what they're after and know how they're wired. Again, I'm not even living there anymore. This data model and Copilot have become a reason, a social conduit for me to become more involved with these friends across the country and it's really awesome. One guy, Joe Mulvey, the other night had a nine point game. Points is goals plus assists, and I wondered where that ranked all time.

Justin Mannhardt (09:24): It's a good game.

Rob Collie (09:25): Nine points is a lot of points. My best game is three points.

Justin Mannhardt (09:31): Good for you.

Rob Collie (09:32): Achieved twice-

Justin Mannhardt (09:33): Proud of you.

Rob Collie (09:34): ... in my 80 something games. For the record, these LLM interfaces to the data model are becoming increasingly comfortable answering the question with the answer that Rob is not a very good player. It's correct, it's correct in that assessment anyway. If I have a dashboard that is greatest game performances of all time, I'm not sure where it is. It's not on the main menu of, I've got probably 20 report tabs in that pivots, but I've only got I think 12 of those report tabs linked as icons on the menu page. And like most people, I can't be bothered to scroll that damn list of tabs, I'm just not going to do it. This is one of the first examples ever. My usage of Copilot with the hockey model up until this point has been trying to explore its capabilities in some cases trying to stump it, but in other cases, asking it questions that are more nuanced than the types of things I would normally try to do with the dashboard.

(10:37): Getting some really surprisingly, we've documented a lot of these results and how funny and surprising they are, and really mind-blowing at times. This is the first case where I had something that can very, very, very clearly be accomplished with the dashboard. It's a layup, but yet it was going to be way faster and easier and lower cost to me mentally to just ask it. Where does Joe's nine point game rank all time? Remember, I'm the person who built the model, I'm the person who built all the dashboards. I'm as close to this as I can get, and I'm discovering a relatively simple example where the fastest and easiest thing for me to do is to ask Copilot, and it's really cool.

Justin Mannhardt (11:21): Which is a very different place from viewing AI as the hassle. When you have a high degree of skill or ability or familiarity to I'll just do it, for you to be in that place is a good example of someone with a lot of reps in, whether it's Copilot or any other AI scenario, you realize that that is the most efficient path you want to start with.

Rob Collie (11:46): Yeah.

Justin Mannhardt (11:46): It's really cool.

Rob Collie (11:47): It's something I've been trying to emphasize on LinkedIn and some of my posts lately, which is that this with data front-end experience I think is going to be a really, really great cultural on-ramp for organizations because rather than thou shalt use AI, which is scary, this interface actually takes something that was formerly scary and formerly difficult and humanizes it. You can just come up to it with the question that's naturally formulated in your head, right? Deconstruct it for a moment. What do I need to do to get that answer? I know implicitly how to get that answer. I go create a table visual that included player name, number of points, but crucially, game date and time, and then sort by points. But most people don't think to put, it's such an obscure concept to put the game date on the table in order to enforce the game-by-game uniqueness so that each row in the table becomes a game. That is a cognitive leap, that's too much. It's too much to ask non-data people to even begin to think that way.

(12:56): I don't even know if I've built that report or not. It's either in there or isn't and I haven't even bothered to validate it, and so this really, really, really, this really encoded it in me right off the bat that we really re-emphasize that take something that is hard and inhuman today that we unfortunately thought wasn't hard, that we thought was human. Dashboards, they're a lot harder than we thought.

Justin Mannhardt (13:22): Oh, yeah.

Rob Collie (13:23): And make it more humane. And that will help people get accustomed to AI and it'll help them think of it as a friend and a helper and not a threat and not scary. That will help an organization onboard its people to more usage, other usages of gen AI. I think this is a really, really powerful example. So then there was a follow-up question though. Some small number of years ago we implemented a rule in the league that was meant to reduce showboating. The first rule of Indy Inline Hockey is don't be an asshole.

Justin Mannhardt (13:58): It's a good rule.

Rob Collie (13:59): I wish it was encoded in Latin on the banner. Spar, if you're listening,

Justin Mannhardt (14:03): That's a good idea.

Rob Collie (14:04): I know that there is Latin on the banner, but there needs to be Latin that says, "Don't be an asshole."

Justin Mannhardt (14:09): I'd buy that jersey.

Rob Collie (14:11): Oh my God, would I. So we implemented a rule that was meant to limit showboating, that basically it's a goal cap. It's how many goals you're allowed to score. I think it limited at three or maybe five. I forget what the exact rule is, but it's a flexible rule, so if the game is close, the limit isn't in effect. If your team's winning by so many goals and you personally have so many goals, you're not allowed to score anymore.

Justin Mannhardt (14:39): Pass the puck, baby.

Rob Collie (14:40): Yeah. We call this the Wheatley rule. It was named for Shane Wheatley after he scored too many goals, a blowout effort, right? Shane, if you're listening, you know this is true.

Justin Mannhardt (14:53): You're ruining the fun.

Rob Collie (14:55): Yeah, I mean Shane's a really, really good guy. I really like Shane, but it's funny to have a rule named after you that, anyway. Since that rule's been implemented, it has been harder to score nine points in a game.

Justin Mannhardt (15:07): Makes sense, yep.

Rob Collie (15:07): It's harder to pass, it's harder to accumulate assists deliberately. A large number of assists of assists is hard to accumulate deliberately even if you're one of the best players in the league. You can pass a lot, but now you're at the mercy of a lesser player to score the goal. If it's me on the other end of the pass, I mean forget it, you're not getting a point. So you have much more personal agency if you're one of the best players to score goals. If there's a goal cap limit in place, it's harder to score this many points.

(15:42): I asked it, I had to ask Ryan Spire to asked the commissioner of the league, "When did we put that rule into effect?"

(15:48): He came back and said, "Summer of 2023." I wanted to know where did Joe's nine point game rank within the Wheatley rule era.

Justin Mannhardt (15:57): Games after that date.

Rob Collie (15:59): Yeah. Well, and the thing is, you said after that date, I don't know what the date is. I don't know what the date is on the first game of summer of 2023. I don't want to know.

Justin Mannhardt (16:11): Oh, right.

Rob Collie (16:12): But my seasons and games table do have a column in there that has summer of 2023 as the label on certain seasons, and then that dimensionally decomposes into dates and everything. Think about it this way, even if I wanted to construct this dashboard, first of all, I'd have to have that other table visual that I talked about building before, which is a pretty big cognitive lift for the average person. But then I'd also have to add a filter.

Justin Mannhardt (16:37): You'd need a new column in your games dimension or something.

Rob Collie (16:41): I mean, I think I have all the columns I need, but here's the thing. I wouldn't be able to put the season description on there and say greater than or equal to. It's a text field.

Justin Mannhardt (16:53): Right.

Rob Collie (16:54): I need to turn it into greater than or equal to a date.

Justin Mannhardt (16:57): Yeah, you need Wheatley era yes, no.

Rob Collie (17:00): I know, that's right. Exactly, right. I could say is, on, or after as a calculated column. You're right, totally, the model could have that feature.

Justin Mannhardt (17:09): But again, if you're the end user, you're not doing that.

Rob Collie (17:12): You're not doing that, right? But even me, if I don't want to crack open the file and add the calculated column, I have to go look up the first date in summer of 2023. Was it in May? Was it in June? The calendar does shift. What a pain. So instead I just go to the chat interface and say, "Hey, we instituted this rule, the Wheatley rule in summer of 2023. Where does Joe's performance rank in the Wheatley rule era?" And it absolutely understands the question. It's almost like the user is allowed to just temporarily inject additional information into the data model that's not available to the data model.

Justin Mannhardt (18:03): Right.

Rob Collie (18:04): It's an off-camera thing, an off-camera change that wasn't captured in the data model. And this happens all the time in business. We made that product change at that point in time. Yes, in theory we could have encoded that into the calendar table somehow. But there's so many things like this that will never make it into the data model.

Justin Mannhardt (18:22): These are the things that if they do get action pre-Copilot, they turn into like, "Oh, we'll add some columns or we need different measures, or we got to build a new report page with different filters on it." And it's just again, examples of things that end users do not, cannot do themselves. They're reliant on the authors of the assets to do that.

Rob Collie (18:45): Even smart people in business who haven't spent some time training their brains around all this stuff, they have a hard time even formulating the goal. They're not sitting there going like, "Oh, if only I could add a column to the model." They're not even getting-

Justin Mannhardt (19:00): Yeah, they don't think that way at all. At all.

Rob Collie (19:03): This is a question that just simply would not get asked is what would happen. It just wouldn't get asked and it certainly wouldn't get answered. I'm taking some information from another source, putting it into my question and letting the LLM process that and it's like an augmentation on the fly of the model and it's really, really cool except.

Justin Mannhardt (19:28): Oh.

Rob Collie (19:29): The damn thing lied to us. Okay, so it got-

Justin Mannhardt (19:35): Wait a minute, you're saying AI made a mistake?

Rob Collie (19:39): This is the second time I've fallen hook, line, and sinker for a hallucination. I'm not proud to admit it, but I did. Only temporarily, but temporarily was enough. The thing is it got the first question completely right, where does Joe's performance nine point performance rank all time?

Justin Mannhardt (19:57): All time.

Rob Collie (19:57): And it's basically tied for sixth. There's been 10 point games and 11 point games, but there's enough people tied with 11 and 10 that it demotes the nine point games to six, so tied for sixth. Then it comes back and says that in the Wheatley rule era, his nine point game ranks first with no ties, he's alone at nine points, which is totally believable. This was my whole hypothesis, right?

Justin Mannhardt (20:24): Because nine points, it's a lot.

Rob Collie (20:25): Which is that it's harder to do now and there have been 20 something seasons, so we've only had maybe six seasons since summer of 23. The vast majority of goal scoring would've happened pre-Wheatley era, pre-Wheatley rule. This fits my priors. I'm just so excited about it. I'm so excited about the experience of it.

Justin Mannhardt (20:45): Right.

Rob Collie (20:45): Just the experience of this being so much easier for me and also knowing that it's a thousand times easier for the average business person because of this, but it's even easier for me. Even I am just like, "This is so much better."

(21:03): So I'm just exuberant about the whole thing. And I text a few people in the league, including Ryan Spar, I texted them pictures of the chat and Ryan was so excited about it that he went to Facebook, logged in with Indy Inline Hockey.

Justin Mannhardt (21:16): The misinformation machine is just off and running.

Rob Collie (21:22): But this happens. But again, it happens so fast. 10 minutes later I'm back at it saying, "Okay, who's second in the Wheatley era?"

(21:32): It gives me two people tied with eight points. And this time I noticed that it's referencing a report page. It did the first time too, when I asked it where his nine point game ranked in the Wheatley era, it gave me a reference, the little one to a report, and I wish I clicked it because the second time I saw it, I clicked it and it took me to a report page that is this season, current season, the leaders in certain categories in the current season. This is a report page that I made, and this is one that's on the menu that has a page wide filter baked in which is is current season equals yes. So it's only showing the most recent season.

(22:14): And sure enough, there are two people with eight point games in this season and those are the two that it's talking about. And I'm like, "Well, that's quite a coincidence, isn't it? That our two second place games also happened this season. This doesn't seem quite right," and it's not. So when I go do it the manual way, the way that I didn't want to do, set the filters myself and all that kind of stuff, I find out that there has been a 10 point game and an 11 point game in the Wheatley rule era. Not only does Joe's game not rank first, he's not even tied for first. In this case, and I want to stress, this is the current version of Copilot. We're recording this episode on August 28th, 2025, okay?

Justin Mannhardt (23:00): Starting.

Rob Collie (23:03): This is a point in time where this version of Copilot was able to get confused by the existence of this report page. I could not in subsequent questioning ever get it to enforce the Wheatley rule filter that I wanted, and not limit to the current season. It would either limit to the current season, or it wouldn't enforce the Wheatley rule filter. Over and over again, I was even telling it, guess what? I went and I looked and that's not true. There's a game, here's a game for the 11 points. There's a game with 10 points and it's like, "Oh, thanks for pointing that out."

Justin Mannhardt (23:42): You're absolutely right, Rob.

Rob Collie (23:43): Right on. And then still, it just had target fixation on this report. It couldn't escape the gravity of this report. It wasn't aware of in some way, properly aware of the fact that there was a filter is current season that would impact this. A little scary, a lot scary. This is not the thing that can be allowed in production. And you can just imagine if I can be fooled by this for 15 minutes with real world like low-

Justin Mannhardt (24:18): Low stakes, real world, yeah.

Rob Collie (24:20): This is another reason why the hockey model is such a great place to be experimenting, and if you want to be stepping on landmines, step on it in the hockey model.

Justin Mannhardt (24:28): Right.

Rob Collie (24:29): But there were real world implications of getting this wrong even for a minute. Most people aren't me and wouldn't ask the follow-up question necessarily and then get suspicious.

Justin Mannhardt (24:39): Yeah, there's a few interesting things going on here. One I find particularly ironic, I'll come to that one in a second. To your point about the little footnotes, the ones and twos and threes it has in its answer where you can click on them and go to the visuals. We were talking with a client yesterday about Copilot and just reinforcing we're going to need to get user behavior into checking those things for these types of reasons. You just know how much of a pain that's going to be, to your point of you going to believe it.

Rob Collie (25:10): I think, and actually, now that you say that, it just needs to take you there.

Justin Mannhardt (25:14): Yeah.

Rob Collie (25:14): It can't wait for you to click, it needs to show you if it's leaning on something. We need to see it.

Justin Mannhardt (25:20): It's such a subtle little footnote, I think a lot of people are going to be like, "What is this? Is this like my collegiate thesis paper? What are we doing here?"

Rob Collie (25:27): Yeah, there aren't any thumbnails. It's just like these little one with a-

Justin Mannhardt (25:30): Little tiny one.

Rob Collie (25:31): A little black one with a square around it. No, it needs to be, "Please click here to validate."

Justin Mannhardt (25:38): And the other, so this is the ironic, interesting thing is that you've explained so well how dashboards were a constraint on the conversation and how AI is going to open those constraints. It's interesting in this example how it was able to understand what you meant by the Wheatley era, but it was also itself constrained by the report in a very damaging way. And I mentioned this and when we were talking about this offline, Copilot has this interesting behavior where it will seemingly be very grounded in an existing visual or report page, and then flip to just running its own query against the model to get answers at times.

Rob Collie (26:23): Yeah.

Justin Mannhardt (26:23): I think understanding that behavior is going to be really important for us going forward because we want that openness in the conversation, but at the same time, why would the LLM necessarily know what that filter meant or was doing? Hopefully maybe some of the improvements that we'll be able to reason through those things. I've had some of my own experiences with Copilot where filters, the idea of filtering or the presence of filters can be a bit of a stumbling block. For example, I was working with some of our other demos and I said, "Tell me about sales in this region."

(26:59): Okay, great, and then tell me about what the top performing category in that region is. You can trend that over time for me, but it loses the filter of the region I was talking about in the next answer. It's stuff like that where I think we're going to need to be very aware, especially in this early, I think a lot of this is going to get a lot better fast, but filters have tripped us up for years, Rob.

Rob Collie (27:24): Yeah.

Justin Mannhardt (27:25): How many times have you had to call someone and say, "Hey man, you got to take that filter off on the right?"

Rob Collie (27:29): Right. Yeah, many. We even had an instance of that in the past 14 days in marketing. Filters can't live with them, can't live without them. They're just so hard for users, for business people who have questions to translate them into filters. It's just like, "Why did we ever think that that was natural?"

Justin Mannhardt (27:50): And not even just simply filters, but someone gets, let's say they get a first answer. They look by looking at a dashboard, they're like, "Well, I want to know the next question."

(28:00): It's like, "Okay, is that on a different page? Do I got to drill through? What do I got to do?"

(28:04): I like the Copilot taking you there behavior a lot.

Rob Collie (28:07): Yeah, I do too. But the thing is, I'm wondering how much we might in the end just have to choose. I haven't tested this, but I'm pretty positive that if I just took all the reports out of that Power BI model and just had Copilot running against the bear model, I wouldn't have had any trouble. I think it would've gotten everything right because, again, it got the initial question right. I guess maybe I'm giving it too much credit. Maybe it wouldn't have been able to translate the summer of 2023 thing into a greater than or equal to filter. I think it probably would have.

(28:42): But anyway, so we want the benefit of prebuilt visuals, when you trust them, the visuals that Copilot conjures in the current iteration, half the time they're completely useless, but that's going to get better. I still firmly believe that human beings who actually understand what's really going on with the situation are going to be able to produce better visuals to explain things like for known cases anyway. We want to keep that benefit, but we certainly don't want those reports, those carefully crafted reports, dashboards to be, I hate that Microsoft, every time I say the word dashboard, I say report. Every time I say the report report, I say dashboard because that's a greater evil than pivot tables not auto refreshing.

Justin Mannhardt (29:29): It's important to maintain perspective folks.

Rob Collie (29:31): Yeah, it is. If those things can be a distraction, if those things can be blind alleys, that Copilot gets caught in and results in bad answers, then not worth it.

Justin Mannhardt (29:41): Right.

Rob Collie (29:41): It has to be right is by far the first criteria. This is why we want this system is because the Power BI model itself is such a trustworthy system of record. It getting distracted by a current season report is just not acceptable. If you were going to deploy a Copilot in production today, I would think that maybe the only "safe" way to do it would be without any reports. To be clear right now, you could probably start adding reports back in, but you need to start from zero, and each report that you add in, you have to be very careful that it doesn't have, how do you spot it, what kind of nuances can be in that report that might trip it up?

Justin Mannhardt (30:23): This experience you've gone through with this sort of passion project is an example of how you want to approach it on your own. Ask questions, find out where it falls down. When we've been talking with clients, we know over the years we've done just creative things, hack arounds to make different behaviors possible and reports, and they involve things like the filter pane and hidden filters and synced slicers and all this kind of stuff. And so, okay, how much of that was done with good intention and it was necessary to get that desired result at that time, but has now become a limiter for Copilot. Those are some interesting conversations that we're having and trying to reach conclusions about. To the point you've always had, we're still going to want dashboards, we're still going to want certain things. So what's the balance between these experiences and how do we rethink about our catalog of content to best service our team or our organization?

Rob Collie (31:29): And so there's two avenues in which Copilot is going to get better. One of them, the obvious one is they're just going to upgrade the LLM models that are working on it, that are behind the scenes. As I understand it, the current version of Copilot has a very one-lane road supreme edict that it will return an answer, it runs to the end of that road and returns the best answer it can come up with, which is not what you want. You want it to be able to explore multiple pathways and reason over them and all that. And I believe all of that is coming, and that in itself might be enough to fix the problem that I had this week.

(32:13): But the other thing that they can do to improve Copilot is just normal software engineering making the information about the report more available to the model, to the LLM. If the LLM had known and understood that there was a filter set on the seasons table that was restricting all of this, it would've had the opportunity at least, and I got to believe that either that information wasn't available to it or it didn't ask, right?

Justin Mannhardt (32:46): It's definitely already extended because a lot of that what you're describing is already available for embedded scenarios where you could understand the state of filters and things like that. So it's surely, it's possible.

Rob Collie (32:58): But how well plumbed is it, right? Because a lot of AI powered solutions, I saw it the other day. It's 5% AI and 95% software engineering. I believe that. You've got to get the right information to the right place at the right time in a format that's understandable. And if you're not doing that, then the AI model doesn't even have the chance to succeed.

Justin Mannhardt (33:21): Right.

Rob Collie (33:22): We've got two parallel tracks that they can follow here. I'm not at all here proclaiming the death of Copilot or it's not going to work. I'm not saying that. I'm not even saying that the only way it's going to work is without reports. There are very, very, very promising near term improvements that they can make, and at least two different dimensions that will improve this. We'll wait and see. But in the meantime, third chapter of the story, we have our own custom chat interface. We've been using a test harness. We have our own ChatGPT style website now where we can have conversations and we can choose which LLM we want to talk to. It's not tied to any of them. We can switch between Google Gemini and Claude and OpenAI models and also has MCP access. We're using one of the community MCP servers to connect to the hockey model.

Justin Mannhardt (34:19): In Power BI.

Rob Collie (34:20): In Power BI. Now this MCP approach, this custom chat interface that we have does not have any access to the report there.

Justin Mannhardt (34:29): Can't see it.

Rob Collie (34:30): Can't see it. Right off the bat, can't be polluted by the report layer. But at the same time, we also are using upgraded more current and more multi path reasoning model approaches with this MCP server to answer these questions. And this version of the tool does not get stumped about the Wheatley rule. It does correctly place Joe as tied for-

Justin Mannhardt (34:56): And that's cool.

Rob Collie (34:57): ... third or fourth in the Wheatley rule era. It gets the all time question right, it gets the Wheatley rule era question, it gets everything right. If I'd only use that from the beginning, we wouldn't have had the bad Facebook post go out. And so this allows us to get a bit of a preview of at least the part of Copilot where it gets access to these multichannel, multi path reasoning approaches.

Justin Mannhardt (35:27): What's neat about what you guys did there with hooking up from our website through MCP is, yes, it can give you a glimpse of, well, how well does something like a reasoning model perform when it's using a semantic model in the experience? I think we both agree and are excited that Copilot is going to get a lot better and it's going to continue to get better for a long time.

(35:53): But this other idea, it's more advanced for apps, it also opens up possibilities, well, what if I want a solution that can talk to my data and talk to other things, and do other things. Copilot's going to keep getting better. And so then you can imagine a larger and larger audience of people gaining value as end users interacting with reports and data sets with Copilot.

(36:14): But then we have these other AI applications where we want to combine our business data with our unstructured data with other things and be able to do things like take action or integrate all this into one type of experience. MCP is a really interesting technology in that regard, allowing us to consider those types of options as well. There's a lot of possibilities out there, and I think it's fun to be able to explore this stuff and kick the tires. I think the thing I'm really excited about is just the reality that semantic models continue to be super valuable and important in this space. We can leverage them in new and interesting ways.

Rob Collie (37:00): I'll tell you about a couple of things that you haven't seen yet that are hiding in our private AI agent playground here at P3. One of them is the MCP powered interface to the hockey model that does not get distracted, that gets both of the questions correct.

Justin Mannhardt (37:15): Yes.

Rob Collie (37:15): Yay. We've got the podcast, Oracle is coming up to speed in that environment as well. We talked about this a long time ago on this podcast about trying to load every podcast episode of all 200 plus episodes into a system so that an agent could answer all kinds of questions about things we talked about on the podcast. That amount of information is too much for an LLM to absorb. It just won't take it. All these hybrid approaches using vector semantic search, which is really cool, and hybridizing that with keyword search and then hybridizing that even further with another kind of machine learning re-ranking of results to get us just almost like God-like search over everything that's ever happened in our podcast. And it's not even search, it's like based on things we talked about in the podcast, what would P3's opinion be on-

Justin Mannhardt (38:07): New ideas, conclusions, opinions?

Rob Collie (38:08): Right, and so it can semantically digest all and find things that would be relevant and then use those to help generate an answer, or anyway, we're doing exactly that now, the thing you're talking about. Some of these agents now have access to multiple different data sources. You take the podcast Oracle stuff and you combine it with the hockey stuff, the hockey Power BI model, and we have an agent that has access both of them, and there's not a lot of cross-pollination there. There's no synergies there between those two, really.

(38:41): My friend Jamie, who is the wizard behind all of this, my old friend Jamie, who's working all this kind of stuff with us, he has a sense of humor and he enjoys ribbing his old friend Rob. He has created an agent in there that's just called Rob. The Rob agent has access to a lot of things. It has access to the podcast archives, the podcast Oracle stuff. It has access to the hockey model, it has access to the databases of instructions about our P3 brand and who we are and what kind of services we provide and what kind of clients we like and how we're differentiated and what our personality is like, but also my writing style. This thing now tries to pretend to be me when it answers questions. It also has access to, he's given it access to an NFL database and a database of Disney reviews, customers who-

Justin Mannhardt (39:40): So basically, he's taking all the little tests that we've been doing.

Rob Collie (39:44): That's right.

Justin Mannhardt (39:44): And he's throwing them all together in one and called him Rob.

Rob Collie (39:47): All in one place, and called him Rob, right? Because Rob, I'm clearly walking around with encyclopedic knowledge of Disneyland reviews, right? Anyway, so this really, really fantastic experience the other day of seriously, just based on these reviews, he's asking the Disneyland review database if there's any patterns in bad reviews and for advice on how Disney might address it. Man, its analysis and its recommendations are just, they're really, really, really credible. Now, I know if you work at Disney, maybe you'd look at them and go not, I don't know. It actually seems right, okay.

(40:24): This was so funny, this was happening because at the same time he was sharing this chat log with me. I was texting with some friends out here in Seattle who were talking about how, I mean, I'm not even kidding. This was in parallel at the same moment, I'm sitting in the chair with my phone in one hand and the laptop in the other, and these things are happening exactly the same time. They're texting about how they like to go to Disney a lot and how they also want to go to Europe.

(40:45): And I'm suggesting to them, "Oh, well, you should probably go to Disneyland Paris."

(40:49): And then 60 seconds later, I'm looking at this chat log and it's saying that Disney Paris owns all of the worst reviews. There are more negative reviews of Disney Paris than there are of all the other properties combined, even though they represent just a small percentage of the overall reviews in the database.

(41:09): So then I'm like, immediately pick up the phone and text them back and say, "Well, maybe not Disney Paris."

(41:16): But then, because Jamie was live chatting with it. So I said to Jamie, "Ooh, ask it if," these are the kinds of follow-up questions that you want to ask. The negative reviews from Paris, to me, they just sounded like the experience of being at Disney. It was too hot. We're standing in line for too long.

Justin Mannhardt (41:35): Too many people.

Rob Collie (41:36): Right? I grew up in Orlando, and so Disney's not a novelty to me anymore. I go to Orlando to visit family, I don't even think to go to Disney. I'm in Orlando and I don't go to Disney.

Justin Mannhardt (41:45): Right.

Rob Collie (41:46): It does not compute to most people. So I wonder if maybe just Europeans haven't been indoctrinated into the art of going to a theme park and standing in line all day in the heat. Maybe they're just too smart. They're just not going to be-

Justin Mannhardt (41:58): This is a horrible way to recreate.

Rob Collie (42:02): Maybe they've got it right, and it's just Americans that have been indoctrinated into this.

Justin Mannhardt (42:07): And so why is this fun?

Rob Collie (42:08): So I told Jamie, "Okay, ask it if Europeans are rating it lower than Americans, because Americans are going there."

(42:16): And remember, this is the Rob agent. The Rob agent comes back and says, really harsh on my hypothesis, "That's a dumb theory."

(42:24): It doesn't say that, but it's like, "Let me tell you how wrong that theory is."

Justin Mannhardt (42:30): Okay, that's-

Rob Collie (42:34): "Because the Americans all rate it really low too. UK visitors rate the Disneyland California way higher than Disney Paris, et cetera. So no, trust me, Disney Paris sucks," is what Rob comes back and says.

Justin Mannhardt (42:49): I like Rob.

Rob Collie (42:51): That's what I said. I said, "Fake Rob was really harsh to real Rob's theory."

(42:55): And Jamie said, I don't know, "Fake Rob sounds nice to me."

(43:02): Every day things just seemed to get clearer to me. Yes, I fell for a hallucination step on the landmines in a safe environment. Don't wait to discover those later. That's a lesson, I'm not going to need to learn that lesson twice.

Justin Mannhardt (43:16): There's so much change in the way we think about things. Just a great many things when you introduce AI to it. I go back, what, two, almost three years now to when I just started using ChatGPT, and it was a process to understand how to use that effectively in a day-to-day capacity. Or we were talking with Jamie earlier and he described the process. He's been a software engineer and he's gone through this process of understanding how to leverage AI to do more and different things. I think the same is true here. There's this process of understanding where's the technology? What is it good at? Where are the stumbling blocks? Okay, we imagine those stumbling blocks go away. How do we adapt to this? This is the journey we're on.

Rob Collie (44:07): And that is the truth, and we also should be careful not to expect too much from others. This has been quite a bit of a journey, and it's not done even for people like you and I who are wired for this sort of thing to begin with. And also, frankly, it's our jobs to make sure that abreast of this stuff, so we're exceptional in that regard. And so to expect the rest of a workforce to just figure it out, it's not even appropriate to think of it as, okay, we're just ahead and everyone else will be slowly grabbing this. There's some truth to that, that everyone's going to be slowly catching up or whatever.

(44:51): But this is why those on ramps I think are going to be so important, safe, low hanging fruit type scenarios. And again, they've got to be safe. They can't be giving you wrong answers. Holy hell, that's not okay. But seriously, this can be so much easier and so much more humane for people to interact with than dashboards. It's going to solve a problem. It's going to remove a negative that they already experience. They're scared of dashboards. They have to be. I am so wide awake to that now about how terrifying dashboards are, and I never would've thought that, because I came from the world of Excel where dashboards are way better.

Justin Mannhardt (45:30): Dashboards, the greatest thing ever.

Rob Collie (45:31): Way better than an Excel pivot table, grid-

Justin Mannhardt (45:34): It doesn't refresh automatically when you-

Rob Collie (45:36): Maybe it doesn't, yeah, I know. Well, once we got to Power Pivot, it wasn't going to refresh automatically anyway, so who cares?

Justin Mannhardt (45:42): Yeah, who cares? Talk to me.

Rob Collie (45:44): When the data model in Power Pivot refreshes automatically. Imagine the recalc. You're waiting and waiting, and waiting and waiting. These on-ramps, these cultural on-ramps that serve as familiarization and training, but as a side effect, familiarization and training as a side effect of them gaining a benefit that just pulls them forward. It's going to sell itself, and so we should all be looking for examples like that that help bring people along. That journey you're explaining is real. You just can't expect the majority of your org to walk it solo. You can't be, again, we've made fun of this now a million times, but you can't be the Fiverr CEO, just telling everyone to figure it out or you're doomed. Bad leader, bad.

Justin Mannhardt (46:31): Boo.