It’s Time to Start Giving Power BI CoPilot a Serious Look (For End User Interaction)

Rob Collie (00:00): We have talked a number of times in this podcast about one of my more recent passion projects where I've really been learning a lot, even in Power BI with the set of dashboards, the data model that I built for indie inline hockey, relearning and learning in some ways new lessons about stakeholders, and even technically speaking is one of the more challenging things I've ever done. Turns out that there were a lot of really complicated problems. The business logic of the questions that you would be interested in about the performance of teams and hockey players led to me some very, very interesting patterns. Don't even get me started on what it's like to power an entire data model based off of PDF files. Yuck.

(00:40): But a tribute to what's possible. Even if your data is that awful. It's in PDF files, and by the way, the PDF files don't even all look the same. They've changed format over the years and everything's just a disaster, and out of that mess comes something so beautiful as this set of dashboards. And they're publicly available, they're on the web. We'll put the link in the show notes like we usually do. But one of the most powerful ahas I've had relating really to everything in our space is again, come from this experience working with the Hockey league. One of the really cool things about having built these for the Hockey league is it's brought me very, very, very close to the consumers of the product. So often when you're working with the client, you're working with the key stakeholder.

Justin Mannhardt (01:25): The owner of it.

Rob Collie (01:27): They're proxying for all of the end users of the stuff. That's just necessary most of the time. You need that efficiency. You can't have 40 or even eight people in the room at all times.

Justin Mannhardt (01:37): Right.

Rob Collie (01:38): As an organization, the Hockey League is a very flat organization.

Justin Mannhardt (01:42): Yes.

Rob Collie (01:44): I was building these on behalf of all hundred people. There was no single sponsor, just me and the consumers. Being asked on Wednesday nights, back when I still lived in Indi, being asked questions like, "Hey, did the dashboards answer this question" or, "Can you make me a dashboard that answers this sort of question?" Was really all at once, both so gratifying. You're in a hockey game, like you're putting on shin guards and-

Justin Mannhardt (02:12): Those people do for fun.

Rob Collie (02:13): Literally a beer league. Being asked questions about dashboards in that environment just felt so good. There wasn't even any money at stake, this community of theirs that they're passionate about. On the one hand that felt awesome. On the flip side, most of the questions were questions that are like, "Oh, I've already built a dashboard for that." They just don't know it, and that's frustrating. My first instinct is like, "Oh, come on, it's already in, it's on page two."

(02:40): I even made a menu, a tab at the beginning to lead off all these report tabs, where you just click on big pictures. It really, really highlighted for me in a way how antiquated dashboards are. They're great, they're not going away, but forcing these people to map their questions in their head into my two-dimensional space of dashboard tabs and also what I chose to call them, the names that I gave these individual report sheets, these individual dashboards. The name makes sense to me, doesn't match up necessarily with the question that's in their head, even though how do you bridge that gap? Well, oh my goodness, has Power BI Copilot really sneaked up on us.

Justin Mannhardt (03:19): In a good way?

Rob Collie (03:20): Yes.

Justin Mannhardt (03:20): Not like in a, "Hey, you scared me."

Rob Collie (03:23): No.

Justin Mannhardt (03:23): And my kids do that sometimes where they sneak up on me and it's a negative. This is a positive experience.

Rob Collie (03:30): Those moments really highlighted for me how an AI LLM style, just natural chat interface is just going to be so powerful and so valuable. People in this Hockey League, if they had that Q&A interface, and I don't mean the old Power BI that was always telling you insightful things like, "Oh, do you know that quantity sold really, really correlates strongly with total revenue?"

Justin Mannhardt (03:56): Thanks.

Rob Collie (03:58): Fantastic. I should just have a natural human question interface, chatbot type interface where I get my answers, and if there's a report or a dashboard that answers those questions, it can take me to them. And if there isn't one, it can still give me the answer because the answer is something that the data model has. I don't have to anticipate every need. And this is so obviously the direction things are going to go, so obviously the direction that things need to go. For a while there, when this need struck me, there wasn't anything to fill the gap.

Justin Mannhardt (04:33): Well, yeah.

Rob Collie (04:34): We could say, okay, obviously they're going to get there. Obviously Microsoft's going to get there, but they weren't there. And then all of a sudden, I mean, it's not perfect, but we're there.

Justin Mannhardt (04:45): Getting close. I remember when Copilot first rolled out, it's been several months, I remember and I know a lot of people that got in there right away also remember being unimpressed. This thing is nowhere near on the same level as the experience we're getting in ChatGPT or Claude or some of these other tools. And so, admittedly I use the Office Copilot all the time and Outlook and Word and all these other things. I wasn't really paying attention to what was happening in the Copilot Power BI other than maybe reading the post. And then I got a chance to play around with the hockey model myself and I was like, yeah, this is in a very different place. Not perfect, but really, really close to being everything we thought we wanted it to be, I think is my fair way to characterize it.

Rob Collie (05:35): Yeah. I would come away saying, A, I'm very impressed. And B, it's gotten better in the last 10 days.

Justin Mannhardt (05:42): Yes.

Rob Collie (05:43): The questions I was asking it 10 days ago that we're giving me unsatisfactory answers are now giving me the right answers today and that, my friends, is a heck of a pace. So you and I have both independently kicked the tires on this thing.

Justin Mannhardt (06:00): Yeah.

Rob Collie (06:01): Your experience with it was this morning.

Justin Mannhardt (06:03): It was,

Rob Collie (06:03): So we're looking at Power BI reports, the hockey reports in the Power BI service.

Justin Mannhardt (06:12): Correct.

Rob Collie (06:13): And it's in a workspace for which Copilot is enabled. So if you go to the publicly available version of these dashboards that's just using published to web Copilot is not going to be on for that, unfortunately.

Justin Mannhardt (06:24): We might get there. We're working on it.

Rob Collie (06:26): I'm really keen on giving that experience to Indy Inline Hockey. We'll learn so much by opening that up.

Justin Mannhardt (06:35): Yeah.

Rob Collie (06:35): Anyway, so let's run through a few of the questions that you asked it and what the quality of the answers were.

Justin Mannhardt (06:41): I put myself in maybe what I think an end user in the business environment. I had never seen your dashboards all that much. I'd seen them when we've talked about them before. So the first question I asked, is what is this report about? And it gave me a pretty good summary of the type of information that was contained in that report and what was in there.

(07:01): Then I started asking some specific questions, and some things that I thought I liked a lot that Copilot in the past couldn't do, and maybe one thing that I saw as a nuance that maybe gets solved or maybe reinforces the idea of Copilot with dashboards. So the thing I really liked is I asked it a question of which player has the highest win percentage who has played at least 50 games, because they're data in there where someone's only played twice and they have 100% win percentage, because they won both of those games. And it nailed it. No problem. It got it exactly right, and I said, that's really great because it can look at that metric and it can filter the criteria against another metric and give me the right answer. That was something that Copilot was not great at early on. I thought that was really cool.

(07:48): The thing that I think I tripped it up on is I asked it who's played the least amount of games and it only gave me one name, but there was a tie. And so, I thought that's a good example where when you get the answer in Copilot, you can click on a reference number and it'll take you to that spot in the dashboard. And then you showed me that later and I said I should have done that because I was thinking, what if I was in a company and I said, who are my top performing people the past quarter? And it just gave me one name and there was a tie.

(08:16): If I didn't go check, I'd be like, "Hey Rob, congratulations." And then over here Kellan's like, "Dude, I did the same as Rob." I think that was a call-out I'd make there. And the last thing I'll mention then we can maybe tease some of this apart is I asked it about defining one of your metrics or it was a ratio. I wonder if it answered the question based on the general corpus of gen AI knowledge, not how you defined it in your report. Because it explained that metric to me in very down to earth terms. It didn't show me the calculation, it just explained this is what this metric means. I asked, can you explain the assists to goals ratio to me? And it says the assist to goals ratio is a metric used to evaluate a player's playmaking ability in relation to their goal start.

Rob Collie (09:02): That is amazing. The fact that it uses the word playmaking, that is a sports term. When you refer to someone as a playmaker, that is a synonym for someone who does create assists. This is a really nuanced phrase. I mean, in fact, it's so nuanced, it's so specific to the sports world that if you weren't in the sports world, you wouldn't know what playmaking meant.

Justin Mannhardt (09:24): Yeah, and so the last sentence in the answer is there's a couple more sentences, and this one I think is also equally as impressive, says a higher ratio indicates a player who is more involved in creating scoring opportunities for others while a lower ratio suggests a player scores more goals themselves. Now, I would've expected the Copilot in the past to be like, Rob has defined this as this measure divided by this measure. It didn't do that at all.

Rob Collie (09:47): It is an amazing explanation. It is semantically correct. Again, it's so semantically correct that it's even speaking the lingo of the business domain, if we call hockey or sports a business domain for a moment. I just reran one that I ran 10 days ago because you said the top five players by winning percentage who've played at least 50 games. You said it nailed it. So that was parallel to a question that I asked it in the past, which was who are the top five players by winning percentage in league history, limited to players who have played at least five seasons, and the top five it gave me were incorrect.

Justin Mannhardt (10:24): Why do you think it was incorrect?

Rob Collie (10:25): I don't know why, but the fifth person it gave me in the list had only played two seasons. And when I asked it, how many seasons has that player played? It said he's played 20 seasons. No, actually he played three. He played three seasons, so I just don't know why it was wrong there, whereas now it leaves that player out, but the new top five, it gave me multiple players have changed in that top five, so I need to verify that it's getting that right. But this top five seems much more believable. I look at this list and I can tell these people have all played five seasons. And yeah, you can click on the little numbers and it takes you to the report that it's using to come to that conclusion.

Justin Mannhardt (11:02): Can we just hover on that reality for a second, that Copilot will answer a question, and then give you a reference link to a specific page in the report and highlight the visual that it got the answer from? You mentioned at the top of our session here how dashboards will always have a place, maybe they'll never go away. I've talked about this in the past where there are certain dashboards that I use that have become central to a process that I complete, and those are very easy for me to understand what they do. But then there are dashboards that I have access to that I don't use on a regular basis and I'm unfamiliar with them. In the past, I would poke around trying to find the thing I'm after. So what Copilot can do is it can answer a question I have and then direct me to where it is. It helps me learn what the dashboard does. Amazing.

Rob Collie (11:57): I'm going to say that the jury is out on our top five. Did you go and validate your top five? Did you really go and dig through to verify all five of them?

Justin Mannhardt (12:06): Well, I didn't do a top five. I just did a top.

Rob Collie (12:08): Oh, okay.

Justin Mannhardt (12:10): Like a singular top rank and a singular bottom rank. I think I did maybe three different types of those. It always got a correct answer, but it missed when there was a tie.

Rob Collie (12:21): Yeah. Mine straight up 10 days ago wasn't really listening to my filter, though that least five seasons played. And so, it's given me different results today and it's dropping the one guy that I know shouldn't have been included, but now I'm starting to wonder if it's making a different mistake. Anyway, the other thing I want to emphasize is that the semantic model behind these reports. I have put at this zero effort into the prep for AI.

Justin Mannhardt (12:48): Tell us, Rob, what is this feature prep for AI?

Rob Collie (12:52): It appears in Power BI Desktop and it's essentially like extra instructions for Copilot to lean on when an end user is interacting with it in the way that you and I are describing. There's three different categories at the moment, and even this is brand new. It's in its infancy. So the first version of this is just paring down the data model, in the same way that you want to hide from report view. Not every column or metric or even table is relevant to the end user of the model, and so there are multiple tables, entire tables that are just used to power other calculations, and you would never want those to be used directly for anything. So you basically just get another version of the field list and you go through and you check and uncheck stuff saying these should be used to form answers or not.

(13:38): And again, I think it very, very roughly parallel the answers that you give for what should be hidden in report view. I wonder if it even starts from that. It would be really smart if it picked it up. When I opened it up, I think it already had some selections turned off by default. Anyway, so then that's one tab, but I haven't published that version yet. All of the things that we're dealing with here, we're dealing with just the out of the box, no tuning, no prepping at all.

(14:04): The second feature here is verified answers. In this case, you go into the report, right click on a visual. I haven't done this yet. Again, this is brand new, but it's particular questions, questions like this, this is where you take them. And it turns out that what we've been seeing is that it's pretty good at figuring that out itself, even without benefit of this feature.

(14:27): I think the most interesting piece of the prep for AI interface is this freeform text box that's just all kinds of extra instructions that you can just write in your language of choice. Write it in English, it is like you're having a chat with an LLM. For those of you who've used ChatGPT and custom instructions, you can just give it all kinds of extra context about you,

Justin Mannhardt (14:53): How you want it to respond to you.

Rob Collie (14:55): And you just write that in your normal conversational English, and it's the same thing here. I'm thinking to myself about putting some Easter eggs in because this is all about fun, right? Right. Now you ask, I've tried this again today, who is the best player in league history and it gives up, it says there's content here we can't work with. That's its answer. It actually returns that very quickly.

Justin Mannhardt (15:17): That's an interesting example for what it's worth of best player in league history. So I assume that probably breaks down. It's like, well, based on what? So that would be an interesting breakthrough where the AI derive based on the context and the domain, what contributes to being the best potentially and make some suggestions. Do you think it'll get there?

Rob Collie (15:41): Well, I did ask it in a chat session 10 days ago to compare me, a bad player, with Brad Denny, who he's the best player in league history and his winning percentage shows it. He's got a 68% win percentage and second place is 57. In a way, it's neat that it does pass on that question.

Justin Mannhardt (16:01): True.

Rob Collie (16:02): Would we be more impressed if it tried to come up with it? But anyway, when I asked a question, I actually asked it who was a better player? Me or Brad Denny, it replied and said, "We can compare and contrast Brad and Rob's statistics," and it blah, blah, blah, blah, blah, and it did say things like, "Brad has a very impressive record in this league." It didn't want to go quite far enough, but nudge, nudge, hint, hint, read between the lines.

Justin Mannhardt (16:25): Like passive aggressive throwing shade at you/.

Rob Collie (16:30): But when you just ask you who's the best player, it gives up, and I haven't done this yet, but I'm pretty sure that when I go into that freeform instructions text field and say things like when asked who's the best player in league history or who's a better player or whatever, judging overall player performance, here are some instructions on things that I would weight heavily. Points per game, probably more important than goals. There's things in this model that when you were asking some of your questions, it's giving some very sophisticated answers.

(16:59): I've got some very intricate calculations that calculate things like how much does Player A improve Player B's winning percentage when the two of them are paired together? How much are you lifting your teammates? One of the beautiful things about that league in terms of data is that the teams change and shuffle massively every season. You have the same teammates for like 10 weeks and then 10 weeks later, boom, all your teammates are different, and so you get all these random experiments. And then you also even get more random experiments, because not everyone shows up for every game. You get to see what happens to Brad Denny's teams when Brad doesn't show up.

Justin Mannhardt (17:34): Yeah, we definitely need to work through the prep for AI stuff. We've given probably a balanced set of examples of what we liked and where it struggled. It did a lot that we liked beyond what we're talking about now, so if that's the state of the art, just by turning it on, getting a fabric capacity, turning on, we needed to go do that and then come back and talk about that.

Rob Collie (17:55): Yeah. No matter what, our takeaway is, this is definitely ready enough.

Justin Mannhardt (18:01): Yes, should be used.

Rob Collie (18:02): It is time. Even if at this exact moment that the time of recording, it's not quite ready for production with your end users, I don't think that's at all a certainty. For a lot of people listening, it might actually be ready, but even if it's not, now is the time to be ramping up. It's not going to reach that production level quality, it's ready for it today. By the time you're done ramping up with this stuff, I think it's going to be there. This is not going to be lost time. This is not going to be lost energy, lost effort for you. This is something that is so close, that is so imminent at this point that learning about it now and prototyping with it now, we're 100% in that zone. This is something we should all be doing.

Justin Mannhardt (18:50): Here's my prediction after playing around with it, is I'm going to use air quotes, the absolute worst outcome. You've got a fabric capacity or you get a small one in F2, you go turn on Copilot. Like Rob was saying, you don't do any of the prep feature stuff. You just get it next to one of your reports and you get going. The absolute worst outcome is you're going to feel like we feel right now and you're going to go, "Wow, this is really cool," and your brain is going to start working a little differently. You're going to realize my users can use my reports in different ways. I can maybe tweak some of my naming conventions so it understands what I mean better.

(19:30): Your brain is going to shift into another spot with this, and you're going to be thinking about different use cases and different ways you can equip yourself and your team, and that is so incredibly valuable right now that you have those types of moments where you shift and start thinking about these things differently. And I say it's the worst, that's an amazing outcome. You're going to get on a path quickly of how do I get this in the hands of people that use these reports?

Rob Collie (19:55): Yeah. I think we're going to be basically talking to all of our clients about this in relatively short order versus then reach. All of the investment that we've been making in these Power BI reports, dashboards, in particular, the semantic models behind them. That investment is the prereq to enabling this AI experience. And you've done the prereq. I wasn't building this hockey model, anticipating this AI interface. I mean, I did see the need for it once I was that close to the consumers. Again, that's the exceptional experience here was that I was how close I was to the consumers. It was like six months ago where I said, "Wow, when we get AI chat interfaces over the top of this stuff, it's going to be a game changer."

(20:39): The investments we've been making in Power BI have set the table for just a magical... Well, number one, an AI experience that's going to blow people's minds. If you're sitting out there going, "What are we supposed to be doing with AI?" Well, here's your first thing you're going to get credit for having quote-unquote, done AI. That's a huge thing. Don't ever underestimate that, but also you're going to get so much more value out of your existing investment.

(21:06): It's become so clear to me how much richness is in the Power BI models we've been building all these years, and what a bottleneck the report layer really is. It's only letting some fraction of the value through. It might triple the value of the data models you're already sitting on, in terms of their accessibility, their consumability, how much they actually inform people. I never would've guessed this until the hockey dashboards experience, that the dashboards as cool as they are as Fisher-Price point and click as they are, that they're actually a primitive way to answer questions.

Justin Mannhardt (21:45): Yes. You made me think of something when you're talking about how the Royal, we made these investments in building our Power BI estate models reports, and now we're ready to triple down on that value. I was in a room recently where I was given a talk and there was maybe 200 people in this room, and I asked him, who's never used Power BI, and this blew me away. Three quarters of the room hands went up, and I live in a world where I expect we all use Power BI, right?

(22:13): Okay, if you're out there and you want AI powered data experiences for your people and you've not already done these things, guess what? You can get there. I'm thinking about all these times where we've gone in for the first time with a client and within a few days they've got dashboards that are telling them things they never thought they would understand, or within a month they've got this whole tool set. It's like that's how far you are away from this type of experience also, which is really cool to think about it that way. If you know someone or you're listening to this and you're not all in on Power BI today, that's pretty cool.

Rob Collie (22:51): I haven't felt legitimately excited about a development in this space in a long time. I mean, I'm always excited about what the tools can do. For someone who's new to them... We were talking about that earlier today as well. For someone who hasn't yet taken that step and understood how different their world can be just with Power BI, that has never gotten old, but it's been a long time since some advancement in the software itself has given me this sense of giddy excitement where I want to go back to people, they've already stepped into the bigger, better world. We've already been helping them for a long time, and I want to go back to them and do it again, give them that experience again.

Justin Mannhardt (23:31): Some of my fondest memories working with clients where we had that moment where it all clicked and we could see things across the business and the dashboards and the models were doing all these things, and if you could go back in time and have that same experience, but at the end and they go, and now we also get Copilot. Or I think even some of the stories you've told going back almost a decade when you were still doing things in Power Pivot that were really exciting and helping people unlock all these things in their business. If you did that same moment, but with, "Oh, and here's Copilot too," that's where we are.

Rob Collie (24:08): Another past experience was like, "I can't get my executives to buy in on Power BI." And we say, "Okay, look. Show them a KPI that they really want to know on their phone." And next thing you know-

Justin Mannhardt (24:19): Done.

Rob Collie (24:20): Got that budget to do the Power BI project, right? This is even better.

Justin Mannhardt (24:23): Much better.

Rob Collie (24:24): You want buy-in? I got your buy-in. Here it is. And like you were saying earlier, this experience is the reason to go and build the data models. Even if you don't have them, this is going to sell people on doing that.

Justin Mannhardt (24:36): Oh, it's funny when you bring up executives, because admittedly, we used to use some of the older, not great Q&A visuals and quick insights. We'd use those to wow people, but we got to do all these things in the model and set up the linguistic schema. You didn't do anything to this hockey model other than build it.

Rob Collie (24:58): No. The linguistic schema is in the LLM. It knows. Copilot knows. Just looks at the data model and cross references like the entirety of human history and figures it out. All right, stay tuned. There's more to come here, but yeah, let this be a galvanizing moment for you, dear listeners. This stuff is ready. It's ready enough that we should be meeting it and we will be.

Justin Mannhardt (25:21): If you can go, turn it on. If you need help, hit us up.