Episode 200: The Road Ahead for Data and AI

Rob Collie (00:00): Hello, friends, and welcome to this, our 200th episode of Raw Data. Justin was out sick this week, which leaves me flying solo for the anniversary, but it did give me the opportunity to sit back and collect my thoughts a bit after a hectic couple of months of learning and experimentation. In fact, having to face the microphone alone kind of forced me to do that, and it resulted in probably my longest solo pod yet, so buckle up. I really enjoyed putting this one together. I sank a lot of time into it and I hope you enjoy it and find it valuable as well. I asked LinkedIn in advance of this episode for suggestions on what to talk about on this milestone episode, and the theme that resonated most was what is the future of data work? A.K.A., what should we be doing as professionals to future-proof our careers in the face of AI? A worthy question indeed, and I'm going to tackle it from the perspective of business leaders and data practitioners because increasingly the answer to the former drives the answer to the latter.

(01:02): But first, let me tell you a short story about something I did yesterday. So my wife, Jocelyn, and I have become what we would call almost like lifers for our gym, Orange Theory Fitness, OTF. We're like unpaid brand ambassadors at this point. We're closing in on each having done a thousand Orange Theory classes each. If anyone from corporate OTF is listening, hey, hit me up, I want access to the full data feeds you all collect from our heart trackers, okay? We'll make headlines. And you also tend to meet people at OTF. You see the same regular people over and over, and eventually you get to talking, which is how I found myself sitting down at Starbucks yesterday after the workout working on AI with a man who is at the top of the real estate game here in Seattle.

(01:45): More than just a real estate agent, he's tightly connected with banks and other institutions holding some prominent official roles. And while he's well respected and well-connected within his industry, real estate, he is definitely not a techie. He's a little older than I, somewhere in his 60s, and he has about average computer skills, like for example, he doesn't know the CTRL+C and CTRL+V shortcuts for copy and paste for instance, but he does put together amazing videos, really movies actually, that showcase certain properties he's listing and he puts those videos on YouTube and his website. And most importantly, he's 100% open-minded and always eager to learn.

(02:22): So after he mentioned to me the other day that he'd had some early success with ChatGPT helping him write some things, I said, "Hey, let's sit down after the next workout and really get serious." So we spent 90 minutes yesterday training an LLM on his specific business, his specific branding and differentiation as a professional, his specific workflows, because as I've mentioned in previous episodes, that's where the magic of generative AI really happens is when you teach it about you, about your business, and even about hyper-specific workflows and contexts within your business. These LLMs kind of come off the shelf knowing everything about human history, but completely ignorant about you and your business, and it's a little tricky teaching them the right things, giving them access to the right information and making it stick so you don't have to do it all over again every time. Now, the net result is I've never seen someone more excited than he was yesterday after we got some basic plumbing set up for that stuff.

(03:21): Remember, I'm in the line of work that I am today because of how excited people get because of the enormous amount of human happy that I was seeing when we helped people see their data in Power Pivot and then later Power BI. That degree of excitement and happy led me to launch P3 Adaptive 12 years ago, and what I was seeing yesterday probably exceeds that level of happy. In fact, I would say it definitely exceeds that level of happy. Now to be clear, this was a personal grade solution. I mean, this is 90 minutes in a Starbucks, right? It's not like in 90 minutes I could be pulling together something for organizational use. It's got rough edges all over the place, it doesn't have a security model, it's not integrated into other tools that he's using. It still requires a fair bit of copy and paste just to give it the inputs it needs. But all that aside, it is still magical.

(04:14): For example, after we taught it about his personal brand and approach, I had him pull up a real estate listing for one of his currently active properties, a nice condo here in Seattle, and my thinking was, "Hey, let's rewrite that three paragraph description of the condo and the listing," but first I asked him, "Do you like the description that's already there?" And he said, "Yeah." He looked at it and read it and said, "Actually, this is wonderful. This is a wonderful description that we've got there." So I changed gears and instead said, "Okay, then let's use this description to further train your new digital assistant on how you like your listings written."

(04:48): So we did, and that went into its data bank brain for future usage. But then I said, "Okay, now let's just ask it if it has any suggestions on how that listing might be improved." Now remember, he had already said this listing was wonderfully written. 10 out of 10. No notes. But what it came back with when he saw it, he was like, "Oh my God. Wow. That is so much better and it sounds like something I would write on a truly inspired day. So let's save that." And so we created a new database on the fly to store write-ups of listings.

(05:20): Now, of course, the LLM has to be a good writer by default, but it only worked because it knew at that point what he was like, how he approaches things. And then we moved on to training it to help him write personal proposal emails. So for example, if he sees a property that was listed and didn't sell or isn't selling as fast as he thinks it should, he'd like to reach out to the seller and offer his services, and this takes a lot of time and energy. But 15 minutes later, we've got this thing in a state where you give it the listing as well as some human interest inspiration coming straight from his brain like, "Wow. They lived in that house for 26 years and did such a great job maintaining and updating it," and boom, it gives him a customized email that he can send that includes all the right links to past movies he's made, showcasing other properties, et cetera.

(06:07): And it's still human, it's still him. It's not auto-generated AI slop because we've taught the system so much about him and his approach, and because we're still injecting the human element into each individual assignment we're giving it. He's himself, but he's able to be himself at a scale he hasn't been able to unlock before now. And he's stoked. His life is changed by just this 90 minutes. Now, we'll come back to this story later because it does help set up some of my points in the bigger story.

(06:40): So back to the question at hand, what's the future of data work and how do we future-proof ourselves as business and data professionals? Of course, I don't claim to know all the answers here, but it is my industry, it is my job, and every day the picture does become clearer in focus. I'll share where I'm at today. First of all, let me say that the term future-proof feels kind of like a defensive stance. It's like how to avoid obsolescence. I get it. But I think a shift you need to eventually slowly make is to change to a more forward-leaning posture and say, "Instead, how do I take advantage of the changes in this space? Not just how do I maintain my current level of opportunity in this shifting landscape, but how do I turn this into more opportunity than I had before?" That's not an easy mindset shift to make because if you don't know how to turn this into more opportunity, you quickly get nervous and fall back into the defensive zone.

(07:37): So I get it, and it took me some time to start to see the matrix, so to speak. I'm happy to say that I'm definitely in the forward-leaning zone today, but I do remain sympathetic to that defensive zone because I was there before. It's okay. It's going to be okay. And of course I would be remiss here if I didn't now also say that we at P3 Adaptive would very much like to be the ones to help you make it okay. So hit me up on LinkedIn or on our website and we'll talk.

(08:05): Okay, with all those preambles out of the way, let's dive in. If you're listening to this podcast, you probably have some sort of association with Power BI. So let's start there. What does the future of Power BI look like as business leaders and as data practitioners? Well, if you've listened to recent episodes, you know where I'm going to start. Far less time is going to be spent on dashboard production because AI front ends, particularly Power BI Copilot with its chat interface are going to replace some large percentage of pre-built dashboards. It's not going to replace all pre-built dashboards, but I'd say like 75% of pre-built dashboards at minimum, and that is a reduction in work. That does reduce the amount of time that you need from skilled intelligent people to do this sort of thing.

(08:56): I don't know what it looks like overall, but if we spend 75% less time building dashboards, what percentage of the overall Power BI work is that considering that the semantic model underneath still needs to be built? I don't know. I haven't done the research yet. I'll get to a number pretty soon. Now, on the flip side, our semantic models are going to be held to a much higher bar. Free form unpredictable usage via AI front ends is going to be very good for the business because it does provide unfettered, convenient and unintimidating access to business data in a way we've never had, but with the intermediate dashboard layer stripped away, it's almost like our data models will have lost like a protective membrane. Those poor data models are now going to be naked, exposed and unprotected.

(09:41): The dashboards used to control what sorts of questions could be asked. No more. It's going to be open season now in a good way, again, because this is what business needs, but whatever sins we've committed in the past in our data models and gotten away with them, well, no, not anymore, and even "worse." The way in which questions are going to be asked itself is unpredictable. See the recent episode, The Dobie Moment, if you want to know more about that. Oh, and then don't forget that it's not just going to be humans asking questions, but other AI agents will be able to formulate their own questions for your data models. So while we're going to be spending less time building dashboards, we're going to be spending a fair bit more time on our underlying semantic models, making them robust in the face of the full spectrum questioning they're going to be expected to handle.

(10:33): And you can be preparing for that now. Whether that means auditing your existing Power BI data model estate, or getting started with your first Power BI project ever, either way you'll be preparing for this bright new future. There's plenty of work to be done to reap the benefits that are coming. I don't have a lot of certainty here about the relative magnitudes of the shift, but I think this probably does result in more overall Power BI work rather than less. Let's just say for a moment that the increased focus on the models probably roughly offsets the reduced focus on pre-built dashboards, because I think that's at least close to the eventual truth, but the fact that these models become more valuable than they were before means that demand will expand. Even in organizations whose Power BI adoption seems relatively mature today, there are plenty of corners of the business that have yet to be instrumented or at least not instrumented well, in part because training the relevant stakeholders and users to use these models and reports seems like an uphill battle.

(11:36): Making them log in on the PC because no one builds mobile layouts, and then making them almost like become data people by using a dashboard for the first time in their lives. I mean, you're trying to almost turn them into different people sometimes, but if they can just pick up their phone and speak into it and then see a chart showing them precisely what they asked for, that meets those people, that basically meets all people where they already are. And again, the users here don't even have to be human. They can be other applications, other AI agents. So the addressable surface area for Power BI goes up and the ROI of every dollar spent goes up. Now, when that kind of thing happens in society, we always end up making more of that thing than we do less of it. So if dashboard creation goes down and model attention goes up, offset each other, but overall demand for semantic models goes up, and that brings us to an overall increase in total Power BI-related work.

(12:40): But that brings us to the next big question. If the total amount of work is going up, how much of that work is going to be done by AI versus by humans? Now, that's the question that I think most of us are most loathe to ask out loud. If you're a business leader, you're probably secretly hoping the answer is a lot, like a of the work can be done by AI, and if you're a practitioner, you're hoping it's well not much, right? You don't want it to be much of the work that can be done by AI. Now, it's kind of interesting that we haven't yet arrived at a confident answer to this question as a community. On LinkedIn just this week I saw Brian Julius interacting with Marco Russo, and those two represent essentially the opposite ends of the spectrum stance-wise. Brian's calling DAX a solved language, and Marco is saying that the newly released ChatGPT-5 still doesn't even understand the language. These are super, super smart people, both of them. It's fascinating to watch.

(13:39): And of course we all bring our own personal biases into this question. Brian's current professional identity is built around AI, and Marco's is built around DAX, and they don't have to even be unconsciously tipping the scales in the direction that they want things to go. It can also just reflect their respective levels of expertise in each. I have no doubt whatsoever, for instance, that AI struggles to meet Marco's high bar for generating DAX, but I also have no doubt that Brian is capable of coaxing more out of AI than Marco is. These are the 99.99 percentile experts in each of their respective focuses, and we can sometimes learn the most from such exceptionally talented perspectives, but also sometimes we can be deceived by them because here's the thing, the world doesn't run on the 99.99 percentile.

(14:32): For example, the 98th percentile is our hiring bar at P3, and that's a population that's already 200 times the size of the 99.99 percentile at population. 98th percentile is super, super elite. It's a much higher bar than the ones at which our competitors hire. The broader Power BI world probably runs at the 90th percentile, even that's maybe being a little generous. So what's going to happen in those broader contexts, not the 99.99? What about the broader real world rest of the world context? That's where things matter. Okay, here's my take. AI is just really, really good at writing code. I see no reason why DAX is somehow sacred. I'm pretty sure it's already good enough at DAX to essentially make Brian correct and that it can match or exceed at minimum the 90th percentile Power BI talent today, and it's still likely to get better even if incrementally. And also just as importantly, maybe more importantly, it's going to become more and more closely integrated into the tools themselves, like able to catch and correct its own mistakes and reason its way through things, all that kind of stuff.

(15:39): AI can go a long way towards writing DAX and building data models and things of the sort without ever matching, even close to matching really what Marco can do, but maybe that's not the bar we need to hold it to. Now, my belief is that 90th percentile technical work, if we narrowly define it as technical work, that's in trouble. It's in trouble in slow motion because the world is very, very slow to adjust to changes like this. So when I say it's in trouble, like 90th percentile technical work isn't going away today or tomorrow or the next day. It may even take years even though the AI is ready already today. But if your career is narrowly defined as I'm 90th percentile-capable at taking specific instructions and delivering some sort of code formula or script to address those instructions, yeah, I think that sort of career now has an uncertain but relatively short shelf life.

(16:32): That still leaves a lot of questions unanswered. For instance, does it stop at the 90th percentile? Does it reach P3's 98th percentile? I think the answer is at least partially. We're already getting faster at our jobs here. We're already doing less busy work. We're already puzzling for shorter periods of time over difficult formulas, and that's going to keep accelerating. It's going to be a while before we reach steady state and see what our business model looks like in this new world. Can we increase the number of accounts we service without increasing headcount to match, and if so, by what factor? Stay tuned.

(17:09): And just as importantly, does the bar go up for what constitutes good Power BI work? I think absolutely it does because as I said, the models we build are going to be held to much, much higher standards than before when both humans and robots alike are cut loose to question them as they see fit. So would 90th percentile Power BI work even be good enough if AI could not replace it? Is what constitutes even 90th percentile work today going to be good enough? Maybe not. Maybe our 90th percentile team will need AI to help us deliver like the 99.99 percentile work is going to be the standard. We're seeing this already in other industries.

(17:53): My son is about to graduate with a computer science degree at probably the worst point in history to do so. The big tech firms have really cut back on junior software developer hiring. Mark Zuckerberg has publicly said they don't even need junior to mid-level software engineers anymore. Yikes. But notice that when he says that he's leaving out the senior engineers, why are they not thinking they don't need them anymore? Because there is an invisible barrier of diminishing returns in all of this when we're talking about replacing skilled thinkers with AI. There's still the interaction with the human plane. Even if we wake up tomorrow and LLMs are suddenly able to produce DAX at Marco Russo level and even Marco says, "Oh my God, it understands it now." Who is operating the AI which is building these Power BI models? It's still going to be a seasoned technical thinker of some sort.

(18:49): For instance, at Starbucks yesterday, we were creating text databases to store the training instructions on how the LLM should write on behalf of my friend's business, and if we'd let my friend drive that process or let the LLM drive that process, we would've ended up with something unwieldy and likely to break down in the future because by default it would've been cramming everything into a single database, which would be messy, but also would have eventually reached the point where it was flooding the AI's context window, leading it to become less effective and more forgetful. So I was thinking several steps ahead about the underlying structures and how they match use cases and all the trade-offs lurking therein. In that process, I didn't hands-on create any databases. I had the AI do that dirty work for me, but I was orchestrating the whole thing, and that was tapping into two critical parts of my brain. One, the technical architectural side, and two, the human factors side. Which brings me to my next big point.

(19:52): I think the line between data work and software work is now blurring in a big way. The thing I was doing yesterday for my Orange Theory friend was not building dashboards. I was essentially helping him build a custom line of business application. He's going to use it to generate output and interact with the broader world, and we're likely to even expand those capabilities over time for him as well. This is software, and yet it is powered by, guess what, databases. In fact, the databases are the unique asset that makes the whole thing go. If he wants to switch from Claude Sonnet, the current LLM he's using, and start using a completely different one, like Google Gemini, we just have to point Gemini to the same training and instruction databases. The DNA that makes this whole thing fit his business is all just data and all portable.

(20:49): For another example of how the line between the two is blurring, when we're thinking about Power BI Copilot over the top of a semantic model, we're needing to think much more about user workflows than we've ever bothered to think about them in the dashboard game. What does a day in the life for a typical user really look like? We're going to have to do our human factors homework, which is much more akin to the world of software design than it is to traditional dashboard work, even though admittedly these two things should have always been more similar, but we know that 99% of the time dashboard design was never really that thoughtful. Should have been, but in most cases it wasn't. And then additionally, when those same semantic models are hooked up to agentic systems as part of a broader and more automated workflow, wow, we are really in the software game then, aren't we?

(21:38): So let's take stock of a few big takeaways here. Number one, the total amount of work in the Power BI space is probably going up, which does increase the demand for skilled thinkers. At the same time, number two, the speed at which we can do that work is also going up, which decreases the demand for those same skilled thinkers. So number one and number two in this list are going to be in tension with each other. We're going to find out kind of where that all sorts itself out. Number three, lower degrees of skill, which are of course the most common degrees of skill are the most at risk, both because AI is better at replacing lower degrees of skill, and because the bar is going up on what constitutes good Power BI work. Four, on net, I think that's going to concentrate Power BI work in a smaller fraction of the workforce than the population who performs it today. The rule here is sort of like be exceptional or be something else.

(22:34): Number five, the worlds of data related development and software development are converging, so everyone needs to start thinking more in terms of the verb of software and less in terms of the noun of dashboard. Number six, that means that both technical, architectural thinking and human factors research and design are each going to become far more important in the data world. If you can hit 98 percentile in those dimensions, whether as a data practitioner or as an organization, you're going to be one of the big winners.

(23:09): Now, with those six now recapped, let's add one more: A seventh key takeaway. Just like the line is blurring between data work and software work, AI is blurring the line between structured and unstructured data. Years ago when I worked on the Excel team at Microsoft and I was trying to recruit talented product managers from other teams in office to come and work with us, I had a pitch I like to use. It went something like this: Sure, you can go work on Word or Outlook, and that's very appealing, I know, because you're already an intensive user of those apps and you understand them quite well.

(23:44): Yeah, Excel is a deeper product than that, and so it's a bit intimidating. I get it, I really do. But Word and Outlook are conveyors of content, sentences, ideas. Those sorts of things don't go through a computer CPU and become better. There isn't nearly as much runway to improve those applications as there is Excel. Again, this is still my old pitch, right? Structured data, says the pitch, can be infinitely improved in mind with the help of a computer, whereas Word, oh, come on, they did the red squiggle underlying spell check feature in 1995. That's their magnum opus. They will never again reach those heights, but we hit new heights every month in Excel.

(24:27): Now ironically, that pitch almost never worked, even though it was 100% true. The intimidation of coming to work on something so deep and as unknown as Excel was always too much, and people always wanted to choose that safer path. One notable exception, of course, is that former Word product manager, Brian Jones, who's been a guest on this show. He's now on his second stint as head of product for Excel. So I guess I can claim a moral victory there, even though I can claim really no part of recruiting him for the role.

(24:59): Now, why do I tell that story? Because it's no longer true. I couldn't really use that same pitch today because AI has turned unstructured content, sentences, concepts, ideas, email exchanges, meeting transcripts, white papers, ad campaigns, everything, AI has turned all of that into data. GPU-backed LLMs can now do things with that kind of information that we used to only be able to do with structured data like databases and CSVs. So you should start thinking of content, unstructured content. You should start thinking of that as data. It's just that when it's time to do something structured with that unstructured content, we're going to be using LLMs backed by giant farms of GPUs rather than the CPUs we're used to using.

(25:51): And by the way, the CPU still reigns supreme for structured data. Oddly enough, GenAI is terrible at ingesting and analyzing structured data, even though it's lights out amazing at unstructured. It seems like the latter, unstructured, is the harder problem, and it clearly was harder because it took us much longer to get there. But getting good at unstructured required a completely different approach than the ones we took to structured data, and as a result, AI can't even always accurately count the number of Rs in the word strawberry, even though it can accurately tell the difference from context between strawberry shortcake the dessert and Strawberry Shortcake, the cartoon character, and then furthermore, it can tell you that strawberry shortcake, the dessert, is often paired with whipped cream and Strawberry Shortcake, the cartoon character is often paired with another cartoon character named Huckleberry Pie. It can do the math on meaning, even though it can't do the math on math.

(26:49): But yeah, now we have tools for both. We've long had the tools to treat data as data, and now we have the tools to treat content as data, and when we intertwine the two into new systems, we can create things that we frankly don't even have the muscles to fully imagine yet. But it turns out that piping unstructured content into AI systems starts to feel really familiar really quickly. It's a lot like piping structured data into analytical systems. The same sorts of problems just with fun, new twists. Storage, storage layout, fast and accurate retrieval, getting the right information to the right system at the right time, writing out new content, whether structured or unstructured.

(27:32): Technical thinkers who are also able to bridge the world over into human factors and who are able to design creative software solutions powered by a mix of structured and unstructured data. Whether you plan to be such a person or whether you look forward to working with that kind of person to turbocharge your business, that sounds like a pretty compelling future, doesn't it? Thank you so much for listening to our 200th episode. It's been quite a ride sitting down at this microphone 200 times. Whether you've been with us since the beginning or if you joined relatively recently, I really appreciate you listening and we'll catch you next week.