Should we be skeptical about Microsoft Fabric?

Rob Collie (00:00): Hello friends. Now I know we've been talking quite a bit about this new thing called Fabric, and it absolutely makes sense that it's occupying a lot of our attention these days. When Microsoft says, "Hey, big, bold, new direction," well, you got to pay attention. We're on that map, aren't we? We're on that trail. But we didn't originally expect today's episode to be about Fabric again because we thought we'd said everything that there was to say for the moment. I mean, the thing isn't even generally available yet.

(00:27): But we have been getting some questions, things we haven't addressed. And the first one is, hey, should we be a little bit more skeptical about this? I mean, this is a huge new direction. It is brand new and these are the sorts of things that don't always go well. So maybe we're just a little bit too fanboy about all of this. And so we take a little bit of a harder look, or as I say during the conversation, we're going to look that gift horse in the mouth.

(00:53): But also, and this came up in our LinkedIn group, which by the way, you can still join if you search on LinkedIn for Raw Data by P3 Adaptive. It came up on the group. What about the human element of all of this? Here comes this brand new direction, and we've spent as professionals in this space, both at P3 and in the community, we've spent so much time investing in Power BI. We've been learning DAX, we've been learning M, we've been learning all the ins and outs of data flows and all the bajillion new features of Power BI that come out every month. And then when Microsoft comes along and says, "Hey, brand new direction," that can be pretty scary, can't it? All these skills that we spent all this time acquiring, are they now becoming obsolete? Do we have to start from scratch and climb that mountain all over again, but a different mountain?

(01:33): So maybe we need a new acronym. Instead of FOMO, fear of missing Out, how about FOBO? Fear of becoming obsolete. Well, a little bit of FOBO is good for you, but a lot of it can be really demoralizing. What's the right amount of FOBO and how much FOBO is warranted? So Justin and I talked about all of that. We solved all the world's problems. We might've invoked Monty Python a little bit in the process. That's how you know we're on the right track. Hey, Luke, do you think that's enough intro? Yeah? Well, let's get into it.

Announcer (02:03): Ladies and gentlemen, may I have your attention, please?

Announcer (02:08): This is The Raw Data by P3 Adaptive Podcast with your host, Rob Collie, and your co-host Justin Mannhardt. Find out what the experts at P3 Adaptive can do for your business. Just go to p3adaptive.com. Raw Data by P3 Adaptive is data with the human element.

Rob Collie (02:33): Welcome, Justin.

Justin Mannhardt (02:34): Hey, Rob.

Rob Collie (02:35): So we've been talking a little bit, AKA a lot, since the podcast reboot. We've been talking about this Fabric thing, haven't we?

Justin Mannhardt (02:42): Indeed.

Rob Collie (02:42): It's what the cool kids are doing. They let us talk about it anyway. Even though we talked about it a lot, it's sort of come to our attention that we should actually talk about it some more, and talk about it maybe from a couple of different perspectives than what we've been using. Maybe you can enlighten us a little bit about what those perspectives are.

Justin Mannhardt (02:59): Sure. Well, we've been singing the praises of Fabric, how much we are excited about it, the things we think are going to be very positive. But it's very fair and appropriate to acknowledge when new technology shows up on the scene, there's a lot of uncertainty and skepticism and questions. Is this the real deal? So I think we want to explore some of those things today, and our take on how we're viewing that skepticism and concerns, and what we like as much as what we don't like.

Rob Collie (03:32): So you're saying we should look this gift horse in the mouth?

Justin Mannhardt (03:34): Let's do it.

Rob Collie (03:35): Yeah. Okay. All right. Well, I am by default a techno skeptic. One of the things that I'm sort of, at least behind the scenes, a little bit known for here at P3 is that I kind of hate software.

Justin Mannhardt (03:49): It's true.

Rob Collie (03:50): So for me to discover software that I really like is a big deal. Excel became a deep true love of mine. And then I actually was able to make room in my heart for things like DAX, and not really any room, for Power Query, yes, M, no.

Justin Mannhardt (04:08): That ship has sailed, folks.

Rob Collie (04:10): That's right.

Justin Mannhardt (04:10): Don't send Rob your tips.

Rob Collie (04:13): Yeah, When I need it, I'll ask. But no, I don't want to be proactively learning M tips, which places me in a weird spot here because, to an extent, I'm having to fill in parts of the narrative for myself from incomplete information, just like everybody is. But I see what they're up to with Fabric, and I go, oh yes. I am betting, it's not a roll the dice type of bet, this is a pretty confident bet based on my experience with the people involved, their past success, their past history.

(04:47): And also, believe it or not, my skepticism about software in general is actually an input to why I think they're going to succeed. My skepticism about all the other stuff that they're trying to make easier, I recognize how much that other stuff, despite its popularity or whatever, it's like perceived prestige. A lot of these other things, I'm pretty sure they suck. And so when someone comes along with the track history of making things that suck not suck, I tend to get on board. So I find myself in a very enthusiastic position about this Fabric stuff, which again runs counter to my nature, but it runs very much in line with my experience. I've been to this particular rodeo, I've seen this movie. So if there's going to be skepticism here, believe it or not, it's going to have to come from you.

Justin Mannhardt (05:37): Wow.

Rob Collie (05:37): I know, right?

Justin Mannhardt (05:38): I'll challenge you on this. So one thing that has come up internally at P3, give the audience some context, when we approach a project like a Power BI project, we use a phrase faucets first. We want to start with the end goal in mind and work backwards from there, as opposed to let's build the data warehouse to know where, we've talked about that in the past. And so a lot of what has come into Fabric is, let's say, data warehouse adjacent. It's the storage, it's the ETL. Yes, there's AI. A lot of things [inaudible 00:06:13]. And so you actually said something that I found really interesting just because of how well I know you, and you said it on our last podcast, which was the thing you're most excited about is OneLake. Which for you, someone that's got bit by the DAX bug, and the data modeling and all that sort stuff, I wouldn't have thought the Rob Collie I knew before Fabric to basically say I'm all in on data lakes, right? Maybe talk about that for a second.

Rob Collie (06:37): What a great question. And I am. I was actually gearing up for you to say, you said this on the last podcast, and me go like, "I said that?" But then when you said what you said, I'm like, "Yeah, yeah, I totally said that." We talked about on the Jeff Jorgensen podcast, the Unfrozen Caveman Lawyer, in a way that's me for technology. I might just be a simple unfrozen caveman, but because I don't enthusiastically adopt technology, new technologies, and part of it is that deep down it's hard. It's not like some noble stance. There are some upsides to me being a slow and cantankerous adopter of new stuff. There's some upsides to it, but it's not all good.

(07:23): I've been hearing about data lakes and data lakes and data lakes. For years it was Hadoop and MapReduce and all of that, and it seems to have just kind of come around to a much friendlier name like data lakes, and it seems like things are kind of settling in a little bit. And yet I still wasn't in on data lakes. I don't really understand them. And I'm sure that a lot of people listening are in the same sort of boat. Not you. You get it. To me, you're an alien species. So a very friendly alien species, the kind that we want to have land on Earth.

Justin Mannhardt (07:52): Take me to your leader.

Rob Collie (07:52): You're the aliens we want to have visit. Okay. Not the other ones. But to be told that building a data lake now is basically, it's really just the same thing as building a data model, oh, I get that. Instead of using Power Query to connect to various sources, and pump them into a PBIX file, I don't know about the internals of a PBIX file, I just trust that it's in there. So if you're just telling me that now it's going to be going into OneLake, and it's going to behave the same, I got the schema view, and I can drag and drop and create relationships, and I can add measures, I can add columns. And oh, then I can flip back over to Power Query and I can add some more. And it just so happens that I'm getting essentially a data lake behind the scenes, chef's kiss. That's amazing. A big part of it is that it brings that foreign prickly thing, data lakes, down to my visceral level.

Justin Mannhardt (08:48): Into your realm.

Rob Collie (08:49): Yeah. That's really great. But then the fact that I gained benefits of it's not just like, see, look, I'm building a data lake, ha, ha, I could put that on my resume now. No, it's not that. It's that that format is now central to their whole platform and everything is able to consume it. Whereas beforehand, it was like you're building this most amazing thing, this data model. And then at the last minute someone said, "Oh, and you know what? Only Power BI reports can access this. That's the only client. That's the only endpoint that can access all of your beautiful work is Power BI reports. Everything else you want to do, you want to make a machine learning model," which I don't really know how to do, "Tough luck. You need to start from scratch, and when you start from scratch, you got to use all this other stuff that's Silicon Valley adjacent."

(09:39): And what I call the Linux cool kids stuff. All of that stuff, whatever you want to call it, the Snowflake, the Spark, the Data Factory, blah, blah, blah, blah, blah, it's all just the newfangled wave, in my opinion, of new solutions to old problems. And old IT technology was bad and hard to use, and this new stuff is also maybe a little bit better, but still bad and hard to use.

(10:09): So it makes even my sort of narrow set of tech skills incredibly more valuable. So yeah, I'm all in on that. I don't really have to do anything, or really even learn all that much new in order to gain all these benefits. Even if I'm not the one building the machine learning model, the person who wants to build it can use it. What you're telling me and have been telling me is that most of the work in building machine learning models, for example, has been getting the data cleaned, formatted, all in the same place, improved with business logic calculations, AKA measures and columns. And having to use all of these esoteric pointy headed tools that aren't as good as Power Query and the VertiPaq SSAS Tabular Power BI experience.

Justin Mannhardt (10:57): I remember when we first got an opportunity to start looking at Fabric, you had asked me something to the effect of, "Justin, I really want to understand how our people will benefit from this." And by our people, you mean people like what you were just describing for yourself, Power BI people, dataset builders, our principal consultants. And initially I was in this camp of, ah, but this is all the plumbing stuff. Quick aside, I'm a bit of a plumber. I have that type of background. And good plumbing does matter at the right stage of a project lifecycle.

Rob Collie (11:33): You're a reformed plumber in the same way that-

Justin Mannhardt (11:34): Yeah, I'm a reformed plumber.

Rob Collie (11:36): ... I'm a reformed software engineer.

Justin Mannhardt (11:38): There you go. So I'm evaluating Fabric, and I'm seeing basically Synapse moving into Power BI. I'm seeing data warehousing, storage, data lakes, ETL, Python notebooks, is this going to be big for our people? That question you asked me. But then I realized, like what you were just saying about the things you don't understand in the PBIX file, how much of that was happening. Even though deploying a data lake in Azure isn't necessarily that complex of a task. We have a script to do it. You run the script and it goes. Now you just click a button, and you can teach a lot of people how to do that. But even things for, let's use Python notebooks for example, so if you're going to write a Python notebook, which is a block of code usually used in an ETL process in some fashion or like an AI model, that code runs on a Spark cluster.

Rob Collie (12:40): By the way, we're taking a tour of things that I will never do in my life. I will not make a Python notebook. I will not spin up a Spark cluster. I will just outright refuse. I'm not going to do it.

Justin Mannhardt (12:54): Well, here's what I think is really interesting. So Python is becoming wildly popular, if not the most popular thing for aspiring analytics professionals to learn. Okay.

Rob Collie (13:03): Boo.

Justin Mannhardt (13:03): Boo. Right?

Rob Collie (13:07): Boo. Yeah.

Justin Mannhardt (13:07): We can talk. Well, that'll be a debate show for us.

Rob Collie (13:09): There's another episode.

Justin Mannhardt (13:11): That's another episode.

Rob Collie (13:13): Change my mind.

Justin Mannhardt (13:15): But again, to do this in the traditional Azure stack, you have to also manage the infrastructure. You have to create the Spark cluster, you have to determine its size. Fabric, none of that. You just go. And in Fabric, that cluster spins up in seconds. Whereas in Azure, it would take several minutes for those resources to be available for you to do what you need.

(13:37): So you see that friction come down for me and you go, we can boot Python, but it's a very learnable language. We have a lot of our principal consultants have taken it on even since they've been employed here and become very good at it. They never needed to worry about all that infrastructural stuff now. So I see the plumbing stuff, your perspective, how it's evolved over the years, and my perspective in looking forward, the animosity towards plumbing is rooted in how difficult and long and costly, and people would start there and never have the right goal in mind.

(14:14): To get to this stage with Fabric where it's like, hey, now it's not hard, it's not costly, it's low friction. So it just came into that realm like you were describing what Microsoft did for OLAP, and now they're doing it for this other stuff. It's like we would still be faucets first at P3. We're not saying that, but we can go from faucets first to production stable so much faster, so much easier. And so there's a huge benefit in that.

Rob Collie (14:49): Well, why? So I was expecting you to say, in the new world, in the Fabric world, that one of our principal consultants could operate the same way they always have with just little tiny aesthetic little differences in terms of what they're doing. I'm writing my DAX measure in a slightly different place than I was before, but it's the same formulator and all that kind of stuff. Really, really, really inconsequential differences in terms of what they have to learn.

Justin Mannhardt (15:14): That's true.

Rob Collie (15:14): So they do all of that. And when they're done, they get the same thing that they always got, like a Power BI model, Power BI reports, whatever, except that now the stage is also set for six, seven, eight other things if there's a need for it.

Justin Mannhardt (15:28): 100%.

Rob Collie (15:29): I wasn't expecting you to say that they're going to get to the same result faster. So explain what you meant there.

Justin Mannhardt (15:36): A good specific example, companies at an increasing rate are becoming more and more dependent on SaaS solutions in running their business.

Rob Collie (15:46): You think?

Justin Mannhardt (15:48): Right? Their CRMs, their ERPs, their accounting systems. That's what's happening.

Rob Collie (15:56): That's all we have, right?

Justin Mannhardt (15:57): That's all we have.

Rob Collie (15:57): We don't have any non-SaaS software [inaudible 00:16:00] business software at P3, right?

Justin Mannhardt (16:02): And so the backends of these systems aren't relational databases. That you're typically calling some sort of API or maybe... Power BI has connectors for a lot of these things, but that's not always the case. So it's fairly common that we encounter a scenario where we say Power Query isn't the best at handling an API service, but someone can still use Power Query to figure out the model, figure out the faucet, all that kind of stuff. And then say, "Hey, data engineer, can you do the thing that handles the API paging and throttling." And that end result can happen much faster. So that's an example.

Rob Collie (16:40): Okay. Okay. Okay. What you're telling me, if I understand it-

Justin Mannhardt (16:44): Let's see.

Rob Collie (16:45): ... let's see, are you telling me that Power Query is getting an upgrade to be able to talk to APIs?

Justin Mannhardt (16:52): Power Query can talk to APIs.

Rob Collie (16:54): Already?

Justin Mannhardt (16:55): And this is one of my main knocks on Power Query, my product team, if you're listening. So API services will commonly paginate the results. So let's say you hit an API endpoint, you want all your customers, and it's going to give you 200 at a time. Power Query doesn't like that. Then an API might also throttle you, like, hey, you can only make so many requests per second per minute, whatever. The thing you don't like to do, Rob. Now we're writing M code to say get 200.

Rob Collie (17:24): Wait.

Justin Mannhardt (17:25): Find out how many more I need. Wait. Go again.

Rob Collie (17:28): Okay. First of all, I'm disappointed. I love it when I'm right, and I wasn't right there. So all right. So that's fine. But now I have the follow-up question. Okay, so where's the magic trick? Where does it get easier? I don't understand.

Justin Mannhardt (17:44): Notebooks. Python notebooks. It's so much easier. It's so much easier.

Rob Collie (17:48): Seriously?

Justin Mannhardt (17:49): Yeah.

Rob Collie (17:49): You're telling me that I might have to do a Python notebook someday?

Justin Mannhardt (17:52): But maybe not, and it's not that much code. I could teach you how to do this, Rob.

Rob Collie (17:56): You'd have to get me into the chair first.

Justin Mannhardt (17:58): This will be one of my traps where at the Ignite conference with Microsoft students, like, and now supporting pagination and throttling, and I'll be like, yay. But this is also why there's a huge market for third party connector services. We use a lot of those ourselves to get data from something and get it into our environment. Because those custom connectors handle that.

Rob Collie (18:20): We use a lot of Stitch, for instance.

Justin Mannhardt (18:21): We do.

Rob Collie (18:22): For running our own business, not just with our clients. All right, so let's assume that I've got some sort of magic wand button that I can press that says Python notebook me, and I point it at this API, and say go, and then that makes my Power Query easier?

Justin Mannhardt (18:40): It just sort of comes out of the equation. So then now you've got that in OneLake, which is a default ingredient in your dataset, which is-

Rob Collie (18:48): I see.

Justin Mannhardt (18:48): You know what I mean?

Rob Collie (18:49): So then when I point my dataset, when I say add new data, new query, I just point it at OneLake and the Python notebook is the thing that's populating that.

Justin Mannhardt (18:58): Yeah.

Rob Collie (18:59): Okay. I don't know, man. I mean I love that. I really kind of want the Python to go away.

Justin Mannhardt (19:05): That'd be awesome.

Rob Collie (19:06): If this is one of those intermediate states where for now that's what we're going to do.

Justin Mannhardt (19:10): This is on the skeptical topic. So there's another feature in Fabric called Data Wrangler.

Rob Collie (19:19): Okay.

Justin Mannhardt (19:20): Right. What Data Wrangler is is-

Rob Collie (19:22): Is dad jeans for data-

Justin Mannhardt (19:22): Is dad jeans for Power Query. So Data Wrangler is like a drag drop point click interface for Python. So it's very Power Query cousin-ish. It's more aligned with data science where you're trying to massage data in specific ways to get different statistical outcomes that you're going to use in a model or something like that. It made me wonder, we can tell Power Query to convert all those steps into SQL code and query folding. When do we get it to also create Python for some of these other scenarios? I remain an optimist on those types of things happening.

Rob Collie (20:07): So speaking for the non-aliens.

Justin Mannhardt (20:11): Which one's the alien? I'm the alien?

Rob Collie (20:12): You're the alien-

Justin Mannhardt (20:12): I'm the alien.

Rob Collie (20:13): ... but you're really, really, really, you're like an ambassador from the aliens.

Justin Mannhardt (20:18): Right. It's an honor and a privilege.

Rob Collie (20:23): Yeah, it's fun. Every new thing that I have to crack open inflicts an enormous cognitive cost on me. And so even if you tell me, oh, Data Wrangler is so easy to use, and then you can make this Python notebook and it's real easy to use, there's still a tremendous activation energy cost for me to even branch out into that. I won't kid myself. I don't think everyone has the same activation energy cost coefficient that I do. Mine's pretty high. But I think a lot of humanity is closer to my end of the spectrum than far away.

Justin Mannhardt (21:03): No doubt.

Rob Collie (21:03): But the ones that don't have a low coefficient, that are willing to just try new things out because they're excited about it, they're the ones that inflict all the FOMO on the rest of us.

Justin Mannhardt (21:14): Maybe an attempt at our mutual conclusion on this point, which is the faucets versus the plumbing, and how we feel about that with Fabric. OneLake is the thing. It makes the data jeaner that's been building data sets that much more valuable. Their work product is now useful in all kinds of places. It was never useful before. They can get in on this game without having to manage infrastructure to deal with all this stuff. It's a point for me where I started being skeptical. I was like, this is just Synapse. And then I was like, well, actually no, this is different because there's a huge population of people that just got a ton more valuable. And so we're not changing our narrative about faucets first at all. We're just saying, hey, our faucets first approach now serves this whole other universe.

Rob Collie (22:06): Just to complete the metaphor, and that requires us to explain it just in case people haven't heard it, but what you really need are some faucets. You need drinks of water. And you go to a consulting firm and they say, "Oh, well, yeah, of course you need water and you need faucets. But really what you need, first of all, is a responsible future-proof plumbing system."

Justin Mannhardt (22:24): Scalable.

Rob Collie (22:25): Right? And next thing you know, there's stainless steel and copper pipe going everywhere. And the costs are mounting, delays are mounting. And you're like, "Where's my water?" And they're like, "Hey, hey, hey. Just wait. Have some patience, young Skywalker." And so right about the time that you're out of patience and out of money, they say, "Fine. Here, have some faucets." And then they put them where you don't need water. That need never materialized for me, but I need faucets here, here, here, and here, and there's no plumbing there.

(22:53): We start with the faucets, and if we need to run a hose to the faucet to verify that it works properly and it's in the right place and it delivers the right kind of water, even if that hose isn't the long-term solution, at least you start to get water and verify that you like the taste of that water. Do you need the mineral content changed? Would you like bubbles? And when it's time to run a more responsible pipe, well, you run a pipe. You're not running a whole system of pipes. It's very, very focused. There's no guesswork, and it's sort of a reactive to reality.

(23:29): Reactive tends to get a bad rep, but no, it's reactive to what's actually needed. That's a good thing. So even we, P3, run responsible plumbing. You go into one of our projects when the dust settles and you will often see copper and stainless steel. But now someone's come along and invented PEX, which if you don't know about it, is pretty freaking awesome. The first-

Justin Mannhardt (23:54): I just redid our bathroom in our basement. Yeah, it's amazing.

Rob Collie (23:57): It's like the equivalent of running ethernet in your house, but it's water, right? It's insane, these plastic tubes that cost nothing, way nothing. They snap together almost like Lego. You can retrofit them into a house without tearing the whole house up. Whereas good luck with traditional plumbing. So it's like, awesome. We have PEX. We will now get there faster. And I guess I understand that.

(24:24): Fundamentally the two things that I am most excited about with Fabric, and they're really the only two things at the moment that I'm excited about, but they're massive, is number one, Power BI people, our skills, without learning anything new, without even touching any of these new things that you're talking about that are being made super, super, super easy, without even touching those, there's an increased demand and an increased utility for the skills that we already possess.

Justin Mannhardt (24:52): 100%.

Rob Collie (24:53): That is amazing, right?

Justin Mannhardt (24:55): Yes.

Rob Collie (24:56): Before I would build a Power BI model for that client. That'd be the most amazing reference, the most amazing information source in their company for those data sources. They've got nothing else. It's like the most amazing oracle that answers all of your questions, but it will only give its information to a Power BI report. Tough. And now it won't.

Justin Mannhardt (25:19): Quick thought there. Some people have asked me about this offline who say, the only place you could go would be a Power BI report. Someone will say, "Well, but what about the XMLA read write point?" It's like, yeah, you can query the data model, but again, you're doing this oddball stuff to get that so much easier.

Rob Collie (25:39): It reminds me of the scene in the Princess Bride. Rodents of unusual size. I don't think they exist. And then he gets jumped by one, right? It's like you're talking about the XMLA for analysis endpoint. I'm like XMLA for analysis endpoint, I think that's just a myth. And then this giant rat with XMLA written on the side jumps me. Do not want. And one thing in the early days of you and I talking about this, this is still falling under my first point of excitement, you were telling me Power Query finally gets its day in the sun, right?

Justin Mannhardt (26:15): Yeah.

Rob Collie (26:15): Because just the Power BI model was sort of like hoarding its information for Power BI reports and this other mythical XMLA thing, Power Query is probably the greatest thing, the apex predator of ETL in this universe, within our observable universe. And better than all of the Linuxy Silicon Valley cool kids stuff for the same.

Justin Mannhardt (26:42): 100.

Rob Collie (26:43): However, when you're done with the Power Query, you can dump the data anywhere you want, as long as it's into a Power BI model. And so Power Query gets a lot more utility, a lot more demand. The value of knowing it, not overnight, but long-term, the value of knowing Power Query probably triples. That's the first category of things that I'm super excited about is the things we already know become more valuable become more important. And who doesn't like that? The other category is more vague, which is it's betting on the people behind this.

Justin Mannhardt (27:18): You mean at Microsoft?

Rob Collie (27:19): At Microsoft. Yeah. So to know that, A, these are the same people that tackled OLPA. Seriously, I think OLAP databases, which is what SSAS Tabular is, OLAP databases are probably the single hardest thing to bring down to a citizen developer in everything in terms of IT technology. I can't imagine anything harder than that. And that was the first thing that we tackled. I had a little bit to do with it, but really very little. So the people who did that are still there. They've grown that team. So it's not just the people that I used to know. There's a whole bunch of new people. When I say new, they've been there for even 10 plus years and I didn't work with them, but that team has been put in charge. They're in charge of the whole data platform now. And if they could beat the OLAP problem, they can beat anything, and they're going to. And so even if the current state of affairs, if I get grumpy at them about like, oh, you still make me do a Python thing, you'll make it easy, but it's not easy enough for Rob, just wait.

Justin Mannhardt (28:29): So there's another idea I have on Fabric. I kind of think of it as the convergence opportunity. So you just mentioned you get grumpy because you don't want to learn Python or all this stuff. Our podcast with Jeff, what we're doing for them involves a lot of Python. But there's some pretty interesting stuff involved in regressions and things I could never even conceive how you would do them with Power Query or DAX. And to think of, well Rob the person that can do data sets and DAX, can now buddy up with someone like a Haas and say, okay, fit this to the curve. I figured out, like you were saying, the hardest part about AI is getting the data into shape. The fact that you guys are so much closer together in your ability to work on that level of a problem, I feel like that's underrated.

Rob Collie (29:24): Yeah, that's a big deal. That's a big deal. And the fact is my investments in the Power BI model will make that easier. By the way, I want to say for the listeners, when Justin says someone like a Haas, you're not using that phrase that Southerners use to refer to some big guy, right? You're talking about Chris Haas?

Justin Mannhardt (29:41): Chris Haas. We capitalize the S at the end of his name, so it kind looks like SaaS, but we call him HaaS.

Rob Collie (29:48): Yeah, Haas as a service.

Justin Mannhardt (29:51): We call it human as a service.

Rob Collie (29:52): Human as a service. That's right. Yeah. I like Haas as a service too.

Justin Mannhardt (29:55): So he's doing that. I think that's one of the things, seeing how he is approaching this very interesting project. He does some things in Power BI with DAX and modeling to get data into a certain state, and then he ports it over into the Python magic that does the things that would be difficult to do, that level of recursion with DAX or in Power Query. That exists very nicely in this new universe. And so I think that's underrated in some of the narrative, like the ability for multi-disciplined people to come together on a problem much easier. Something I'm excited about.

Rob Collie (30:30): It's like the old story that was really important to Power BI's success, which is that in the old days, a solution built by the business could not be upscaled and taken over by IT without being completely rewritten because it was implemented in Excel and Access by the business. And it's a completely different language. Like Access to SQL is a relatively tiny little transition compared to Excel to anything else. So to have the business start out with languages like DAX and M that end up being 100% bulletproof for IT, really takes the ceiling off. And that's important. So we've now had that dynamic for so long that we've kind of forgotten how amazing it was. So to have that same dynamic coming for all of this pointy data sciencey whatever else stuff is also a really big deal. And I think it'll be a while before we fully absorb its significance. So we're going to be skeptical, and I'm just more in than ever now.

Justin Mannhardt (31:38): So we could probably pull on this thread for a while if we brought the team into the conversation.

Rob Collie (31:44): I like that. Pulling on this thread.

Justin Mannhardt (31:46): We can have all sorts of detailed things we're skeptical about, and some of it's going to get solved in the release plan. But for example, Azure Data Factory is a component of Fabric. Power Query moved into that. That's one of the things we're really excited about. Traditionally, Azure Data Factory is very useful in certain scenarios, but at the time of the preview, it couldn't connect to on-premise data. You had to use the Power Query part to connect to on-premise data. And everybody's up in arms because it's like, but we have all this stuff we want to move over from Azure into Fabric. And that's going to get solved. So that was a thing we were like, what the heck's going on here? The direct lake concept where you don't have to refresh your data set as soon as it's in the lake, everything's refreshing. Ed Hansberry, who's been on the pod, he's done some performance testing about large models. Where does that seem to fall off, and where does it seem to be really good, and what's the sweet spot? Where is it improving?

Rob Collie (32:42): Can I throw in a plug here real quick?

Justin Mannhardt (32:43): Yeah.

Rob Collie (32:44): If you haven't heard the episode that had Ed on, we did a DAX function draft.

Justin Mannhardt (32:48): That's right.

Rob Collie (32:49): I got destroyed. Us taking turns drafting functions from the DAX language that we get to exclusively use. Yeah, let's just say that it didn't go well for me. But a great episode.

Justin Mannhardt (33:02): Now that the release plan is public, we're able to sort of draw the lines between, okay, that's going to ship in Q1, that's going to ship this year. But overall, of all the new stuff I've seen released from Microsoft, I'm remarkably impressed by how well it all works. The frequency by which I'm encountering bugs and showstoppers is very, very low. This is more like your believability and the people behind this, I'm like, they're figuring this out.

Rob Collie (33:31): And any sort of feature X isn't quite there yet skepticism isn't really skepticism. That doesn't qualify. They've got time, they've got resources. It's hard to even remember how much Power BI has advanced over the years.

Justin Mannhardt (33:49): It's insane.

Rob Collie (33:50): Imagine that same marching monthly dynamic. Give it 12 months. I brought up like, oh, Data Activator. It won't do cross joins across multiple dimensions to scan for outliers and changes. Yeah, I think that's a problem. It's not going to take them long to implement it. It might take them a while to realize that it's important. They're not just going to come ask me what I think. I've already told them. They didn't change their roadmap. They'll figure it out. It's just waiting. That's almost like disingenuous skepticism. It's like being handed a chocolate cake for the first time ever. It's like the first chocolate cake in the history of chocolate cakes, and going, "Where's the sprinkles?"

Justin Mannhardt (34:35): Or better, it's actively watching someone make the cake, and while it's in the oven being like, "The sprinkles aren't on it."

Rob Collie (34:42): And again, you've never had sprinkles, right? You've never actually had them.

Justin Mannhardt (34:46): You're making a cake. It's not even frosted. Who is this guy?

Rob Collie (34:56): Bro, do you even cook? Do you even bake?

Justin Mannhardt (34:57): Do you even bake, bro? I agree. Those things aren't criticisms.

Rob Collie (35:02): I'm feeling like an utter failure as a skeptic.

Justin Mannhardt (35:04): So the one other thing I wanted to throw in here is the type of people I am most excited about are the type of businesses we most like helping. The companies in the mid-market or departmental groups in large companies that just never had a chance. They couldn't get the resources, couldn't get the budget, couldn't get the people, couldn't get the talent. Some of these companies, they don't need some of the fancier data gymnastics I was talking about with the Python and stuff. They don't need any of that. They can get so far with their goals by all the stuff you were talking about, probably the first time ever. Sure, you could go buy a Snowflake or buy a Databrick. So guess what? You're still looking for someone that knows SQL or Python or a combination of both. And then you're porting that over to your reporting platform. No, everything you need to help your company through analytics and AI, you can get it and you can do it.

Rob Collie (36:12): Yeah. Just like most organizations, and when I say organization, you can think of that as company, you can also think about it as departments of larger companies, most organizations, the overwhelming majority of organizations, were priced out of OLPA data models forever. And then along comes Power BI, and it's like this gold rush.

(36:35): And even though I'm a hardened skeptic of the old approach to OLAP data models, was really just barely better than not having it. Even for the people who could afford it, it was kind of a nightmare and almost never delivered on its promise. At least they were in the game. And the people who were never in the game, not only are they in the game now, but they're in the game at a point where the game is actually enjoyable and fruitful to play for the first time ever. It's not just, oh, now we can behave like the enterprises. No, you can behave in a way that's better than the way that they were behaving by orders of magnitude. That's massive.

(37:16): And it's not like the Power BI revolution is anything close to done permeating out. It's really still early. The next wave of gold rush is coming behind it. It does not supersede or replace the first gold rush wave. It doesn't invalidate anything that's coming before it. How humane and good can it get? That is what you want. I'm sure that Fabric is a source of FOMO for a lot of people. Just the pace of things coming out seemingly. This is one of the questions we answered in our Q&A episode. The pace of things coming out can seemingly invalidate you. You feel like the things that make you valuable are under attack. They're now threatened because these new things that are coming to replace it. That's not what's happening here.

Justin Mannhardt (38:04): That's right. When you were talking about this, I thought of something I saw today, and I do think it's a bit of a risk. And so I want to throw this out to the audience. Because Fabric is this umbrella term for seven different workloads, some of them are still rather technical, the Power BI blog was swallowed by the Fabric blog. And so there was an announcement today about some capacity data warehouse auto something, flux something else. When you see something like that, I think it can contribute to you like, oh my God, am I falling behind? There is this convergence of multiple audiences that's happening into this product. And so don't let that stuff discourage you. It's going to be part of it.

Rob Collie (38:51): Power BI had the same thing. We had the data warehousing crowd, we had the ETL crowd, we had the OLAP crowd. These were all priesthoods in a way, and they all had their ivory towers. And so there was, and still to an extent today is, a little bit of friction there.

Justin Mannhardt (39:08): So an example that I think for me is one of the reasons Fabric's amazing, do you remember your Crushinator report?

Rob Collie (39:16): Yes. The Crushinator.

Justin Mannhardt (39:19): Right? So debate me on this, but I think essentially what you were trying to do is predict which cars to crush on a junk lot.

Rob Collie (39:29): Yep.

Justin Mannhardt (39:30): You were trying to do predictive analytics.

Rob Collie (39:34): I mean, I would say it was more of a decider. We weren't going to predict what was happening because in the end we were going to decide what was going to happen, which car got crushed.

Justin Mannhardt (39:40): So the general situation was you were using a combination of inputs to decide what to do.

Rob Collie (39:48): It's an optimizer.

Justin Mannhardt (39:49): It was an optimizer. Okay.

Rob Collie (39:50): It was an optimizer.

Justin Mannhardt (39:52): So when you do this in DAX, it was very doable. You had to make decisions. You had to decide which variables to include. You had to potentially decide on their weighting and their relevance. You made choices is my point. So all the work you did to build the data model that provided those inputs, okay, now let's make the leap over to AI. Because what a machine learning model can do is that technology can help you decide which inputs matter and in what ways, and it can adapt over time. Now we've got access to a tool that might be better fit for the problem I'm trying to solve.

(40:34): So I've seen data models where people are trying to predict something like customer churn or revenue, and they're doing this in DAX, and they're having to make these choices. How important is this? How important is that? It's just a different thing altogether. Now you can say, okay, instead of us trying to make all these decisions in a DAX measure, let's hook our data model up to the machine learning model that's designed to help us make this type of prediction.

Rob Collie (40:58): So you said debate you on this. So I'll debate you on this. Now, part of what I'm going to do is sort of nitpick the example you chose because I actually think in the example that we're talking about, which cars should go to the compactor today versus staying on the lots to let people come and buy parts off of it, the solution we arrived at there, I think in DAX was damn near optimal simply. Because it was understandable and there weren't a lot of hidden nuances in the DAX.

(41:31): So all we did was agree on what the top five factors were. Top five measurements in terms of their importance to which cars to crush. Now you're right, there's some human instinct there. There's a lot of tribal knowledge, but it's also known as business knowledge that you don't want to be blind to. You don't want to sideline all of that. Now, of course your point would be what if you're wrong about what those five are? Totally fair point, right? Our degree of confidence of what the five most important factors were was really high. And all we did was rank each car according to those five. So there's the rank X measure, five rank X measures, one for each of these metrics. And then we averaged the ranks.

Justin Mannhardt (42:09): Sophistication.

Rob Collie (42:11): And sorted by that average. So now at least if the top row of the report wasn't the right answer, the right answer was still going to be in the first few rows.

Justin Mannhardt (42:20): The consequences of being wrong were very low.

Rob Collie (42:23): That's right. And there's no trusting a system at that point that can lead you astray. Like the hallucination problem in AI that we're talking about. Once it becomes complicated enough to deliver value, oftentimes it can be wrong in ways, and you don't know why and you can't detect it. How many times have you followed a GPS? And this hasn't happened very often lately, but in the early days of GPS systems, the longer you sustained your faith in that thing, the later you could be to your destinations. It's like, no, no, it must know something. Okay, I'm getting less and less confident. I don't know why we're on these back roads and not on the interstate, but it might know something. You get there an hour and a half after the other car. There's a lot of danger there. So again, in that particular example, I would say that the DAX solution might remain optimal forever. It'll resist the AI invasion much more durably than many, many, many other problems for which what you were saying is completely valid.

Justin Mannhardt (43:29): Right. Well I think another perspective put on this is let's say that the team or the person that learns the DAX solution to that type of problem, they then try and mature it to a seemingly similar problem. Oh, I've got variables and inputs and I can build my rank. And you realize you're maybe in this different ballgame. I guess my whole point is on some of these things where we're trying to use measures to predict outcomes or be suggestive, maybe machine learning, which is different than large language models, that's maybe another episode for us. Maybe there's an opportunity to leverage that. You haven't had that opportunity.

Rob Collie (44:09): All right. So we came at this, let's be skeptical. I was in on Fabric when we started this conversation. I'm twice as in now. It's good though, right? To pick at it and to understand why we're excited about it and get precise about it. And I think one of, this word's too harsh, but I think one of the mistakes that I've been making in how I talk about this is talking about it as Microsoft is looking to bring the citizen developer type experience to everything else. But the core of what they're doing to accomplish that is to make the Power BI tool set the answer to everything else.

Justin Mannhardt (44:51): That's a good way to think about it.

Rob Collie (44:52): When we think of it that way, I could describe it the first way and there's all kinds of skepticism and fear and FOMO, and I got to learn all this new stuff, but you describe it the second way to the same people and they're like hot damn. And it's a good clarification I think to how we think about how we talk about it is, sure, there's going to be elements of that first story, bringing brand new experiences in. If you instead kind of ignore those for a little bit and just focus on the other thing, that's probably the place to be focusing your attention is on the extra value and impact that your existing skills are going to have.

(45:31): And I think that will also naturally lead you into some of those new experiences more organically, less fearfully, lower the activation energy, lower the barrier, the mental barrier, the cognitive barrier, the one that's in your head, will come down a bit when you approach it the other way. All the things I already know are becoming more valuable. Let me explore what that looks like, what that even means. I think that's the positive and most productive way to map this.

Justin Mannhardt (46:03): Agree.

Rob Collie (46:04): All right. Meeting adjourned forever. Have you seen that, the Monty Python Royal Society for Stacking Things on Top of Other Things?

Justin Mannhardt (46:12): No, I haven't.

Rob Collie (46:13): It's one of the greatest Monty Python skits in history. It's so good. I'll describe it briefly, and then we'll link it in the notes. So some starchy high society get together like a club, but it's the Royal Society for Putting Things on Top of Other Things. They come in, and they go, "Meeting called to order. It's been a great year for us putting things on top of other things. And everyone here has done a great job except for," and I forget what the city is, "our Devonshire branch this year has not stacked a single thing upon any other thing. They've not stacked anything this entire year. I call upon the good gentleman from Devonshire to explain this terrible behavior."

(46:55): I think it's John Cleese. He gets up and he looks embarrassed. He's really mumbling under, "Yeah, yeah." He identifies himself. "I'm from Devonshire." He says, "Well, we all just kind of thought that it all just seemed a bit silly." The leader of the group goes, "Silly? Silly?" He gets all indignant, and he goes, "I suppose it is a bit silly, isn't it?" And then he goes, "Right. Meeting adjourned forever." Bangs the gavel. And I even printed, there was a time when I worked at Microsoft, where I was so just blown away at the absurdity of everything, just how it all worked, that I printed out the whole script for that segment, and I had it taped to my window of my office. Right?

Justin Mannhardt (47:41): That's great.

Rob Collie (47:42): Royal Society. Every now and then it just seemed like, what are we doing?

Justin Mannhardt (47:48): Seems a little bit silly.

Rob Collie (47:49): It seemed a bit silly. I don't think the things we're up to these days are silly. And I think that's one of the reasons why I've enjoyed this chapter of my career so much more. I enjoyed the Microsoft chapter. I mean it was brutal in some ways. And you wouldn't think it, right? Building the technology that enables all of this. Why would that have ever seemed silly to me? It was just we were just such a great distance from the value, and it was just so hard to viscerally connect with it. And I think that was sort of why half the time I would just kind of sit around and laugh. It was like, I don't know, maybe this works. We're doing our best was kind of the way I looked at it. Whereas out here, we can tell.

Justin Mannhardt (48:30): Feels good.

Rob Collie (48:31): Lives change.

Justin Mannhardt (48:33): Yeah.

Rob Collie (48:33): It's not silly. You will not see the Royal Society for Stacking Things.

Justin Mannhardt (48:38): The meeting will continue.

Rob Collie (48:40): You will not see that script taped on my office door.

Justin Mannhardt (48:43): Right on.

Rob Collie (48:45): Great chat. We'll do this again soon. Same bat time, same bat channel.

Speaker 3 (48:48): Thanks for listening to The Raw Data by P3 Adaptive Podcast. Let the experts at P3 Adaptive help your business. Just go to p3adaptive.com. Have a data day.