What We Learned at the Fabric Conference

Rob Collie (00:00): Hello, friends. In today's episode, Microsoft Fabric is once again in the spotlight. Microsoft recently held its first ever Microsoft Fabric conference in Las Vegas. P3 sent a contingent of three our employees to that conference, and in today's episode, I sat down with the three of them, Mark Beedle, Chris Haas, and Will Gillingham to get their trip report.

(00:21): What are the questions that they went in with? What are the questions they came out with? What were the answers that they got along the way? And I know that you've heard Justin and I talk about Fabric in a number of episodes to date, but Mark, Chris and Will are thoughtful people with their own unique perspectives, their own unique sets of experiences, and their own unique current professional contexts.

(00:43): And when we're trying to make sense of something as new and as almost deliberately vague as something like Microsoft Fabric, we benefit tremendously from incorporating that kind of diversity of opinion and background into the conversation. You don't really know something until you've seen it from 17 different perspectives, 17 different angles; so in that vein, we ended up discussing things that we really haven't talked about before.

(01:12): For example, Fabric sounds like a lot of the Linux cool kids stuff sometimes, like Python and Spark and PySpark and Notebooks and Data Factory and all that kind of stuff, things that if you're coming from the Power BI perspective, kind of sitting there scratching your head wondering, "Well, what does this do for me? Do I need to use Fabric?"

(01:33): On the flip side, though, if you're already completely up to your eyeballs in that same cool kid stuff, what benefit does Fabric bring you if you already know all those things like Python and PySpark and stuff?

(01:45): Another interesting question that came up was, in the same way that Power BI brought a large contingent of the Excel crowd into a place that was more front and center with the business, what fraction of the data engineering crowd might also follow a similar pathway using Fabric as the gateway?

(02:06): And along the way Will gave the best definition of medallion architectures that I personally have ever heard, so if you've ever wondered what medallion architectures are, what they mean to your business, and whether they have anything to do with the Beastie Boys wearing Volkswagen hood ornaments as necklaces in the 1980s, you get to find out today when we get into it.

Announcer (02:28): Ladies and gentlemen, may I have your attention, please.

Announcer (02:32): This is the Raw Data by P3 Adaptive Podcast, with your host Rob Collie and your co-host Justin Manhart. Find out what the experts at P3 Adaptive can do for your business. Just go to p3adaptive.com. Raw Data by P3 Adaptive is data with the human element.

Rob Collie (02:59): I am here with Will, Mark and Chris. Will Gillingham, Mark Beedle, Chris Haas. The three of you were the P3 Adaptive representatives at the recent Microsoft FabriCon. Very clever name. Fabric Conference, but FabriCon, right? Is that a nickname for the conference or is that a real official name?

Mark Beedle (03:19): I saw a lot of FabCon,

Rob Collie (03:23): So Fabricon is something I made up?

Mark Beedle (03:24): I think so.

Rob Collie (03:24): Fine. Well, I think Fabricon is better.

Mark Beedle (03:26): It is. It's a missed opportunity, right there.

Rob Collie (03:28): Why don't you each briefly introduce yourselves. Chris is the only one that's been on the pod before. Let's start with Will.

Will Gillingham (03:34): Thanks, Rob. My name's Will Gillingham. I'm a principal consultant specialized in Power BI, and I've been with the company for about two years, now.

Rob Collie (03:43): All right, thank you. Mark?

Mark Beedle (03:45): Thanks for having me. Mark Beedle. I, too, am a principal consultant here at P3 Adaptive. I've been here since the end of 2020.

Chris Haas (03:52): It is great to be back, Rob. I'm Chris Haas, solution architect here at P3. I've been with the company just coming up on four years, now.

Rob Collie (04:00): Wow. It seems like longer than that, doesn't it?

Chris Haas (04:02): I've known you for eight, so that's really where I consider the start.

Rob Collie (04:05): That's the trick, isn't it? All of you, thank you so much for taking time out of your busy days to be here. I have talked quite a bit on this podcast about the thing that is Microsoft Fabric. Justin and I have been spending some time evolving our collective impression of it, but the three of you, going into the Fabric Conference, you took your own sort of questions and opinions and things with you. I know that we all went in with your own individual pictures.

Mark Beedle (04:34): A little bit of background to contextualize it: I have a background in old school on-prem SQL. That's not the popular thing anymore. Lakehouses and parquet files. That's the hipster stuff. I've worked with and had success with SQL Server. Just in general, I was hesitant to buy into the lakehouse world. We're taking big data concepts, thinking back to Hadoop and all this big data, and then applying it to data that's not necessarily big data. Why wouldn't I just continue doing what I normally do and put it in a SQL Server? Azure SQL database in the cloud?

Rob Collie (05:19): It's good enough for me.

Mark Beedle (05:20): Yeah. What do I need Fabric for? So I went in with "Prove to me that I need Fabric." That's my attitude going in. So a little bit of a skeptical attitude. I see the vision of Fabric. I think Microsoft's going to knock it out of the park. I needed convincing that my old school ways needed to evolve.

Rob Collie (05:40): Yeah, that's a 100% valid thing. That's how I approach things, definitely even a more stubborn attitude than what you just described. I don't think you're being stubborn. I'm stubborn. It's less common for our Power BI principal to have come up with a SQL background at P3 than for them to have come up with an Excel background. Now of course, sometimes there's an Excel and SQL hybrid background, et cetera, right?

(06:07): SQL is not necessarily the first love or first technically competent language that our team has learned. Really, Power Query, M, DAX are oftentimes the first professional-grade languages that our people have learned, so you bring a history and an opinion to this that is even, I wouldn't say unique, but it's not the norm, even at P3.

(06:30): What you just described, this transition in the cool kids space from the rectangles of SQL land to the squishy, blobby, curly data, storage of data lakes, blah, blah, blah, how much of it is just this is what the cool kids are doing versus how necessary? How did that question and opinion evolve as the conference unfolded for you?

Mark Beedle (06:55): I would say after the conference I'm less skeptical, but I'm also not like, "Everything I do has to go into a lakehouse first." I did mention you I've got a background using SQL, but Power Query and DAX are my day-to-day tools. Sometimes Power Query and Power BI Desktop are still the only tools I need, so my eyes were opened that there are huge use cases and value to be seen in Fabric.

(07:25): I don't have to start using all of them. If my data doesn't need to scale and I can import with Power Query and build my semantic model in Power BI, it's still in that Fabric umbrella, I don't need to go out and learn PySpark to get value out of this new umbrella that is Fabric.

Rob Collie (07:44): Yeah, and for you, Fabric mostly means OneLake. Honestly, for me, Fabric mostly means that, as well. Fabric is sort of like, "It's a philosophy, it's a way of life and not just any one technology," and I say that tongue in cheek, but I actually mean it. I think it'd be a mistake for Microsoft to just say, "This is just to push this new wave as if it's just OneLake," because it's more than that.

(08:13): This is a really fundamental concept that I'm still hazy on, and before we're all good with Fabric, we need to internalize this in our bones in a really intuitive way, which we're not there yet. But I agree with you. There's this history of Hadoop and curly data that's not in nice rectangles.

(08:37): But then a Power BI model, when we're dealing with creating a semantic model in Power BI, it's still rectangles. It doesn't matter how the data was stored originally. By the time it gets into the Power BI model, I get to think of it as rectangles. On a recent podcast, Matt Allington was saying, "Underneath the hood, it'll blow your mind, man." And I'm like, "No, no. No. Uh-uh. No, no. Stick my fingers in my ears don't care." It's rectangles. And it is conceptually, right?

(09:04): Okay, so here's the crux, and this is where maybe Chris can help us out: now they're telling us that OneLake, "Just [inaudible 00:09:13] OneLake from a moment, lakehouse, blah, blah, blah," you're thinking curly, weird, semi-structured, non-rectangular storage. But then they're also saying, "And by the way, the OneLake storage format is the Power BI storage format."

(09:29): So when you're building a Power BI model, you could just flick a switch and turn it into a OneLake storage, the data model, and or the reverse where I'm just dumping data into OneLake. Guess what? It's already making you a Power BI model. All you need to do now is start drawing the relationship lines and adding the DAX.

(09:49): So given that Power BI is conceptually rectangular, the semantic model is rectangles. That seems like the SQL tradition is winning out in OneLake, but it's got that word "lake" in it, which is curly squirrely. So is it more like SQL under the hood than they're letting on? If I'm just dumping all kinds of weird, funky, oddly shaped XML files into Lakehouse. It's just automatically, without any Power Query, that's the old way.

(10:20): Hadoop was like a garbage disposal. It'd eat anything, didn't need to be pre-processed, whatever, just stuff it in there and think about how it's stored later. I'm very confused about this point. I'm not even sure if my question makes sense.

Chris Haas (10:34): It does. I really think what's happening is OneLake is removing a lot of the friction to make those rectangles possible. It knows that rectangles are what's needed eventually, but it's going to let you just get data in. We'll figure that out and we're going to make even better rectangles than you could have imagined, which are going to give you better performance. It makes rectangles without you recognizing that you're making rectangles.

Rob Collie (11:03): Okay, so the way I described the curly data storage formats for a while, like Hadoop and stuff, was they were the perfect example of faucets first. Rule number one of data warehousing was don't lose anything. You have all these line of business systems that they don't care, they keep one year of history; or they're overwriting their own history every day, so you lose history.

(11:25): Data warehouses, really the reason for them to exist was to capture history, and then it spawned this Kimball priesthood of SQL storage. Okay, fine. Given that SQL was our tool for capturing history, you needed that, I guess. It just seems like this weird perversion of the original intent, which is just don't lose anything.

(11:44): Hadoop was really good. Having a storage format that just dumps stuff in there and it won't lose it, we promise, and then you can figure out how to structure it later. You don't have to anticipate all of your needs in advance, which is the whole data warehousing schema design and everything, which is why data warehouses never finish. They're never a success.

(12:00): So I've loved this idea. What does this mean from a business perspective, this uniting of the two worlds, the really clean and effective Power BI semantic model world with the old garbage disposal, "Don't worry about it, just capture it" world?

Chris Haas (12:19): When I talk to clients and customers and try to understand, "Do we need all of this data in there?" And I just start with a yes, the first thing we do is put data in the lake. Why? Because that answers the question. "Where's the data?" "Oh, it's in the lake." That's easy.

(12:36): What do we do with it? Well, now we can decide what we do with it, but we already know where it is. And with OneLake, the ability to just get the data into the lakehouse and then not think about it, it's easy, it's simple, and then the ability to, "Hey, I think now we actually want to take a look at that data. Now how can we make sense out of it? How can we get value out of it?" Well, the best part is it's already there. We can already just get Power BI in to say, "Oh, yeah, there's a there there; let's pull on that thread a little bit."

Rob Collie (13:09): Okay, I can get behind this so I can play this story back. Fabric, AKA OneLake, offers you the following business benefits: it's very low cost, low barrier to get data into it and to start capturing history, for instance. The original purpose of data warehouses was just to make sure you're not losing things that might be valuable later. A lot of it's going to be valuable later, but you don't know which pieces of it, and you certainly don't know how it needs to be structured and twisted and turned to be valuable.

(13:39): If you capture the raw level of detail, you're safe; and to get to that level of safety without a lot of effort, without a lot of cost, is a wonderful, wonderful first step. That, of course, would've been true of all of the lakehouse type of formats ever, all the curly storage formats, is that they're low cost, low barrier to get things into. Although with Microsoft, there's probably even lower barrier.

Chris Haas (14:03): 100% easier.

Rob Collie (14:05): All of the cool kids Linux West Coast startup-y stuff is pointier than it needs to be. Okay, but even if we think that they're equal, now OneLake brings the second benefit, which is that once it's there, it is so much closer to providing value because OneLake is structured differently. It is ready to provide those high performance, really easily, quickly, presto chango, reshapable rectangles, and all analysis ultimately runs on rectangular shapes of data.

(14:40): I kind of like that. The same low cost of entry, but now it's also 100% more primed; because what happened is with the old data lake formats is that the cost of entry was low, but then when you came back later and had to extract rectangles from the mess, you still had a lot of work to do, and I think OneLake doesn't do that. It makes that incredibly painless.

Chris Haas (15:01): Not only does it make it painless for any individual to extract data and play with it, but multiple personas using multiple pieces of technology can still access that same data and share it with each other in a way that reduces the friction all around, so data scientists can use PySpark and Databricks the same way that someone could just use Power Query and make a separate table, or someone could use SQL and make a separate table; and now the fact that everyone is on an equal playing field to consume the outputs and inputs from each other is just incredible.

Rob Collie (15:44): Me personally, I don't care about the PySpark crowd. I have a hard time getting excited about that benefit. Now, if I was working with a bunch of people who are PySpark aficionados, this would be a huge boon for me. "Oh, my Power BI models are now good for them," and we'll come back to this topic, but I want to change gears here just for a moment and say we haven't engaged Will yet. Will, what were your thoughts like going into the conference

Will Gillingham (16:14): Going in, I think I told Justin that I wanted to come out as a Fabric evangelist. I think that morphed into during the conference, "What is the business value for our clients?" And that thread that we were already on.

(16:29): It's low barrier to entry. It gives people like me who don't have a lot of data engineering experience an area to explore for potentially just low code solutions for specific problems that our clients are facing. Yes, I can go knock that out in Fabric now, or this is something that we can design and implement in Fabric and we don't have to necessarily recruit a data engineer for 100% of what you're trying to do.

(16:59): And just going off of that previous slide that we were talking about with the business value, I sat through a session where another consulting firm had basically showcased this project that they'd worked on for a client, and they started out with this huge schematic of multiple data sources.

(17:17): You've got Hadoop, you've got SQL Server, you might have Snowflake, Oracle, a bunch of different systems, all hosting data that you need to consume; and their solution was, instead of figuring out the path from whatever storage system you're in to Power BI was, "Let's just consolidate everything in this one drive for data, the OneLake system," and from there you've got these semantic models, these rectangles that have already been pre-built or have the capability as a Power BI developer to go in there and say, "This is the exact semantic model that I want to build."

(17:55): All of my data's in this one place. I don't have to worry about getting access from it to a bunch of different backend data systems. I also don't have to worry about building views in SQL Server for tables that have already been built because what I'm looking for has probably already been named and created in OneLake, and I can just go over and take this object, drag it into the semantic model that I'm trying to create online, and boom, I've got all my data right there.

Rob Collie (18:22): You hit on one of my core themes. One of the things about P3 that makes it unique, especially compared to traditional consulting firms in this space, is what I call the decathlete model. So many roles that traditional consulting firms have a single person or one or more persons for each role. Every time you need a new role, you need a new person.

(18:44): The ETL specialist, the data engineer that does the backend plumbing, you've got the designated person who talks to the client and tries to understand what it is that they need and then go back and translate that to the rest of the team. You've got the data modeler who's building the star schemas or whatever. Then you've got the formula writers that are...

(19:02): Everything is like its own little silo, and so you end up with a team of a half a dozen people, not to mention report designers and report developers; and the cost of dragging that team around, putting a single dot on a single chart requires a tremendous amount of coordination. It's just an impossible model. It's shocking that this model ever was believed to work. We consolidate so many of those roles in single consultants, why we have the term "principal consultant."

(19:32): And the decathlete. Maybe you wouldn't win the Olympics of DAX, but if there was an Olympic sport that involved talking to clients, understand what they need, report design, DAX, Power Query, M, data modeling, star schema, now, you're going to be on the podium, and that allows us to provide a level of service to our clients that's just magical by comparison.

(19:54): When I first heard of Fabric, I'm like, "Oh, look, they're going to take all of the things that are still out of our reach, out of our principal consultant reach, all those really, really, really sharp, pointy things that are just slightly beyond our fingertips, and they're going to make that stuff doable by our principal consultant. I was like, "Giddy up. Yes." But then I'm kind of like Mark. I'm like, "Now, what are those things actually valuable for? Those things that were outside of our reach, what are they, really?"

Mark Beedle (20:20): The same thing that Power BI did: you had multi-dimensional; you had to be pro-dev; it took years to implement; and then you had MDX grossness. Then Power BI democratized BI for the data analyst. Fabric is going to do the same thing: democratizing data engineering.

(20:40): You're saying this makes P3 principal consultants even better decathletes, but it also makes the business analyst in accounting able to get their squishy data from their software as a service that isn't stored in... They don't even have a source system, it's just a cloud service that's curly and squishy, that accountant can do data engineering work with the point and click Power Query, and they have a lakehouse. It really does open the door for citizen developers to do stuff. That was so much harder before,

Will Gillingham (21:19): But even the reverse side of that, Mark, too: I talked to a client there who was a member of the data services team within the organization, and he worked entirely in Snowflake and Oracle, had never built a Power BI data model before, was not familiar with what it is I did for them; but we had this ongoing conversation during the course of that contract, where he wanted to understand more about what I did with the data once I got it into a Power BI report and started to model it.

(21:48): And now that you've got the ability to create this semantic model in Fabric, it lets that data engineer work more closely with me; or if they want to have the agency over the creation of the model based on the engineering work that they've already done in the back end, then I can hand that off and still let him have a say in what the data is that I'm bringing in, but now they're getting a lot better understanding of, what does this end state semantic model need to look like for the Power BI report to have an impact?

Rob Collie (22:24): There are a couple of points there that I think are really fascinating. First of all, it hadn't even occurred to me that this might be opening the door for some data engineers to come out from underground. All that plumbing that they're digging and drilling and everything, like dwarves in Lord of the Rings that are subterranean species, if they want, they can come out into the light where this stuff actually meets the humans.

(22:49): In the same way that Excel people grew into BI through Power BI, a lot of data engineers might find, "Oh my gosh, I've been held at arm's length from the business value side of things." By the way, it's always really, really good for one's career to get closer and closer to the business value side of things.

(23:09): Now, a lot of people, frankly, especially in the tech world, who derived tremendous comfort from being at great distance from the business value, this is a very comforting world where building the lakehouse is what you do. Seems simple. "I've walled myself off from all that scary, unpredictable human shit," and it turns out that you haven't, you just don't have any say in it. I imagine that some percentage of data engineers will be excited by this, whereas the majority of them probably will by default stick to pure data engineering.

(23:42): Aside from that, the second thing that you mentioned that was really striking to me was that on a larger team where everyone does still kind of stay in their lane, the more each role understands each other and can anticipate each other's needs and understand things quickly, the better.

(23:59): A data engineer, even if they do want to stay firmly rooted in the plumbing, if they're willing to peek over the wall periodically to where the faucets are, they're going to gain a greater understanding and they're going to be a better partner. And likewise, you learning more about the data engineering side will be a more informed consumer.

(24:17): And we do hire data engineers here at P3. The ratio of principal consultants to data engineers is really high. We don't need many data engineers. It'll help our internal relations between our data engineers and our principal consultants.

(24:32): So you were mentioning, Will, activities, like data engineering style activities that would've been out of reach. Are there any examples that come to mind after going to the conference? Like, "Oh, here's two or three things like that that are now coming into range for me, Will, that wouldn't have been in range before."

Will Gillingham (24:49): Yeah, the main thing was seeing that you could build a medallion architecture in Fabric that previously would've been out of reach for me. We had a client who built their medallion architecture in Azure, and Chris was a big part of scoping that call. Chris would still be a part of that call, because most of the time, I'm not going to really be able to ask those technical questions to get to, what should the end state be? How should we set these different stages up for you to bring in your raw data, go to the silver layer and rename columns or set data types, but that you can move to an end state would be considered the gold layer, but that is now or potentially going to be a capability that you can implement in Fabric.

(25:42): That was exciting to me, because that was previously an area where I have no Azure experience. I don't feel comfortable going into SQL Warehouse building pipelines and setting up this type of structure to get data from raw to consumable for a Power BI report; but if it's in OneLake, I feel like I know enough now, because the system with Fabric that they've set up is software as a service and it's low barrier to entry. I think with enough clicking, you can get to a reasonable solution for medallion architecture in Fabric. Previously would not have been able to do that,

Rob Collie (26:18): And why would I want a medallion architecture? Kind of sounds like Flavor Flav with his giant clock he was wearing around his neck, right? Is that what we mean?

Will Gillingham (26:28): Yeah, you can certainly take it that way. I will say, it's going to be for the benefit of me, as well. We'll see if I can phrase it correctly. Essentially, with the medallion architecture, what you're doing is you're taking data in a raw state, and by raw we mean with no transformations on it or as little as possible. It's what the source system brings in. You could have an ERP system that just spits out a table or has a table available that you can reach for that has a convoluted name of the table, convoluted columns. There's no set standard data types as you bring them in, so that would be your bronze layer.

(27:06): And from there, you have a connection to this silver layer, and the silver layer is where you start to put in some transformations in that connection from bronze to silver and maybe rename some columns. You could set data types. You could correct errors in the data if there are any. But you're doing some of these ETL or extract, transform, load processes from Bronze to silver in order to get this silver data into somewhat of a understandable format.

Rob Collie (27:36): A human being could look at the silver layer and say, "Okay, I kind of know what's going on here," whereas looking at the bronze layer, "No way."

Will Gillingham (27:43): Yeah.

Rob Collie (27:44): Then take me to the gold layer. What does the gold layer do? Because the silver layer sounds good enough to me.

Will Gillingham (27:49): The silver layer is good for developers. Us as Power BI developers, when we're not reaching for this raw data or this Bronze layer of data, it's typically the silver layer where you grab the tables to model them. You might have four different tables that you're taking from the silver layer and you're going to put them all together into one dimensional table in your semantic model for reporting purposes.

(28:14): Now, the Gold layer is where you can finesse it a little bit more. My transformation for a gold layer table: taking four tables from the silver layer that's already report consumable, I could do all the merges, all the SQL commands, all the joins that might be needed. I could put together columns that calculate metrics that I need, and I put them all into one command that moves four silver tables into a Gold table.

(28:42): That is my one final source dimensional item that I need to bring into my semantic model, and then that's just the single table that I need to grab in my Power BI report and use for my reporting purposes.

Rob Collie (28:55): That makes sense. So there's silver layer, which is I can actually understand the data, but it's still not assembled in its most convenient format. From silver to gold transformation, it would be sort of hoarded by the Power BI model. It would only be in there and not available to the rest of the world. If I ever wanted to do anything else with it, it would be hoarded.

(29:17): And I think this is the other really, really, really, it might even be I think the most impactful business pitch for Fabric is that the Power BI models of literally yesterday, we put so much effort into them getting the data right and then improving them with business logic, relationships and formulas and metrics and expressions, and these Power BI models became the single most valuable reference point about what's going on in that corner of your business, whatever data is involved in it.

(29:54): If this is your digital marketing model, this is the far and away most valuable source of information in your entire business about your digital marketing operations, and the problem is that that information could only really be used by your Power BI reports. There were no other consumption endpoints that it was available to.

(30:14): Most notably, if you ever wanted to do anything with AI, machine learning, or LLM, anything on that data, you'd have to start from scratch and recreate all of that beautiful, beautiful, beautiful business logic and data richness in a completely parallel format, which is going to be, by the way, feeble by comparison, because there's nothing like a Power BI semantic model. It is the richest, most flexible possible endpoint or source point for this kind of information.

(30:44): Even though you could go spend all that parallel effort and still end up with something that sucked by comparison, but only because other consumption points, AI for instance, couldn't talk to the Power BI model. We've talked about a lot of benefits here, but I think this one's the apex predator benefit. Do you agree? After coming out of the conference, should I keep saying that?

Chris Haas (31:09): Absolutely. The whole idea that now we have information to make a better decision, and now we can share all of that information with other parts of the business so that they can make better decisions and they don't have to redo all of that work, it's just already there to then build upon; and that we're able to build upon that again and again, there's definitely a force multiplier in play that just didn't exist before.

Rob Collie (31:41): The thing that triggered that thought was that Will's description of the medallion architecture, which by the way, Will, I thought when I asked you to describe medallion architecture, I was really setting you up to fail. I was setting you an impossible task. I think you just threw down the best definition of medallion architecture I've ever heard. In fact, it'll be the first one that I remember.

Will Gillingham (31:59): Thanks. I was looking at Chris the entire time waiting for him to jump in and rip me apart.

Rob Collie (32:04): Could you have given that answer before going to the Fabric Conference? Did you already understand medallion architecture that well, or did you crystallize it as a result of being at the conference?

Will Gillingham (32:13): I understood what it was before I went only because I had Chris explain it to me, but once I got to the Fabric conference, it was like, "Oh, I know that this is something that our clients can use."

Rob Collie (32:24): So Chris gave you the silver layer description of medallion architecture and then you just turned around and gave it to me in the gold.

Will Gillingham (32:31): I did my best. I gold plated it.

Rob Collie (32:33): Maybe that was the platinum level, even. I don't know.

Mark Beedle (32:36): I'm going to try to one-up will real quick.

Rob Collie (32:39): Oh, yeah, do it.

Mark Beedle (32:40): I can't take credit for it. It was in one of the sessions at the conference. They were talking about medallion architecture, and we throw around bronze, silver, gold, and then what about platinum? What about something before bronze? And the presenter was like, "They don't mean anything. It's just like a progression." He was like, "Maybe you name it Pikachu and Raichu. Your gold layer is Raichu." So I thought of you whenever...

Rob Collie (33:06): That's a low blow, Mark, bringing Pokemon into it. There was a backstage conversation, folks, that we're not going to get into. I see you, Mark. Was that just an elaborate ploy to get Pokemon into the conversation?

Mark Beedle (33:24): That really did happen, though. That really was in the presentation.

Rob Collie (33:27): I like that raw, semi-raw, and then enhanced description of medallion architecture. I'm also dying to write a blog post where I find some late '80s, early '90s picture of a gathering of the Beastie Boys and Run DMC and Public Enemy. Remember there was an era, maybe you don't, there was an era when hood ornaments on cars were constantly being stolen to be turned into things that you wore around your neck? Like dinner plate sized. Hood ornaments used to actually be detachable and eventually the auto industry just decided, "No, we're just going to burn these into the grill of the car." They'll become a structural element so no one can steal them and wear them.

(34:14): Anyway, the medallion conversation and the fact that the gold layer now being in OneLake as opposed to being trapped in a single Power BI model really underscores this end point you have for consuming this information about your business. It's going to be available and not locked up. It just seems to be a huge theme all the way around.

Chris Haas (34:40): I think that's what Power BI was when it first came out. It was finally that piece of accessible software and a tool that could connect multiple dots into a single place and provide incredible value. Now, we've gotten to the point where the fact that it's its own [inaudible 00:35:03], which doesn't talk back to other things, was the limiting piece. And now Fabric says, "Nope, that's just part of the whole pie." It gets to talk to everything, and because everything can talk to each other now, that's where the intrinsic value is created.

Rob Collie (35:19): And imagine you've got hardcore software engineers on staff and there's a particular custom application that you want to build. Amazon dynamically adjusts the price of products on their website all the time. There's no human being doing it. It's all algorithmic. And if it needed source data on the history of pricing and consumer behavior and everything, and let's say that it made sense to capture that in a Power BI model.

(35:50): And maybe that's not a sensible example, but it might be, and now your software engineers want to take advantage of that richness to power the algorithm that is adjusting pricing in real time all the time, the tools that they use, the tools that they're familiar with, those software engineers, can now access this data without them having to go learn some funky XMLA or ADOMD or something that has nothing to do with anything they've ever learned and probably isn't even convenient if they understood it, probably doesn't actually fit their ecosystem for good reason. Now, even something completely bespoke, it's not even another piece of the Microsoft ecosystem, it's something they're going to build from scratch, can access this stuff.

(36:35): I just use that as an example because that means all points in between are also in range. That is the most farthest out there example, is to build something at industrial scale that requires heavy duty software engineering and that can take advantage of the efforts of people like us who are building these rich semantic models. And I love that they're calling them semantic models again. It's pointy headed, it's nasty, it's a little too nerdy, but I think it's good. I think on net, the benefits outweigh the drawbacks, here. I love it. And then Fabric, I feel like there was a better name than Fabric,

Chris Haas (37:14): And imagine how the Azure product team that supports Azure Service Fabric feels.

Rob Collie (37:21): Yeah. As we've mentioned on the show, it's even confused Google. Microsoft Fabric, Google's like, "What are you talking about?" It's like the disambiguation at the top of a Wikipedia page. Google needs that. Like, "Okay, look, we're just going to level with you dear search user: Microsoft doesn't know what it's doing with naming, so we need you to provide a little bit of metadata to your query. Are you looking for the thing in this category or that category?"

(37:47): All right, so going into the conference, Chris, you had, not skepticism, but a different set of questions. You're one of the handful of people at P3 who can do the data engineering things. You don't need no stinking Fabric. What was your experience coming both into and out of the conference?

Chris Haas (38:06): Well, I'm just excited that apparently I'm one of the cool kids in Mark's eyes because I do write PySpark.

Rob Collie (38:12): Let's be clear, when Mark and I talk about the cool kids, we don't really think they're cool.

Chris Haas (38:16): Rob, don't take this from me. This is my chance to be the cool kids.

Rob Collie (38:19): Okay, but you're like our ambassador to these cool kids. These cool kids we're talking about, they're dressed west coast slick. They only carry Macs. They would never use a PC. If you're one of the cool kids who uses a PC, you're actually a cool kid, not the air quotes cool kids.

Chris Haas (38:35): Got it, okay. I've been using Synapse for I'd say a solid two years now, and High Spark Notebooks have been my weapon of choice. I've really been able to help customers understand, that, "Hey, we know what we want to do in Power BI and we have all this other stuff over here. I kind of feel like the Underpants Gnomes in South Park, and I've got an idea of what that phase two is with the big question mark to get to profits."

(39:05): And that's where Synapse has let me bridge that gap into providing what was needed for the model to actually provide value, because at the end of the day, it's not about making the model: it's how do we make a model that provides that value to the business? And Synapse was, for the most part, doing it, doing everything it needed. So I came in where Fabric says, "Hey, we can do all of that too, but better." Yeah, color me interested.

(39:34): I was focused on the two or three differences in feature parity and I said, "Oh, well, because I can't do this, I don't know if that's even worth it," and then as I stayed in the conference longer, went to more sessions, talked to more people, I realized, yes, Fabric does all of these things really well. It also does all of these other pieces that Synapse can't and Power BI can't, and that's really where the value is.

Rob Collie (40:02): Can you give me an example of something like that?

Chris Haas (40:04): Sure. Getting data from the raw sources just into the lakehouse in that first step, there's still a couple pieces where Fabric is getting up to speed with Synapse. So connecting the on-premise data, it's getting a lot better now. Just in public preview, they've announced more connections into on-prem data. Before that, Synapse had an easy win. I'd say, "You know what? I'd rather just do it in Synapse."

(40:33): Now, I have the ability to have that cake and eat it too. And talking to a lot of people, other participants in the conference and even some from Microsoft, they said, "Do that part in Synapse, that's fine. As the next step, push it into Fabric into the lakehouse, and now you get all of the benefits of it being in Fabric with the ease and additional functionality which will come to Fabric," it's just not there yet. But it's also a case of, "Hey, you don't have to rebuild everything that you've already done. Just go chase the value."And the value is once that data is in Fabric, now we're able to have that force multiplier.

(41:17): It really helped me to kind of re-embrace that faucets first idea of I don't care about fixing everything. Eventually I will, but I'm just going to add value to the data closer to the business, because that's where the value of the data is actually realized.

Rob Collie (41:33): Delay that level of investment until the point where you actually know what the requirements really are.

Chris Haas (41:39): Precisely.

Rob Collie (41:40): That move saves you 95% of your effort, because so much of it is wasted when you're trying to anticipate in a vacuum. All right, so just closing out here, vibe at the conference. Amongst the attendees, excited? Then there's different flavors of excited. There's excited like fandom, right? They're just going to eat it up no matter what. The vibe amongst the people who were actually really being thoughtful about it that you ran into, what was that?

Mark Beedle (42:10): It was a very high energy conference. You could tell people were excited. Being in Vegas, I think, helped some of that. Who doesn't like being in Vegas? But in the sessions I was in, you heard skepticism in the questions. What was cool about all the sessions, at least I attended, there were always two presenters: one from the Microsoft product team that was being presented on, and then usually an external consultant.

(42:36): You could hear some bits of skepticism, so I was like, "Oh, they're my people. They're skeptical, too." Without missing a beat, usually the person from the product team would answer, "Yeah, we know that's a gap and we're fixing it next month," or, "It's on the immediate roadmap." I saw skepticism and I also saw Microsoft showing, "We see that and we're addressing all of those things."

Rob Collie (43:02): That's so cool. Any other reactions like that?

Will Gillingham (43:05): Every question was given an answer in a very quick turnaround. It wasn't like, "Let me check in on that and I'll send you an email," or, "Let's speak to each other after this session." I think there was just a very keen sense to learn from everybody there. I felt a lot of fanaticism in the opening event, right?

Rob Collie (43:27): Yeah, of course.

Will Gillingham (43:29): There was definitely a good amount of skepticism. I think Mark and I went into a session with a healthy amount of skepticism when they were talking about visual level calcs, and both of us came away like, "All right, there's an application." We're not just going to die on our horse that there's no use for these.

Rob Collie (43:47): Which hairless cat is that that we hear?

Will Gillingham (43:49): That's Steve howling in the background. I'm sorry about that.

Rob Collie (43:52): No, it just adds a little flavor. All right, last question along those lines: breakdown of attendees. What percentage of the attendees were business leaders versus hands-on techies?

Mark Beedle (44:07): I could be biased because of the sessions I attended, but it looked like hands-on techies were the majority of the attendees.

Rob Collie (44:15): Yeah, I would expect them to be the majority. I'm just wondering if it's like 95/5, 75/25, that kind of thing. For sure they're going to be more than half.

Chris Haas (44:23): I will say probably two to one techie to otherwise. Mark's beard was still top three.

Rob Collie (44:30): Top three beards. Yeah, that's the 11th event in the decathlon. Mark just cruises to victory every year. Oh, who's going to get silver this year? It's a race for a second.

(44:50): Listen, I really appreciate you taking the time to do this. I'm happy you all got to go. Sounds like you came out of it, all of you, just thoroughly changed, just transformed as individuals. Just kidding. It does seem like your opinions evolved quite a bit. That's why we do these things. So the next time they throw a FabCon, FabriCon, we should have a larger presence, it sounds like.

Speaker 2 (45:12): Thanks for listening to the Raw Data by P3 Adaptive Podcast. Let the experts at P3 Adaptive help your business. Just go to p3adaptive.com. Have a data day.