Fabric – It’s GA (GO) Time!

Rob Collie (00:00): Hello, friends. A departure from our usual Tuesday schedule, here we are coming at you on Wednesday afternoon Eastern Time. I'm here with Justin. Big News out of Seattle, the Microsoft Ignite Conference. It appears that talking about Fabric as something in the future is a thing of the past. See if you can untangle that sentence. I think we're just going to do a quick episode, just live breaking news, sort of reactions. Luke, can you give us a quick sort of abbreviated version of the intro?

Announcer (00:30): Ladies and gentlemen, may I have your attention please.

Announcer (00:31): This is the Raw Data by P3 Adaptive Podcast with your host, Rod Collie, and your co-host, Justin Mannhardt. Find out what the experts at P3 Adaptive can do for your business. Just go to p3adaptive.com. Raw Data by P3 Adaptive is data with the human element.

Rob Collie (00:42): All right, Justin, exciting day today.

Justin Mannhardt (00:45): Big day.

Rob Collie (00:46): Big day in our corner of the woods.

Justin Mannhardt (00:48): That's right. Long awaited.

Rob Collie (00:49): Today at the Ignite Conference, it's official, Fabric; you can pay for it, you can use it. And when they announced all kinds of new things that were heretofore unknown, we should do a quick rundown, shouldn't we?

Justin Mannhardt (01:04): There's a lot to cover.

Rob Collie (01:08): And I'm glad you've been digesting it in real time. So obviously the biggest thing right now is that you can buy it, for real.

Justin Mannhardt (01:14): Generally available for purchase and also announced is something we weren't super clear on, just to follow up on pricing. So there's two models for pricing, a pay-as-you-go model and a Reserved Instance model. Reserved Instance model is going to be just over 40% discount from the pay-as-you-go model.

Rob Collie (01:34): What is the smallest Reserve Instance model I can purchase? Do you happen to know?

Justin Mannhardt (01:38): I believe it's around $260 a month.

Rob Collie (01:41): That's great.

Justin Mannhardt (01:42): Actually, I think that's the pay-as-you-go price. So it'd be more like $190... I don't do math in my head.

Rob Collie (01:51): Yeah, 40%. [inaudible 00:01:50] somewhere. Somewhere between $100 and $200.

Justin Mannhardt (01:53): Yeah, somewhere less than that. So the Ignite Conference has started, there's three days of sessions going on. Some's online. There's actually a conference in Seattle going on. And in all of the announcements that are being posted to the blogs, there's literally over a hundred pages of new things.

Rob Collie (02:11): A hundred pages.

Justin Mannhardt (02:12): But there's a few things that really caught my attention. First is one that we've all been kind of waiting for and weren't really clear when it would happen or how, is Copilot is coming to Fabric. We've seen the announcements about Copilot for Sales or Copilot for Microsoft 365. Copilot is now available for public preview. I believe there's going to be an opt-in still, so there'll be a form you can fill out. But I'll just a little snip on the announcement, you're going to be able to use natural language to create data flows, pipelines, write SQL statements, build reports, develop machine learning models, et cetera, et cetera, et cetera. This is wild.

Rob Collie (02:57): All right, so I can just tell it, "Go do X, Y, Z for me," or, "Write me code that does this," or whatever. Right? That was a long list of things that you rattled off there. The only one I had seen was the one that helps me build reports against an already-created data model. You were saying things like it will write me SQL code, it will develop me machine learning models. I didn't hear DAX in there.

Justin Mannhardt (03:19): So on the DAX front, specifically in the Power BI space, a lot of our listeners are probably familiar with the smart narrative visual. So that's being rebranded to the narrative with Copilot. So now Copilot with visualization and written summary of the data is coming together. It's already actually been in Power BI Desktop for a few months. What used to be known as Quick Measures got replaced with Copilot. So that DAX authoring, it's kind of hidden in there. And this is really interesting and exciting time to just think about how far does this go with natural language and how practical it'll be. We've talked about the hallucination situations and things like that.

Rob Collie (04:05): Well, it's certainly something that also generates a lot of unease. So in the last podcast, I introduced a new term and I think I want to make this a thing. So can we make it a thing? FOBO.

Justin Mannhardt (04:16): FOBO.

Rob Collie (04:17): Fear of becoming obsolete. All of this new stuff has two different ways in which it triggers FOBO. One of them is, "I'm not learning enough." There's all these new things that are coming out and if I'm not learning them, I'm not learning them fast enough, I'm falling behind." And that's bad enough. But then Copilot comes along and says, "Maybe even the things you already know, maybe they're not valuable anymore. Maybe everyone can do this now." Whether we admit it or not, every person in this space feels a little bit of unease about this sort of thing. Right?

Justin Mannhardt (04:50): Totally.

Rob Collie (04:51): What's ideal? We're kind of rooting for it to be helpful and good, but we're also rooting for it to not quite be good enough to completely replace all of the intelligent human beings that are doing this work. And I've been to the rodeo long enough to know that it's really hard to replace the humans. But where? Not knowing where.

(05:12): If we all knew sort of confidently where this Copilot stuff was going to land three years from now, I think we'd be comfortable with whatever that story was. We probably shouldn't try to get into that today. That probably is its own episode, the Copilot FOBO episode. Today, we want to focus on just sort of the new stuff. So a lot of Copilot stuff.

Justin Mannhardt (05:32): Yeah, a lot of Copilot, public preview.

Rob Collie (05:35): Here's another one that you did clue me in on. I can take an existing import mode, Power BI model and just OneLake it.

Justin Mannhardt (05:43): That's right. I was not expecting this.

Rob Collie (05:46): Yeah, so I don't have to start over.

Justin Mannhardt (05:49): To me, it's awesome. Because we've been talking about how much more valuable the Power BI type person is in this scenario. Before I'm like, "Well, man, if I've got these import models that I've invested all this time in, where does Fabric come into that?" Now it's like, oh, it does. That's great. You've exposed this data that's been enriched and turned into truth, and now it's available to all these other workloads and possibilities. I don't have to start over.

Rob Collie (06:19): And is this capability in the product today? Did they just lit it up, or is it coming?

Justin Mannhardt (06:24): This will be a preview feature. So Fabric's GA, right? But there's going to be features within Fabric that are in preview. I believe this is going to be in preview. It's a theme that's become more and more evident to me with Microsoft Strategy with Fabric is to allow customers and people to use their data where it is with Fabric. There's a lot of other interesting things in the announcements like integrations with Databricks and Snowflake and Synapse and Google and Amazon. Wherever your data's at, we're cool.

Rob Collie (07:02): This is again, not the Ballmer and Gates Microsoft that I grew up with.

Justin Mannhardt (07:06): Yeah, I kind of combined two things there. So there's the Power BI, convert that into OneLake.

Rob Collie (07:14): Can I push you just a little bit more on that thesis for a second?

Justin Mannhardt (07:15): Yeah.

Rob Collie (07:16): So it's cool, right? I can take my existing Power BI thing and I can get the fancy new hotness. And the reason why that's exciting is that now lots of other things can access that same data, that same data model with all of its semantic richness, with all of its relationships; the business logic, expressing calculations, all of that is now available to other things like AI, for example. So that's cool.

(07:41): I assume that it also gives me, suddenly I have Direct Lake capability, but my refresh is still going to be on a schedule. I'm not sure I'll notice the difference between Direct Lake and other mode yet. There's some really weird stuff, and I assume that all of my reports that were pointing to that Power BI data model will now point to the OneLake version and they won't just stop working. What do I gain and what do I lose, is going to be really interesting to find out. And so, again, given that this is a surprise feature, we haven't played with it yet.

Justin Mannhardt (08:14): We haven't played with this yet, and I'm not sure on your specific question, and I don't think it clarifies in the announcement either, this is one of those things where I'm like, I assume they're going to have an option for that at some point.

Rob Collie (08:25): You assume they're going to do the rational thing.

Justin Mannhardt (08:26): I'll assume. Maybe it'll take time. I don't know. But we've worked with several customers, is they'll have this legit data model, good reporting, but then there's all these ad hoc needs that the company has. I can't always drag, drop my way through this. Sometimes someone wants to do something with SQL, someone wants to do something with Excel, somebody wants to do something in AI or machine learning. It just makes that asset of a data model more useful, more valuable in more places and more ways.

(08:59): In addition to some of the Copilot-oriented features that are in Fabric. So there's another feature, I think it's called Explore, where it makes it like an end user can get around the data model much easier in using this net. So it's like having the data in OneLake seems really useful.

Rob Collie (09:18): I was looking at it going, I don't get it. But the way you just described it, I understand it a little bit better. I look at a data model today and I have no idea. If I didn't build it, if it was someone else's. I've got to go through a process of looking at the diagram view, and I've got to do all kinds of things to wrap my head around how I should use this field list.

Justin Mannhardt (09:39): It can be intimidating if you're not familiar with it.

Rob Collie (09:42): Well, especially if that field list doesn't give me access to the scheme of view, the relationship view. How many people in the world is that relationship view going to be helpful to? I'm one of them. It would be harmful, actively harmful to a lot of people. So having a guide to what the data model can do for you, holy hell, could have used that forever.

Justin Mannhardt (10:04): There's a tieback in my brain that just occurred on the FOBO thing. So we're talking about Copilot and this explore feature and a lot of other things. All of these features, their effectiveness have always come back to the quality of that central asset. And getting that right, sure, there's a lot of technical capability that goes into getting that right, but there's also a tremendous level of business logic and rationale that goes into getting that, right?

(10:31): Yeah, there's new tools, there's new features. The need to solve the business problem doesn't go away, I guess is my point. There's still going to be a lot of human input required to make all these tools useful. It's not like companies are going to have a push button for their data stack top to bottom.

Rob Collie (10:48): Having things like Explorer, you were just mentioning, right, means that the data model can become more valuable. This is one of those themes that we keep seeing, right?

(10:58): Let's say you were in the business of mining and refining aluminum. Before soft drinks became a thing, right? Before refrigerators were a thing in grocery stores, before the aluminum can had been invented. One day, along comes all these different ways in which aluminum can be used, which we never thought of before. Airplanes and all that kind of stuff. Digging aluminum out of the ground and refining it is the same amount of work, but there's a lot more ways to use it, A lot more demand for it, so the price per pound of aluminum goes up, the value of your service goes up.

(11:33): So even something like Explorer, even that, even something as simple as that... And I remember building a data model for a massive, massive, massive company, and it was amazing. It was like the thing that couldn't exist. It was so amazing what it gave them. I also had to go back and train a whole floor of analysts on how to connect to it with Excel in a field list and get the information out it they wanted. It wasn't about teaching them how to use the Excel wizard to connect to this SSAS tabular server. It was about the field list, what you can and can't do. It's like, you got to use something from this table. When you use something from this table, you think you might be able to use it from over here, but you can't because they can't see the relationship view, and if they did, they wouldn't understand it.

(12:18): What other new stuff? A hundred pages of updates, what does that mean?

Justin Mannhardt (12:21): No, that's the page count. Those are from the November updates on both Fabric and Power BI put together.

(12:28): So there's updates to the typical stuff like, oh, there's new visuals and all that kind of stuff. So there's just a lot all at once. I'm going to categorize this. A category of features that I wasn't expecting or anticipating them and excited about fall under customers that are asking the, "How do I migrate to Fabric?" question.

(12:47): So there's a feature in there - this is preview, it's new - called mirroring. This is sort of the shortcut thing. You've got a MongoDB database or a Cosmos database or an Amazon S3. There's a list of what these things are. You can mirror them to Fabric, and without any movement of your data, it looks like a Fabric lakehouse and warehouse and default data set.

(13:14): So if people are excited about, "Oh, my end users like to write SQL or they like to do DAX, or they like to do this. Oh, okay, I don't have to move off of my current data warehouse to benefit from Fabric. I can just go." The migration conversation used to be a really pain-in-the-ass conversation. And so there's all these features that are revolving around this topic of, just use your data where it is.

Rob Collie (13:40): But I have a hard time believing that doesn't come without some significant trade-offs.

Justin Mannhardt (13:45): There probably are.

Rob Collie (13:46): If leaving your data where it is just as good, why did they spend so much time investing in the new OneLake storage format?

Justin Mannhardt (13:54): That's what's interesting is what happens in OneLake, this is technical, is you get the Delta Parquet metadata mapping to the other thing. So there is probably some level of performance trade-off.

Rob Collie (14:09): There'd be a massive gain to not leaving it in Mongo, right? There'd be a massive gain to actually ingest.

Justin Mannhardt (14:18): I'm excited to find out, because I don't know.

Rob Collie (14:20): Either way, it's still huge, right? Because it allows you to start gaining access to these features and functionality and all that kind of stuff, without... Guess what? Without doing much of plumbing work first, which sounds very much up our alley.

Justin Mannhardt (14:32): Exactly.

Rob Collie (14:33): And then once you've sort of proved it out, made sure that you like it, it's configured the right way, and you can validate that there's ROI here. You say, "Oh, now we can make it faster if we want," or gain these other additional incremental benefits, and here's the cost of doing it as opposed to going and paying that cost before you learn anything else. And I completely respect that sort of approach. I'm very suspicious of them saying, "Yeah, yeah, no problem, no problem." Right? Yeah, well.

Justin Mannhardt (15:02): It's still servers, Rob. It's still servers with CPU and RAM, and-

Rob Collie (15:08): Again, the philosophy and the amount of work that went into it is admirable. They didn't get this for free. They had to go build this. They had to go build the behind-the-scenes adapter to make all the pass-through work. They put real engineering effort behind this. We want our data overlords to be so merciful.

Justin Mannhardt (15:28): Yeah. So that's a compelling feature set, especially if they could figure it out. Because it is about the plumbing. That other database isn't just some database. It's not like the backend of an application. It's like the thing that's getting fed and processed and there's all this stuff happening, and somebody might say, "Well, we'd really like to do things with data activator. We'd really like to do something with empowering more of our company." And it's like, "Okay, well let's eliminate this huge move as a prerequisite to that."

(16:00): We'll see, I got to try all this out. I heard a story about Snowflake in this scenario and how impressed they were with how they went from wiring up Snowflake to building reports in Power BI in a matter of minutes without moving any of the data. Okay, that seems kind of cool. But again, where's the trade-off? Where's the ceiling?

Rob Collie (16:23): And I bet there's cases like your Mongo, your Snowflake, whatever. Let's say you're on a pay-to-use model with those services, and now you wire Fabric up to it. You start generating all these queries and activator, for instance, in particular kind of needs to see everything, doesn't it? It's constantly asking about all these things, because it doesn't even know what it's looking for. What is your Mongo bill going to look like that month? The last episode, we talked about Fabric. We were attempting to be skeptical, and I think we largely failed. I did.

Justin Mannhardt (16:58): You're doing a good job at the moment.

Rob Collie (16:59): I'm smelling some marketing sleight of hand here. There's going to be some horror stories with this pass-through mode, and everyone's going to try it and come to the conclusion that it's going to be better if you bring it into the OneLake. But they're wise, right? They're not going to come out and say, no, no, no, you have to play our way or you can't play.

Justin Mannhardt (17:20): And this is true today. Power BI plays better with certain things. It just does. I think that'll continue to be true. So I'm sure Fabric playing with Azure Synapse compared to Snowflake. Right? There's probably same core technologies. I think it's just going to be different. We'll have to see what it's like in terms of the Delta. Is it material enough that it makes a difference for a given situation? I don't know. I'm excited to dig into this stuff, obviously.

Rob Collie (17:50): All right, anything else jumping out at you?

Justin Mannhardt (17:52): Just some honorable mentions and I think would just encourage people to go read the blogs. So there's a cool integration called Microsoft Graph Data Connect. So this will be more appropriate for system administrators to get a very cohesive view into your Microsoft 365 landscape. What's happening with my SharePoint and my Outlook and my systems? Because that's now integrated with Fabric and they can use this whole utility.

(18:23): And there's an interesting read for software vendors about using Fabric in the build of their platforms. So today, there's Power BI embedded, and they kind of describe these three major ways you could incorporate Fabric into another software offering. You could build your backend workloads in Fabric now. You could build apps on top of it, some interoperability between your app and Fabric.

(18:49): So that was a neat read. So if you're in that space and you're looking for some analytics-oriented solutions for your own software, that might be worth a look.

Rob Collie (18:59): Yeah, I bet the possibilities there are damn near endless.

Justin Mannhardt (19:03): Well, and so an example on that is a company like Informatica. They build ETL type tooling and stuff like that, and so they can develop software that does stuff in Fabric or building an entire application that runs on top of Fabric.

Rob Collie (19:18): That's fascinating.

Justin Mannhardt (19:19): Yeah.

Rob Collie (19:19): So four or five lifetimes ago for me, four or five complete different personalities ago for me, I worked on Windows Installer, the MSI technology at Microsoft. The Windows 32, Windows 64 software installation setup is the most boring and thankless and incredibly complicated and tedious thing.

(19:41): One of the things that bothered me until a light bulb went on was this was being given to Windows as to become the standard for how software should be installed on Windows. The InstallShield team, not a team external company, InstallShield, which most of the world's setups, most software setups for most software in the world for Windows was InstallShield. That was their customers. And so they came to visit.

(20:09): On the face of it, this thing we were building was like the Death Star to them. Bye-bye. But here's the thing, we weren't building any authoring tools. We weren't building any tools that helped you develop an MSI setup. And they're like, "We don't care about the engine." They were actually excited about what we were doing. This was good for them. It took me a little while to wrap my head around it. You would've thought, again, that we were obsoleting their entire business. They're like, "Oh no, no, no, no. You're taking some of the work off of our plate. And in fact, you're giving all of our customers a reason to buy the new version." And so they were super happy about what we were doing. Super, super, super happy.

(20:59): So on the face of it, someone like Informatica, why would they do that? But it might be this is a godsend for them. I don't know. You never know.

Justin Mannhardt (21:06): Or just areas where it opens up more market potential for some of these other customers, where it was difficult for their customers to cross pollinate, or whatever. It's really interesting to think about the jump from like, "Oh, we do Power BI embedded stuff." Fabric could do all kinds of stuff in the context of software.

(21:26): So the hundreds and hundreds of pages, there's a lot of technical stuff in there. A lot of it's important for enterprise workloads. You want to be aware of like, okay, does this thing have all the bells and whistles and security and nuts and bolts that I need to be comfortable? So a lot of announcements in that category. It's time. It's GA.

Rob Collie (21:44): It's GA time.

Justin Mannhardt (21:45): It's GA time.

Rob Collie (21:46): It's GA time, people. Enough of this, then. Let's go dig in. Right? Let's go turn one of those things, one of our existing Power BI data models, into a OneLake instance and see what we get and what breaks.

Justin Mannhardt (22:00): Let's do that, and then let's ask Copilot to analyze it for us, and then let's spin up some MongoDB and get Ed Hansberry to figure out when the query performance starts to suck wind.

Rob Collie (22:14): I'm going to guess immediately.

Justin Mannhardt (22:16): Immediately.

Rob Collie (22:17): There's my skeptical nature, alive and well.

Justin Mannhardt (22:20): There's a ton to chew on out there. We're excited to just start playing with some of these new preview features and understanding where it plays and what we like we don't like, and I'm sure there'll be a lot to talk about down the road.

Rob Collie (22:32): Well, stay tuned.