The Biggest Improvements Announced at FabCon, and the Ever-Widening Gap Between Adoption and Capabilities

Rob: Hello there Justin. It's been a minute. We've been seeing plenty of each other

Justin: In the flesh.

Rob: But not seeing any of each other over the top of podcast microphones. People listening to this, they're aware of this, that it's been about a month since we've had anything.

Justin: Sadly, this is true. We apologize. We've had a number of business matters, off-sites, meetings, strategic summits where we discussed the strategery of P3.

Rob: Yeah. I got on a plane and visited you. You got on a plane and visited me. [00:00:30] So now, that business thing getting in the way of our true love of podcasting.

Justin: It's true.

Rob: The business makes the podcast possible. So here we are back to the pod and I thought we would start with a head exploding revolution for me.

Justin: Okay.

Rob: Which is that around 2013 when I was heads down in consulting work for clients who by the way, never ever, ever once had a business calendar that followed the normal month on the wall calendar. I'm not even talking about fiscal [00:01:00] calendars where the year starts on a different date, right?

Justin: Right.

Rob: No, they didn't have the concept of month. They had truly custom calendars. There was no such thing as a real world calendar in any of these clients. And I'm writing in a land before the switch function, in a land before variables. I'm writing just increasingly tedious, if this then that, if this then that type of like you're at this level of the report, this level of filter context, do this and this level of filter context, do that. Some of those formulas are again, [00:01:30] like pre-variables are just a page of nested ugh.

Justin: Did you adore this work product or did it make you cringe?

Rob: A little of both. I adored it when it was completed. It felt awful teaching it to others when I was teaching classes. So how many of you all have real calendars and how many of you all have custom calendars in your business? And everyone raises their hand for custom calendars. And I'm like, okay, here we go. It would reach the point where I'd be just like, "Look, I'm just going to give you the formulas. Okay? Take them with you, these page-long formulas, and if [00:02:00] you need help adapting them to the columns in your model, let me know." And it occurred to me that many years ago, that this was a solvable problem at Microsoft, or at least it seemed like it might be.

Justin: Sure.

Rob: I had this 50:50 feeling that they could give us custom calendar versions of the standard dates functions, prior period, previous year, YTD, whatever, right? And allow us to designate which columns of the model correspond to year, which columns of the model correspond [00:02:30] to quarter, whatever. Right? And I felt like they could handle that behind the scenes. But I wasn't positive and I didn't really like get up close and personal talking to Jeffrey Wang or Marios Dimitriou about this. I just sort of like floated the idea and moved on. And so many years have elapsed that I came to the conclusion intuitively that just wasn't possible.

Justin: Right.

Rob: Because clearly, if it were possible, it would've been done by now. But suddenly, I see in Kristyna Ferris's blog about FabCon custom calendar decks. I'm just like, "What?" [00:03:00] When I first saw it, the title of that section in the bulleted list of her blog post, I thought, oh, this is another version of calendar auto that generates a calendar table for me, like another version of that. When I read it, I was like, oh my God, I almost felt plagiarized. Like, I was talking about that years ago. And so this is amazing. Not something you'd expect to happen on a 12-year delay. And I'm not taking any credit at all for this feature existing. No one remembers that I even said anything about it, and I'm sure that they thought [00:03:30] about it before I even said anything. There's nothing original about me here, but it's just like my own personal relationship with the story.

Justin: Yeah.

Rob: It's so twisted and tortured. And now I'm sitting here going, oh man, kids these days, they're never going to have to do the page-long formula that behaves completely differently at every single level of filter context.

Justin: Just to emphasize this for folks that maybe haven't caught the news yet. Yes, there will be the ability to create things like a 4-4-5 or a 4-5-4 or even other custom calendar [00:04:00] tables more easily so the table itself, you can create more easily. I was thinking about when I read that, Rob, when we would teach classes and you start teaching people time intelligence, like as soon as you don't operate on a normal I guess like [inaudible 00:04:13] bust out the calculate.

Rob: Yeah. [inaudible 00:04:16] if you're lucky, this is magic. How many of you are lucky? None of you are lucky. All right, moving on, moving on.

Justin: Moving on. Let's talk about context transition. So now you'll be able to use things like dates YTD on these [00:04:30] custom calendars. Awesome.

Rob: Amazing, right?

Justin: Boom.

Rob: Amazing. You could do it for like NFL season calendar. And by the way, I've done that. Believe it or not, I have written custom time intelligence-

Justin: Sure.

Rob: ... measures for an NFL team around their NFL schedule, like season to date and like prior week is really prior game, right? And then, even like within the context of a week, what day of the week is it? It's wild how different that has been on what day of the week you play it on, whole chunks of my life [00:05:00] no longer were necessary.

Justin: I'm just thinking about now how many... It's probably like a seven-figure number of models laying around planet Earth that are just like, I missed the party. Like nobody's going to go back and rewrite all that DAX.

Rob: Is at least one model on the planet that's still running to this day that I built in 2011 that I promise you still has the extra all in the greatest formula in the world pattern, and it's the place where I [00:05:30] figured it out.

Justin: It's like an artist's signature, Rob.

Rob: Those poor people. It's still working, so why make it better? I'm sure that we could now write it in a way that would run like 10 times faster, too. Again, didn't have variables, didn't have switch, didn't have anything.

Justin: Yep.

Rob: And of course, now with this custom calendar stuff, I would say that in all of the news around the Microsoft data ecosystem, this one sort of quietly might make the biggest difference. [00:06:00] It's going to be top five in terms of like making the most difference on the ground in the real world, probably to little fanfare. But I mean, it's just incredibly significant. Like you said, there's a million plus models laying around that probably have edge cases that don't even really work very well because it's so hard to get it right in all circumstances. Anyway, this one pulled on my heart in multiple directions simultaneously. Like, it's great. Well, why did it take so long? And sort of like desiring that everyone else be hazed [00:06:30] in the same way that I was hazed.

Justin: Yeah. We're excited now, but wah wah.Public preview is probably like a couple months away still. We await the roadmap.

Rob: So we still have a few months inflicting misery. The way that it should be. We do it the hard way.

Justin: We can still sling it.

Rob: If anyone's doubting this. No, I think this is great. I'm not the least bit resentful of this.

Justin: It'd be interesting to go, like dig back through the user feature request boards and stuff like that because a calendar table is so fundamentally [00:07:00] important to a model. And not even just the ability to work with custom calendars, but things like, hey, when I have a calendar table and I mark it has a calendar table, turn off the auto summarization of my years and months numbers because that's really annoying. Like just syntax sugar stuff that almost everybody who builds a model has to deal with.

Rob: I mean, again, this is a long time ago, this is dated, but I remember like most of the time, if I forgot to mark it as a calendar [00:07:30] table, all the dates functions still worked. There were like one or two examples that I found that didn't work. I'd be like, "Hmm, why isn't this working?" And I'd just go mark it as a calendar table and it clears up. And when was it necessary? When was it not necessary? The platform sort of self-gatekeeps at times, you know? Like, just to-

Justin: I did quite a few projects where part of the work was to assess the quality of a model and how to make it better. Without fail this was just the first thing I'd be like, "Oh, you have 17 automatic calendar [00:08:00] tables in your model because you have auto date time turned on." Microsoft, why are you doing this to us? Help us please.

Rob: Yeah. There were a number of things announced at FabCon.

Justin: Totally.

Rob: This is my emotional leader of all of them. Doesn't mean that it's the most impactful, but it is the one that to me really stands out, just because of my history with it. What else we got? You're going to walk us through some things.

Justin: If this one didn't catch your eye, I wonder if it's a close runner-up for you. Little naming convention [00:08:30] challenge here, which is UDFs for DAX, user-defined functions for DAX.

Rob: I did see that. I know what it means. I don't have a sense of like how thoroughly they're going to implement this, and so I kind of hung back.

Justin: The promise of the idea is pretty interesting to me talking at P3 since the beginning about concepts such as like measure trees. Like, we build our little base measures, our sums or mins or maxes or averages, and then we can use those over and over again in other [00:09:00] measures for different contexts or whatever. UDFs are... The thing that I'm wondering about is I think about more complex measures where I'm writing variables that materialize like tables of data on the fly, and then I'll have a measure that's real similar to that, but I'm rewriting the same code that does the variable part. And so UDF would potentially replace the need, I could write a set of code that for example, finds like active customers in a period, [00:09:30] and I can just call that function within other measures because I'm not returning a scalar, I'm returning maybe a list or a table or something like that. At least the promise made was, and then we'll be able to use those UDFs across the model and potentially across other models, so it could simplify the real estate quite a bit.

Rob: That's a big deal. Many years ago working on Excel, I wrote a specification that we didn't have time to implement in Excel 2007, so we ended up dropping [00:10:00] the feature. But the idea at the time, we called it codeless UDFs. In Excel for many, many years, you were able to write, even like in VBA, you could write your own function that you could then call from the formula bar in Excel and it was just called UDF. So you could write them in C++, you could write them in whatever language you want. Call them from the formula bar just as another function, just like the SUMIF function, right? You've added your own.

Justin: Exactly. Yep.

Rob: But you couldn't construct your own UDF by writing [00:10:30] formulas in the grid, which of course, is the most powerful and intuitive thing in Excel, and it'd be a real game changer. Now, eventually this came to light as Lambda functions in Excel, which I still don't understand. I just don't get it.

Justin: It's M-ish.

Rob: But in theory, not in theory, actually in academic practice, Lambda functions serve this role for Excel. I look at it and go like my caveman nerd brain looks at it and goes, but really I just want to codeless UDFs, right? I just want to be able to-

Justin: Right.

Rob: Like, why did we do it that [00:11:00] way? Anyway.

Justin: Yeah.

Rob: So the idea of being able to use DAX to write my own custom DAX functions is pretty awesome.

Justin: Yeah. There'll be a little bit of structural wrapping that's going to be non-DAX kind of M-ish in nature, but the actual function building them itself is DAX. So you can specify like an order of operations and to do this or do that. So when I read that I was like, yeah, how many times I've gotten tripped up because I'm doing different permutations [00:11:30] of calculations that I need this to materialize like these same variable table sets? You can't write a base measure for those things.

Rob: Right. It solves a different problem. Similar problem, but it solves a different problem. Is it measure groups?

Justin: Calculation groups.

Rob: Calculation groups. Okay. So it's like a cousin of the problem the calculation groups solve.

Justin: Yes.

Rob: But it's different because it's not about the measure in the middle. It's not about the thing that returns the number. It's about the table that you're [00:12:00] calculating before you do it. In my hockey model, I have like this whole family of measures that go and identify the games that I played with another skater or the games I played against another skater. Holy hell, if I ever needed to change the definition of what meant games played with games played against, like take out the playoffs or something, I'd have so many places to go and make that change. And what are the chances that it's written slightly differently in my model today just because of copy-paste errors or-

Justin: You forgot, [00:12:30] yeah.

Rob: It's a small percentage chance that something's already off. But would I stake my life on it being 0%? Hell no.

Justin: No. Well, that's the thing, is like these little kind of intermediary steps and more complex DAX functions, it's the same pattern, but like a filter criteria is different. So being able to pass that criteria into a function as a variable, for example, and then I'm actually looking forward to seeing how that one helps people.

Rob: As people who [00:13:00] have written formulas and built models. Many, right?

Justin: Many, many.

Rob: You and I can completely appreciate what these features do for us, but it turns out you and I don't write many formulas these days. Right? Like-

Justin: I wrote some DAX this morning. Just saying.

Rob: You did not. You did not. You dirty liar. I wrote some last week, but again, it was hockey. Right? In fact, I wrote some really cool DAX-

Justin: Okay, my DAX wasn't cool.

Rob: ... about hockey. It's just to unite the clans of different [00:13:30] eras of stat keeping in the league. It turned out like get you can't double count because there's some overlap, and like it was just really so easy to do today and it would've been damn near impossible in 2013.

Justin: Yeah.

Rob: For people who listen to this who do write formulas, like you're getting it. If you're a business leader listening to this, if you and I just had our other jobs, the ones that take us on airplanes and the ones that prevent us from doing podcasts, the real running of the business type stuff, we wouldn't think these were significant. Like what does this mean if you're a business leader, these two features? Like, where would this surface for you as [00:14:00] being a good thing?

Justin: Kind of the layup answer is hopefully this makes the teams that are developing solutions that you use faster, more effective. You could parachute into any company using Power BI today and you spend enough time, you'll find discrepancies across reports because of these types of problems. It's like Rob worked on the model and then Justin took over and now there's like a, oh crap, the measure variation. So hopefully we can see a tightening up [00:14:30] of consistency and accuracy in reporting where things do get advanced and hopefully you see your team struggle less when you ask to kind of take things to the next level.

Rob: Yeah. I know I'm the one that asked the question. It's unfair. Right? Like, I make you answer it first, but then I listen to your answer and come up with mine.

Justin: So we build a complete answer, Rob.

Rob: That's right. So I like your answer and I'll add mine, which I think you were trying to hint at. This is not just about efficiency, it's [00:15:00] also about making things feasible that weren't. I think what you're going to find more often than not, finding the DAX team that was willing to do battle with custom calendars, for example, all the way down to the nth degree, that is a test of stamina and skill and intellect. And the people who are smart enough and talented enough to do it often get worn out. I suspect that there are cases out there right now where the business runs on a 4-4-5 calendar, but the reporting [00:15:30] is not. The reporting is running on like weeks. And so like they're constantly having to translate in their heads or our favorite, export to Excel.

Justin: Yeah.

Rob: And do the last mile in Excel for lining up budget versus actual. So I think there's probably a very sizable footprint out there. It's not like, oh, we're going to be so much cleaner and we're going to have fewer errors and it's going to be easier to maintain. It's like, oh, we're going to do the thing that we didn't do.

Justin: I've had these experiences [00:16:00] on a great many projects where you're excited about the progress you're making and the insights you're having and what you're learning and what you're able to do, and then you're like, "Oh, could we do this?" And your initial response is like, "Yeah, that would be awesome." But then you get in there and you're like, this is actually hard and it's kind of finicky. Can't deliver it in a way that the end user's not going to be constantly like, oh yeah, Rob, you got to clear the filter pain. Arg! Like, I think we'll be able to do things we previously just maybe decided not to do because [00:16:30] they presented those types of concerns.

Rob: I think Stalin famously said, "Quantity has a quality all of its own." There's a point at which, and this, of course, is a man that sent millions of people to die in humane ways.

Justin: Right.

Rob: So like, he's not a good person.

Justin: Not a good role model.

Rob: But he can still be right about something, which is when something gets easier and faster and more efficient, it's often the difference between doing it and not. Right? So there's a qualitative difference that emerges, a significant qualitative difference that emerges, even though you might look at [00:17:00] it and say, "Yeah, but this is just a quantitative improvement," blah, blah, blah. It's like, yeah. That's the whole story of the entire platform. In theory, everything we've done with Power BI was possible. Everything we've done with SSAS Tabular was possible. I mean, not really, but arguably possible with the previous wave of tools. It just never actually happened.

Justin: Yeah.

Rob: Don't sleep on the quantitative incremental improvements if they're dramatic.

Justin: Marco, his quote on this one that UDFs, it will [00:17:30] have the biggest long-term impact compared to all the other announcements at FabCon combined. Of course, he's the DAX guy, but he's just seeing like in a year, we'll be very different in how we approach things.

Rob: So two things. Number one, how cool is it that we can use a man's first name and we all know who we're talking about?

Justin: Yeah.

Rob: No one's out there saying Rob said. Right?

Justin: Right.

Rob: Those two gentlemen, like their names are just unique enough and their stature is amazing, so we speak their names in hushed reverential tongues.

Justin: We are not [00:18:00] worthy.

Rob: Yeah. Here I am saying that the custom calendar thing in the upper tiers of DAX development, absolutely, it's the UDF thing. But in the like dirty street fight, the custom calendar might be bigger.

Justin: Yeah. So those things are super cool. Kristyna was out of the conference. She gave a session on real-time intelligence that was really cool. She recapped it in our blog a bit. We sent three of our other team members out there just to be there, take it in, learn some stuff, and [00:18:30] I had a chance to meet with them when they got back and they were all just super excited about all the things that are coming. We don't need to dig deep into any of them. Some of them are more exciting to the people that build solutions. Some of them are more exciting to the people that are going to consume this stuff.

Probably the biggest announcement that got a lot of legs, because it was one of the first things that came out. I think it was Amir and Arun maybe together announced that all of the Copilot AI feature sets that are in Fabric, those used to [00:19:00] be reserved for the big Fabric capacities, like an f64 and up, which is going to run you $5,000 a month or more. Now, all of those features are going to be shipping to all of the paid SKUs. So if you've been sort of on the sideline just reading about this stuff and you haven't been able to put your hands on keyboard with it, soon you'll be able to. You'll be able to get a little f2 and spin it up for a couple hundred bucks a month and at least see what these features can do, what you think about them, so that's pretty neat.

Rob: [00:19:30] So first of all, this is good news. We want to be kicking the tires on all this kind of stuff more rigorously, more frequently than what we have been because again, do we spend up a $5,000 a month capacity just as a playground? Like, hm-

Justin: Not for very long.

Rob: Not very long. Right? Turn it up, turn it down. Right? The punchline here is like, okay, so now, everyone can start complaining about Copilot. Right? We can all start seeing this is not there yet. [00:20:00] And again, we've got to be really careful. When we see things that aren't there yet, the word yet is really important.

Justin: It is. I'm still very much betting on some of this stuff coming together. I've worked on a couple Bootstrap-y AI projects recently where we've tried to build like a Copilot-type solution that can interface with documents, web data, and like actual hard data. It's been tough to get something that really performs well with the hard data. [00:20:30] One of the announcements that came out at Fabric is soon, you'll be able to leverage your Fabric data estate. So if you've got your data in a lake house, in one lake, you'll be able to leverage that data in Azure AI Foundry, which is Microsoft's like main, if you want to work with the big daddy models, that's where you would go, and I've been really wanting to be able to do that because I've been able to export some data and go to like a Claude or a ChatGPT and do some really cool stuff, but I want [00:21:00] that more integrated. I can run through semantic link over my model, interface with that, and so they tease some of those things that they're working on. So I'm kind of like, okay, yeah, work on that and bring it to us.

Because that's why I think the game for the end user starts to change. You give people a more natural interface to understand their business's data. Maybe that rides within a dashboard, alongside it, or in some cases, like instead of. There's still a little bit of a [00:21:30] future promise to be fulfilled on that kind of stuff.

Rob: We'll be watching that feature with great interest, Palpatine said.

Justin: Yeah. And there was a ton of other announcements. I mean, things that are improving security and fabric, things that are improving the data engineering workflows, more on creating more parity with the tool sets in Azure, for example. So a lot to be excited about and candidly, some things that we've been waiting on for a while. One Security was something that was out there from [00:22:00] the very beginning and like, oh, here it's finally coming.

Rob: Yeah, but were you waiting 12 years on it, Justin?

Justin: No. No. It's been a struggle for ages. Well, how is my data secured in a database and then how is it secured once it gets to the model? To not have people racking their brains on that stuff, that'll be a good thing.

Rob: Yeah, and I can understand why behind the scenes that is just a lot of work.

Justin: Oh, it's got to be really hard.

Rob: Even when it's not like the smart kind of work. Got to wire it all [00:22:30] up, got to pass it all through. It doesn't happen overnight, so I get it. I can't resist the joke. They've probably been working on the custom calendar DAX stuff for 12 years, Justin. That's how long it takes.

Justin: Yep. The advancements they've made, they being Microsoft, in the code structure of like a Power BI desktop file, so that now everything is exposed in basically adjacent style notation. The visuals, where they're at, the definition of the whole model, the report pages, [00:23:00] like the whole kit and caboodle. Well, why is that so important in the age of AI? Because the AI writes the code. Right? It needs to be able to generate the code that says, put a bar chart here effectively.

Rob: Yeah, the clock is ticking down to ta-da!

Justin: Yeah. Yeah.

Rob: What else do we got here?

Justin: Oh, there's going to be a command line interface for Fabric. Thanks. Yeah, a lot of nerd stuff. Ooh, a new SDK.

Rob: There's [00:23:30] really no end to what you can do with CPUs and GPUs in terms of like extracting value from data. That's an endless ROI, software can always be improved.

Justin: It's funny to reflect, even in my pretty short career so far in analytics, let's call it a decade-ish, you start doing things in Excel, and then you are writing SQL, building SSRS reports, and then you discover Power Pivot and Power [00:24:00] Query, and then you're out in Power BI and there's all this Fabric stuff and now AI's here. You just reflect on how much things have changed in terms of the tools we are using, both to create analytics and to consume analytics. The importance of that, if anything, has increased. And so where will we be 5, 10 years down is like, who knows, really? Sort of wild.

Rob: Here's an interesting thought. Let's say you take the average of [00:24:30] how far the world has come in its application of data technologies, the application of it, like on the ground, actually using it, like it's delivering value in businesses today.

Justin: Sure.

Rob: Take the midpoint example of that and compare that to the art of the possible, how far the technology has come. Those two points aren't moving at the same speed. Like the technology is advancing outward into space at its pace, and the world's uptake of [00:25:00] it is, I think, moving more slowly. In this list of announcements here, we're into improvements in features and functionality that hasn't even been touched by most people in the ecosystem yet. Even if you're on the Microsoft platform, they're improving things you haven't gotten to yet, and replacing things you haven't gotten to yet. Do you have that same sense that the adoption and uptake lags? There are absolutely customers for whom each one of these updates [00:25:30] is a game changer and they've been pounding the table for it in executive briefings with Microsoft, holding hostage, saying like, "If you don't implement this, we're going to go to Snowflake" or whatever. Right? There are bleeding-edge adoption scenarios for every single one of these things.

Justin: Right.

Rob: It's just that the median ain't there.

Justin: No. I would agree with that. The advice I have here is if you are in that state of like the median example, you see all this stuff and you can get [00:26:00] pause really fast. Just don't.

Rob: Yeah. Just relax. Relax. It's okay.

Justin: The technology is still already just so good.

Rob: Yeah.

Justin: So if you're debating like, oh, should we wait? No, there's not going to be a big benefit in waiting.

Rob: And also, should you feel lesser? Should you feel overwhelmed? Should you feel not up to it because you don't understand why this new feature XYZ is a big deal? Like even the ones we've talked about, like the UDF thing. If you're saying like they're talking about this as like a revolution [00:26:30] and I don't get it, that's fine. That is one of the biggest hurdles to getting started on something, is just feeling the distance between you and others, feeling the distance between you and the art of the possible. If you're looking at this list and thinking like, I'm the only one in the world who doesn't understand why these things are big deals. No, you're not.

Justin: Even here in our business, we have our core models that we use to understand and manage our own business, [00:27:00] they've been in place for a long time. They don't have fancy calculation groups. Could we go back and re-implement things in a new, fresh, better? Sure, but we don't need to. We're getting tons and tons of value out of technology that is technically dated, in a way. Like, it's okay.

Rob: Yeah. I'm all right with that.

Justin: Yep. Works for me.

Rob: Project UDFs, everything we run our company on goes in three, two, never.

Justin: Yeah. Go ahead and put that on the backlog.

Rob: [00:27:30] But it's just better, Justin. I do want to circle back to that 2011 model though, and take out the extra alls.

Justin: Nah. I feel like that's like a signature. We're going to be doing some forensics someday and just be like, "Oh, yes, Rob was here."

Rob: Oh, the calling card., Yes.

Justin: The extra oh.

Rob: Completely harmless. No big deal. Just a little embarrassing.

Justin: We used to drop some ASCII art of the P3 logo in like [00:28:00] SQL code and stuff we would write, too. It's just like-

Rob: Oh, wow. All right. Well, good to be back in the podcast chairs. More comfortable than airline seats.

Justin: Agreed. Let's go back to back. Let's make it two for two.

Rob: Let's not get carried away now.

Justin: Until next time.

Rob: Until next time.