A Most Generous Mentor, w/ Microsoft’s Dany Hoter

Rob Collie (00:00:00): Hello, friends. Today we have a grizzled veteran of the data industry by the name of Dany Hoter. Now, when I say grizzled veteran, you immediately start to think a little bit unfriendly, right? Can you be grizzled and friendly? Dany is. Imagine your favorite grandfather, assuming you have a favorite grandfather who has been around the data industry forever, and that's Dany. Dany is in his fifth stint at Microsoft. I met him back in the early 2000s when I thought he was in his first stint at Microsoft. But no, it was his second already and he has seen it all. He began his data career working on IBM mainframes in 1974. And then eventually he got into SQL, and then he got into OLAP and MDX. And today in his current incarnation at Microsoft, he works on something called Azure Data Explorer, which you might know by its code name of Kusto, and a query language called KQL. KQL sounds like it's one letter short of a west coast radio station.

Rob Collie (00:01:09): Doesn't it? KQLA, where data meets the interface. And he's just such a good person. In another piece of trivia, Dany and I conspired to take probably the most expensive business trip and Microsoft history taken by non-executives. Make sure you listen to the details of that little vignette. And one of the compliments that I would like to pay to Dany is that, in my years at Microsoft, I knew a lot of high performing individuals, knowledgeable individuals, intelligent individuals, wise individuals, and most of the people that I knew that fit all those descriptions, they were running fast. They were climbing the ladder and that was their focus, was climbing that ladder. What do high performers do? They climb ladders. Dany though, always wanted to help. That was always clearly his number one overriding drive was to help. Back in those early 2000s, when I still was learning how to spell BI, Dany was incredibly generous with his time and not just to me, he was generous to anyone who would ask. He would sit down and he would start from the basics.

Rob Collie (00:02:14): And he would explain to you exactly what was going on in this world, this foreign world of data at the time. Happy to field any question, you never felt dumb asking him really fundamental questions. And having that kind of friendly non-judgmental, informed resource was incredibly important to me in particular, when starting into this new world of BI, which of course has been my career now for the past 20 years. So if we were going to make a list of like a half a dozen people who shaped my future, my career, Dany would absolutely be on that list. That's still true of him today, just an incredibly warm and friendly person. And on my relatively short list of human beings that when I see them I will immediately hug them. We're long overdue getting him on the show. I'm thrilled that he's back at Microsoft, but this is his first stint on Raw Data. So let's get into it.

Announcer (00:03:11): Ladies and gentlemen, may I have your attention please?

Announcer (00:03:15): This is the Raw Data by P3 Adaptive podcast with your host, Rob Coley, and your co-host, Thomas LaRock. Find out what the experts at P3 Adaptive can do for your business. Just go to P3adaptive.com., Raw Data by P3 Adaptive is data with the human element.

Rob Collie (00:03:41): Welcome to the show, Dany Hoter. One of my absolute favorite people. It is so good to see you again. How are you tonight? It's tonight for you, right?

Dany Hoter (00:03:51): Yes. It's tonight. It's 8:30 PM for us here in Jerusalem. Few rough days, we are doing some work on the apartment. So there's a lot of noise, now it's quiet, but it's very hectic. Other than that, I'm pretty good.

Rob Collie (00:04:04): You have no idea. The chances that this podcast recording will be interrupted by some sort of major chaos in my house, but it could be coming for me and I just don't know it yet. I got bit in the neck by a dog in the middle of a podcast recording one time. Yeah, Tom, you missed that. That was during the dark time when Tom was on sabbatical, and not checking his email and not showing up for recordings.

Thomas LaRock (00:04:25): Okay. So most of that is true.

Rob Collie (00:04:29): So Dany, first of all you look great. You look younger than the last time I saw you. Are you reverse aging? How are you pulling this off?

Dany Hoter (00:04:37): Well, people say that to me. I can't say I don't feel my age. I'm in my eighth decade, so I'm going to be 71 in December. But I'm 100% active, busy working very hard and enjoying work like I don't remember when-

Rob Collie (00:04:55): What are you doing for work these days?

Dany Hoter (00:04:57): I am now in my fifth period at Microsoft. This is the fifth time, if I count a period of a few years that I was working with Microsoft as a vendor doing all kind of Excel work. And then eventually as a vendor, I met GPM of the team I'm working with, which is the Kusto team.

Rob Collie (00:05:17): Oh, okay. What is Kusto?

Dany Hoter (00:05:20): Azure Data Explorer is the official name. Kusto is used to be the internal name, but most people know it is Kusto. And I'm surprised that you ask me what is Kusto, it means that we are really-

Rob Collie (00:05:32): I know what it is.

Dany Hoter (00:05:32): You know? It's a very well kept secret, but this is something really I'm very excited about. Kusto is just in a few words, it's basically database for big data, runs an Azure. We have clusters of customers with 1000 machines on Azure running together. So let's think about 16,000 cores trying to give an answer for a single query beginning with billions, tens of billions, hundreds of billions. And we have single tables with trillions of pros in them. So well, it's worth a whole session about it, but I don't think that this session is about going very deep into one product. I don't know. Maybe you tell me.

Rob Collie (00:06:14): We'll definitely going to talk about it. I did not know that this is what you were working on. I had not heard of it. That doesn't really mean as much as it used to.

Dany Hoter (00:06:23): And actually more specifically I'm working on the interface between this beast that I was talking about and Power BI. So I'm the man responsible for the relationship between Power BI and Kusto. And we have lots of action going on in this area. How it's all about direct query and about doing things a little bit different than what you do regularly with Power BI. But we have some nice success.

Rob Collie (00:06:53): I was going to ask, is it ever make sense to do anything other than direct query against a trillion raw source store?

Dany Hoter (00:06:59): No. In some cases you may want to aggregate to... I don't know, 20,000 rows of very aggregated data. And then still do a lot of playing around slicing and dicing on this very small set of aggregated data, so you might want to import that. You might get some benefit from that. But other than that, it's all about direct query, that's for sure.

Rob Collie (00:07:24): Cool. All right. So Kusto is a pretty cool name I think, the vastness of the ocean that now you're going to go boil.

Dany Hoter (00:07:32): Boil?

Rob Collie (00:07:33): You've got to boil that ocean to get any value out of it. It's just a metaphor all the way down. Okay. So this is your fifth stint at Microsoft.

Thomas LaRock (00:07:40): That's amazing.

Rob Collie (00:07:41): Recently had a guest on who had retired twice from Microsoft,

Dany Hoter (00:07:46): Went into retirement?

Rob Collie (00:07:47): Twice, retired, went back, retired again. But they don't have nothing on you.

Dany Hoter (00:07:52): Actually, I quit twice to go to a startup. I was fired once and-

Rob Collie (00:07:58): I don't understand that. I don't understand why would they ever... It makes so much sense to me that you are back at Microsoft, I can't imagine how you could ever get fired.

Dany Hoter (00:08:08): Yeah, it was 2008, at the time I was a product planner with Office.

Rob Collie (00:08:13): Oh, that's right. Yeah.

Dany Hoter (00:08:14): Working remotely with a team that was all in Redmond at the time, because at the beginning when I started this Office product planning, I was in a team that was completely international. So every one of us was in a different country. So this was back in the early 2000s. We learned how to work remotely back then, so working remotely is nothing new for me. It was pretty good. But then I moved to a team that was all in Redmond. I was the only one remote. And this was the thing, this doesn't really work. If you ever been in a meeting when everybody is in a room and you are remote-

Rob Collie (00:08:47): It does not work.

Dany Hoter (00:08:49): It does not work. But generally it didn't work.

Rob Collie (00:08:53): Because you were already back overseas at that point, not just remote, you were at a completely different time zone. Like just other side of the world.

Dany Hoter (00:09:02): 10 time zones away from anybody. Well, once I was working only with people in Redmond, as I said when we were working with this team with which international, we had people in different capitals. It was fun because every three months we would meet either in Redmond or in one of the capitals. So we had team meetings in Paris, in Dublin, in Athens, in Stockholm and Rome, so many cool places

Rob Collie (00:09:30): Everywhere you went, you ran into people that you knew just randomly, right?

Dany Hoter (00:09:34): Yeah.

Rob Collie (00:09:34): That's one of Dany's signature moves. One of, yes, it is a Dany legend. And we're going to get to that for sure. So Tom, Dany here, and we made jokes about Donald Farmer as Yoda. Dany has served as the Yoda of BI for more Microsoft employees than anyone else, including me. He came in with an acquisition, the Maximo acquisition.

Dany Hoter (00:10:00): This was the second time. First time, 1998, I just was working for Microsoft in Israel, working on SQL related stuff, SQL server.

Rob Collie (00:10:09): Yeah. I didn't know you yet.

Dany Hoter (00:10:10): Then I met the whole idea of OLAP coming with the SQL server seven back in 1998. I really liked it. Then I met the guys who were the founders of Maximo. And after a year at Microsoft, I left for the first time and joined this guys.

Rob Collie (00:10:28): Joined Maximo, and then you came back. So yeah, you're right. I did not know that was already your second stint. But yap, I missed it. From my perspective, Tom, all these people come in with the Maximo acquisition and they're all already up to their eyeballs in the BI market, and not the reporting services corner of it. They're into the analytical OLAP end of the pool. And then there's people like me who are on the Excel side being said, "Hey, now you're going to be in charge of a lot of this, not these people but a lot of this for Excel." Most of the people that came in with this acquisition were so focused on going and doing the job, Dany loved teaching people.

Thomas LaRock (00:11:13): And he still does.

Rob Collie (00:11:15): Yeah. He loved sharing what he knew and not that the others were stingy, but Dany was an outlier in this regard. And so a lot of what we all came to learn us non BI folks, a lot of what we came to learn we came to know it through the vector known as Dany Hoter. I learned a lot from this guy and I'm super grateful for all the investment he's made in me over the years.

Dany Hoter (00:11:43): We're talking 2001 actually, we came to Redmond one month before September 11th.

Thomas LaRock (00:11:51): I feel a real connection to Dany because he's worked at Microsoft five times, and I've interviewed with Microsoft five times.

Rob Collie (00:12:00): Six times a charm.

Thomas LaRock (00:12:02): Maybe.

Rob Collie (00:12:03): You've probably heard me Tom multiple times say things like, when I was at Microsoft working on this stuff, every six months I would get my courage back up and say, "Okay, now I'm ready to learn MDX." And then someone would come in and start teaching me MDX. And we'd get into all the complexities of writing an IF, and I'd say, "Oh, right. I forgot about this. Yeah. This is terrible. I don't want to know this." It was Dany basically every time who was on that six month merry-go-round with me.

Dany Hoter (00:12:32): Yeah. The funny thing about that is that actually after they had MDX, and they realized that MDX is very difficult to learn. There's a really difficult learning curve. And they created docs and said, "Oh, we'll do something which will be so intuitive for the Excel people." This was the whole idea. I tell it this way usually I say, why did they invent a new language? First, because they love inventing new languages. This is the main reason.

Rob Collie (00:13:02): Fair.

Dany Hoter (00:13:03): And second, because they claim that it's going to be so intuitive to sell people because it's kind of resembled somehow. That's simple.

Rob Collie (00:13:11): Oh, last for me.

Dany Hoter (00:13:13): People struggle with-

Thomas LaRock (00:13:13): You don't think DAX is simple?

Rob Collie (00:13:15): Yeah, exactly. Right.

Dany Hoter (00:13:16): Marco or Roberta, what do they say? Easy-

Rob Collie (00:13:20): Simple not easy.

Dany Hoter (00:13:20): Easy-

Rob Collie (00:13:21): It's simple but not easy.

Dany Hoter (00:13:22): Whatever that means. Do you think that it is simple?

Rob Collie (00:13:25): I think it's wildly successful at its goal. We got to be really careful not to look at things in two binary, a light. That's the SEOG DAX Formulas for PowerPivot book.

Dany Hoter (00:13:38): How hard can this be? Look, the book is so thin.

Rob Collie (00:13:41): It's written by me. It can't be hard.

Dany Hoter (00:13:43): Look, what's so hard about the ISBLANK function. Is this blank?

Rob Collie (00:13:48): Not hard.

Dany Hoter (00:13:48): Not hard.

Rob Collie (00:13:49): So a couple things, Dany, data's never easy. It's just never going to be easy. Excel can also be called simple but not easy once you really get into it. Building a complex Excel model, it uses all kinds of primitive concepts. But next thing you know, you're out there doing index match and offset and array formulas.

Dany Hoter (00:14:13): Not anymore. Now you have the XLOOKUP and you have this fantastic array functions.

Rob Collie (00:14:21): Definitely getting easier. Yeah.

Dany Hoter (00:14:22): Remember the old array functions, they were really something for the concatenate for-

Rob Collie (00:14:27): I agree. I believe old style array formulas to be the most technically complicated thing in Excel, even more than VBA.

Dany Hoter (00:14:35): And also you meet 15 fingers to do the control shift and whatever.

Rob Collie (00:14:42): Yap. Growing that third arm, the future species of human that's mutated to use array formulas would in fact have a third hand, yes.

Dany Hoter (00:14:49): This one and the data tables. Data tables again, it's still there. It's haven't changed a bit. The UI haven't changed. This is another thing that few financial people know it. If you know it, you love it. It's very powerful but very rare.

Rob Collie (00:15:09): Yeah. So let's go back to DAX.

Dany Hoter (00:15:12): Yeah.

Rob Collie (00:15:12): One of the things that I have noticed about DAX, is that the community around DAX has migrated towards the high end. The average DAX formula that you find on a technical blog these days is written in a dialect of DAX that is intended to be like optimal. I think that does a disservice to the accessibility of the language. There's so much that can be done in DAX. I don't know if you remember this, but I used to give the old analogy of carpenters versus physicists. Does this ring a bell for you?

Dany Hoter (00:15:48): No, maybe it's too long.

Rob Collie (00:15:50): So it's fine. Maybe this metaphor came along in a four or five periods of time that you weren't at Microsoft. So when I was working construction in the summers in college-

Dany Hoter (00:16:02): This I remember.

Rob Collie (00:16:03): We were building temporary ladders to actually support the rest of the construction crew going up and down from floor to floor. And I'm driving nails into this two by four, the two by four kept splitting as I was driving this nail in, big heavy nail. And they said... The guy who was sort of had been saddled with me as an assistant, he showed me and he said, "Hey, look, watch this." He took the nail and he turned it upside down and hit the point of the nail with the hammer to dull the point. And then flipped it around, drove it into the wood and it didn't split. I'm like, oh my God, that's magic. So I try it. It's not going to work for me. I'm just positive it's not going to work for me. I do it and it works. It totally works.

Rob Collie (00:16:47): Okay. Now, the nerd in me is sitting there fascinated like, oh, why does this work? What is really going on? But the ladder's got to get built. And so I get like two seconds to contemplate that before I'm right back to dulling nails and driving them in. And no one there was sitting around contemplating why this worked, just follow the pattern and it works. By comparison, you can be Einstein and probably suck as a carpenter. Like you're going to have to learn those patterns whether you understand them or not, to continue that metaphor. I think that most of today's DAX blogging, most of today's DAX content available in the community is physicist DAX. You need to understand like every possible concept.

Dany Hoter (00:17:37): Like for me, physic of the Italian persuasion.

Rob Collie (00:17:43): Well, I think they have a lot to do with it. Marco and Alberto understand every last little nuance of the language, probably better than all, but like one or two people even at Microsoft. And I think their brand is an appropriate brand for them, which is be the best. Be the Jedi masters of this stuff. But I think there's a natural social dynamic that occurs and leaks out into the rest of the community, is that if you're not sort of also holding yourself to that same standard, you're not up to snuff. And so there's sort of this arms race to become the most physicist like, when what we need is carpenter blogging. If you're getting hit over the head from day one with row and filter context, transition and cross join and all this kind of stuff, it's going to build a moat. It's gate keeping.

Rob Collie (00:18:32): And I always approached it from the other end, which is the carpenter end. And not because necessarily I'm high and mighty about it, it's really the only way I could learn it. You tried to teach me MDX like four or five times, never took. And it wasn't because of you Danny, it was because that language was evil. And then I managed to sort of muddle my way through DAX before there were ever any resources on it. And honestly if there had been resources, I probably wouldn't have muddled my way through it because the resources would've been written in physicists speak. I kind of got through in this window where it was possible for me to learn. I don't think I would learn today. I think if I saw DAX today from scratch, I would think, this is as bad as MDX and I want nothing to do with it. I'm positive I would react that way. But how much you can get done with a simple calculate and all and dates YTD, it's magic. And you don't even need to understand why it works.

Dany Hoter (00:19:34): It's magic. If you can explain it to an Excel person and they can do this jump, for them it could be a real magic because this is the kind of thing that they were just totally impossible for them to do with Excel. You see many people from the Excel camp, more and more people learning DAX and getting into it.

Rob Collie (00:19:55): Absolutely. And our business is proof of it. The majority of our hires at our company for consulting come from that Excel background much more than they come from an IT background. We have a handful that did come more from an IT background, but most of us came up through the Excel ranks originally. We have about 50 consultants now that came in this way.

Dany Hoter (00:20:19): How many do you have in total?

Rob Collie (00:20:21): We're about a 60 person company now, so that includes back office, people like me. I don't do any consulting anymore. Everyone else is better at it. I don't do training or consulting anymore.

Dany Hoter (00:20:34): You miss it?

Rob Collie (00:20:34): I do miss the training. I miss lighting people up like you. Like you love teaching. I didn't really know that about myself back when you and I were together on Redmond. But I shared that same passion I think.

Dany Hoter (00:20:47): For me it runs in the family. What we know is at least four generations of teachers in teaching, my grandmother and my grandfather on my maternal side were both teachers. My daughter is a teacher. My brother is a teacher. And we know that back then in Europe before they migrated to Argentina, they were known to be a family of teachers back in Ukraine, in the 19th century something or.

Rob Collie (00:21:15): Did you say Argentina?

Dany Hoter (00:21:17): I was born in Argentina.

Rob Collie (00:21:19): That's right. I'd forgotten about that. You speak fluent Spanish.

Dany Hoter (00:21:22): That's right. I can teach in Spanish and I did. I did full courses in Spanish.

Rob Collie (00:21:32): I had totally forgotten that, but yeah, absolutely. This is a man of the world. This is an international man of mystery, but there's not much mystery to him because he'll tell you anything you want. He's very transparent. He's the most transparent international man of mystery you'll ever meet.

Dany Hoter (00:21:46): You remember the thing we were doing with Microsoft with these colors? Are you red and blue?

Rob Collie (00:21:52): Yeah. So I did this I think a couple of times.

Thomas LaRock (00:21:55): Insights.

Dany Hoter (00:21:57): So one of the things they tell you is there is what they found about you, and there is what you think about yourself. And differences between the two. So in my case, I was reading yellow and exactly the same internally in externally. Exactly the same, no difference.

Rob Collie (00:22:17): The way that I had that presented to me when I was at Microsoft was, this is your sort of your resting state, like who you are naturally. And then here's what you're like at work. So how do you change when you come to work in order to sort of fit the workplace. And the greater the distance between those two, the harder work is on you.

Dany Hoter (00:22:39): You're kind of faking all the time.

Rob Collie (00:22:41): Yeah. They said it takes its toll. The consultant that was delivering this stopped and underlined that twice, it takes its toll. And I'm sitting there looking at mine, I had the single greatest distance between my personal and professional. The total the Pythagorean difference distance between was greatest for me. Although I was tied with Marius.

Dany Hoter (00:23:03): Marius?

Rob Collie (00:23:04): He and I not only had the same distance, we were in exactly the same spot on the map. The two of us were like off on this little cluster by ourselves. The rest of the team was elsewhere in big bunches. And then it's like me and Marius huddling over in this one corner. My distance between them has shrunk. I did it again three years ago. My base personality had sort of come up towards my work personality, and my work personality had also kind of receded back.

Dany Hoter (00:23:32): Was it deliberate? Did you do something deliberately or just naturally happened?

Rob Collie (00:23:36): I think it naturally happened. First of all, I found myself in a job where my base state was considered more valuable. Green was my dominant color, like emotional togetherness type of vibe. That's helpful at Microsoft, but it's not the main thing. I think consulting and recruiting people and leading a company of consultants, the heck even blogging which is how it all started, green plays a much bigger part in all those things. But at the same time like running a company, holds you to a set of standards. There's realities of running a business that I was shielded from at Microsoft. The buck sort of stops with me and my executive red nature had to come up, which is incredibly low by default at Microsoft.

Dany Hoter (00:24:22): So for the audience the summary of the green block is show me you care.

Rob Collie (00:24:28): Yeah. That's me.

Dany Hoter (00:24:29): Red, which is I read red because it's on top of here. Red is be brief, be bright and be gone.

Rob Collie (00:24:37): Yeah. And that was my lowest when I was at Microsoft.

Dany Hoter (00:24:40): And that's my highest.

Rob Collie (00:24:42): It's still probably my lowest, I think. But it had to come up quite a bit. Also, maybe just getting older, it's like I have less time left. Whatever you do don't waste it.

Thomas LaRock (00:24:53): I have a question for Danny because he's been with Microsoft five times, and because he is in his eighth decade and still working. Is there advice he has to a listener like what's the secret to enjoying your work?

Dany Hoter (00:25:07): For me, I enjoy the constant learning, so the teaching and learning. So all the time learning new things and teaching, I enjoy very much working with people. I don't enjoy having to make decision about people. So hire and fire for me is just as remote as can be. I never fired a person and I very rarely hired or help in hiring. I enjoy the feeling of I help someone and he said, "Oh I am unblocked now, I can go on. I can now continue on my own." And this is happening a lot in my current job. That's why I enjoy it so much because it's happening a lot, pretty much every day.

Rob Collie (00:25:51): There's this Venn diagram going around the internet right now and I don't remember where it is. But there's like these four overlapping circles, one of them is the things you're good at, another one is the things you enjoy. A third one is things that you can get paid for or something like that. And then there's a fourth one. And ideally you find yourself at the intersection of all four. Dany just told us that his professional and personal readouts are identical, zero distance. So we're clearly talking to a human being that's right in the middle of that Venn diagram and just very happily right at the intersection of all those things. Tom, will you find that for us then?

Dany Hoter (00:26:35): I'm on it.

Rob Collie (00:26:36): I've got the third and the fourth one wrong. Tom, any luck?

Thomas LaRock (00:26:39): I did. I put it in the chat.

Rob Collie (00:26:41): Oh, okay.

Thomas LaRock (00:26:42): I found an article on a Japanese thing called Ikigai.

Rob Collie (00:26:45): Yap. That's it.

Thomas LaRock (00:26:47): And I'm probably not pronouncing right.

Rob Collie (00:26:48): The four circles are what you're good at, what you love, what the world needs and what you can be paid for. So those last two I guess you could in some sense argue that they're kind of the same thing. But it's kind of like what the world needs and what the world knows it needs.

Dany Hoter (00:27:05): A lot of people are working on things which the world does not need and getting paid for it. They're working on kind of stupid applications, they're [inaudible 00:27:15] social system or whatever and even a little bit more of advertising they're getting paid, or gambling or other things.

Rob Collie (00:27:26): That reminds me of the show of Silicon Valley where he decides to change jobs, ecosystem startups. And they're like all of the startups are doing is putting mustache filters on people on video in real time. So you're right, he could be paid for that. But the sense of meaning in it of what the world needs is missing. So however you pronounce this term Ikigai or whatever, Dany hoter is an Ikigai.

Dany Hoter (00:27:49): Yeah.

Rob Collie (00:27:52): No, that doesn't sound right. Does it?

Thomas LaRock (00:27:54): It's an Ikigai guy.

Rob Collie (00:27:57): Ikigai guy not a... Right. Okay. Definitely not an Ikigai. Okay. Find your way into that overlapping Venn diagram into the middle and you're going to be happy. And I think I got a lot closer to the middle in the second act of my career than I was at Microsoft, that's been really good for me.

Dany Hoter (00:28:14): So you were with Microsoft like 14 years?

Rob Collie (00:28:18): Yeah. From 96 to 2010, but I didn't quite make it to my anniversary, so it's like 13 and a half years, long run. How many total years have you been at Microsoft now over your five stints, do you know what it adds up to?

Dany Hoter (00:28:29): One of this periods I received my 15 years thingy. You know that you get it for five years, 10 years, 15 years, you get this block of whatever it is, glass or something.

Rob Collie (00:28:39): Crystal glass.

Dany Hoter (00:28:42): It keeps getting bigger and bigger. And so I got my 15 and I didn't get yet my 20th. So I'm somewhere close to the 20th, I think. And they are doing a very good job at combining, so counting when I come back they know all the history and they keep counting. And eventually one day I will get a notice, oh, you are 20 years.

Rob Collie (00:29:04): Microsoft has gone so soft in the Satya era. In the Bill era, no, you leave and come back, you start at zero. That's how it used to work.

Dany Hoter (00:29:15): One of the stories of my many times at Microsoft is about the employee ID. So at Microsoft you have employee ID. When I came for the first time was 1998, at the time there were 30,000 employees at Microsoft. And for some reason which I don't know to this day, I got the number 9820. So four digit employee ID that was either skipped or recycled, or I don't know where the person not only left, but told everyone that he was never going to come back. I don't know. This number was free, I got it, I left. I came back, I got it again. I left. I came back, I got it again.

Dany Hoter (00:29:53): So I still have it to this day. So people now are walking around with seven digit employee IDs, but I have four digits. Now, I have to tell everyone when I show them because I do show this as one of my moves. And I tell them that I was not one of the first 10,000 employees at Microsoft, because if I would have been one of the 1000 first employees at Microsoft, I wouldn't be working now. So this is limit to everything. I will find myself something else to do, but I have this number. And for people who understand for people who understand about employee ID numbers, it makes a lot of impression on that.

Rob Collie (00:30:39): For context, I joined Microsoft two years before that in 96, and my employee ID was 44291.

Dany Hoter (00:30:48): Oh, you see.

Rob Collie (00:30:50): I was in the 44,000, you got given 9,000 two years later.

Dany Hoter (00:30:53): Go figure.

Rob Collie (00:30:55): They knew you'd be around for a while. They assigned that as basically as a position of influence.

Dany Hoter (00:31:02): Also, what I find that actually... Because I've been in this business as you can imagine a long time. So I started in programming and so at my age we didn't have home computers or hobbies. So I encountered computers when I was already a grown up. And I started in 1974. So if I continue for a little bit more, I can reach this 50 years host. I'm not sure, but maybe it'll happen. There were lots of technologies, I can tell you all kind of stories about the technology we had in the '70s from my first seven years I was in IBM mainframes. So this is an area in which I almost didn't took anything. So I can say that I'm using now anything that I learned when I was working as a system programmer on an NVS operating system on a mainframe somewhere. Other than that, from their point on if it's about financial system, if it's about Excel of course, if it's about... Nowadays, actually I'm involved with a customer which trying to move out of multidimensional and OLAP. They're struggling and investing tons of money of trying to get out of it.

Dany Hoter (00:32:14): And I thought at some point that I was able to actually do a lot with our tool, with the Kusto and the language we have which is called KQL and was very happy. And then actually today I found out that I'm pretty far away from what this old technology can do. Basically, almost everything that I learned on my way, it's actually at some point I find that it's useful, which is amazing because some of it is so old.

Rob Collie (00:32:40): Certain concepts just keep repeating themselves. By the way, so your career was born in the same year that I was born.

Dany Hoter (00:32:47): Hmm.

Rob Collie (00:32:49): The universe was like, oh, this Rob guy, he's going to need some help. Let's get Dany started.

Dany Hoter (00:32:57): Yeah, I was 23 at the time.

Rob Collie (00:32:59): So a number of just bouncing around things, I definitely want to make sure we talk about a number of little vignettes. So no particular order here. Let's talk about SQM. The original Microsoft telemetry service that collected data on how customers were using the product. And this long predates the cloud. This was way before 365.

Dany Hoter (00:33:24): If you would ask me what was the name of this system? I wouldn't remember SQM.

Rob Collie (00:33:29): Well.

Dany Hoter (00:33:31): I would never remember. You did some research.

Rob Collie (00:33:33): No, it's just that... Remember my memories of Microsoft are like frozen in a time capsule. I didn't keep layering more and more Microsoft jargon on top. In Microsoft years, SQM it was still kind of there when I left, it was kind of current for me in a way. I know that they've moved on from it, but this was my first experience running into the right only database. The concept called the right only database. SQM was an engineering Marvel. One of those things that you just say, now that couldn't be done. This software wasn't running on our computers, on Microsoft's assets, it was running on customer computers. 100% desktop stuff. And I guess to over time also on servers, on-prem servers. And this is sort like a big data type of thing before we knew to call it big data. It's like every click going on, now only certain customers opted in. And so it was a skewed percentage, a small percentage of overall audience.

Dany Hoter (00:34:32): Yeah. The enterprise was not presented enough because many large enterprises just blocked their users from sending anything to other softwares.

Rob Collie (00:34:41): Yeah. And the whole country of Germany was against this. But even with that tiny fraction of the audience opting in, you just think about every single thing you do in the app was being instrumented and sent back. Not the contents of your documents and things like that. It's like if there was a macro recorder turned on while you were doing everything in Office, we were being sent the code essentially. This was all being stored in SQL.

Dany Hoter (00:35:05): Yes.

Rob Collie (00:35:06): Because there wasn't anything else.

Dany Hoter (00:35:08): But it was too much. If you remember, we were getting too much. So for every application there was a factor of taking like one in eight, or one in four, or one in two if it was a small application. So for example, we couldn't take all the information. So we had to take for specific users, your ID, your anonymous ID is module eight something, then would take you. Otherwise, we thank you but forget you.

Rob Collie (00:35:34): Yeah. Even of the subset of people who had opted in, we still took a subset of those. But here's the beautiful memory I have of all of this, is that all this effort went into collecting and recording this data. And the whole point of doing it was to inform product decisions. We weren't going to sell this data. We weren't turning into an advertising network. It was 100% meant to be used by people like me to make better decisions. However, when I'd go to their system to consume the data, I couldn't consume it.

Dany Hoter (00:36:05): Didn't have any tool for that.

Rob Collie (00:36:06): And I have some friends that were on that team at the time, so I got to be gentle here. But it also did not seem like any of them were concerned that there wasn't a good interface for it. They were so focused on the engineering effort, that the Herculean effort of collecting, storing, et cetera, that making it remotely digestible for their fellow program managers was not even really on their radar. And so none of us used it. None of us were using it for a long time until Dany Hoter showed up with his Excel interface, these documents. I don't even remember how you did it. Suddenly, thanks to Dany and this Excel layer that he created between us and SQM. Suddenly we could start seeing things. We could start learning what was going on in SQM. SQM became useful because of the Dany front end.

Dany Hoter (00:36:59): I must admit that I don't remember it this way. I don't remember something else, but I don't remember exactly this. But if it's okay I would tell you another story about SQM. At one point I worked with reporting services and I was analyzing the data on SQM with reporting services. And my idea was to create a day in the life of specific office users, because we had the same anonymous ID for different applications. So I was able to show that this person came to work at eight o'clock, started Outlook, which was actually active across the entire day. Then he started Excel while he was in Excel he started PowerPoint for a while, then closed it. Then opened something else. So you can see by combining the data from different application, actually a whole works day of this office user. I don't think that this was ever used for anything or anybody. I showed it to some people, they were impressed but didn't really amount to any useful thing. But I was impressed with myself of being able to do this on top of the SQM data.

Rob Collie (00:38:12): The lesson that I took away from the SQM experience and I saw this repeat over and over again, I still see it repeating to this day. And it's a broad overgeneralization, so of course there's exceptions. Here was the conclusion I came away with, teams or people who are good at collecting and storing data are not good at analysis and vice versa. The data warehousing priesthood at the time did not care about consumption, did not care. It was being judged by its own standards of storage and like, look at this glistening tower on the city on the hill that no, you're never allowed to go there. No, don't be silly. And SQM was like that. And then when I got to Bing, the cosmos system was also like that. Again, Herculean effort, storing everything that ever happened on the Bing servers meant to inform people like me. And there was only one person in the entire Bing organization who was able to query it and get results.

Dany Hoter (00:39:16): And it took hours, for a single thing to run hours.

Rob Collie (00:39:18): Oh, days. The joke I would make was that you'd spend eight hours writing your query trial and error. Eventually kick it off, come back 48 hours later to get your runtime error.

Dany Hoter (00:39:34): This is almost as bad as when I started, when we were writing these programs in whatever language, someone had to punch them on cards. And you have to run the program and then you waited in queue for someone to run it. And after a few hours you get your listing and you say that you have a syntax error somewhere. And then you have to punch this specific line, place this one card, run it again and so on. So this is about same experience as what you're describing.

Rob Collie (00:40:04): I think that whatever the Psql or whatever we had to write, it was like a combination of batch files and XML. That was my equivalent of punch card. I kind of did my punch card time.

Dany Hoter (00:40:16): Interesting thing is that Cosmos is still a thing. Cosmos is still a service in Azure.

Rob Collie (00:40:21): But I don't know if it's the same thing. Is it inherited from the same tech DNA, or it's just again like a cool code name that got recycled just like your employee ID.

Dany Hoter (00:40:30): I don't know. I never used it. I'm glad that I never, I was tempted a few times or people tried to pull me in and to write the scripts for Cosmos and I never did, which is a good thing. Actually, people were also trying for years to get me excited about Kusto. And I also thought, why would I need because Kusto has a separate language which is not SQL. And many people are skeptical. And I was in the skeptic side also saying, why do we need something else when we have SQL? I had never learned it but only when I joined the team. And I'm one of the biggest fans ever of this technology and this language, which is so much fun. It's just fun to work with. It's amazing.

Rob Collie (00:41:14): How is it different from SQL?

Dany Hoter (00:41:16): Think about Power Query. Not the language, God forbid, right?

Rob Collie (00:41:22): Not M, okay.

Dany Hoter (00:41:23): Not M but the idea of doing it in steps. So you start by, if you want to write select star from table, you just say table. We start from table. Now you start adding steps to it with a pipe. You say table pipe and now you say where column equals something. Now you have the table filter to something. Now you say, you want to add a calculate column, you say pipe extend column, name equals formula. Now you want to sum to do a grouping. You say pipe, summarize sum of sales, buy this and this. And now you can continue. And now maybe you want only the summaries, which are greater than something you say, okay, where the name that you gave to the summary is greater than something. And you can continue forever. You can continue adding steps forever and ever. You just have a long list of these steps. One inheriting from the previous one.

Rob Collie (00:42:25): What's the name of this language again?

Dany Hoter (00:42:26): KQL.

Rob Collie (00:42:27): KQL.

Dany Hoter (00:42:28): And if you search for KQL, you'll find that there is like at least four or five different languages which have the same acronym. But if you say Kusto, you have tons of examples. You can see many demos and examples and people are doing just amazing things. I'm kind of on the intermediate level I would say, with it there are always people who are... Well, they're working with it for many years. I just started less than a year ago, but it's really very cool. One of the effect of this, there's an environment for creating the queries. So there's an environment which is a Windows app that you can install. And there's an environment which is on the browser, but both share a lot of common features. Now, at every point you get absolutely accurate intelligence, which is not the case with SQL.

Dany Hoter (00:43:23): When you write a SQL statement, you start with select. Now I have no idea what you can select, what columns because you didn't tell me yet what tables are you going to use. So I don't know what columns are you going to use. So you can write any garbage there. Once you start writing from it can go back to the list of the columns. And actually now I will get some help about because they have to come from the tables in the from. But the way KQL works, at any point so if you start with the table with the 100 columns, and now you say project column. So this is like you're limiting yourself to this column. So from this point on, your intelligence will include only those. Now you do summarize, after the summarize you'll have only the columns which participate in the summaries like group by the actual aggregations that you created so that every point you get very helpful intelligence. And the people who created this they attach a lot of importance to this feature, the fact that you can get very close intelligence.

Rob Collie (00:44:25): But didn't we establish earlier in this conversation, that the number one reason to create a language is because we like creating languages.

Dany Hoter (00:44:32): Yeah. Well, I say that about the people in analysis services. I don't think it's true for... But for the people who created there were four people who were working at the time. It was like seven years ago, eight years ago. They were actually in Israel, working in the development [inaudible 00:44:51] in Israel. And they were doing the porting of the analysis services engine, our engine to Azure. So they did the work that actually now is Power BI analysis services Tableau running in Azure. They worked on this porting and they were pretty bored that's to tell the truth, because there are very bright people and this was not a very exciting... Was very technical thing of moving port-

Rob Collie (00:45:18): Just time to make the donuts every day, come to work.

Dany Hoter (00:45:22): And they saw the need for a good tool for managing exactly this logs that we are talking about. I never actually connected it in my mind to SQM what you is telling about SQM. But every service everyone has logs, and logs are becoming bigger and bigger. Think about logs of Azure, logs of SQL and logs of Office. All these products, every product you can think of are putting their logs in Kusto. I don't know if you know that. But if you know log analytics, application insights, Sentinel, Azure monitoring, all of these tools are using as a backend tool. So they saw a need for something like that. Microsoft at the time was using Elasticsearch and other third party products, which were also very expensive. And I don't know them, so they say they were not very good. These four people they had some ideas and they started this product as a kind of a startup within Microsoft. With a few years, they managed to convince almost every engineering team at Microsoft. And then a few years ago they started to sell it outside of Microsoft. This is a story about-

Rob Collie (00:46:26): Okay, so nerdy question about KQL. I get it what you were saying. So an SQL statement starts with the verb, select. KQL starts with the noun, the table, because you start with a noon and then you go verb, so it's like your SQL is verb, noun. KQL is noun verb. Okay, so that's what allows the intelliSense to always work. I think that's awesome. But was that the only reason for creating KQL? It seems to me that SQL could have been written that way if we had thought about it in the beginning, if we thought about in intelliSense from the absolute start which we didn't as an industry.

Dany Hoter (00:47:11): One thing is the word didn't exist though. The word intelliSense didn't exist and there were no-

Rob Collie (00:47:16): Right. No one could anticipate that. I completely agree.

Dany Hoter (00:47:18): Also, the SQL statement is a very rigid construct, which everything has its place. So as you say, you start with select, you name a few columns that you want to get. Then you say they're from where with the joints and everything, then you have group and you have order. Order is at the end and that's it. You can't just now after the order say that you want now a second where you have having, you have this having something so you can filter the group list. But that's about it. So you use everything once, one workload, one list of columns. With the KQL it's just endless. And so you can change and combine things in much more flexible ways.

Rob Collie (00:48:05): So the two things I'm trying to tease apart here is that on the one hand I would tell you that if I had been introduced to KQL instead of SQL... If that magical world could've happened that way, I would've taken to KQL much more than I did to SQL just based on what you're saying. It appeals to me. I think there's a usability approachability to it. Lots of people have learned SQL, it's not an impossible language or anything like that. It's just I hated it so I didn't do it. KQL I would've been excited about. But there's another reason to create a language like this is that, the entire storage infrastructure of something like Kusto is completely different than the storage structure of a SQL database and so-

Dany Hoter (00:48:49): Well-

Rob Collie (00:48:49): Here we go. Now we're getting-

Dany Hoter (00:48:55): Kusto present itself very much like a relational database. You have cluster, in the cluster you have databases, in the database you have tables, tables have columns. Now columns can be much richer. So you can have a column which is adjacent. If your schema will fluctuate will just change, you need to introduce so in different timeframe or different areas or different branches have different columns. So instead of having all the columns for everyone, you can have just adjacent structure that in theory can be different for every row. So in one row, one element of adjacent exist and the other way it doesn't exist and so on. So, but it's richer than the content of a SQL database. Now internally, it's very different. It's a columnar. Of course, it's columnar database like every large database.

Dany Hoter (00:49:44): Database for big data is usually columnar, compression, tons of indexing. We index everything to death. Every column is indexed. You don't have to specify indices because everything is indexed. And if you have a free text column, we index words in it. So terms and words in the... So there's tons and tons of indexing going on automatically you don't have to weather. But it wasn't clear that relational will actually win and will become the gold standard.

Rob Collie (00:50:17): Back in the '70s or something like before I was paying attention?

Dany Hoter (00:50:21): Yes. There were some hierarchical databases which were not built on relational technology at all. And gradually they just disappeared. IBM had one, Digital have one. Look at IMS, I remember IMS was one of those it was an IBM product. So it was not always the case that relational was everything. Now, after SQL and relational we started to see things which are no SQL and doing things differently. But first it was the era in which SQL just became completely dominant. And also SQL as a language. When I started using relational, the first relation database I used was called RDB. It was from Digital Equipment Corporation. You remember this name?

Rob Collie (00:51:08): I do.

Thomas LaRock (00:51:08): Yeah. They're just down the street from me or they were.

Dany Hoter (00:51:12): They were in Maynard. At the time, they were the second largest computer company in the world. Second only to IBM, much smaller than IBM but still second. Very technological advanced company and so on. And they created also this database, which was state of the art relational. And at the beginning, they didn't use SQL as a language to query this relational database. They used something else which I can't even remember what was it called. Then gradually they started to say, oh, we have two ways to query. One, it's called SQL, which was identified with Oracle. Oracle was the one that actually from day one started to promote and use SQL as a language after he came out of the academia. But Digital took some time to adopt SQL, and then gradually they abandoned the other thing and SQL became everything. This is the history. So I'm taking you into the '70s.

Rob Collie (00:52:10): Yeah. Let's come back to today. So Kusto, it's the thing that you would use for massive log file storage for instance, the billions of rows, trillions of rows. Let's say I was building some new CRM app from scratch, like a Salesforce competitor, but a lightweight one. Would it ever makes sense for me to use Kusto as the storage platform?

Dany Hoter (00:52:35): It's not transactional is a pen only. So not for maintaining the transactional aspect of an application, that's for sure. That's not at all in way. But let's think about the BI side of things. So would you use it for a classic BI instead of Power BI or migrating from cubes or some other technology. And the answer I give is yes, I've been involved in implementations. I don't want to mention customer names, but let's think about very large retailer, worldwide, global, known to everyone. So this classic BI was the same analytics. The only thing unusual about it was that it was in the billions. So they have like three, four billion rows of history that they wanted to analyze. So there are still big users of Power BI, but with Power BI you have and they were importing everything into a very large Power BI capacity, thinking P4 something very expensive.

Dany Hoter (00:53:42): Even this was at the limit of what they can do. So we actually managed to get the data into Kusto connected with Power BI and get the performance with direct query, get the performance, get the logic. So large scale BI makes sense. Of course, if you're doing time series, log analytics, connected cars, we work with a lot of automotive companies. So when a lot of IOT, wind turbines, solar panels, all kind of things that would produce insane amount of data, for sure. But I also I'm a big believer in the large scale BI. If you have BI in the billions, if the word billion means something to you, you should look at it.

Rob Collie (00:54:33): Okay. So I was getting ready to fight because you were saying it replaces Power BI, but you're saying now works great in tandem with Power BI.

Dany Hoter (00:54:41): Yeah. Power BI as a front end, interesting to talk about role of DAX versus doing the calculations on the backend, because you can create measures in the same way that when you do Power query. So you create your query and then your intention is to use direct query. So every transformation you do, some people just write a page long of KQL with all the steps and everything they need. And they just page this query into Power query. And that's it gives them what they want and they don't. But you don't have to do it this way, you can just start with the table and start doing transformations. Like you filter some, you remove some columns, you filter some data, you add some calculated columns and maybe you aggregate them maybe not. Everything you do, every step on the way in Power query is immediately translated to KQL.

Dany Hoter (00:55:34): So you can go to the step say right click, same thing with SQL, right click, view native query. So you see the representation of all the steps you did so far, how they look in KQL. If you happen to use something which the connector doesn't know how to fold into KQL, you'll get a warning banner saying it's time to go to import. It's kind of a stupid banner because it says, now you have to move all your tables to import and you don't want to move to import. You started that query for a reason. So what you need to do is just delete your last step and do something else about it. For example, if you try to do something which looks like kind simple, add an index column. There is a type of new column in Power query called an index column, which will just number your rows one to three which sometimes it's useful.

Dany Hoter (00:56:29): We don't know how to do that in KQL or the connector doesn't know how to add such a column. So if you try to do that you'll get this error. But other than that, it's translated all the way and then you have the query and you run it. Now, you create on top of that you go to the model, and in the model you can make it even more complicated because you might add even calculated columns, and you can add measures. Now there's a need to translate this measures into KQL, so that everything can run and be sent to the backend. They did a pretty good job about it.

Dany Hoter (00:57:04): But if you go a little bit overboard with your DAX, what you pay is multiple queries. Multiple queries are being sent to the back end, then they do some calculation inside the engine. So if you do time intelligence or something like that, they will do it. But you might see some performance issues with just sending too many queries to the backend. You have to know your way. It's a new path that you have to take. It's not that you take everything you know about Power BI and you just implement it as it is.

Rob Collie (00:57:41): I'm looking up a picture here. Hold on a second. You know the picture I'm looking for Dany, do you still have it?

Dany Hoter (00:57:50): No, I must say I don't. Actually, it may be... I have a friend at Microsoft who used to work at the Excel team now he's actually in the security group, and we were sitting in the same room. And when I left Microsoft for the ends time, he kept this picture in his room.

Rob Collie (00:58:07): Yap. And then he came to... I found it. He came to the US and joined me on a site visit and when we met, he said, "Hey, you won't believe this Rob, but I have a picture of you hanging in my office." I'm like, what? He had very carefully kept all of the story from me so that he could lead with that sentence. Very entertaining fellow.

Dany Hoter (00:58:39): His name is Nathan.

Rob Collie (00:58:41): Here's the picture. Hold on a second. We're going to have to launch like a Patreon version of this podcast where people get access to the video.

Dany Hoter (00:58:49): What the...?

Rob Collie (00:58:51): That's the album cover. That's the picture on the back of an EDM album. It's got some fat beat that's about to drop. So that's me, Dany and [inaudible 00:59:03].

Dany Hoter (00:59:03): And in the background you see a lot of wind turbines, which can give you a clue about the location, because this is a country... We can do a trivia quiz, this is a country that gets 70% of their power from wind. And it's the only country that gets to this levels. And it's also a world leader in producing wind turbines. And this is Denmark.

Rob Collie (00:59:28): Dany did you know, it's so funny how clueless I was for so long about certain kinds of interpersonal clues. So Dany, I became infamous after that trip because I was the one held responsible for it. And it was the most expensive business trip like ever taken in Microsoft history by minions. I took shit for years about how expensive that trip was. And so here's how it worked. You and I Dany, conspired to make that trip the most expensive trip in the history of Richard Bank [inaudible 01:00:06]. Part of it was me insisting on American style hotels. I had just been to Europe like the previous summer and it was hot and there was no air conditioning. And I'm like, F this we're going to stay like in Hyatts. So I insisted on Hyatts and that's what Allan booked for us. But then we got over there, and Danny's like well, we're in Copenhagen. Let's go to the Michelin Star restaurant. And I'm like, right you are Danny. Let's just go.

Dany Hoter (01:00:37): What about the car? This was the trick that we had the Kia Shuma and-

Rob Collie (01:00:41): We had a Shuma and it was bad, and we took it back. We got another car.

Dany Hoter (01:00:46): We decided that this car is no hire. Remember the term no hire?

Rob Collie (01:00:50): Yeah, no hire.

Dany Hoter (01:00:52): This is a no hire, which is an internal joke at Microsoft. So when you don't hire someone you say it's no hire. We returned it and we went to the office and said, the car is in the garage and we don't like it. And they say, "Okay, take this key." They don't even go with you to the car. They just send you and say, "Okay, there's a car there. Just go." Then the second one was much nicer.

Rob Collie (01:01:14): And probably a lot more expensive we'd ever even found out. But I would sit there in team meetings where Richard and Rachel would be shaming me over this trip. And it was just like washing over me. It was like they were talking about someone else. And it's like 20 years later I go, it all lands on me I go, oh, I'm so embarrassed.

Dany Hoter (01:01:35): Actually, I remembered another trip we had a guy... I don't remember his name. He was originally from India. Anyhow, he was educated in the US and he couldn't drive stick shifts. He insisted on having automatic. And the only automatic was a Mercedes.

Rob Collie (01:01:52): There you go.

Dany Hoter (01:01:54): Yeah. So we rented a Mercedes which also was pretty expensive.

Rob Collie (01:02:00): Yeah. The intersection of Dany and Rob is like the Ikigai of expensive business trips.

Dany Hoter (01:02:05): My business trips overall at Microsoft took me to on one hand of the globe, Argentina, Brazil, Chile. And on the other hand, Australia, twice in New Zealand.

Rob Collie (01:02:17): Oh and to add to it, we were even on the executive floor of the hotel, so we had access to this lounge on the top floor. Where they would serve these really great breakfast buffets and cocktails at night. We go in there one day and suddenly Dany's talking to three other people, old friends of his from forever ago. They happen to be staying in the same hotel. And just completely random like he flew in from Israel to go to Copenhagen. We flew in from the United States and he just runs into a bunch of old friends. And it is not even a big deal. He's like, this happens wherever I go. All right, I have saved a very important question for you, Dany. Right now as we speak I am being roasted on Twitter.

Thomas LaRock (01:03:01): Oh yeah. You are. Well, you're not being... Okay.

Rob Collie (01:03:05): I'm holding my own.

Thomas LaRock (01:03:06): Yeah, you're doing great.

Rob Collie (01:03:08): I've got a lot more people after me than defending me.

Dany Hoter (01:03:11): Do you need my help?

Rob Collie (01:03:12): Yap. Sure, wait in. Here's the topic, Dany. Every time I tell an Excel MVP type, when I let it slip that I had a hand in changing the default of pivot tables to be compact access.

Thomas LaRock (01:03:29): Greatest move ever by the way.

Rob Collie (01:03:31): They look at me like I'm Satan. Now it's so clear to all of them that never should have happened. That tabular should have always been the default. Now that there's an option to change the default for all new pivot tables, they all use it to go back to tabular.

Dany Hoter (01:03:48): They also go to classic or not that much.

Rob Collie (01:03:49): I don't think anyone goes that far. I think they still like the field list without the capsules, the lozenges hiding and the-

Dany Hoter (01:03:56): But they don't like the cursor changing to a little blue rectangle up or down or the left side or the right.

Rob Collie (01:04:03): I think they probably don't care for that, but my counterpoint is that the masses that aren't opinionated. Let me not lay out my case. What do you think, was compact access by default a mistake?

Dany Hoter (01:04:16): No, I don't think so. Although there are cases in which I find myself changing to tabular when there is a need to actually remember what are the columns and it's difficult to remember. But the problem was that there were too many columns and screens were not that big, the solutions were not that good. And you were just having to start to scroll over just to see the numbers because everything took so much space. No, I think that compact was a good idea. You make a default, you make a stance. So default are very strong and most people don't know how to change the default. The expert know so they find the way. And now I think the feature of having a default that is sticky it's a good thing. It makes their lives easier. But in general, I think that it was the better... So is of course the current way versus the classic, which of course was an improvement. Yeah, I remember the discussions about the ribbon. How, how many people hated the ribbon?

Rob Collie (01:05:22): Yeah.

Dany Hoter (01:05:23): I was going to customers and getting so much negative feelings about the ribbon and how much space it takes and oh, it's terrible. And the first question was, how can we go back? So they want to do a classic option for the ribbon versus the old menus.

Rob Collie (01:05:42): Yeah. The UX team in office anticipated this and made it impossible.

Dany Hoter (01:05:47): To do classic.

Rob Collie (01:05:48): Yeah. They're like we're not going to even give you the option. Nope.

Dany Hoter (01:05:54): Yeah. But one of the other decisions people hated was about get pivot data. At some point, I don't remember it was 2003 or 2006, what version exactly, people were used to reference inside a pivot and just get a regular reference, which will break eventually. You do something on the pivot and your reference is broken. It's now pointing to some random place and so on. But they knew what they were doing so they were okay with it. And at some point with some version they were doing the same thing, but now instead of just having equals G3, they were getting this ugly thing that they didn't understand. And they were really angry about it.

Rob Collie (01:06:36): The three reasons I get that I've distilled out of the pushback on this, number one is that compact access makes it very difficult to do LOOKUPS into the pivot table, to do exactly what you're talking about. The formulas that reach into a pivot table to extract values become a lot harder because you don't know what level of the hierarchy you're on, to which I say we don't do that anymore. We've developed a whole bunch of tools to not have to use pivot tables as data sources for formulas.

Dany Hoter (01:07:03): If you reference into a pivot, you are already in trouble. So there's something wrong with what you do. I know that you've been doing this for years and so on, but you are in trouble.

Rob Collie (01:07:14): Yeah. Okay, so you and I completely agree on that one. The second one is like a data integrity point, which is the idea that there are now multiple different kinds of values in a single column. Which of course would be terrible in a database, like completely different semantic meanings in a database in a single column. But this is a report and so I'm not as worried about it there. And then the third objection is the subtotals at the top of the group instead of at the bottom, which I find to be an alien take because I like the totals that align with the header. Like there's USA and here's the amount for USA. I don't want to say here's USA and scroll down to find the amount for USA. But I think there's an accounting sort of historical reason for it.

Dany Hoter (01:08:00): I think that the fourth one to help your opponent, and this is about filtering. Filtering is also more difficult because you have to look for the right column and only then-

Rob Collie (01:08:11): There's a dropdown within the dropdown, I'm not proud of that but that's the best we could do. And I think it was worth it for compact access. And the point I'm trying to make to them is that some percentage of the people who were even objecting to this got into pivot tables sooner than what they would have, because compact access made them look so much more approachable. They had to come into the tent in order to get upset with us. And the whole goal zooming way back, the mission I was given was expand the usage of pivot tables. The person who did most of this work, the person who really deserves the credit for all of this amazing overhaul is Alan Faulting who reported to me. But I paid very close attention to pivot tables with him. And ultimately if a default behavior for the application for pivot tables got changed, I was just as responsible for that, on the hook for it. So I don't get to dodge it.

Rob Collie (01:09:05): But we were told to go make pivot tables more widespread, more readily adopted. And one of the conclusions we came to is that people much less building pivot tables, people struggled to read them. Those same people in usability testing and we did a lot of research, those same people that they really understood what the compact access pivot was telling them much better than they did the tabular, even the improved tabular. And so there's a selection bias in the people I'm arguing with. And that they're like in the absolute deepest end of the pool to begin with, and this change wasn't for them. And it's just a default, they can change it. Now they can change it back permanently. I kind of feel the same way for different reasons. I feel the same way about compact access as I feel about the ribbon, like it was a painful change that was on net a positive for everyone. But it's sort of like a cruise missile designed to piss off the advanced types, while broadening the tent of people who use it.

Dany Hoter (01:10:02): Some of the technology behind the regular people table in Excel is actually pretty amazing. The people with cash and the amount of data it can hold, even in this old fashioned machines with I don't know what the processor they have, and how much memory they have. And the speed that they can process the data is amazing. I don't know who is actually the person behind it.

Rob Collie (01:10:29): There's approximately 412 people who claim to have invented pivot tables. And they all introduce themselves as the inventor of pivot tables.

Dany Hoter (01:10:39): But it was definitely invented within Microsoft and within Excel. It was not taken from any other source.

Rob Collie (01:10:45): Those are arguments about whether Improv invented it. Improv was more of a twist attorney spreadsheet than an aggregator.

Dany Hoter (01:10:53): I just tell people you invented it, Rob.

Rob Collie (01:10:58): Oh, yeah. Well, that's fine. Go ahead. I won't say that.

Dany Hoter (01:11:02): You are a power pivot pro.

Rob Collie (01:11:04): Well, that was the moniker for a while. Yeah. I did not invent pivot tables. I'm not like Austin power's father.

Dany Hoter (01:11:11): I tell people something I believe in that I suggested IFERROR. But I know that Charlie thinks that he is the person behind it. But I remember that I am the person behind it and we'll never know, I think it's too long ago. But there're many people they believe it because I taught them, and usually I'm considered a person that tells the truth.

Rob Collie (01:11:35): You're not prone to just brash claims. I'm prepared to believe you, Dany Hotter, the father of IFERROR. I might have invented Concatenatex in DAX. I don't know if I did. I might have. We talked to Tim Rodman on this show and he was the one that I was at the bar with when we were talking about... And we go, oh my God, this would be Concatenatex, that we need Concatenatex function.

Dany Hoter (01:12:01): You know that I was once in a conference in Australia, and this guy was actually giving a speech about things which are evil about Excel. And one of them was IFERROR, IFERROR is part of the evil things. Can you think of why? Why would someone think that IFERROR is bad.

Rob Collie (01:12:24): I'll guess because it hides errors of all kinds and flattens them out into a number that so you don't even detect that there's an error.

Dany Hoter (01:12:32): Exactly. So even if you have a pound name error, if you misspell the name of a function you'll get the value for the error. You think that you divided by zero? No, you actually completely misspell the thing. And so for people who care very much about auditing and error detection and automatic error auditing, IFERROR is a problem.

Rob Collie (01:12:54): All right. So we've solved the world's problems, compact access, good.

Dany Hoter (01:12:59): So this is what you are being harassed on Twitter?

Rob Collie (01:13:03): Well, I asked for it.

Thomas LaRock (01:13:04): He did. He's getting what he asked for.

Rob Collie (01:13:07): I antagonized the-

Thomas LaRock (01:13:08): He really did. He called everybody out. He's like, come fight me.

Rob Collie (01:13:12): Yeah. I'm like, yeah.

Thomas LaRock (01:13:15): Just flat out told everybody, please attack me on this.

Rob Collie (01:13:18): I swear though, if Dany had said that he thought it was a mistake my stomach would've sank, because that would really mean something to me. That Dany agrees doesn't surprise me. But that was a risk in asking you. Bill keeps using Howie against me. Years later Howie told Bill that he thinks that compact was a mistake. Howie wasn't part of the original decision making process. He wasn't part of all that research we did, he was doing something different. And Howie's also a bit of an elitist on the user scale. He's in that deep end of the pool whereas I said on Twitter, I'm more for the masses kind of guy.

Dany Hoter (01:13:55): The father of Cube functions.

Rob Collie (01:13:57): Yes. And thank God he was working on that because they did a great job. He and one developer and one tester did that whole feature that we're still using to this day.

Thomas LaRock (01:14:06): What feature is that?

Rob Collie (01:14:07): Cube formulas.

Dany Hoter (01:14:08): If you go in Excel, you say equals cube, you get a list of seven functions which can pull data from the data model or from a cube or tabular. I'm a big fan.

Rob Collie (01:14:20): Dany is next level. Like when I wanted to create my calendar chart view in Excel, turn the Excel grid into a dynamic calendar, I needed to pick up the bat phone multiple times and call the Hoter.

Dany Hoter (01:14:35): Now another anecdote about Cube functions, which actually I saw an example of this today. I was working with a French colleague and a French customer, and this is the case in which they're trying to move out of OLAP and we are trying to suggest Kusto as a replacement. I was telling them that they can actually... And he didn't know about Cube function. They never use it. They are big users of Excel connected to OLAP and never find about... Now, they're using a French version of Excel. So all their functions are in Excel. So it's not equal sum, it's equals somme and so on. Now, you know that French, Spanish, Italian, German, all those languages have the functions translated into their native language.

Rob Collie (01:15:24): I did not know.

Dany Hoter (01:15:24): You didn't know that. Oh, of course internally they are maintained the same. So if I'm working with the Italian version I send to you, you'll see it in English, no problem.

Rob Collie (01:15:34): That makes sense.

Dany Hoter (01:15:34): But on translating the names of the cube function, someone did a big mistake. Instead-

Rob Collie (01:15:41): Oh yeah. Bring it.

Dany Hoter (01:15:43): For example, we have cube value they translated it to valeur cube. So the right syntax in the language is to reverse and say like value cube. But the effect of this is that when you miss intelliSense, if you start writing equal cube there's nothing there. Every cube function starts with a different word. So they're not combining together, and I saw this years ago in Spanish. I notified the people. I saw actually in the office in Argentina, I saw some people using a beta version of Excel. I started to shout about it. It was too late, decisions were made. I wasn't able to change it. And now I see that it's the same with in French, valeur cube.

Rob Collie (01:16:30): I thought this was going to be like a Chevy Nova story. Where they translated to a... It means does not go in Spanish. I thought it was going to be that kind of story. But Dany, that story brought back another one for me. There was a point at which I think Power BI, they had rushed bidirectional relationships into the product. Do you remember this?

Dany Hoter (01:16:55): No.

Rob Collie (01:16:57): I had been part of like a SWAT team that had been sent to a customer in Pennsylvania, that was debating adopting Power BI versus Tableau. And because they didn't understand data modeling, they didn't understand the idea of a star schema powered model as opposed to flat tables. They just kept complaining about why is this so hard? And they had a direct line into the executive layer at Microsoft. So they started getting all kinds of like concessions made in the UI and in the capabilities. So like we want relationships to be bidirectional. So suddenly bidirectional relationships showed up in Power BI. And I was like, oh my God, this is a terrible idea. It's going to change the semantics of DAX and everything.

Rob Collie (01:17:45): And I got ahold of you Dany, because I was not an employee anymore myself. I got to a hold of you Dany because this was during one of your times when you were on at Microsoft and said, "Have you seen this? And like, it's going to change the results of formulas." And you went and played with it and said, "Oh my God, it even breaks time intelligence." Like the day intelligence functions actually start erroring out with these bidirectional relationships.

Dany Hoter (01:18:07): Yeah, now I start remembering.

Rob Collie (01:18:10): Yeah. And you and I got this feature blocked. Like we saved the world, Dany. Now we have them, one team did them without properly vetting it through the DAX crew. It was like done in a separate universe.

Dany Hoter (01:18:25): Do you find yourself using it nowadays?

Rob Collie (01:18:28): Bidirectional relationships? No. There are occasional situations where I do, the many to many pattern. But per true many to many is so rare. And so to this day, bidirectional relationships are way too easy to turn on in Power BI. Like if you're uninitiated and you're being asked, do you want one direction or two, you're instinct is, well, I want more. Two's better than one. And so people I think are very often... I don't think I know that people are very often activating bidirectional relationships to link two fact tables to each other. And so they never learn the value of dimension tables. They get set off down this completely the wrong route.

Dany Hoter (01:19:12): You want to hear another Kusto Power BI story?

Rob Collie (01:19:15): Sure.

Dany Hoter (01:19:15): Connected to exactly what you're talking about. This is more of a technical point, where you create something like a star schema with all tables coming in direct query from let's say Kusto. And you try to create a relationship. So you drag the column to the other side and you see something very weird. You see it says single, directional one to many and then after two seconds it changes many to many, both. Why?

Dany Hoter (01:19:49): What's going on? The reason is that it's starting to do the validation to see if the columns are unique on both sides. So it's very quickly realized that the fact table the key is not unique there, so this is relatively easy. Although it could take time because you imagine if the table could be 50 billion rows. So looking for uniqueness on a 50 billion rows it could be a challenge. But now it comes to the dimension and it tries to see if the key in the dimension is unique. And the way it does it is that it creates a distinct count versus count of the rows. So this distinct count of the column must match the count rows. Okay, so far so good. Problem is that distinct count in a tool like Kusto is an estimate, is not accurate because you cannot do a performance distinct count on a data which is distributed to 500 nodes without bringing all the data to one place.

Dany Hoter (01:20:56): So there is some very smart algorithms that actually I think were develop in Google and published to the public domain, that are used for this kind of calculations of distinct count. And you can even say, distinct count and you have another extra parameter to say how accurate you want it to be. So you pay more if you say I wanted to be closer to the actual value. And there are also techniques to get the exact value, if it's very important for you can actually pay more in performance and in resources, and do it some somewhere in a somewhat different way. But-

Rob Collie (01:21:31): Do you hear that? You can pay more for the right answer?

Dany Hoter (01:21:35): Yeah. So with Power BI-

Rob Collie (01:21:37): So subscription service isn't the right answer.

Dany Hoter (01:21:39): What it tries to see how many compare the number of rows to the... It sees that it doesn't match, the only conclusion is it's many to many. And the default for many to many is both directional, bidirectional. But the good thing is what I tell people is ignore, you can leave the many to many because it has no effect on... Actually one effect on one little effect on the behavior when we're talking about this direct query thing. But you have to change the both to single to the right direction, don't leave it as both. But you can change the many to many, to one to many for example with tabular editor. You can go with tabular editor and edit the relationship property, and then there's no validation. So you can do whatever you want and you can say that the many to one is the one to many or whatever, it's up to you.

Dany Hoter (01:22:33): So you can change it there or there are other ways to do that. But this is something that happens to a lot of people. They're scared because they were told in their course when they learned about DAX and modeling, that many to many is rare and should avoid it. And in that the joints which are created between the two tables are using inner joints. When it's the many to many the joints are inner, and when it's one to many the joints are left out there.

Rob Collie (01:23:01): Okay. Yeah, so I don't want any of that, what it comes down to. Hey, this conversation so far has been very friendly, but I have a real reason to resent you, Dany. Do you want to know what it is?

Dany Hoter (01:23:13): Yes.

Rob Collie (01:23:14): Dany Hotter has probably cost me... I don't know, $200,000 maybe more.

Dany Hoter (01:23:20): What?

Rob Collie (01:23:21): Yeah. Okay. So I had created a power pivot video class back when the company was still just me and Jocelyn. And we were charging like $350 per class for this video class.

Dany Hoter (01:23:35): Oh. And then I did my edX Score.

Rob Collie (01:23:39): That's right. This thing was going crazy, Dany. I was making so much of income out of this. The trajectory we were on it was like, this could be retirement right here. And then one day the signups dropped by like 90%. And it was exactly the time that Dany launched his incredibly good and free similar class on edX. And that was it. That was the end of that video business for me.

Dany Hoter (01:24:06): I did two of them and last that I looked it was while ago, the first one went up in 2015.

Rob Collie (01:24:13): That sounds about right.

Dany Hoter (01:24:15): Last I looked at the numbers and this was already for a few years ago, now they don't give this archive. They don't give it anymore. There were like a 1.2 million people that took this class.

Rob Collie (01:24:26): Right. See, there you go. So Danny, it could have been $350 million for me.

Dany Hoter (01:24:31): Wow. That's also the power of free. The fact that it was free.

Rob Collie (01:24:36): Yours was just the first. Then there were all the U2ME classes that were being discounted to like $5 at the holidays, because U2ME owns the pricing and can do whatever they want and everything. So it was a race to the bottom. But you were a little quick, if you had given me like another six to 12 months. Remember how you were talking about if, oh, if I'd been employee 10,000 at Microsoft I wouldn't be working. If you just waited another six months, Danny.

Dany Hoter (01:25:02): Well, you know what actually contributed a lot to the success of this course is something that I didn't have any control on. And this is that someone there at edX, they decided when it was published first to put it in their homepage. So it was highlighted and this made a huge difference. A lot of people found it. In the beginning it was like going like crazy. And then when it started to go down to the list of courses, then it slowed down because it's so important how easy it is for people to find you. So they were going at edX just to look for something and here it is, analyzing data with Excel.

Rob Collie (01:25:42): And probably while it was on the homepage, it was picked up by the search engines. And probably it wasn't as picked up by the search engines as it fell off of the homepage. It's like-

Dany Hoter (01:25:50): I'm sorry.

Rob Collie (01:25:50): It's okay. It's like this is stealing a Will Ferrell joke from Saturday night live. But the chart of signups for my class comes with its own built in slide whistle. There's this downward slope that goes whoo. That's okay. It just sort of accelerated and hastened our transformation from being just a quote unquote 'mom and pop' operation to being a consulting firm. And we still do training. We still earn a fair amount of revenue from training. Whereas it used to be like 95% of the company's revenue it's now I think under 20%, and we're primarily an implementation company. We still help people. We're happy to teach people as we build. And actually I found that's far more effective for teaching people than training. It's just that training is sometimes the thing you do in bulk. We're very open and transparent, we love sharing knowledge.

Rob Collie (01:26:48): It's kind of like you Danny, if someone hired you to go build something, you would want to be explaining to them how it works. You wouldn't be able to help yourself. And that's how we are too. Sometimes the client's like, no, we don't want to know. Just built it. Okay, fine. This podcast is at least 50% an excuse just to catch up with people or meet people. We meet some new people this way too and that's fun as well. But so enjoyed speaking with you. And you're the best tour guide I've had on any vacation of all vacations I've ever had. Tom, this guy took us on the comprehensive tour of Old City, Jerusalem.

Rob Collie (01:27:31): This is not an experience you get in America. At one point, the calls were going out for the Muslim prayers, and we were walking upstream against the crowd of everyone flowing in to prayer. What a different experience. But we also did things like Masada. We floated in the Dead Sea. We did it all, man.

Dany Hoter (01:27:54): Yeah. You should repeat it.

Rob Collie (01:27:55): I think so, the city's only getting older. So Dany, thank you so much for making the time.

Dany Hoter (01:28:00): You're welcome.

Rob Collie (01:28:01): And also, thank you so much for everything you've done for me and others like me over the years. You truly have influenced so many people. There's not many people in your same category or your scale anyway. And so many of us out here that owe a great debt to you, we all know it. So thank you very much for being you, and looking younger than the last time I saw you. Fantastic.

Rob Collie (01:28:22): Thanks for listening to the Raw Data by P3 Adaptive podcast. Let the experts at P3 Adaptive help your business. Just go to P3adaptive.com. Have a data day.