Setting Up For Success w/David McKinnis

Rob Collie (00:00:00): Hello friends. Let me ask you a question, and be honest. How often do you think about software installation? Basically never, right? That's healthy. It's not something that you really would think about. But like so many things in tech, it can be invisible most of the time, and still have tremendous implication in the human plain.

Rob Collie (00:00:22): Today's guest, David McKinnis, well he and I, many years ago, spent several years thinking about nothing but software installation. Because he and I were working together on the first version of Windows Installer, MSI. Which is, I think to this day, still the most commonly used installation technology for installing software on Windows. So fair warning, we talk a lot about setup.

Rob Collie (00:00:49): I sincerely think you'll actually find it very interesting, to hear something so invisible discussed as such a tremendous challenge, and it absolutely was in the late '90s and early 2000s. I like to think that our efforts as a team have something to do with the fact, that you don't think very much about setup today.

Rob Collie (00:01:07): The world of setup was my first real exposure to the world of IT, and also to this concept of thankless jobs in tech. Where you only get noticed when you fail. Which seems to be a lot like IT, right? So even though I left the world of setup behind, I carried with me some very, very valuable lessons, that are relevant to me, even to this day.

Rob Collie (00:01:29): Those were also some incredibly early and formative years in my career. I found myself in a position as a youngster, way over my head. As a 24 year old, I had a ringside seat for Windows, Office, inter-division politics. Even at the highest levels. It is not the most dramatic technology I was ever close to at Microsoft, but absolutely the most dramatic events of my career did take place in that era, and we talk about some of those.

Rob Collie (00:01:58): David was a long time veteran. He had already been around Microsoft for more than 10 years when I showed up. The way he treated me in those early years, I think was crucial in me choosing to stick around. Those first few years at Microsoft were very hard. I didn't know whether I would even want to stick around. Having the support of someone so senior and influential as David, I just can't imagine having made it through without it.

Rob Collie (00:02:25): We don't just talk about setup though. We also talk about teaching, because he has spent a fair amount of time, both in his Microsoft career and subsequently, as a teacher of technology.

Rob Collie (00:02:36): Here's another one you might not have thought about very much. When a software company wants to buy another software company, who do they call as the inspector? Think about it as buying a house. You call an inspector, to inspect the house you're about to buy. Well, they do that in the software world as well. They call it technical due diligence. That's a huge part of David's day job these days, and has been for a while. I find this fascinating, so I asked him a number of questions about it.

Rob Collie (00:03:01): It even led to an example of me telling a story, about a technology that Microsoft decided to purchase, even though the due diligence came up very poor, and we've redacted it from the show. It just felt dirty, sharing the name of the product. But if you guess it on Twitter, hit me up. I'll tell you, if you get it right.

Rob Collie (00:03:19): It was a wonderful excuse to catch up with an old friend, and just a really warm, friendly, and caring person. Those are the kinds of people we like in tech, right? Heck, those are the kinds of people we like, period. So let's get into it.

Announcer (00:03:34): Ladies and gentlemen, may I have your attention please.

Announcer (00:03:38): This is The Raw Data by P3 Adaptive podcast. With your host, Rob Collie, and your co-host, Thomas LaRock. Find out what the experts at P3 Adaptive can do for your business. Just go to P3adaptive.com. Raw Data by P3 Adaptive is data, with the human element.

Rob Collie (00:04:03): Welcome to the show, David McKinnis. It's been a long time. We've talked periodically over the years. I haven't seen you since Redmond, right?

David McKinnis (00:04:13): Our paths haven't crossed in person since Microsoft.

Rob Collie (00:04:15): That's right. But you've been really good at keeping in touch. The reason we're still in touch is because of your efforts. I really appreciate that.

David McKinnis (00:04:23): It's always interesting to talk to you, and you're always doing interesting things. I think you have a good outlook on life as well.

Rob Collie (00:04:27): I appreciate that. Well, it was forged in the fires of Microsoft. Other adversity. You were there for some very formative years in my life. In talking about this podcast ahead of time, backstage one of the themes was, do we really want to revisit those old years? Those versions of ourselves, that were quite a bit lesser than today's versions?

David McKinnis (00:04:50): Well you learn more when you make mistakes, than when you're successful. Sometimes when you're successful, you're lucky.

Rob Collie (00:04:55): Yeah, that's right.

David McKinnis (00:04:56): Even though you did the wrong thing. But it's when you make mistakes.

Rob Collie (00:04:59): Obviously I know you. Let's give the audience a little bit of background on you. Were you a straight from college to Microsoft individual?

David McKinnis (00:05:05): I was straight from college to Microsoft. I actually interned at Microsoft in 1985. The Microsoft intern program, we used to say it was a great bait and switch program. Because you're there in Seattle, between May and some time in August. Boy, those are just beautiful months in the Pacific Northwest. Seattle weather is okay, but it's definitely not the same in those other months.

David McKinnis (00:05:31): Being from North Carolina, the natives from Seattle getting worked up when the temperature finally broke 90 degrees, one summer day when I was there as an intern. I'm thinking, "Boy, this is nothing." Yeah, so I was there as an intern in '85. Then I came out when I graduated from college in 1987. Got to work on Word for the Mac, and had a lot to learn.

David McKinnis (00:05:52): What you learn in college today is more about software development, than what was taught in college back then. You came from learning how to write code, but not really develop programs. So I had a ton to learn about how you actually write features that people use, and are interested in, and that sort of thing.

Rob Collie (00:06:09): I certainly also had a computer science education that didn't remotely prepare me for what I was going to be doing. Less so even than yours, right? Because I ended up in program management, which required very little to zero actual programming knowledge, and much more knowledge about people, and project management, and all those sorts of things. I was a weird kind of nerd.

Rob Collie (00:06:32): I remember when I joined in '96, I joined the Office Configuration Management Team. CAPM, we called it CAMP. Why was it called CAMP? How do you get CAMP out of Configuration Management? Do you remember?

David McKinnis (00:06:45): Who knows? Probably someone needed a good acronym, and-

Rob Collie (00:06:47): CAMP was the closest we could get?

David McKinnis (00:06:48): That was it.

Rob Collie (00:06:48): So I joined a team called CAMP. You had already been there nine years. I thought of you as a veteran. In hindsight, holy cow. You'd been there nine years when I showed up.

Thomas LaRock (00:07:00): What was his employee ID number? Was it only four digits, like our last guest?

David McKinnis (00:07:03): I don't remember what my employee ID number was. Probably, because there were only, I think there were around 1200 or so, when I started.

Thomas LaRock (00:07:12): Yeah, in the mid '80s? I was just thinking, yeah.

David McKinnis (00:07:14): Actually, the story I'll sometimes tell people is, when I was there as an intern, the building was Highway 520 in the-

Rob Collie (00:07:20): 405?

David McKinnis (00:07:21): ... cuts on the east side. Yeah, 405. There was an office complex there. But then when I started in '87, they had moved out to where their current campus is. I think we only had four buildings when I started. They may have been building the fifth and the sixth ones. It was back in this office park. So I actually got lost my first day, because I was driving around. There wasn't any big sign at the corner that said, "Microsoft, this way." You had to go back in the office park, and find the little Microsoft sign.

Rob Collie (00:07:49): Yeah. Whereas today, Microsoft signs almost replace the traffic signs. It's like it's got its own gravity.

David McKinnis (00:07:56): Yeah, now the problem is probably, which of the campuses are you supposed to report to?

Rob Collie (00:07:59): Yeah. Are you even on the right campus? That's right.

David McKinnis (00:08:01): Yeah.

Rob Collie (00:08:02): Now you mentioned that the weather in Seattle is okay. I take exception to that. I've found it to be oppressively sad in the winters. Everyone talks about the rain. The rain's nothing.

David McKinnis (00:08:13): It doesn't really rain.

Rob Collie (00:08:14): It's just a constant mist that never stops. Okay, fine. However, the mile think blanket of clouds that stacks up against the Cascades, and blocks out the sun forever? That takes its toll.

David McKinnis (00:08:27): Oh yeah.

Thomas LaRock (00:08:28): I didn't realize, I think I knew, but I didn't really realize it, and I don't think I've ever really told you this. So if you started in '96, and you were living in Seattle, I was five hours southeast of you, in Pullman, Washington. Because I was at Washington State from '95 to '97. So I had my own weather, in Palouse.

Thomas LaRock (00:08:50): So everybody's like, "Oh, you're going to Washington state? It's going to be like this, this, this." I get there, there's nothing but wheat, and lentil fields. It was beautiful in its own way. But I had to drive through central Washington, before I could get to the pine trees, and everything else that you imagine, and the mountains, and then get into the Cascades.

Thomas LaRock (00:09:09): But yeah, what you're describing, for me? It got gray, maybe about Halloween, and it stayed gray until about April or May. It was just overcast, 40 degrees, and rain, every fucking day. It was miserable.

David McKinnis (00:09:25): Yeah. I distinctly remember, so my wife graduated a couple years from college after me. We carried on a long distance relationship for those two years. She moved out to Seattle, and I very distinctly remember, very shortly after she moved out in September, the cloud cover came in. I have this very clear memory, on her birthday, in June, I finally saw the sun. I'm like, "There is absolutely no way she is going to let us stay here in Seattle. It's been all these months of clouds. She's going to say, 'Look, love you. But you can either stay at Microsoft, or come with me back to the east coast.'"

David McKinnis (00:10:02): Very thankfully, she didn't do that. She put up with it for quite a while. That was one of the problems with driving. Because if the sun ever came out, everyone was like, "What's this big orb in the sky?" Or Mount Rainier is visible, and it's like, "Oh, and there's a mountain there." Everyone would slow down to look at it, because they were so not used to seeing it.

Rob Collie (00:10:18): I interviewed in February, out there. It happened to be the rare clear day. So I saw the mountains on all sides. It was crystal clear. It was just like, "Oh, this place is gorgeous." So even in February, I got a little bit of a bait and switch.

David McKinnis (00:10:33): My father in law though swears, when we were out there, he swore that we were making it up with the clouds and the rain. Whenever he came out to visit, it was nice. So yes, some people can get lucky.

Rob Collie (00:10:45): How did you find yourself from Word, over the nine years you were there before I showed up, how'd you find your way from Word, onto something like Office Setup?

Thomas LaRock (00:10:56): Who did you piss off?

Rob Collie (00:10:57): Yeah. I've never asked you that question, but now I'm struck by it.

David McKinnis (00:11:02): Well, I'd had a chance to work my way through the ladder. I had started off as a junior developer, and then worked on Word 5 as a tech lead, with David Lubert, who was a wonderful lead for the team, who was much more experienced than I. We got that, it went out the door. Then Mac Word 6 came along, and I was the development manager for that project. We had a few bumps. For anyone who remembers back to Mac Word 6, it was not the best release of all. Those were some of those failures that you look back on, and you realize-

Rob Collie (00:11:31): That we learn from, yeah.

David McKinnis (00:11:33): You learn quite a bit from them.

Rob Collie (00:11:34): Learned a lot on Mac Word 6, it turns out.

David McKinnis (00:11:36): Learned a lot in the first release of Mac Word 6, yes. Then it felt time to move up, partially because just the way, I think, other companies have changed over the years. As far as, sometimes you don't need to push people up to higher management levels, so they can do other things, and be good technical people. But the logical progression at that point was to be a development manager.

David McKinnis (00:12:00): Looked around, and this was also a very interesting problem to solve. We were trying to solve it at the time actually, for both Windows and Mac at the same time. The goal initially, actually with the first version of the team, was to create one setup script that ran for both products. Both the Windows and the Mac version. So a really interesting and complex problem. There were some neat people working on it. Of the possibilities for a development manager spot, this seemed like a good place to go, so off I went.

Rob Collie (00:12:34): There's actually some parallels in that for me as well. We'll get to that. You mentioned this idea of a single script, that could be used to install Office. Now, people don't even think about this anymore, which is good news, right? People don't think about software installation nearly as much as they used to, because it now mostly works. You take that for granted, because is used to very often not work. Dave, you and I have made our little contribution to that world. We'll talk about that in a moment.

David McKinnis (00:13:03): Yeah, I believe some of our work actually has contributed significantly to that-

Rob Collie (00:13:07): Part of that new world. I completely agree. This idea initially, that the logic that would be authored to install Office, a very complicated product even at the time. Already, incredibly complicated. Byzantine is a word you could use. To author one script that could run on both platforms, there was no code that you could compile that would run on both platforms, right? So is this where those STF scripts came from? Is that the reason why we had this new language?

David McKinnis (00:13:36): That was some of it. But very quickly, we figured out that this was going to be quite problematic. If you want to call it the same script, they ran through a pre-processor. So you had a bunch of if-def, Mac if-def Win sorts of things. Saying it was the same script, in the end, probably was a bit fanciful. Although it did I think force people to think about components that were similar. Try to keep them in some sort of synced-ness. It quickly became clear that this was not going to work out too well.

Rob Collie (00:14:05): When I showed up, all of this was just the bedrock that I was introduced to. I wasn't exposed to any of the history, or why it was like this, or anything like that. I didn't have enough perspective to even ask these sorts of questions back then. But Tom, this is akin to how data transformation, ETL processing, such a very specific and nuanced space, that they basically had to invent a new language to do a good job with it. This is where the M language for Power Query came from.

Thomas LaRock (00:14:34): Not this again.

Rob Collie (00:14:36): I know. No, no, no. No, listen. Listen, we're going to revisit this. Dave, we've recently recorded a software hall of fame episode, where-

David McKinnis (00:14:46): Oh, okay.

Rob Collie (00:14:46): ... we've created an inaugural class for the software hall of fame. There's rules for it. We'd love to have you back on to argue about it. I think that's going to be, some percentage of our future episodes will be having people on, who disagree with our selections, think we missed some things. That's going to be a shtick we can wear out for a little while. Anyway, so Tom's talking about a product. He doesn't think it's a product, but anyway, it doesn't matter. If it passed the product test-

Thomas LaRock (00:15:09): It's not.

Rob Collie (00:15:09): ... it would definitely be a solid hall of fame candidate. So in the same sense as that, okay, yeah. It totally is.

Thomas LaRock (00:15:15): No, it wouldn't.

Rob Collie (00:15:18): I've got the editor on my side, Tom. We're going to splice together snippets of Tom saying, "Power Query is the best product in the software hall of fame. It's a travesty that it didn't make it in in the inaugural class."

Rob Collie (00:15:37): So STF, I don't think any of us would look at it and call it optimal, looking back. Right? But it was the same sort of thing. A language of sorts, invented for a very specific purpose. Now Tom, as a software tester, my first job at Microsoft, software test engineer. Assigned to this team, that is creating both the engine that reads these STF scripts, and then applies them to your computer as the install. They're also the team that's writing these scripts.

Rob Collie (00:16:06): As a tester, what do I end up doing? There were two things that chewed up the majority of my days as a tester on Office '97 Setup. One of them was running Windiff on half meg script text files. Half a meg of text, running Windiff on it. Oh, but then you have to pre-process it, Tom, to take out all the little line numbers and things like that, that would lead to extraneous diffs that you didn't really care about. Then pour through the diff all day, to hopefully find two or three things you could flag as a bug, and then have the dev team say, "Nah, won't fix."

Rob Collie (00:16:47): The other thing I did, was wipe machines down to bare Windows, over and over and over again. Install software. Oh, snapshot the machine. Every file and every registry key. Install software. Then run a diff of that state of the computer, versus the basis. Then sometimes even run a diff of diffs. Oh my god.

Thomas LaRock (00:17:12): Of course.

Rob Collie (00:17:13): This is brute force labor. But somewhere along the way, y'all had learned that this engine and its scripts, the ACME engine, that was the internal codename. This, Tom, is where Swedish ACME comes from. Which was the name of our running team. I remember one day seeing the build report come out. Something was wrong with Swedish ACME. I was like, "That's some Looney Tunes shit if I've ever heard it." New band name.

Rob Collie (00:17:44): In parallel, y'all had been incubating this Skunk Works project called Darwin. This is where the story really gets interesting. How much had you been involved, David, with the impetus for there being a Darwin?

David McKinnis (00:18:01): That's a good question. I'd have to think back quite a ways on that one. My recollection is that John DeLowe was clearly the one who saw that need. Or at least saw the direction we could go. I think everyone felt like, "Gosh, this is lousy." But I'm not sure that anyone had a clear sense of what direction we could go, and how we could get ourselves out of this pickle, before he was really thinking about it.

David McKinnis (00:18:25): My recollection too is that he probably had more bandwidth than anyone else at the time. Because I seem to remember that he came on, when the rest of us were knee deep in just trying to play whack-a-mole, and getting things released. It's one of those, it's hard to remember why you're doing what you're doing. You need to step back, and really re-evaluate where you are.

Rob Collie (00:18:47): There is simultaneously nothing more valuable, or more dangerous, than a software engineer with some free time.

David McKinnis (00:18:53): Oh yeah.

Rob Collie (00:18:55): It was both. Let's be clear. This is not a binary thing at all. They key difference, so Darwin was meant to be, and it ultimately was, the successor to ACME. A replacement for ACME. So y'all know Darwin as Windows Installer, or MSI, .MSI files. The originally internal codename for that entire project was Darwin. The Evolution of Setup was the subtitle.

Rob Collie (00:19:23): The key computer science-y observation driving Darwin was that ACME, and the STF scripts, were all imperative language. Like, "Go do this." Each line of the script says, "Go do this. Go write this file to the disk. Go write this registry key to the disk." Darwin's move, the key shift in Darwin was, don't worry about how it happens. Specify what you need. What is your product made out of? What are the parts of your product? It's not an imperative script. The thing you're going to develop as a setup developer, is a description of what your product needs installed in order to function.

Rob Collie (00:20:00): So it was more of a declarative thing. You described the end state that you need. Then the engine handles all of it for you. The Darwin engine was a lot more sophisticated than the ACME engine, because it had to know how to do everything. Whereas the STF scripts had a lot more control over what happened in the ACME world.

Rob Collie (00:20:19): So this shift to declarative, I think we can say, was a good move. Because one of the things, Tom, that was really amazing, about the imperative thing, is that you had a path, almost like a choose your own adventure path. Line items call line items call line numbers call line numbers. It would go through this STF, to install the product.

Rob Collie (00:20:38): But then if you wanted to uninstall it, if heaven forbid you wanted to uninstall the product, there was a completely separate choose your own adventure pathway that would bounce around through this STF file to remove things. Well, guess which one got tested the most?

Thomas LaRock (00:20:54): The install.

Rob Collie (00:20:55): No, none of us were uninstalling. We were wiping our machines clean, because we didn't trust that thing.

Thomas LaRock (00:21:02): Right, exactly.

Rob Collie (00:21:02): So uninstall would be very buggy compared to install. Then it was also reinstall, which was repair. Guess what? It also had its own path through the STF file. That's the path not traveled. That's off the beaten track. So it doesn't get nearly as many eyeballs either. Whereas in Darwin, if you describe what makes up your product you don't need to describe uninstall, reinstall, and install separately. The machine understands that. In theory, you've got reliable uninstall. You've got reliable repair. But really, honestly, the biggest win is that you've got reliable install. This guy, John, he was challenging to work with. But he was also incredibly smart.

David McKinnis (00:21:39): Yeah, the vision of declarative versus imperative was really important. The other thing that was clear, and I forget actually now in the STF world, how we kept track of this. But the other issue was just keeping track of, what other programs needed those things that you had touched? Because you didn't want uninstall, for example, if you didn't install the C runtime version four, if that was already on the machine when you came along, and you needed C runtime version four, at uninstall time you didn't want to yank it off. Because if you removed it on uninstall, then suddenly, now some other program doesn't work. So uninstall was quite dangerous.

Rob Collie (00:22:19): Yeah. This notion of ref counts, right?

David McKinnis (00:22:20): Right.

Rob Collie (00:22:20): How many people need this file? If you're uninstalling, and the ref count, the number of people who are using that file, is greater than one, don't remove it. But subtract one from the ref count.

David McKinnis (00:22:33): If I remember correctly, I think that was part of the problem with the reinstall world, with STF files as well. That reinstall might-

Rob Collie (00:22:39): Yeah, bump the ref count.

David McKinnis (00:22:41): ... change that ref count. Then that library would never get removed.

Rob Collie (00:22:44): Right, and this is back when hard drive space, relative to your software, was relatively critical. The chances of running out of hard drive space were higher back then. So a really thorough uninstall, a safe but thorough uninstall, was quite an asset. Yeah, this was a mess.

David McKinnis (00:23:01): Well, then you also had to take a step back further. The other issue that we had, this was the time when we were getting a lot of pushback from companies, because they wanted to install Office on 500 machines overnight. That script based STF, you might have five questions in it. Of course, we don't ask those questions upfront. You ask that question when you get to it. So it's really hard to create an automatic script that says, "Go off and run ..." You have to have someone who goes in, and touches each of those machines, and answers the questions as they get to them.

David McKinnis (00:23:36): I know we had some hacks to try to make that less painful. It was really not tenable for large companies, so that's where we morphed into, the team was called the TCO team, the total cost of ownership. Which was interesting, because it foreshadowed, do you buy open source software? Or do you buy a license? Do you get free open source software? Or do you pay for software? The arguments around, "Well yeah, you may pay something to buy this software. But if it's easier to use, and it's actually cheaper than the stuff that appears to be free."

Rob Collie (00:24:04): Yeah. This is right around the time, so this would be around 1998, 1997. We shipped Office '97 in early '97, I think. I got there in summer of '96. I think we were done with '97 maybe even late that year. I don't remember.

David McKinnis (00:24:21): Yeah, usually we tried to number them ahead of time. Because it's like car models, no one wants to buy Office '97 in '97. They want to buy it ahead of time.

Rob Collie (00:24:29): Office '97 definitely wasn't done until it got gloomy in Seattle. It was definitely during one of the dark months where we released it. Right about now was when Windows itself was coming under a tremendous amount of criticism. This was when Gartner, I don't know if this was the first time Gartner had appeared on the landscape or not. But Gartner, first time I'd ever heard of them was with their very damning analysis of the total cost of ownership of Windows. Saying, "It costs a lot more to own Windows than the license, and it's many, many many times the license cost."

Rob Collie (00:25:03): Most of this came down to, not all of it, but a lot of it came down to software install on corporate machines, and what a pain that was. So in a daring move, a member of the Darwin team went over and talked to Windows, and convinced Windows that, not only should Darwin be the Office installer. It should go and become the official installer for all software on Windows. Solved this problem, right?

Rob Collie (00:25:29): That deal was signed before I became a program manager on the Darwin project. So the joke I was going to make earlier was, you're looking around to move up as a development manager, as you're leaving Word. Some jobs are more open than others, right? The setup job is more open.

Rob Collie (00:25:48): Same thing with me, when I wanted to leave testing. From the moment I arrived at Microsoft, I felt like I had been miscast as a tester. Because a really good test engineer needs to believe the product is broken. You need to be a pessimist. I'm fundamentally an optimist. So to come in every day, and project myself into a curmudgeonly, "It's broken," space, was just really difficult. It was really tough for me. Being on the creation side was much more my thing.

Rob Collie (00:26:11): So looking around, not many people wanted to hire me as a program manager. "You just got here. You're a tester, who wants to be a PM. There's thousands of people like you here, Rob." But then the universe conspired to make the Darwin program management job available. No one on the planet wanted that job, except for me.

David McKinnis (00:26:39): Well I can't say how you were hired, but I-

Rob Collie (00:26:40): Oh, I know.

David McKinnis (00:26:41): I don't know if that's entirely fair-

Rob Collie (00:26:42): I know how I was hired.

David McKinnis (00:26:43): Okay. Okay.

Rob Collie (00:26:49): There was a point at which I'm just, I'm sitting in these kangaroo court interviews, to get into that job. There's a point at which I'm looking at them, thinking to myself, "We both know there's no other candidates for this job."

David McKinnis (00:27:04): Well you did a good job, so-

Rob Collie (00:27:06): I appreciate that.

David McKinnis (00:27:07): Glad you were interested and available.

Thomas LaRock (00:27:08): So it sounds like at that time, the idea of an unintended install was just this foreign concept. Because you guys, you're installing software. So it's interactive-

Rob Collie (00:27:18): Oh yeah. Yeah. Oh yeah.

Thomas LaRock (00:27:19): "Click here, this, that." The idea of just, "Here you go," that's one of the things you do with database servers. It's like, "Here you go." The fact that SQL Server, I didn't work at Microsoft like you guys did, so I'm not trying to be critical. But I can recall, at the end of installing SQL Server, they're like, "Hey, here's a script for everything that we just did, in case you'd ever want to just install this software, with the same exact configuration, and intended install."

Rob Collie (00:27:49): Oh, yeah. Like a macro recorder for your installation. That's kind of cool.

Thomas LaRock (00:27:53): So that's the only life I've known. Only because I never did desktop support. I never had to install Word or Office for 500 people. But if I had to do 500 database servers, we had that.

David McKinnis (00:28:04): Yeah, you have to think back. I was trying to look at this. I have a copy of the Mac Word box on my wall, that was signed by everyone who worked on Mac Word 6. Certainly, four years before this, you're shipping out software on 3.5 inch floppy disks. How can you do an unintended install? You need to swap the disks every few. So unintended install caught up to us. We didn't really see the market.

Rob Collie (00:28:30): LANs, just having a network at a company, is taken for granted today. But a lot of people I knew in '96, they didn't have anything like that at work. You could share a video with other people at Microsoft, by using the internal LAN. Putting it on a network share somewhere. Most people didn't have that ability at their offices. Just even having a network at work. File servers, or whatever. That wave crested and broke pretty quickly. We were playing catch up a little bit.

Rob Collie (00:28:58): I would say that the net effect of Darwin, Windows Installer, in Office, looking back, that was the more manageable part of the job. It was still hard. We still had a lot of bugs, a lot of challenges, and a lot of problems. But that relationship with Windows, I don't know if anyone even cares about this anymore. But it used to be that software, you really wanted to have the logo on your box, for software that said, "Designed for Microsoft Windows." Which was really saying, "Blessed by Microsoft, as this product is good."

Rob Collie (00:29:27): So there was this logo program. In order to get the logo, you had to jump through all these hoops. I'm talking about a non-Microsoft company. If Borland or whatever wanted to claim, "Designed for Microsoft Windows." They had to conform to all these requirements. Using MSI, using the Windows Installer, was going to be, and it is, it was, part of that logo program. We were going to force the entire world to use this thing. Was that an entanglement?

David McKinnis (00:29:58): Well then you need tools to create these. A whole bunch of things that that leads to.

Rob Collie (00:30:03): Yeah, so multiple times a week, I would walk across campus, and sit in a room full of relatively heavyweight Windows people. People from the Windows team. I was significantly junior. Even the second most junior person in that room was a lot more seasoned than I. Who else from Office wants to go over there? No one. No one wants to go over there. Send the youngster.

David McKinnis (00:30:28): Then we can disavow whatever you say-

Rob Collie (00:30:30): Exactly, right? Yeah.

David McKinnis (00:30:31): If you agree to anything, we can say, "No, Rob doesn't-"

Rob Collie (00:30:33): Plausible deniability. Yeah, yeah.

David McKinnis (00:30:34): "Rob doesn't know what he's talking about."

Rob Collie (00:30:35): That's right. Expendable.

David McKinnis (00:30:39): Yeah.

Rob Collie (00:30:39): The thing is, I was so blissfully clueless about the whole thing, in some ways. If I had known the gravity of everything, I don't know if I could have functioned. I held it at a distance, emotionally.

Rob Collie (00:30:50): So we got into some really, really nasty situations. This is where things really get interesting. Dave was talking about that reference count thing, the ref count thing, earlier. You install something, and then you remove it. Should you remove the file? Well, how do you know if someone else is using it or not? Okay, well now we were going to be the standard for that entire process, for every file ever installed on Windows.

Rob Collie (00:31:12): It's one thing if you're just Office. That's bad enough. But now you're going to be the standard for everything. That means that every other product at Microsoft also has to play ball. They also are going to need to follow the logo. So I end up in all these meetings. This was sort of a John DeLowe scheme, that I was sent to represent to the world. How we were going to distribute software within Microsoft, in these re-dis packs.

Rob Collie (00:31:34): You need the C runtime? You're not going to just figure out how to do it yourself. You're going to go grab this re-distributable pack off of some internal server, and merge that into your MSI file. Then that's going to go out and install things. But in order for our system to work, the Achilles heel of our system, was that a re-dis pack could only have one file in it. There was some really limiting thing.

David McKinnis (00:31:57): I actually seem to remember, it was the versioning issue, right? This has been solved more recently. But it takes a lot of effort to solve this properly. I seem to remember the issue was, so let's say that you have a program that's been tested, and runs against version 3.0 of a re-distributable library. Then version 3.5 has been released. We know that there are cases where 3.5, programs that ran with 3.0 can't run with 3.5, but you're installing a new app that has to have 3.5. It doesn't run with 3.0. This creates all sorts of complexity.

David McKinnis (00:32:29): The biggest issue was that people would screw this up. The obvious solution then is, you just name the library something different. You don't name it foo.dll, you name it foo35.dll. But human nature being what it is, no, people name it foo.dll. Now you have two incompatible versions. You have to somehow account for that, and you have to make some trade-offs, is what it comes down to.

Rob Collie (00:32:54): Looking back, this is fundamentally a problem that can't be solved retroactively by an installation program. When the operating system is already set up to have this problem. But we tried. I went out, naively representing our solution as the solution, and everyone just needs to get onboard. Again, this is a job that really no one wants to do. Other people were busy doing things, so send the youngster.

Rob Collie (00:33:15): Sooner or later, there's a tremendous blow back. The entire company, it's really the Visual Studio team, they really got upset with us, and started complaining, all the way up to Jim Allchin, who ran Windows. Then the dreaded email comes back. Jim Allchin replies all to this email, and says, "It's time for me to take a look at this technology. I need to do a full technical review of this." Oh my god.

Rob Collie (00:33:40): Everyone knows the import of this. This is gravity. If this engine needs to be rebuilt, if Darwin needs to be rebuilt in some way, Office 2000 is not going to ship on time. That's a terrible, terrible problem.

Thomas LaRock (00:33:53): That's a problem.

Rob Collie (00:33:53): Right? Billions of dollars of revenue lost. Can't happen. If the Windows team says it's not good enough. There's this nerd showdown scheduled. It draws out, like the line in Braveheart, "This must be a fashionable fight. It's drawn all the finest people." We had, heavies from both organizations were there. From the Office side, we had Steven Sinofsky. We had John DeVon. From the Windows side, we had Allchin, and all of his lieutenants, including Rob Short, and all these folks.

Rob Collie (00:34:24): We sit down in this giant conference room, to have this half day review of this technology. About the time it's about to start, the Visual Studio people, who have grown to hate us, walk in the door with handouts of why we suck. These just get passed around the table. So this is where Dave and I, our memories aren't so good. We weren't sure if the Visual Studio guys were coming or not. But we definitely didn't expect handouts about why we suck. It was very well put together, this handout. This was an ambush.

David McKinnis (00:34:57): It pointed out, we definitely had real problems-

Rob Collie (00:35:00): It was a problem, for sure.

David McKinnis (00:35:00): I see this as, it's one of those, are you going to let the perfect kill the good? But the problem is even worse. There's no perfect solution. It's not like we could have worked on it for another, well maybe, I guess if we waited until the global assembly cache. But that came out 10 years later, at least.

Rob Collie (00:35:18): There were taskforce established at this time, that again, I got sent to. "Run once, run forever," was the mantra. So Tom, in the middle of this meeting, where we're being ambushed, this could go really poorly. Allchin was very temperamental. If he just came down hard on us, that could have been a major problem, that immediately would have had to have been escalated up to Bill. So this ambush lands.

Rob Collie (00:35:40): It's this well laid trap. Then that happens to be the morning where the feds sued, to block the release of Windows 98, because of the Internet Explorer bundling antitrust case. So they're blocking the release of Win 98. All the executives in the room, especially the Windows executives, their cell phones keep going off. Most people didn't have cell phones back then, but the executives did.

David McKinnis (00:36:04): Were they pagers, or cell phones? I don't remember.

Rob Collie (00:36:08): That's a good question. I don't remember. The floor was handed to the Visual Studio people, to walk us through the handout, and explain why we sucked so bad. But at the moment that they are given the stage to talk, all of the executives are gone. All of them are out of the room. They're now, it's a lampoon of the whole thing. It's just us and them again, looking at each other, and them explaining back to us why we suck. There's just no one else in the room. It was so funny.

David McKinnis (00:36:35): Except maybe some of the Office higher-ups, who were on our side.

Rob Collie (00:36:38): Yeah, who were going to go to the mat with us. Sinofsky is in the room, asking things like, "Who even uninstalls Office?" So eventually, the executives come back in. The ambush is over. They've finished their presentation. No one's heard it. The Windows executives are like, "Listen kids, children, please. You just have to go figure this out on your own. We've got much bigger problems than your little kerfuffle, that has apparently become a storm in your world. It's nothing. Get away from us."

Rob Collie (00:37:12): So then David went off, one on one, with Robert Whaby, who was a bigshot on the Visual Studio side. I remember the issue just kind of going away. The two of you were assigned to figuring it out. I don't really remember anything ever-

David McKinnis (00:37:27): I honestly don't remember the details. I'm sure we made a couple of small changes to help placate them. But I think in the end, it was just one of these cases, where you couldn't solve it. The system just didn't help. It wasn't like other systems at the time, where you could pull specific libraries for specific programs, and they were running in their own address space.

Rob Collie (00:37:49): Yeah. In hindsight, the whole problem wasn't our fault. We did, and I'm not saying you did this, David. I did this with John and Mike Year. We over-represented ourselves as a solution to the problem. That I think is what largely created the whole crisis in the first place. It wasn't that we were hiding anything.

Rob Collie (00:38:08): It's just, this is the danger of being a computer scientist in the real world. You're enamored of these logical structures that you've created. You think they're magical. No, they're not. They've got to go out and deal with real world dirty reality, and how well they succeed is going to be tested by real world dirty reality. We just underplayed that. If we'd played it more modestly from the beginning, I think the whole thing would have been avoided.

David McKinnis (00:38:30): I would accept that we probably did not clearly point out the edge cases that we didn't handle properly. Because yes, I think we probably were thinking of the mostly perfect world, that scientists, computer programmers, like to live in.

Rob Collie (00:38:43): Frictionless world, yeah.

Thomas LaRock (00:38:44): So if I could relate one install story, and we'll just say, "I heard about this manager at a company once." Because you had mentioned the filename. Like, "Why would you go rename the file name?", and all that. This manager, they have a system that was running Sybase. Sorry, the artist formerly known as Sybase. They were looking at using SQL Server 2000. So he decided that he would install SQL Server, because they wanted to test it.

Thomas LaRock (00:39:17): So he goes over to the production server, puts in the SQL Server disc to install the software. You two might know immediately, what happens is that the installation overwrites a whole bunch of Sybase DLLs, because they're named the exact same. So when you mentioned that file name thing, that's the first thing I thought of. "What could go wrong? I'll just install this. It has nothing to do with these other applications." Except Microsoft decided to just reuse the same DLLs.

Rob Collie (00:39:50): Well the real question, Tom, is, was the resulting SQL Server install usable?

Thomas LaRock (00:39:55): I didn't hear the rest of the story, because this was now a production down scenario. So I'm pretty sure they had to reinstall everything, and-

Rob Collie (00:40:10): Oh, yeah. Yeah. Pave the machine, yeah. Yeah.

Thomas LaRock (00:40:13): Pave the machine. Recover from a backup. Get the accounting team back in here.

David McKinnis (00:40:21): I don't often test software. But when I do, I test on production hardware.

Rob Collie (00:40:26): Yeah.

Thomas LaRock (00:40:27): I think the manager lost access to the data center at that point.

Rob Collie (00:40:30): Do you remember the dark day, David, when a Darwin bug blue-screened every developer and Office computer all at once? We deleted some system DLL, and not one of those machines was usable afterwards. Every single developer in Office had to pave.

David McKinnis (00:40:46): Had to be paved.

Rob Collie (00:40:47): It was a full day of productivity lost. That's a tremendous loss. Again, at the time, I put myself in this emotional cocoon. Actually, I think probably I had always had that emotional cocoon. I had to come out of it over time. I look back in hindsight at some of those things that happened back then, and go, "Oh my god, I had no idea how bad that was." I just thought it was funny at the time. It's not funny. It is funny, but it's not.

Thomas LaRock (00:41:13): It's all right.

David McKinnis (00:41:15): Now that it's a couple decades away.

Rob Collie (00:41:19): Yeah, the statute of limitations has expired.

David McKinnis (00:41:19): Yes, exactly.

Rob Collie (00:41:19): We talk about setup a lot. Lessons learned, right? One lesson we just mentioned was, don't over-represent yourself as a solution to a problem. Be realistic about what you bring to the table. Exceeding lower expectations is better than failing at higher ones.

Rob Collie (00:41:33): Another thing that I learned, that was really important for me, in that I think I would have never learned this if I hadn't been on that project, is the nature of certain kinds of problems that only get noticed when you fail. Setup is exactly that. No one's going to call you a hero when the software installs properly. It doesn't matter how hard it was. You are never going to be celebrated for that. But you are going to be noticed when you blue screen every computer in the org.

Rob Collie (00:42:00): So many jobs end up being like that. Most of IT is like that. "Oh, you want a medal for keeping the lights on? No. Oh, the lights went out? Now we're going to fire you." Tom, I'm sure you relate to that as a DBA. Some jobs are just defense.

Thomas LaRock (00:42:18): You mean you're always on defense?

Rob Collie (00:42:19): That's right, that's right, yeah.

Thomas LaRock (00:42:20): Yeah.

Rob Collie (00:42:21): Mm-hmm.

Thomas LaRock (00:42:21): Okay. Yeah, you're always behind. You're always playing defense, yeah.

Rob Collie (00:42:24): Yeah, you're never a hero, but you're often a goat, and not the good kind of goat. That's helped me empathize with a lot of where I find myself today. Again, out in the real world, where we have friction, and we have wind resistance. Any others that jump out at you from back then?

David McKinnis (00:42:42): Honestly, I wish we had better testing. The whole idea of test driven development. I think we did actually a tiny bit of that. We definitely didn't do as much as we could have. But that's more of where the-

Rob Collie (00:42:53): An industry. Industry change, yeah.

David McKinnis (00:42:55): ... was at the time.

Thomas LaRock (00:42:55): Yeah.

David McKinnis (00:42:56): Personally though, I don't feel that when I was significantly younger, that I was as good a manager as I would be later. But again, that's just the test of time. I've probably learned more between now and then, how I'd lead a team like that.

Rob Collie (00:43:09): It's like the old Mitch Hedberg joke. "Here's a picture of me when I was younger. Every picture of you is a picture of you when you were younger." When I was less skilled than I am today? Yeah, that's what the past is, until you start reaching a certain age, sadly. Then you can start looking back at your peak, I suppose.

Rob Collie (00:43:24): But being a relatively early Microsoft-y, you had options, right? You had options ... Hold up, look at that. I didn't even mean to say options. Yes, you had stock options-

David McKinnis (00:43:33): Little play on words there.

Rob Collie (00:43:34): But what I really meant was, you had options with what to do with your career. When I showed up in 1996, there were test engineers who had been there for seven years. It's a less lucrative job to be a test engineer at that time, at Microsoft, than to be a developer, like David was. Program manager was somewhere in between the two.

Rob Collie (00:43:51): Test engineers who had been there for seven years were retiring. Because the stock had done some amazing things. It had doubled each of my first three years there, so the price had 8X-ed in the first three years that I was there. I didn't have enough options, and they weren't vested enough, and all that kind of stuff, so I never really realized a whole lot of money from stock options. Because the bottom fell out shortly thereafter. But David was able to get out while the getting was good, and moved back east. What year?

David McKinnis (00:44:18): After we shipped Office 2000, one of the things, a strand through my life has been teaching. I've found that I really enjoy working with younger developers. Helping them come onto the team. We had this internal technical education organization at Microsoft. But the problem was that no one in that organization had actually shipped a Microsoft product at the time I started interviewing.

David McKinnis (00:44:39): I actually joined at the same time a friend of mine, named David Johnson, joined for the money team. But one of my thoughts was, A, I like teaching. I saw the need for getting new developers up to speed, and I felt like my time as a developer, we were working really hard, all the time. Which was fun, but can lead to burnout.

David McKinnis (00:45:00): I think also, I mentioned my ability as a manager. I think I was feeling unsupported. I don't know that I would have used these words, but I didn't feel I had the right skills as a manager. In hindsight, I think it was that Microsoft didn't have a good method at the time, of taking people who were good developers, and turning them into good managers.

David McKinnis (00:45:18): So I made a move over to the technical education group, which was wonderful for the two years I was there. We had some new tools come into play, which I helped educated the teams on how to use. I did have a Welcome To Microsoft course, that I really enjoyed teaching. Then my wife had graduated from medical school at the University of Washington, and we were really fortunate for her to do her residency training also at UW.

David McKinnis (00:45:41): The first thing that happened was, after our daughter was born, my wife got an offer to come back to work. We had always joked that I would put her through medical school by working at Microsoft. Then I would retire, and she would support me in the manner to which I was accustomed. It came down to this question of, are we actually going to follow through on that? We could certainly afford wonderful childcare for our daughter, and we could both work.

David McKinnis (00:46:08): But at that point, I'd been training people for about two years. I was feeling a little rusty, as far as, my whole point had been that I was a person who had shipped software. I was getting a little out of touch with the development group. I felt it was either time for me to hop back in and work on product, or perhaps just leave.

David McKinnis (00:46:28): I decided I would leave, and take care of our daughter, which was a wonderful time. My wife, she had been off for about a year, after the birth of our daughter. So she could put that medical training to use. Then we did a lot of, our families are both east coast based. We were doing a lot of flying back and forth to the east coast.

David McKinnis (00:46:47): At one point, we looked at each other, and realized that we were fortunate that her job and my job was something that we could do in any place in the country. It wasn't like, I have friends who are university professors. You go where the job is then. So we decided to move back to the east coast, to be closer to our families.

David McKinnis (00:47:07): Yes it's true, Microsoft, I feel very fortunate that I was at Microsoft when I was. It has given me some flexibility to pursue what I wanted to pursue, rather than what I had to pursue. So I've done a number of things since then. You talked about IT. I did actually help form an IT company, that was modestly successful. We actually just finished selling it off. Because I think one of the problems you point out the, Rob, is that when everything's working, it's like, "Why are we paying you so much?" Then when things break, people get all upset.

David McKinnis (00:47:41): Then the other thing I've done is, I do a lot of reading of software now, not so much writing of software, with a group that does tech due diligence, by the name of Tech DNA. One of the guys I worked with at Microsoft, actually he worked for me at the time, Mike Kelly, formed. I feel very fortunate. I've worked from home for about 15 years now. In fact, my kids teased me when the pandemic hit. They said, "Dad, your life hasn't changed. You're still in your office in the basement."

Rob Collie (00:48:07): That's Dave McKinnis's music.

David McKinnis (00:48:10): But I have three kids running around the house.

Rob Collie (00:48:13): Yeah, it's a little different now. You've done some teaching as well, right? You've been involved with high school kids?

David McKinnis (00:48:19): I do. I feel like I have an itch to teach, so I have done a number of ways. Most seriously, I've volunteered through a group that was actually started by a Microsoft-y, called Teals, that brings computer science education to high schools around the country. With people either in person, or remotely. To try to increase both the number of high school students who can have a computer science course, but also diversity and inclusion of underrepresented groups in computer science. So I did that remotely about five or six years ago to a school in Minnesota.

David McKinnis (00:48:52): Then actually, we're going to have just one kid at home next year. So I have re-upped, because there's a school in the community next to us, that is involved in the Teals program. So I'm going to be a lab assistant in person in the fall.

Rob Collie (00:49:05): Neat. What does the curriculum in a program like that look like? What are you involved with teaching?

David McKinnis (00:49:09): The program has several ... The really neat thing I like about it is, one of its goals is to expand the capacity to teach high school computer science. You're always paired with a classroom teacher. The goal is for them to be able to teach it independently in a couple of years.

David McKinnis (00:49:27): There are various courses that teachers will offer. The first time I did it, it was just an intro to CS course. We used Scratch at the time, and it was just a semester long course. The course I'll be helping with next fall is AP Computer Science A, which will be in Java.

David McKinnis (00:49:43): One of the things I really did like, is that you are paired with a high school teacher. It's not just, have the volunteers come in, and then you have to keep turning that crank, and getting more and more volunteers if you want to expand. In fact, the teacher that I volunteered with in Minnesota, really by the start of the fourth semester, he was pretty much doing it on his own. The other volunteer and I, who were both remote, listened in. Then when they did labs, and needed more individual attention, and to get around to each of the students, we would hop in and help. So yeah, it's a wide range. Some do AP computer science principles as well.

Rob Collie (00:50:17): That's interesting. I never really thought about that. In terms of the AP courses in high school, advanced placement courses, AP physics today is probably exactly the same as AP physics was when I was in high school 30 years ago. But AP computer science isn't like that. Java didn't exist when I was in high school. If it were conceived from scratch today, it would probably be Python, would be what they would teach.

Rob Collie (00:50:42): We're probably, one of these days, the college board or whatever is just going to decide, by stroke of a pen, "We're moving to Python." I know that some of the fundamentals obviously are the same. But basically, every last piece of tangible course work needs to be changed. That doesn't happen in the world of chemistry and physics. Or calculus, or-

David McKinnis (00:51:02): Generally not, yeah. Even though you didn't have the luxury of quite the same Microsoft stock, you've had quite an interesting path, from Microsoft to-

Rob Collie (00:51:12): Yeah, that I have.

David McKinnis (00:51:13): To post Microsoft.

Rob Collie (00:51:15): A little bumpier maybe. Leaving Microsoft without the big nest egg. The situation in our family was pretty difficult. Let's not talk about the difficult things, shall we? I'm just going to roll with, David, as if it's 100% the same. It's funny. By the time I was actually really good at my job at Microsoft, which I would say was definitely long after we worked together. By that point, I had reached the point where I didn't enjoy it. All the perspective that I needed in order to become good at it, was also perspective that led me to become disenchanted with the whole thing.

Rob Collie (00:51:51): We had a recent podcast guest on, who was a Microsoft MVP in the community, for the data platform, and describes herself as a technophobe. I'm like, "Yeah, that's me. Yeah, that's me." I'm a technophobe, who ended up at Microsoft, with a CS degree. I'm not into it in the same way that most of my fellow Micro-softies were. I'm into the humanity of things. I love what the tools can do for humans. There's plenty of opportunity for that.

Rob Collie (00:52:22): But it just narrows the set of technologies that I tend to care about. This role, what I'm doing now, is a much better fit for my personality, and my talents frankly, than what I was doing at Microsoft.

Rob Collie (00:52:35): Dave and I probably spent, I can't even imagine how many hours. Easily hundreds of hours, if not 1000 plus, just in triage of bugs. Bring up the list of all of the bugs that were filed today, on Darwin, and preemptively decide whether we're going to engage with them or not. This is a common thing that is done in software. Every fix you make introduces 0.6 new bugs. So you've got to be really careful which bugs you fix. Especially as you're getting close to some sort of release date.

Rob Collie (00:53:12): But we went into this mode 18 months in advance of release. It was a very unusual step that we did. Because so many bugs were getting fixed, in sometimes grandiose ways. So we had to institute a policy, that every single bug needed to go through the star chamber. Me, Dave, and Mark Hopwood. I've complained about how boring and interminable Spanish class was in high school. I spent more time in triage for Darwin than I did in Spanish class. Just, ugh. Just over and over and over again.

David McKinnis (00:53:50): We're glad we're out of that.

Rob Collie (00:53:51): Oh yes, that's right.

David McKinnis (00:53:52): But one thing I've been impressed with you, and I didn't realize quite how big P3 adaptive was until I started looking around, after we had chatted. You've taken this idea, and really grown with it. Grown a fairly, from what I understand, a pretty healthy company. Both in terms of what it does, but also just the culture itself.

Rob Collie (00:54:13): It is the single best thing I've ever been associated with, is this company. It's so good on so many fronts for our clients. They get miracles, essentially. Whereas the traditional industry setup, to just extract tremendous amounts of cash from them, and deliver a little bit. Our people get meaningful employment that values them properly. Whereas there really wasn't a space that valued these new citizen developer types, is what Microsoft calls them. If you're a really good citizen developer, you're more valuable in the BI space than a whole team of siloed experts from the past.

Rob Collie (00:54:50): The world's not really set up to value these people properly. Our business model is, A, deliver the best thing possible to the client. Lowest efficiency, zero friction, results for the client. While simultaneously being the best place to work, for this new breed. It just feels like this old fashioned thing. Like, "That's what companies are supposed to do." They're supposed to provide a service to the world, and employment, and compensation for their people. It's an honest trade, in terms of an exchange.

Rob Collie (00:55:16): It's hard to feel that when you're working for software, because you're just so far removed. Certainly, Microsoft has advanced the status of humanity tremendously. You can't argue with that. It's a productivity tools company. It's so hard to feel it. This is much more visceral. You know all the people.

Rob Collie (00:55:35): There's an episode of this podcast with Kellan Danielson, who's our president and COO. The short version is that I had this vision of what this company should be, and I think that vision's been accurate. I saw what it could do, and I believed in it. I give myself credit. I called that one right. What I didn't call right was that I also thought, because I was the one with the vision, that I was the one who could build the operational organism that would do it.

Rob Collie (00:55:56): I'm not wired that way, it turns out. I learned the hard way that that's not my skillset. Kellan is this operational genius, who was also willing to believe in this crazy idea. Most operational experts, you can imagine what the average operational expert looks like. Stay in the line. The pencils are straight on their desk. You wouldn't expect them to be in the land of innovation. It's like asking, "What's more responsible for the area of a rectangle? The length or the width?" When you ask, who's more important to our company? Kellan is one of those. He's either the length or the width, I'm not sure which, but it doesn't matter. It is something. Tremendous amount of complexity has been solved. It was much harder than I expected it to be.

David McKinnis (00:56:39): I actually did listen to that podcast. I listened to several-

Rob Collie (00:56:41): Did you really? Oh wow.

David McKinnis (00:56:42): Yeah. Yes, I listened to several-

Rob Collie (00:56:43): How flattering.

David McKinnis (00:56:44): Once you said I was going to be on, I-

Rob Collie (00:56:45): He did his homework.

David McKinnis (00:56:46): It was a good interview. It seems to me that one of the keys then, is that you've brought your personal values into the business, and made sure that the business reflects those. Rather than having separate personal, run the business in a different way.

Rob Collie (00:57:01): My wife often comes to me for social advice. The best person she can go to, to ask for social advice, is me. Poor her, right? Very often, I find that when my wife is wondering what to say to someone, my advice is always, "Just tell them the truth." If you tell this other person exactly what's going on in your head, they're going to completely understand you. If you try to shortcut it, some people are good at that I suppose. But it's a form of deception in a way.

Rob Collie (00:57:24): I've never cracked the code, being able to shortcut that, and be effective. Again, I know that there's lots of people in the world who have. Think of your average person who is really, really, really good at high value sales. They're a maestro of this. It's not even a moral or ethical thing for me. I just can't. I don't even have the choice to act that way.

Rob Collie (00:57:43): It's leaning into your own strengths. That transparent approach. Let's see how far we can get with that. Turns out, pretty far. I appreciate you saying that, Dave. That is how we've operated, and Kellan has absolutely been on board for that. Everyone in our company has been on board for that. To go and build something that does reflect your own personal values, and have it be a success, I can't imagine anything more satisfying.

David McKinnis (00:58:08): That's very cool, and that you followed your values, and what's important to you. In both business and personal life. I think that's key.

Rob Collie (00:58:15): Yeah, it's been an adventure. When you're doing due diligence, so let's be clear for the listeners, right? If Company A is interested in buying Company B, and Company B is a software company, or has some sort of tech assets, they often will bring in a team of experts, to evaluate Company B's work. What's the level of quality? Is this real, or is it an apparition? What does that analysis look like? Are you just pouring through source code for days on end? How does that work?

David McKinnis (00:58:43): Yeah, so usually, one of the things I like about the work I do with Tech DNA is, the projects are pretty limited. They're two to three weeks in length. Because this is generally the time between, a letter of intent has been signed, and the deal gets finalized. The other neat thing is that I'm working with a lot of really smart people. In some ways it reminds me of Microsoft. To do that quickly and reliably, you need people who have done software projects.

David McKinnis (00:59:07): So we have folks that have similar experience as mine, as far as being involved with the software industry, and some who are even clearly much more experienced, and have worked on even cooler things than I've worked on. We look both at the source code, and then we also interview the team. We ask lots of questions. Find out how they go about developing the software. Do they actually keep track of bugs? What's your disaster recovery scenario? Backing up.

David McKinnis (00:59:35): Part of me is always amazed, just how often these guys are living on the edge, as far as doing the right things. A lot of these are startups. So you have to make the decision. Are we going to be in multiple Amazon regions? Or are we going to add another feature, to get more customers? The decision is almost always, get more customers. Because then we'll get bought out by large company Z, who will then care about being in multiple Amazon regions, and give us double our development staff, so we can actually do that properly.

David McKinnis (01:00:08): It's really interesting, because I've discovered all sorts of areas, that you wouldn't have thought that there would be software written specifically for that niche. Maybe if you think about it a little more, it's obvious there's going to be software to do that.

David McKinnis (01:00:20): For example, you walk into these places that have big digital displays. Those displays, there's software that runs that. There's software that runs your car wash. There's a set of various companies that do that. One of the benefits that we get to be involved with, because we're an independent group from whoever is doing the acquisition, is we do get to look actually at the source code. Because we're independent, if the deal were to fail, they can't actually come to terms. We just stop work, and then there's no IP taint. Because someone in the company actually saw how it all worked.

David McKinnis (01:00:55): We're often called in by lawyers, that also help make sure that our reports are clean, and there's nothing in it that shouldn't be. It's really interesting, finding out about these different businesses. Having to put your spidey-sense on, and you start asking questions. You get some weird answers, and you're like, "Okay." Just being able to use that from your experience, finding out various holes.

Rob Collie (01:01:21): I know a couple people from my days at Microsoft, who ended up at Microsoft, because they were brought in as plants, to pretend like they were working in the office when Microsoft came to visit, to make their team look bigger than it was. When Microsoft bought the company, they were like, "Well, we need all your people, including that director of test that we met that day." So they're committed to the charade, all the way to the end.

David McKinnis (01:01:46): Oh god.

Rob Collie (01:01:49): Now they've been at Microsoft 20 years, and very successful and everything, because they were smart.

David McKinnis (01:01:55): Oh, that's great.

Rob Collie (01:01:55): They were smart people.

David McKinnis (01:01:55): As I said, it's short projects. One of the things I feel very fortunate about is, as I do get to work from home, my wife has a much more rigid schedule, and so I've been fortunate to be the one who can go pick the kids up from school. I often take some time off between projects, in order to put everything back in order, and catch up on the things that didn't happen during the project. Because these projects are often pretty intense. As I said, the acquiring company is trying to figure out what to do. You want to do as thorough a review as you can, in the limited time you have.

Rob Collie (01:02:25): One thing I want to say, and one thing I want to ask you. You're talking about how you get all this exposure to so many thing that you wouldn't know about, as a result of this. These short projects, and you're bouncing around between industries. You're getting this tour of the world of tech, that you would never get in any one job. Kellan likes to say that, in six months working here, you tend to get a similar type of crash course exposure.

Rob Collie (01:02:47): I felt the same way, having left Microsoft. The same thing with me. Do you know how many companies there are, that make $1 billion in revenue a year, or more, that you've never heard of? Thousands. It's unbelievable how many companies are out there. The question I have for you is, what does the work product of one of these due diligence exercises look like? Is it like in the coliseum at the end? The emperor stands up with his thumb, and goes thumbs up or thumbs down? How binary is it?

David McKinnis (01:03:17): So our job is to identify risk. Honestly, in my experience, I don't know of a time when we have done something that has scuttled any deal. But mostly what we're doing is just identifying risk. Generally, provide both a final report out, where we meet with the client, and go over, "Here are our top concerns. Here are our top takeaways."

David McKinnis (01:03:39): Sometimes the client has specific questions about either key personnel, they want to make sure they know about and keep. Or they do sometimes ask, "We're going to try to integrate this with this dashboard. Our timeline is X." The answer is almost, "Yeah, no. That's too aggressive." Then we have a very detailed written report.

David McKinnis (01:03:58): But generally what we're trying to do is just point out risk. One of the reasons that we're helpful, is that companies are going to come to some agreement. A couple months down the road, they're going to take on this company. So what we do is, we help jumpstart the process of, the two development teams meet. They can just go over and say, "Okay. Well we know these are the top things we need to work on."

David McKinnis (01:04:19): The other thing we help do with some clients is, there's some remediation that needs to happen. We'll point out and say, "The company you're acquiring has this data store house, that they don't have consent from the customers for." That's the sort of thing the acquiring company will say, "That's going to get deleted before we ..." Obviously when you acquire the entire company, you take on any liabilities from that company. Some of the time, it's remediation steps.

David McKinnis (01:04:47): The other thing we see a lot, and we're generally, there's a group called the NCC Group out of the UK, that we are usually working with, that does most of the security review. If they're not involved, then we're including security review as part of our work. A company that's small, and no one really notices. If they have a few out of date libraries in their cloud platform, it's not a big deal. But if a well known company in Silicon Valley puts out a press release, saying, "We just acquired Joe's Software Company," suddenly all of the hackers, something like that needs to get remediated before they get acquired.

David McKinnis (01:05:23): So it's generally about risk, more than up and down. Sometimes we have talked to the client about, "I'm not sure there's anything here. We think that this needs to get re-written, and that's out of date. Your plans were to scuttle this, so are you sure?" By the time we're called in, there's usually enough understanding of what they're getting, that the deal's probably going to go through. As long as the target doesn't think they're worth 20 times what the client thinks they're worth, and they just can't come to any agreement on valuation.

Rob Collie (01:05:57): Right, and you probably wouldn't be privy to this. I would expect that your report does end up ultimately influencing the final transaction price in certain ways.

David McKinnis (01:06:06): Probably. But yeah, we're totally reviewing technical. We don't try to say how much anything is worth.

Rob Collie (01:06:13): Sure. No, it reminds me of a home inspection. The intent has been declared. We've signed something. But it's pending inspection. Usually the deal still goes through, but there's some negotiation over certain items. How much you're going to give me for this. Also, it gives the new homeowner a one up on all of the things that they need to be concerned about when they move in. "This is going to need to be repaired in a few years," or whatever. It's a very, very close parallel. Not a sexy parallel, but maybe illustrative.

David McKinnis (01:06:43): Part of the thing is that we get to talk to the builder-

Rob Collie (01:06:45): That's right, yeah.

David McKinnis (01:06:46): ... and understand, "Did you put in all the beams?"

Thomas LaRock (01:06:48): So that being said, Rob, in real estate right now, you can forego the home inspection. So the question is, in this transaction, are you allowed to ... I can't imagine somebody was finding, unless a company like Tech DNA walked in and did their due diligence.

David McKinnis (01:07:06): Oh, there are plenty of deals that don't get any due diligence.

Thomas LaRock (01:07:10): That's crazy to me, that one tech company would buy another, and be like, "Yeah, we're good. Whatever you've got, we'll take it."

David McKinnis (01:07:16): Sometimes I think you're correct, it is crazy. There are other times when it's simply, "We don't really care. We want that business," for whatever reason.

Thomas LaRock (01:07:24): I can't imagine taking on that unknown risk.

David McKinnis (01:07:26): Yeah, we have been called in at times, the deal is going to happen. It's finalized, everything's been agreed to. But as essentially hired contractors, that we can look through these programs a lot faster than the client's staff can get to all of them. It just depends on the business need. Sometimes the businesses feel like, "We want the business that they have." It does in some ways slow things down, and feathers can get ruffled.

Rob Collie (01:07:55): Yeah. There's ego involved. Of course.

David McKinnis (01:07:57): If it doesn't matter, you might as well just do the deal.

Rob Collie (01:08:00): Yeah. Like Facebook was going to buy Instagram no matter what. Instagram's numbers, they were what they were. Take it or leave it.

Rob Collie (01:08:14): What just happened there folks, was a redacted story about one such acquisition made by Microsoft, where the quality of the technology and the people acquired was very, very substandard, but Microsoft's going to buy it anyway. However, we've edited it out to protect the innocent, and the not so innocent.

David McKinnis (01:08:31): Sounds good. Sounds good.

Thomas LaRock (01:08:33): Yeah, that's what you go with.

Rob Collie (01:08:35): So are you still running? You still run?

David McKinnis (01:08:37): I do, I do. I've also taken up basketball as a big thing. Even though I'm not particularly skilled at it, and my experience isn't that great.

Rob Collie (01:08:43): I can't imagine playing basketball. I go to Orange Theory Fitness three to five times a week. The wear and tear on my tendons and ligaments is next level. I can't imagine what a game of basketball would do to me. I do play roller hockey.

David McKinnis (01:08:57): I see your posts about roller hockey. That seems to be actually quite a bit more dangerous than a game of basketball.

Rob Collie (01:09:02): Just playing a basketball game without calamity would still take a tremendous toll on my body. A hockey game without calamity does not, because it's very laminar. It's very smooth. It's actually a low impact sport, until you run into someone. Or run into the wall or something. I try not to do that. It's not a checking league. If it was a checking league, then it would be just a disaster. We try to not hit each other.

Rob Collie (01:09:26): I'm involved in a lot of collisions, because I'm less skilled. I'm the biggest collision risk. Basketball, I'm impressed. Actually jumping and landing repeatedly, that's something else. So you're still running, you're still playing basketball. Tom, Dave and I used to be part of a five person relay team.

David McKinnis (01:09:42): Which was a lot of fun.

Rob Collie (01:09:42): It was a lot of fun.

David McKinnis (01:09:44): I still have, those tee shirts are still in my closet.

Thomas LaRock (01:09:46): What relay are you talking about?

Rob Collie (01:09:49): At the time, it was called Falls to Gasworks. Way out by Snoqualmie Falls-

Thomas LaRock (01:09:55): Oh, I get it. Okay. Yeah, okay.

Rob Collie (01:09:55): Then it ended at Gasworks Park, in Seattle. It was 52 miles. But it's split across five people. I always took the shortest legs. Dave would usually be the one running the eight mile legs.

David McKinnis (01:10:07): We didn't get Roger to run those?

Rob Collie (01:10:09): We got Roger to run some of them. We also had Richard. We had a ringer.

David McKinnis (01:10:11): Oh, that's right. I forgot about Richard.

Rob Collie (01:10:12): College track, and cross country athlete, Richard Moreno.

Thomas LaRock (01:10:16): There you go.

Rob Collie (01:10:16): We were very often in the absolute lead after the first leg. We'd have Richard lead us off, and Richard would come running into the first station to hand off. Our second person would be gone over the horizon, before the next person ever even showed up. Then we'd finish middle of the pack. But for that first brief leg, we were always in the lead. What would you say you're doing as a mile pace these days, Dave?

David McKinnis (01:10:40): I'm probably in the 8:30. I do try to run a couple half marathons a year. Although I might only get one of them in this year, because of travel plans, and other things.

Rob Collie (01:10:53): There was a brief moment in time where I could run about as well as Dave. It was right when I got to Microsoft. It was when I was still super, super young and lean. Many pant waist sizes ago.

David McKinnis (01:11:04): I managed about four years ago to do an eight minute mile, just under eight minute mile, half marathon. That was my-

Rob Collie (01:11:13): That's amazing.

David McKinnis (01:11:14): That was my peak. I've wanted to try to replicate that, but I have not put in the training, and I'm getting older.

Rob Collie (01:11:21): My wife and I are pretty good skaters. We'll go rollerblading in the summers. I wear the GPS monitor when we do it. It's calling out to me after every mile, what our pace was. We'll do an eight mile circuit, and there's one mile at the end that's nice and flat. That's the only time where we even get into the neighborhood of the pace that the Olympians run the 26 mile marathon. It's unbelievable.

Rob Collie (01:11:51): We're four minutes and change for that last mile, while we're on skates. We're on skates. We're passing bicyclists. Wind resistance matters. Whichever one of you is in the front is working harder. It's wearing us out. We're not able to sustain it for more than that final sprint home. That's the time when we're moving at about the pace that these human beings run for 26 miles.

David McKinnis (01:12:18): It's pretty impressive.

Thomas LaRock (01:12:20): Yeah, but the thing is, when your body gets to a certain point, so they're doing five minute miles for two hours. There's a few things about that. But at some point, your body, doing an extra mile just doesn't matter. It's a weird thing. You just train. You're like, "Yeah, I can run for 40 minutes at a great pace." They're like, "Okay, so today, why don't you do 45?" "Yeah, I'll do 45." It just is-

Rob Collie (01:12:45): You can put me on the starting line against one of those people, and give me my rollerblades, with the 90 millimeter wheels, and clean bearings, and perfectly low friction. Even if it's a flat course. If it's a hilly course, they're really going to beat me. Climbing in skates is actually harder than climbing running.

Rob Collie (01:13:03): But let's say it's a flat course, 26 miles. They are going to smoke me. I have the ability to stand up and breathe, and take a rest, and still be moving, and they don't. It's just unbelievable. They'd just be flying away from me the whole time.

Thomas LaRock (01:13:18): I've heard sometimes that they're kind of resting while running. They slow down a little bit, catch their breath, while they're still running.

Rob Collie (01:13:27): I guess. I guess they have no momentum maintenance like I've got.

Thomas LaRock (01:13:30): Their bodies are perfect cardiovascular machines. That's all you need to know.

David McKinnis (01:13:36): A lot of practice. I don't know if either of you watch any of, the world championships were in Eugene, Oregon this year. I caught a little bit of it on TV. It sounds like to some degree, for those types of races, like the 5000 and 10,000, you've just got to be willing to hurt a lot, in order to do what those guys are doing.

Thomas LaRock (01:13:57): Yeah.

Rob Collie (01:13:57): The pain tolerance is a big thing, yeah. 26 miles at a five minute pace. Yeah.

David McKinnis (01:14:03): I've done two marathons in my life, and it was quite a while ago. Before the turn of the century.

Rob Collie (01:14:07): Have you seen those giant treadmills, where they'll let you simulate running a five minute mile? You get on there, and you run. Sooner or later, you just get thrown off the back of it.

David McKinnis (01:14:17): Yeah, no. I haven't seen this.

Thomas LaRock (01:14:17): You just try to keep on-

Rob Collie (01:14:19): Yeah, there's a lot of videos. They've got similar things for swimming. A swimming wind tunnel, where the water is being jetted at you. You dive in, and you try to swim it, and you just get washed against the back wall. Those were fun to look up. Well David, this podcast, among many things, it's an excuse to carve out two hours of our day and chat. It was really good to see you again.

David McKinnis (01:14:39): Yeah, good to see you.

Rob Collie (01:14:40): I really appreciate all the patience and tutelage that you offered me over the years that we worked together. There was a lot going wrong with me, and you could have been a lot harsher. My experience working with you was a very, very, very positive thing for my career arc. So much appreciated, sir.

David McKinnis (01:14:56): Great, thanks.

Speaker 3 (01:14:56): Thanks for listening to The Raw Data by P3 Adaptive podcast. Let the experts at P3 Adaptive help your business. Just go to P3 Adaptive.com. Have a data day.