Paul Vixie grew up in San Francisco. He dropped out of high school in 1980. He worked on the first Internet gateways at DEC and, from there, started the Internet Software Consortium (ISC), establishing Internet protocols, particularly the Domain Name System (DNS). Today, Vixie is one of the few dozen in the technology world with the title "distinguished engineer," working at Amazon Web Services as vice president of security, where he believes he can make the Internet a more safe place. As safe as before the Internet emerged. "I am worried about how much less safe we all are in the Internet era than we were before," Vixie said in an interview at the Open Source Summit in Dublin earlier this month for The New Stack Makers podcast. "And everything is connected, and very little is understood. And so, my mission for the last 20 years has been to restore human safety to pre-internet levels. And doing that at scale is quite the challenge. It'll take me a lifetime."
Paul Vixie grew up in San Francisco. He dropped out of high school in 1980. He worked on the first Internet gateways at DEC and, from there, started the Internet Software Consortium (ISC), establishing Internet protocols, particularly the Domain Name System (DNS).
Today, Vixie is one of the few dozen in the technology world with the title "distinguished engineer," working at Amazon Web Services as vice president of security, where he believes he can make the Internet a more safe place. As safe as before the Internet emerged.
"I am worried about how much less safe we all are in the Internet era than we were before," Vixie said in an interview at the Open Source Summit in Dublin earlier this month for The New Stack Makers podcast. "And everything is connected, and very little is understood. And so, my mission for the last 20 years has been to restore human safety to pre-internet levels. And doing that at scale is quite the challenge. It'll take me a lifetime."
So why join AWS? He spent decades establishing the ISC. He started a company called Farsight, which came out of ISC. He sold Farsight in November of last year when conversations began with AWS.
Vixie thought about his mission to better restore human safety to pre-internet levels when AWS asked a question that changed the conversation and led him to his new role.
"They asked me, what is now in retrospect, an obvious question, 'AWS hosts, probably the largest share of the digital economy that you're trying to protect," Vixie said. "Don't you think you can complete your mission by working to help secure AWS?' "The answer is yes. In fact, I feel like I'm going to get more traction now that I can focus on strategy and technology and not also operate a company on the side. And so it was a very good win for me, and I hope for them."
Interviewing Vixie is such an honor. It's people like Paul who made so much possible for anyone who uses the Internet. Just think of that for a minute -- anyone who uses the Internet have people like Paul to thank.
Thanks Paul -- you are a hero to many. Here's to your next run at AWS.
Colleen Coll 0:00
So welcome to this special edition of the new stack makers on the road. We're here at the Open Source summit in Dublin, Ireland, discussions from the show floor with technologists giving you their expertise and insights to help you with your everyday work. Amazon Web Services is the world's most comprehensive and broadly adopted cloud platform, offering over 175 fully featured services from data centers globally, millions of customers trust AWS to power their infrastructure become more agile and lower costs.
Alex Williams 0:46
Hey, everyone, I'm here today with Paul Vixie, Vice President and Distinguished Engineer at Amazon Web Services. We're live from Dublin, Ireland at the Open Source Summit. Paul, thank you for joining us in today's discussion about your life and your career and how it relates to where we are with security. I'm very interested in our discussion.
Paul Vixie 1:10
Well, thank you for inviting me.
Alex Williams 1:12
You bet you bet. So your Digital Equipment Corporation, the late 80s and early 90s. In 1994, you founded the internet software consortium. And for those who don't know about it that supports the infrastructure of the universal central self organizing Internet by developing and maintaining core production quality software, product protocols and IP operations. So you develop at the ISC, there are several core Internet technologies that were developed, including bind ISC, DHCP, and Kia. So I'm curious about the transition from DC to the Internet software Consortium. In particular, how did that happen? And what were you talking about with people at the time? What were some of the conversations that you're having?
Paul Vixie 2:04
Well, I think it's, it's obvious now, when you look at what happened with digital equipment, in the next five years, there were sold to Compaq, a shadow of their former selves, they were simply moving too slow for me. And I could see that the internet was going to be a really big thing. And I knew the only way I could really get involved in making a difference and how the Internet evolved, would be to leave Dec was hard to do. When I got out back into the real world, there were this, this DNS things, this Domain Name System had come into vogue. And it was a big deal. And it was responsible for a lot of the internet's scale, but it was not mature.
Alex Williams 2:53
So a question about Dec. Why did you join Dec? Had you come out of school? And what were your hopes and aspirations that you saw change while there?
Paul Vixie 3:06
Well, okay, so the story begins in 1980, when I dropped out of high school, I apparently spent too much time playing with computers, not enough time doing my homework. So the writing was on the wall. And after eight years of sort of working in the field, as you know, late teenager, early 20s, something, I was ready for a challenge, I was ready for something that would be teaching, teaching experience or learning experience for me. And Dec was the company, they were responsible for almost all early ARPANET nodes, and almost all early Unix servers. Dec was the first company to do any of that. But it was really their customers, not the company, the company was not so much aware of the internet or Unix. But when I had a chance to join, I jumped at it.
Alex Williams 4:00
So you began to understand the Internet, I expect during that time, through the work you're doing there, but also just do your own research. So then you joined the internet. So you started the internet software Consortium? Why did you start it? What was really the genesis of that?
Paul Vixie 4:21
I'd like to say there was a vision statement. And then we got a lot of buy in and it was sort of well organized, but no, it was quite organic. While working at DEC digital equipment, I was responsible for their corporate Internet Gateway, which was a big deal even in the around 1990. And the software that I was using was not good and the protocols that we were speaking were not good. And I have gotten involved in maintaining that and contributing to that. So when I left Dec and started the internet software consortium, it was to continue that work specifically on the domain name system and And the bind implementation of the domain name system. So, you know, ultimately I think I started in the software consortium so that there could be someone other than me to own the copyright. But then it snowballed, and the Internet got big. And so we hired a lot of people and did a lot of things.
Alex Williams 5:18
That time was very special now, when you look back at it, right? And, but even at the time, you know, you you're quoted in Wikipedia, yeah, as saying. And yet, it's a lot harder to break 500 Duct Tape systems of mixed and varied heritage than to break one well designed system. monoculture is intrinsically brittle, no matter how strong the genes themselves may be. That to me is really interesting, because I'm curious to know how you view those systems of yesterday, and how they've really spread so rapidly. And what do you look back on now to think, wow, what does this reflect upon what I learned? And what does this reflect upon today and where we are headed?
Paul Vixie 6:16
Well, I think that quote in my Wikipedia page shows its age in two ways. The first was it was before scale, right? At that time, there were only technical people and technical companies connected to the internet. And it was likely that every one of them was kind of science fair projects duct tape together, trying to somehow get to a place that we could not see, inch by inch. But what is true for 500 systems is not true for 500,000 systems. And so you know, if you want to break into somebody's enterprise and steal their data, or do ransomware, attacks, whatever you need to know about their weaknesses, and the chances of them having weaknesses are much greater in a 500,000 enterprise population than they were in a 500. And so that's why the monoculture that were that I was trying to avoid, turned out to be both inevitable and not as bad as I thought, because at scale, you have to manage this stuff. You can't just have everything be duct tapes and bandits.
Alex Williams 7:29
What's an example of the monoculture that has that has evolved and emerged?
Paul Vixie 7:35
Well, certainly, it's much harder to change the Internet Protocol Suite now than it was then. So there was a time when, let's say, if TCP IP itself had a problem, like when there was a lot of congestion control work that had to be done once things got to scale, you can actually change it, you could say, All right, let's everybody speak a new protocol. And we'll all switch to that protocol. You know, next Wednesday, you can't do that. Now, the internet protocols are cast in stone at this point. They're embedded everywhere. And so there's a very long tail of systems that will continue to operate, no matter what changes we introduce. And that changes how we can think about the things we'd like to do differently. So I would mention that the IETF, Internet Engineering Task Force is very busy trying to make everything more secure. But we all know that it's going to be one or two decades before the work that we're doing now has a broad impact on sort of digital society.
Alex Williams 8:41
So what does that work that you are doing now that will have an impact in 10 to 20 years, just as much the work he did in the early 90s, has had an impact on today for particular DNS.
Paul Vixie 8:56
So I personally have not introduced any new technology, either in my last startup or so far in my first six months at AWS security. But the last things that I worked on had to do with privacy preserving data collection, that would be compatible with student privacy regulations, national privacy law, and so forth, that would collect enough information about the way the internet was being used, that we could begin to home in on the people who were abusing it. That was led to a lot of important advances in my previous company. And I think the DNS field is now very, has adopted almost everything that we did, and I think that's cool.
Alex Williams 9:43
So what were those things that you did that have been adopted?
Paul Vixie 9:47
Well, one thing we did was called DNS tap tip. And it was just a tiny piece of middleware that you would attach to your name server software, so that you could monitor the transaction flow sort of this is the question I received, this is the answer I gave this is why. And that became a sort of a standard middleware protocol by which a whole analytics industry has sprung up so that people can use their DNS transaction flow to help them detect that maybe one of their employees has been compromised, or that there's malware running loose or something like that. That was never going to happen. If the only way we could tell what the DNS server was doing was by looking at, you know, packets on the wire. So DNS tap would be my my favorite example. We also introduced the first DNS firewall least the first standardized one, so that if you wanted your DNS server for your enterprise, or your university, whatever, to answer untruthfully, about unsafe things. In other words, somebody asked for, let's say, Tell me the address of that goes with a certain name. And you know that that name is only going to be used by malware, you might prefer to just not answer that question for your customer for your internal system, just as a matter of security policy, so we introduced a way to make DNS service part of your security apparatus to deliver security through DNS service. And we made it kind of a lingua franca so that there could be a vibrant community of producers of this type of DNS threat intelligence, and consumers where it wouldn't matter what server you were running, you still had a choice of any of the vendors who were selling into that space, and that was called the RPZ. That's stupid, we should never let programmers name things. But that is also now a de facto standard across all open source name servers, and that came from us.
Alex Williams 11:46
You know, you sold you sold your company that you started this company Farsight. And it's basically about a lot of the work that you did at the at the ISC. How was to go from that various standards oriented work that you're doing to developing a company with Farsight? And what were kind of like your goals and intentions in doing that? And what did you see resulting from that when you have to use all that?
Paul Vixie 12:12
Well, I think whenever you take some intellectual property out of a nonprofit, and commercialize it, there is fear in the community, that you've somehow stolen this and that you're inappropriately monetizing something that the community helped you develop. That was not true. When I moved to Farsight, when I started Farsight, and pulled some intellectual property out of my previous company, the which by now was the internet systems Consortium. It was because we were stuck. We were delivering DNS, or excuse me delivering security through DNS, as well as we could as a nonprofit. And we were going to need funding and growth and a large customer base if we were going to solve the real problem. And, you know, to our credit, we paid for the intellectual property that we received. And that helped ISC produce more versions of bind and so forth. So everybody won from that deal.
Alex Williams 13:12
You're now that you're with AWS, what's still, what do you think is still needs working on there's probably any number of things but one of the things you said, you know, prior to getting on camera here is that you really have always to think about your focus and your energy, and how you really think about pinpointing issues that really relate to what you believe is the tough problem the problem that needs to be solved. So what are what is on your Mind
Yeah, we'll take a lifetime and more. Right. And so why join AWS then? And what are some of those challenges, you're seeing that open source plays a role in helping resolve
Paul Vixie 15:28
after I sold first add security to domain tools, November of 2021, took a few months off. And I was speaking to the AWS security team. And they said, you know, why don't you come here? And I said, Oh, but I've got this mission, I've got to get to restore human safety to pre internet levels, you know, so I don't know what I'm doing next. But it has to be that. And, you know, what, they looked at that, and they asked me what is now in retrospect, an obvious question, gee, AWS hosts, probably the largest share of the digital economy that you're trying to protect, don't you think you can complete your mission? By working to help secure AWS? The answer is yes. In fact, I feel like I'm gonna get more traction now that I'm able to focus on strategy and technology and not also operate a company right on the side. And so it was a very good win for me, and I hope for them.
Alex Williams 16:32
Great. And so on your mind, you must be thinking about, for example, projects you've already been, you've played a huge role in DNS, for example. How are you thinking about how AWS approaches Internet security? And, you know, what do you think is the role that, you know, the organization should play in influencing the work that we all do every day, because we all live in this at scale world now. And we're so dependent upon it, and that AWS plays such a big role in it. I'm curious on how you're thinking of approaching these types of really big, really big challenges and issues?
Paul Vixie 17:26
Well, part of the attraction to the challenge of working for AWS security is that it's the biggest game in town, it's, it's a big thing. So for a more medium sized company, your biggest threat is probably coming from the outside other people's networks, other people's unpatched servers, other people's sort of ancient technology that should not still be connected anything. But when you are one of the big dogs, then the biggest thing you've got to worry about us how well organized, are you what's happening with your systems, and AWS has been around long enough. And it's gotten large enough that we have built a lot of things that would in a more medium sized company, they would be bought. And it gives us a huge advantage about how we get to think about both detection and response to security events, right, everybody is under attack all the time, we know that there's really nothing that can be done to change that. What you have to change is how often your systems will be affected by those attacks. And so I think the way AWS is thinking about internet security is, you know, we are the internet in some ways, we need to make sure that we have a very clean house. And, you know, we make a lot of investments to make sure that what we do is well understood and well curated and well managed.
Alex Williams 18:58
So you talk about organization, you talking about things being well curated, you're talking about AWS as the Internet, what are those organizational map matters that you want to address, in particular?
Paul Vixie 19:19
So since I came recently, and since I was fairly well known before I came, I have a big advantage is that people all over the company. Hear that I'm working here, and they want to talk to me about their worries, their unique peeves. Their concerns, especially as it relates to open source software, where I was well known and the Internet protocols and the domain name system. And so I'm getting to hear firsthand what a lot of people are thinking around the company that may not have otherwise filtered its way into the executive ranks. So I don't, you know, I don't have a grand solution. But I do know there are some voices that I can help amplify. A lot of what you do when you're new employees, especially at the executive level, is listen, and let the company solve the problems that already knew how to solve but didn't know how to talk to itself about.
Alex Williams 20:23
So what are the themes that you're hearing from people? One of the themes that we hear a lot about, and just generally is about the software supply chain and the concern about API endpoints and their security. I though, seems you're hearing about AI? And if so, are there any particular issues that relate to those that are relevant to our discussion
Paul Vixie 20:48
with the software supply chain, at least, we have, we note to sort of excellent problems in the community. The first is, you know, programmers want to be efficient, and they don't want to reinvent something that's working pretty well, the way it is, are working well enough. It affects your time to market, it affects your maintainability, and so forth. And so we tend to say, Alright, I gotta, I gotta deliver a service and it needs a certain thing, let's look around, see, if there is a thing like that, oh, there is I'm going to import that I'm gonna add that as a dependency. And the problem is, it will have dependencies of its own. And so when you're working with the software supply chain, you have to be very careful about knowing who you're depending on, and ideally, who they're depending on. So that if some vulnerability or bug is announced somewhere, you can make sure that you patch it, and also that your upstream supply chain patches it, you know, we saw in around the December January timeframe, a very famous event. Now the log for J problem. And the problem with log for j is, you know, it was it had features that people didn't know were in there, which turned out to mix badly up with the data handling in its own call chain. But the bigger problem was, most people weren't using it. Most people were using some package that had a dependency on log for J. And so you didn't necessarily know whether you were affected. And we're going to have to change that. So just what I mentioned earlier, curation is very important, just sort of a an understanding of what you're publishing, and what you're depending on. And then if what you're depending on changes, you have to work very quickly to get a patch into your system. And that's actually one of the advantages of scale, a small company with a small development staff who understands their business workflow very well, is maybe not staffed well enough to do this, in addition to everything else. And I think that's one of the big advantages of the cloud is that you can depend on your your cloud partner, your cloud operator, to sort of say, Look, we're gonna be responsible for this part, you're gonna be responsible for that part. And, you know, let's, let's talk from time to time, you know, ultimately, security begins at home, every team has to spend a certain amount of effort thinking about how to secure their their stuff. And I guess one of the points I'd like to amplify, is that security sounds important. It sounds like it involves big locks on doors, new crypto algorithms, large keys, you know, just written guns and so forth. And that, I'm sure is involved. But if you have a great lock on your door, but you don't manage the key carefully, then it won't matter how good the lock was. And so a lot of success and security just has to do with discipline. And again, at scale, you can apply discipline over a larger defense surface.
Alex Williams 24:07
I think about the movies and like how always is like this, like the hero is able to get through the fortified, you know, armed guards and the walls and the guns and, you know, and the here always has, like, these karate skills or, you know, martial arts skills or, you know, can throw knives and eventually gets through, right, you know, to save the day, right? You can almost think of that as like hackers, right? You know, they're like, you know, they're getting through all these kind of ways. And so, I was actually reading an article on this topic where, you know, the whole idea here is that actually software security, you know, has a premise that the more open it is the better. Right, in some ways, right? Yeah. And, you know, is much better to have I've, in many respects opened in cryptography then not. Because if you do have closed cryptography, you know, only a few people are going to be able to really kind of determine how it is. And so how do you want to? Is that something that you agree with? Is it something that you that you think is important? And how do you think that affects kind of like, you know, customers at AWS and how they view this office supply chain and how AWS views it itself?
Paul Vixie 25:25
The answer has changed over time. So in the 90s, I remember sharing a stage with Mad Dog, John Hall, were one of the points that we discussed on a panel was the fact that proprietary software might have bugs that are only going to get studied by the people trying to attack them, because they're willing to sort of disassemble it and carefully look through everything. Whereas open software gets looked at by everybody. And so if you have more eyeballs looking at something, then you're probably gonna find more things sooner. And so there's a possibility that open source will have a structural advantage over proprietary software, just because it is inspectable and is therefore widely inspected. And I think that was our position going into the great expansion of the internet, and open source. But it's not always true now. So you think about the shell shock announcement from I guess, almost 10 years ago now. Turned out that the popular the almost universal shell used on Linux derived systems, including all of the little embedded ones, called bash, had a problem, or were against a little bit like log for J and had a feature that nobody knew about, and that feature mixed badly with the management of the data that you handed in. Yeah. What fascinated me about that is that it was committed two years earlier, there was a connector that have the feature, there was exactly the right code review, according to all the rules. And it was there for anybody to inspect. But it took two years. All right. So what that means is the structural advantage is not as obvious as it used to be open source software, being word inspectable only helps you if you have enough people to look at it all and there is so much open source software now. But I don't think we could possibly have enough people to look at it all. Yeah. And so if you bring this back to how AWS treats open source software, before we ship our own software, it goes through this application security group, who sort of squeezes it hard in every direction and tries to make it misbehave. So that we're not in the position of publishing something that has a vulnerability that we could have discovered not doesn't, there's no guarantee that you'll do that. But we try pretty hard. We have also got teams who will look at our dependencies and maintain relationships if we're possible with the people we're dependent on. And of course, we work with the cert organizations around the world CRTs. So that when there is a vulnerability being reported, you know, we will hear about it and we'll have a chance to manage it. You know, what this comes back to is open source software can be safer, but you have to do a lot of work. Right? So I guess the other thing I'd like to amplify is that when you get something for free, it probably means that you're going to have to do something. And in this case, what you have to do is make sure that you hear about patches, subscribe to whatever the GitHub repository, whatever so that when things go wrong with the things you've decided to import, you become aware and you're able to take action, that action may involve working over a weekend. But again, if you're getting something for free, chances are you're going to have to do something to get it. That's what you have to do with open source software.
Alex Williams 28:52
And that's really a good lesson for your customers, isn't it without software's open source software supply chain.
Paul Vixie 28:59
We would like all of our customers to pay close attention to what they depend on, make sure that they have the resources to respond quickly. And also make sure that if they themselves have a problem and they are producing software that other people are using that they can again, act quickly publish the patch, publish the vulnerability disclosure, and help the world stay a little bit safer than we would be if we had 500,000 unpatched enterprises out there each running their own duct taped version of software. A private fork for example, where somebody makes a copy of some software and modifies it and you got to make sure that if the thing you started with gets a bug report on it, that you go look and see if it if it affects your private fork. That's that's like work that you have to sign up for in order the open source software do you more good than harm?
Alex Williams 29:56
Well, thank you so much, Paul, for your time today. It's been really interesting to hear your story and how you view Internet security today. And how that has. Story has been built over many years of hard and distinguished work. So thank you for for your time.
Paul Vixie 30:17
This has been great. Thanks for sitting with me. If you liked
Alex Williams 30:19
this video, please give us a thumbs up. And if you'd like to see more videos like this, you can always subscribe to our YouTube channel. We're on all the major social media platforms. You can always find it at the new stack.io We hope to see you soon
Transcribed by https://otter.ai