LOS ANGELES — When you’re deploying a business-critical application to the cloud, it’s nice to not need the “war room” you’ve assembled to troubleshoot Day 1 problems. When BOK Financial, a financial services company that’s been moving apps to the cloud over the last three years, was launching its largest application on the cloud, its engineers supported it with a “war room type situation, monitoring everything” according to BOK’s Andrew Rau. “After the first day, the system just scaled like it was supposed to … and they're like, ‘OK, I guess we don't need this anymore.’” In this On the Road episode of The New Stack’s Makers podcast, Rau, BOK’s vice president and manager, cloud services, offered a case study about his organization’s cloud journey over the past four years, and the role HashiCorp’sVault and Cloud Platform played in it. Rau spoke to Heather Joslyn, features editor of The New Stack, about the challenges of moving a very traditional organization in a highly regulated industry to the cloud while maintaining tight security and resilience. This episode of Makers, recorded in October at HashiConf in Los Angeles, was and sponsored by HashiCorp.
LOS ANGELES — When you’re deploying a business-critical application to the cloud, it’s nice to not need the “war room” you’ve assembled to troubleshoot Day 1 problems.
When BOK Financial, a financial services company that’s been moving apps to the cloud over the last three years, was launching its largest application on the cloud, its engineers supported it with a “war room type situation, monitoring everything” according to BOK’s Andrew Rau.
“After the first day, the system just scaled like it was supposed to … and they're like, ‘OK, I guess we don't need this anymore.’”
In this On the Road episode of The New Stack’s Makers podcast, Rau, BOK’s vice president and manager, cloud services, offered a case study about his organization’s cloud journey over the past four years, and the role HashiCorp’s Vault and Cloud Platform played in it.
Rau spoke to Heather Joslyn, features editor of The New Stack, about the challenges of moving a very traditional organization in a highly regulated industry to the cloud while maintaining tight security and resilience.
This episode of Makers, recorded in October at HashiConf in Los Angeles, was and sponsored by HashiCorp.
In late 2019, Rau said, BOK Financial deployed one small application to the cloud, an initial step on its digital transformation journey. It’s been building out its cloud infrastructure ever since, and soon ran into the limits of each cloud provider’s native tooling.
“Where we struggled was we didn't want to deploy and manage our clouds in different ways,” he said. “We didn't want our cloud engineers to know just one cloud provider, and their technology and their tech stack. So that's when we really started looking at how else can we do this. And that's when Terraform was a great option for us.”
In 2020, BOK Financial began using HashCorp’s open source Terraform to automate the creation of cloud infrastructure. “We made a conscious effort to really focus on automation,” Rau said. “We didn't want to do things manually, which is really that traditional data center, how we've done things for decades.
In tandem with adopting Terraform, BOK Financial’s teams began using GitOps processes for CI/CD. But doing “everything as code,” as Rau put it, “required a lot of upskilling for some of our staff, because they've never done version control or automation capabilities. So in addition to learning Terraform, and these other cloud concepts, they had to learn all of that.”
The challenge, though, has been worth it: “It's really empowered us to move a lot faster, and give our application teams the ability to deploy at their pace, versus waiting on other teams.”
It took about a year, Rau said, to get BOK Financial’s developers comfortable using Terraform, largely because many were new to version control procedures and strategies.
Because the company works in a highly regulated industry, handling customers’ financial data, security is of utmost importance.
“We had users credentials for our clouds, and we had them separated out based on the type of deployment that [developers] were doing,” said Rau.
“But it wasn't easy for us to rotate those credentials on a frequent basis. And so we really felt the need that we want to make these short, limited tokens, no more than an hour for that deployment. And so that's where we looked at Vault.”
HashiCorp’s secrets storage and management tool proved an easy add-on with Terraform. “That's really given us the ability to have effectively no credentials — long-lived credentials — out there,” Rau said. “And secure our environment even more.” And because BOK’s teams don’t want to manage Vault and its complexities themselves, it has opted for HashiCorp Cloud Platform to manage it.
For other organizations on a cloud native journey, Rau recommended taking time to do things right. “We went back to rework some things periodically, because we learned something too late,” he said.
Also, he advised, keep stakeholders in the loop: “You need to stay in front of the communication with business partners, IT leaders, that it's going to take longer to set this up. But once you do, it's incredible.”
Check out the podcast to learn more about BOK Financial's cloud native transformation.
Colleen Coll 0:10
Welcome to this special edition of the new stack makers on the road. We're here in beautiful Los Angeles had hacky calm logo, discussions with technologists, giving you their expertise and insights to help you with your everyday work. Infrastructure enables innovation hashey Corp provides consistent workflows to provision, secure, connect, and run any infrastructure for any application.
Heather Joslyn 0:41
Hello, and welcome to an on the road episode of the new stack makers Podcast. I'm Heather Jocelyn Features Editor of the new stack and we're coming to you from the scene of hashey. conf global here in sunny downtown Los Angeles. Today's episode is brought to you by hashey Corp, makers of TerraForm vault console and other solutions for consistent cloud operations. Today we're going to be sharing with you a case study from Bok financial a financial services company, Bok financial created an as code operating model in a tightly regulated industry while strengthening security, increasing resiliency and driving efficiencies. To do this, they've used hashey, corpse TerraForm, Vault and hashey Corp cloud platform, and we'll learn how that's worked out for them and also find out about any challenges they may have encountered along the way and how they've resolved them. Our guest today who will tell us the story is Andrew RAO of Bok financial welcome, Andrew.
Andrew Rau 1:36
Thank you, Heather, thank you for having me.
Heather Joslyn 1:38
Can you tell us a little bit about your role at Bok financial? Sure.
Andrew Rau 1:40
So I've been with the company for about three and a half years now started out as their cloud architect and then moved into a leadership role, you know, manage managing our cloud services DevStack ops strategy as a company,
Heather Joslyn 1:54
as we start to start, let's set the scene a little bit UK is a financial services company, which means it's in a heavily regulated industry. What are some of the challenges inherent in that for a company that wants to move to cloud being in a regulated industry like that?
Andrew Rau 2:09
Yeah, I mean, the regulations and regulators really are focused on managing risk, right, we want to, we want to ensure that, you know, customers can get their money, that it's accurate. Those are, those are the core things, right. And so it's all about the controls that you have in place to ensure that that it's accurate. A lot of times, you know, in our in a traditional manner. And how we were over three years ago was separation of duties. You had a team that built, you had a team that deployed, you had a team that handled security network, so forth. And then from a cloud perspective, there's a lot of different ways that we can pursue that.
Heather Joslyn 2:46
So when did the okay begin to move to the cloud? So we
Andrew Rau 2:49
started playing with it and 2018 and then really started focusing on in 2019, roughly, you know, the time that I joined, we really started to gain a lot of momentum. Our first application, it was it was a small application doesn't get much usage, it was kind of pilot was in late 2019. But then we continued our build out since then, it just seemed
Heather Joslyn 3:12
like when you move to the cloud, a lot of best practices seem to be start small start with something right, right. Don't go all in Right, right off the bat. So I was at your presentation this morning, which was terrific in mentioned that the company started using TerraForm. And two years ago, 2020. What was the issue you were trying to solve with that? And how did how did you arrive at that is,
Andrew Rau 3:33
it really came down to you know, we started out with one cloud provider, we use a lot of their native tooling, it worked great. But then we had a need to deploy and build out our second cloud provider for different capabilities. And their tooling was was not the same. And so really, where we struggled was we didn't want to deploy and manage our clouds in different ways. We didn't want, you know, cloud engineer to know, just one cloud provider and their technology and their tech stack. So that's when we really started looking at how else can we do this. And that's when TerraForm was was a great option for us.
Heather Joslyn 4:12
So and then get ops as part of what you do. And how does that all work together?
Andrew Rau 4:18
Yeah, we made a conscious effort to really focus on automation, right? We didn't want to do things manually, which is really that traditional data center, how we've done things for decades. So we wanted to do everything as code, it required a lot of upskilling for some of our staff, because they've never done version control or automation capabilities. So in addition to learning, you know, TerraForm, and these other cloud concepts, they had to learn all of that, but it's really empowered us to move a lot faster and give our application teams the ability to deploy at their pace versus waiting on other teams.
Heather Joslyn 4:56
Yeah. And you mentioned in the discussion this morning that people were about, you know, because they have to wait for their request to be approved, and so on. But in a regulated industry, things don't always, there are obstacles to
Andrew Rau 5:08
there's, there's a lot of checks and balances in a regulated industry. And the beauty of Git Ops is because it's in a version control system, you have all of that history of who made it, who approved it, when it ran, you know, all of that is built into it. And so what we ended up doing was essentially taking away access for every user to our cloud environment. So really, most of the staff has View Only they can't make changes, has to go through this process, which gives us that full audit tracking capability.
Heather Joslyn 5:40
I just want to go back to you mentioned the upscaling that was needed. There's a you know, saying that technology problems are often really people problems or, or people challenges, how did people react to, you know, they've got to upskill, we've got to, we got to learn this new stuff.
Andrew Rau 5:53
We started off with, you know, with people that were wanting to learn cloud and wanting to work in this new model, and built that team. And, you know, it took us probably a good year to get everyone comfortable in it. You know, because they're not just learning. Like I said, they're not just learning TerraForm, most of them have never done application coding. And so when you start looking at infrastructures, code, TerraForm, version control, they have to learn how to do branching strategies, and even merge request approvals, looking at differences and things like that. And so that was all new, that they had to learn in addition to everything else.
Heather Joslyn 6:30
Yeah. And get his eye having learned get it's it's his own. Its own hill to climb.
Andrew Rau 6:36
There's a lot Yes, yeah. But it's very powerful once you know it, and seems very easy. So
Heather Joslyn 6:42
yeah, so obviously, you know, security and secrets management is especially important when you're dealing with people's financial data and so on. How did you arrive at you added vault to that your stack? What was the process there?
Andrew Rau 6:56
Yeah, I mean, we essentially had, you know, we had users, right, or deployment, users credentials for our clouds, and we had them separated out based on the type of deployment that they were doing. So there was still separation. But, you know, it wasn't easy for us to rotate those credentials on a frequent basis. And so we really felt the need that we want to make these short lift tokens, no more than an hour for that deployment. And so that's where we looked at Vault, it was a great solution, it was an easy add on, you know, with TerraForm. And it just made sense. So, you know, that's really given us the ability to have effectively no credentials, long lived credentials out there, and secure our environment and even more, that sounds
Heather Joslyn 7:44
great. If there been any challenges in dealing with TerraForm and vaults and getting other than the upscaling mentioned.
Andrew Rau 7:50
Yeah, you know, the biggest challenge we really found was with our TerraForm modules, you know, we we went that route so that we could ensure that things were deployed securely, right and effectively, like we didn't want to deploy an application in a static environment. So we want to force our application teams in a way to build resilient application, right, automatic scaling and things like that. The biggest challenge we had was, and this kind of goes back to coding practices was you start having these modules dependent on other modules? And so what is that design pattern look like? We started out having these modules separate them, we put them all in one. But our deployment, we ran into a problem where we were running on containers, and the container filled up with memory. So we had to basically now start peeling back and saying, you know, it doesn't make sense to put it all on one that's basically like a monolithic application, right? So there's been a lot of learning along the way, just like anything else you're doing with technology, right? You you implement, you learn, and then you want to go back and redo what you already did at the beginning. Did you go
Heather Joslyn 9:01
through like hashey Corp to get the support for this or the Learning Support? Or we did?
Andrew Rau 9:05
Yeah, we went through different training programs that had hashey Corp on there. We did work really closely with our hashey Corp team. Even recently, I think, beginning of the year, we had them come on site, we did a Git ops day where we educated more, we did a hands on lab. And so that was really geared towards our security team to learn these concepts and the basics and actually deploy something through TerraForm at the end of that day. So that was another great session to just bring some awareness.
Heather Joslyn 9:38
Yeah. Even using hashey Corp Cloud Platform. Can you tell us a little bit about
Andrew Rau 9:42
Yeah, so you know, hashey Corp. We have TerraForm cloud. And then we did hashey Corp cloud platform with with vault we didn't want to deploy and run vault ourselves. So we went that approach, so that we could have connection to our cloud environment and basically then manage it. It's been great. It's one of those things, it's like Kubernetes is powerful, but could be complex. And so supporting it went out really want to do. Well let them manage it. So that's been great. And then TerraForm cloud through hashey Corp cloud platform has been phenomenal with just the features that they bring out the run tasks, the variable sets, all of that.
Heather Joslyn 10:22
So the big news out of this conference is the availability of hashey Corp boundary, you know, setting a spinning up a zero trust network. I mean, what do you think that kind of impact will that will have on your industry, it's something
Andrew Rau 10:34
that we're going to explore I mean, zero trust, when when you look at a regulated industry, again, managing risk, you don't want to have a big data center where it's just wide open, right, we do have segments that we we've isolated out. So this is something that we're going to definitely look at, because it could help us become more secure and reduce our risk overall. So I think it's, you know, a very promising product that we want to explore further,
Heather Joslyn 11:02
if you could give advice to maybe a parent who's also in this industry about this sort of journey. What sort of like, what do you know, now that you wish you'd known maybe three years ago, two years ago about this? What advice might you give,
Andrew Rau 11:14
definitely take the time to do it, right. I mean, we went back and reworked some things periodically, because we learned something too late. And you need to stay in front of the communication with business partners, IT leaders that it's going to take longer to set this up. But once you do, it's, it's incredible. I mean, we we can spin up as same environment in a day, right, complete, full stack everything in the cloud in less than a day through this process. That's, that's incredible. And we've had to do that.
Heather Joslyn 11:46
So it's had a good impact on your business. And
Andrew Rau 11:49
absolutely, when we launched our largest application, you know, I'll say they were ready for the launch week, to be in a room, you know, war room type situation and monitoring everything. After the first day, the system just scaled like it was supposed to did what it was supposed to. And they're like, Okay, I guess we don't need this anymore. You know. So that complete shift from what they're used to in the past to this new model and new way that we designed and deployed everything. So big impact.
Heather Joslyn 12:19
Terrific. Well, thank you very much for joining me today. Andrew, that's a wrap on our discussion about the cloud journey at Bok financial and its use of hashey corpse solutions in that journey. And again, thank you, Andrew, for joining us. And we think this episode sponsor hashey Corp for sponsoring today's podcast. I'm Heather Jocelyn for the new snack. And this has been an on the road episode of the new stack makers. And thank you for joining us, we'll see you next time.
Colleen Coll 12:50
Infrastructure enables innovation. hashey Corp provides consistent workflows to provision, secure, connect and run any infrastructure for any application.
Alex Williams 13:03
Thanks for listening. If you liked the show, please rate and review us on Apple podcast Spotify, or wherever you get your podcasts. That's one of the best ways you can help us grow this community and we really appreciate your feedback. You can find the full video version of this episode on YouTube. Search for the new stack and don't forget to subscribe so you never miss any new videos. Thanks for joining us and see you soon.
Transcribed by https://otter.ai