Expert Talk: Software Security

Intro

John Steven: Hello, good morning, good afternoon, good evening. My name is John Steven, and I'm the Founding Principal at Aedify Security. I've spent the last two and a half decades helping define the software security space and helping organizations build their security practices, as well as helping vendors build some of the tools you may have used to do security in your codebases. I'd like to introduce you to Jim Manico who I've known for the better part of two decades. 

We met each to fight about what the correct answer to cross-site scripting was, and then further spent the day fighting about whether or not developers could fix their own code, or if we as security people had to do it. And 20 years later, a lot has changed in terms of our perspective on that and that's what I'm here to talk about. 

Jim Manico is in his own right what I like to call a security magpie, somebody who has circled security spaces and developer spaces and picked up all of the good bits off of that chaotic beach of good and bad advice!  And he is the CEO and founder of Manicode, where he delivers, in my opinion, some of the best training out there in terms of developer-centric, building security across languages and spaces. Jim Manico, do you wanna say anything else about yourself?

Jim Manico: I'll say something about you, John. John was one of the principals, internal CTO at Cigital. It was one of the biggest application security firms in our industry. And John did a lot of work behind the scenes empowering people around him, working with some of the biggest companies in the world with application security. I'm just a fan of your career, John, and the number of people you've mentored in the last 20 years. So, it's a pleasure to be here with you. And I'm looking forward to your questions!

John Steven: Likewise, in reverse, which probably won't come through as we fight like cats and dogs.

Jim Manico: Bring it on, John!

How has the role of developers evolved when it comes to security?

John Steven: So, I wanna start with kind of what...you know, as I began to speak in the industry, in communities like the OWASP community, which was mostly security practitioners. I was a big proponent of you having to understand the code better. If you don't understand the dev frameworks, then you're not going to provide decent remediation advice and you may be flat out wrong. You recall that in the beginning, I spent a lot of time trying to explain to the OWASP community that fixing cross-site scripting wasn't an input validation problem, but it was about output encoding and those things, and those were controversial. Now, the way I look at this space, I see something of a sea change over the last three years and it's undeniable now.

Developers in many organizations have said, "We're done waiting for security to solve these problems. Your tools were too error-prone and too painful to use and your advice was too hand-wavy. We got this now. We're gonna build the security controls, we're going to define the security activities, we're going to score and figure out whether to release. And you can record what we do, and report it however you want."  Security has changed that much. You've been there the whole time, we both have a lot of gray hair. What do you think? What do you think about that trend? And what is the developers' role in an organization's security program? How are they to think about this sea change and participate in it constructively?

Jim Manico: The word that keeps coming to mind when you ask this question, John, is DevOps. I think that's the convergence of what we did good in security, removing what we did bad insecurity, and getting developers more involved to build that capability. And what I mean by that is, there are a few things I think are critical. Number one is blocking me at check-in from finishing that check-in, from merging it completely until I finish security testing. And DevOps-style security testing is meant to be very lightweight. It runs in under 30 seconds. It's meant to be low fidelity, multiple low fidelity tools that run fast, but give me at least a little assurance at this critical juncture of checking in code.

Recommended talk: Secure by Design – the Architect's Guide to Security Design Principles • Eoin Woods • GOTO 2016

Those who are mature at DevOps also build pre-commit hooks, where I have to run certain tools locally before I'm even allowed to commit. And very often, this toolchain is built by developers - for developers. I think that's the best when I'm using Jenkins or GitHub automation or something else to maintain and automate my testing environment led by devs, assisted by security. So now, I can build, scan, and deploy code on my own and get security out of the way. Now, there's fragility with this, there are problems with this. But that kind of thinking where I want to do security, I want to have reasonable rates and get security people out of the way, I think that's a really big piece in how application security has matured, still gives me significant capability as a developer to get my job done, and stops me when there's a problem. That's a big piece of how things have gotten more serious in the last three years.

John Steven: So, this is security FUBU, you know, security FUBU. This is security for us and by us as developers.

Jim Manico: Exactly.

John Steven: And it's going to help me move quicker, and I understand my code, so maybe I make better decisions. But isn't this wolf watching the henhouse?

Jim Manico: A little bit. This is why I think if developers only established the pipeline, it's going to be weaker than the security influencing it. Now, here's where developers like to do pipeline work. "I'll automate the scanners you need me to run, and I'll automate the live deployment, I'll automate all these pieces." Do you know what developers don't like, though? They don't like pre-commit hooks, "This is forcing me to run certain security tools before I'm allowed to check-in." And developers hate being blocked at check-in, they hate being stopped from completing their merge. And so, usually, if devs build a pipeline by themselves, pre-commit hooks, and commit locking at security testing is gone. And when...

John Steven: I'm a little bit confused because now I think you're talking out of both sides of your mouth!

Jim Manico: Absolutely.

John Steven: So, I wanna get the commit, but I'm gonna get mad if I get the commit. And so, there's subtlety. I think the point you're making here is that to get this correct, culturally, you need to understand the way people get flows to work, how they operate inside of the repository workflows, and what the specific phases of branch strategy are. This branch means it's gonna be automatically promoted by orchestration software in a production list. So that one I get. Whereas my private branches, I want to be able to play within my playground, and I'm going to be highly aggravated if you want to beat my progress.

So, I mean, I think that's something. You and I were a little bit of a standout inside the security practitioner space early on because we would always introduce ourselves with the caveat, whether it was a badge of honor, or a badge of, you know, shame that we're developers in this space, and we feel like visitors. So what is the security person's...how do they understand this nuance? How do they get enough literacy to participate in those DevOps cultures and practices and hang? Because, honestly, I feel like that’s one of the biggest challenges in the cultural imperative of DevOps that said, you know, "You two brothers need to play together." But they don't necessarily know how to play together well.

Jim Manico: Well, I've seen the most success with developers and traditional application security working together to accomplish these security and Dev goals. It's strange, like a lot of the original DevOps thinkers would say, "Oh, you don't want to have DevOps as a service." And I look back ten years and I'm like, "Well, that's exactly what we need, in my opinion." DevOps as a service is outstanding because I don't want 50 teams all building their pipeline from scratch. That's a huge waste. I like having a service-based DevOps team that includes developers and a security person who collaborate. Usually, it's a security person who's leading the DevOp pipeline management and creation team, but this team is a team of developers who need to work on the pipeline. By the way, programming and scripting the pipeline is more difficult than writing the application itself. There's so much subtle nuance in this. And it's not even a lot of code, it's just really specific code.

John Steven: Highly parametric, highly meta, yeah. So, I almost...you can tell the maturity of a security initiative. And, if you look at the study, which I've helped author, which is a survey of what people are doing, there are sort of factions of survey response.

Jim Manico: Yes. Absolutely.

What can a single developer do to improve security?

John Steven: And the faction of survey respondents that says, "We implement security through our platform team, which does repo as a service and pipeline as a service." You get fundamentally different physics. To your point,  I think as a security executive, I'm not sure how you report effectively across 300, 3,000 application teams without some kind of standardization on that reporting, which isn't gonna come out of your security tools as easily unless you've standardized the way your pipelines can, you know, execute and consume the results from them.

So, I think that there's a challenge here, and I think this challenge is kind of a proxy of the challenge we had two decades ago. Two decades ago, security people didn't understand enough about code to understand the root cause and the fix and developers didn't understand the security issues. Now, the tooling has become more developer-centric. The security people have understood it's about the code a bit more. But there's still that language barrier, okay? And so, how does development and security...what are things that they can do tomorrow to get if they don't have a platform team or if they're just an individual contributor to make that?  Maybe a team, maybe their peer, more effective and get this turned on or get a prototype and get progress? What is the thing you look for as, "Okay, they're gonna learn how to share the toys in the pipeline"?

Jim Manico: That's a hard question. Let me see if I can...your question is, I'm an alone developer without a product team. There are a couple of AppSec guys maybe trying to get me to scan stuff. What can we do as a single contributor to help security? I think a good first step is;  a lot of developers are under the influence of assessment techniques that they can skip.

John Steven: Lovely. That's good.

Jim Manico: It's just that as a developer, when I'm confronted with non-blocking application security assessment technology, I gotta use it. And, first of all, as a developer, I shouldn't have that choice. I should have a pipeline that's set up for me, that forces me to go through security gates. But guess what, no one likes gates. So very often, as a developer, I have assessment technology as a service available to me. So the first thing I should do is use that service. I should suck it up and run the scan.

John Steven: So, pick up the toys off the floor and start putting them into my automation pipeline and saying, "This is my telemetry, this is what I see. What do you guys think?"

Jim Manico: And a lot of developers don't want to do that. They think it's a waste of their time. And I want to convince...and it is a big hit of time to start to mature in tune, even the most basic application security tools. So, number one is, to work with your security team, go through the assessment technology, it's brutal, but do it. I think the second thing is to make recommendations on tooling. Like, I don't want to use Fortify anymore John, not a DevOps lifecycle, I want to use Semgrep. John, I don't want to use tools that take me 20 minutes to run, I want tools that run fast for the initial pipeline, and take additional tools like Fortify and put those post deployments so they can run all day if it needs to.

Recommended talk: AppSec: From the OWASP Top Ten(s) to the OWASP ASVS • Jim Manico • GOTO 2019

And guess what, that's hard for a developer. Let me rewind from that. Let me go back to your question. What can I do as a developer? I would start in two areas. I would start with a simple vulnerability that is easy to identify and fix like SQL injection and go through all of my SQL queries and look at that. That's tangible, it's basic, assuming I'm even doing SQL. But that's one vulnerability I can thoroughly investigate and fix and understand my ability to find, fix, and deploy a single bug. And that's an easy one to fix. On the other end, I want to get my authentication code dialed in. If I'm using OpenID connect, I want to make sure I got that connector done right. If I'm doing my manual password storage, I want to review, like, the app. Like, one of the projects I work on John, is the application security verification standard.

It's a little messy, don't get me wrong. But I would look at the ASBs requirements, or NIST 863 special application, and look at my authentication layer. That's the harder place to start. SQL injection is a simpler place to start. But I would try to address individual vulnerability categories and try to root cause them. One last thing and I'll stop. There is no good answer to this question. If I'm an lone developer with no product security team, weak AppSec, and no real support, there's a heavy limit to what I can do and be effective because AppSec is a team sport. I need resources and compatriots and expertise to do it.

How to make the right security choices in the cloud?

John Steven: This goes back to the aphorism of don't try to boil the ocean, right? So, if you can implement a freely available tool, and you can find that tool, not because you're an expert, or versed in the space, but because you Googled something that helps you with a particular problem you're concerned about or feel like is endemic, then you can campaign that problem, and you can build towards a comprehensive approach with individual tools and individual campaigns. And I think that's a great way to do it.

Switching gears a bit, we think about code, and you mentioned things like SQL injection, you mentioned ASBs, which focuses on the kind of problems that developers can introduce by coding. Ten years ago, we recognized, "Well, wait a minute, a lot of what developers are doing is downloading and implementing open source. So maybe we should look at that open source and see if it's the latest version, or if it's got malware in it, or if it's patched, or what the deal with that is?." And so we built composition analysis tools. And, you know, that kind of shifted our software security down the stack. And, you know, recently, we've started using things like Kubernetes, and then service mesh orchestration like the Istio and others.

Not only did we have open-source self-service, coding by downloading somebody else's stuff, but we have the self-service infrastructure in terms of infrastructures code For example Terraform, Ansible, and then self-service workload orchestration, Kubernetes, self-service application, you know, orchestration with Istio. All of that ends up being configuration and code. And so, in some sense, just as the security vendors have clued in and said, "Oh, my gosh, shift left, it's the code." We've been telling them this for decades, but they finally figured out that they need to be able to read, understand, and fix source code. Now, the source code is migrating to the right. And so, more and more of what we do to make applications happen in production is source code. And there are all of these different formats popping up like mushroom patches everywhere in the DevOps in that middle ground and to the operations and maintenance pieces. What does that do to what we thought about as application and software security? And how does a developer respond to that? And how does the security person respond to that?

Jim Manico: I believe we're in the sixth wave of migration when it comes to moving to cloud services John. The first wave was like, "I'm using basic Amazon infrastructure. I do a lot of the work myself, still. I'm using services." You know, first getting into the cloud, but we're at a whole new level now, where we see like in all three of the major providers, GPC, Azure, and AWS, I got full infrastructure capability scripted now. And I have to do a lot of that myself, right? I have to maintain a script or deliver my code. And there are a lot of the same problems as traditional software because I have to manage it myself.

But, John, I'm going to change your conversation. We're beginning to move into the third wave of migration for cloud services, and here's the difference. I can go to Azure right now, set up a master key, click, and I got a master key now in my hardware security module. I then go and build a data level key with Azure storage, click, and I'm done. I then encrypt the data and have it go to Azure storage while the actual encryption data key is encrypted by the key encrypting key in a HSM at Azure, will go back to my vault and click, click, click, I have an extremely mature key management lifecycle and extremely mature data-level encryption mechanism, and I'm two minutes in.

Until the cloud gives me control at that level, which is happening now, it is all coming into place, and that's gonna save us, John!  That's going to give me the ability to build a security layer that a million developers can benefit from, click, click, click. And so, we're between that second and third wave of migration. If you're stuck in the second wave where you're managing scripts, managing code, and building it all from scratch, you got the same traditional AppSec problems in a different location in your YAML file, in your infrastructure scripting file. And all the scanners can start to understand a little bit of that now. But until we move to standardized architectures, which is what unstandardized security capabilities, which is happening all over the place now, we're in the same morass problem of scanning, fixing, assessing because...

John Steven: The enabling security controls that are provided in particular around identity and encryption…

Jim Manico: Off the chart.

John Steven: ...and around like, centralized login and monitoring are really valuable. Each of these cloud service providers is doing a great job, and I don't wanna denigrate any of them, but each of them has gaps.

Jim Manico: Absolutely.

Recommended talk: Introduction to OAuth 2.0 and OpenID Connect • Philippe De Ryck • GOTO 2018

John Steven: And so what you described with Azure, in particular, is something I've experienced, which is very easy, very easy to segment least privileged compartmentalize, persistence-based keys from my data stores in Azure, maybe not as easy on something like AWS. Maybe AWS does something else better. How does a developer interface with the thousands of lines of infrastructures code or externalized cloud configuration? As a flat-file and figure out where their CSP and their use of the CSP is providing them a huge lift that they need to replicate team to team as that secure template, and where it might be a gap that reflects in an open situation in their architecture? How do they navigate that?

Jim Manico: When you say CSP, I always think of content security policy. I know you're using...

John Steven: Sorry, cloud service provider. I'm sorry.

Jim Manico: I'm sorry, I'm stuck in a web security world. So, is your question how does the developer navigate all the complexities and nuances available at a cloud service provider?

John Steven: That's a shorter way to put it. And how do they figure out what the cloud service provider has done well, that they, I mean, this is the thing, right? So, with Azure, the example you gave, the best thing a developer might be able to do is to say, "You know what? I'm gonna push all of my key management and my persistent encryption to the Azure controls, and that's going to take a ton of work off my plate and give me a really hard posture." With another cloud service provider, they may not have that same kind of ease and strength. All the cloud service providers are doing a great job across the board. But, of course, each of them does it sort of differently and there's sort of rough edges and gaps in each...they just don't match up. So, how does the developer consume what is in essence, hundreds of security controls and get themselves to a level of literacy which they can tackle?  What techniques do you suggest they use to navigate that?

Recommended talk: Serverless Security: New Risks Require New Approaches • Itay Rozenman • GOTO 2021

Jim Manico: Take my classes.

John Steven: Take your class?

Jim Manico: That's part of the answer,  I'm not saying take my class, I'm saying get educated, take a modern AWS security class if you're an AWS shop, but let's go back to Azure for a second. Even in Azure where I can do really good data cryptography with key management in a click, you can still choose to use the user-managed HSMs in Azure. You could still use their managed HSM, but you can write keys in HSM as opposed to having Azure managed keys. So, if you go Azure managed and Azure created keys, turnkey easy, you don't get a copy of the key. And that's best practice. Some companies hate that as they want to manage the keys themselves. Some companies want to manage their own dedicated HSM in the cloud. So, the point is, even when you have the secure option, click, click, I got data storage and key management. Other options are gonna be less secure because these cloud providers need one to provide flexibility. And that's the problem. There's a secure option, but every cloud provider gives me insecure options.

So how do I make the right choices architecturally and get security when the cloud is not mono, it gives me an infinite number of configuration possibilities.  And that, John, there's no easy answer for that. You need to have a solid architect, you need an Amazon architect, or be willing to take Amazon education, which hundreds of companies offer. And you got to have real modern knowledge of what the cloud providers provide at a software engineering level. And I see some of my customers where they have that expertise in-house, and somewhere they're fumbling along figuring it out as they go. And that's the problem. Last thought here. If I want good AppSec, if I want good development, I need a framework and cloud expert embedded with my team if that's what I'm doing every day. And that's...

John Steven: Yeah, let me contextualize this a bit for a second here because we started in pipelines with tools. And we said, "You know what? DevOps is helping me out because I can use these freely available open-source tools, things like Semgrep, and others I can push that into my pipeline, I can pick those for myself, I can campaign the vulnerability types that bother me, or that are prevalent, and I'm gonna make progress." We talked about code moving into infrastructures, that shift to the cloud, and we said, "Look, the strength and quantity of the controls that the cloud service providers are providing means I don't have to worry about those implementation bugs as much maybe. But I've got a design problem. I need to know if I'm solving against somebody driving a forklift into the data center and pulling the discs, or if I need to manage an escrow of the keys for my customers on an individual basis."

So I need design-level competence inside the cloud and knowledge. And I've got my pipeline, and I and my other peer developers are building sort of a pipeline as a service or repo as a service that I feel good about with those little checks on it. You talked about the OWASP community and what security practitioners have done with things like ASBs, writing an encyclopedia of checks that one needs to make. But this, I do think the CS, the space of the cloud design does change that game to your point. Terms like Threat Modeling come into play, you know.

Jim Manico: I was just about to go there.

How do you think about design-level challenges as developers?

John Steven: How do we, as developers, think about the design-level challenges? I mean, because getting certified as an Amazon security architect takes months, right? Like, this is not a small undertaking, right? So, what do I do in the next 12 months to feel good about myself and to make progress incrementally as we talked about campaigning the individual bugs with the free tools?

Jim Manico: Let me go back to the bugs and free or cheap or even expensive tools. The tools are getting good at understanding YAML file problems. Technical bugs and cloud configuration, those tools have come out. All these cloud visibility tools, even Kubernetes, there are multiple products. My buddies say hey, Jim Manico, I'm part of a Kubernetes security company that gives deep analysis into esoteric cloud configuration. So, that technical cloud scanning, it's there now. It's all there. But to your point, that doesn't help me make good decisions in assembling the architecture of my software.

And your question is, how do I do that well? Well, good luck. There's no easy answer to that, John. We talk about threat modeling, but that's not even helping me pick something good. Threat modeling and design review help me understand if the design that I chose has reasonable security. What threat modeling doesn't do is help me understand all the components available at AWS, or other components available at Azure, and how to assemble them. Threat modeling does not give me that. Threat modeling doesn't help me understand the AWS inventory. Again, I'll make my choices, and then the threat modeling design review will vet that those choices I made, in theory, were decent. So, we're back to the problem of, "I got AWS, I got Azure, I don't touch GPC, it's crap. Sorry. But I'm gonna use non-GPC Azure or AWS..."

John Steven: I think GCP is doing some really interesting things with security. I think...

Jim Manico: Yeah.

John Steven: I think that they're an interesting challenge but... Go ahead.

Jim Manico: It's unusable, though, compared to AWS. This is my opinion. AWS is built for every man...every man or woman, sorry. Azure is built for the Microsoft community and GPC is meant for Google PhDs who are real pedantic and don't care about smooth usability as AWS or Azure does. And you know what? GPC has its place. If I was a scientific community, a group of senior developers, some of their capability is outstanding, but it's unusable crap for nicer developers. It's unusable John!. PhDs, I know you love that stuff. That's your crack. I know, John. Every man and every woman, John Steven, and I will not touch GPC because it's not for everyone. It's for PhDs only.

John Steven: Your opinion is noted. So, what I liked about what Phil and others are doing at GCP is that they are trying to transition from a shared responsibility model, which I affectionately referred to as like you're screwed and it's your fault, right? And to a shared-fate model. And they're coming up with security blueprints that try to solve the problem of, "Here are the security controls that we provide, and here's how we want you to build systems using them." And so, they're doing it in a very prescriptive opinionated fashion. And think about Linux distributions, right? There are Linux distributions that are not opinionated, and there are very opinionated ones. People who have a preference. But GCP is going to be an opinionated secure cloud.

Jim Manico: I'm interested.

John Steven: The cloud is going to provide people a paved road to deliver quickly on and we'll see, but each of them has their things to love and hate about each. Anyways, let's go back to the question at hand here. So there was a lot in application security that was art, and we bemoaned writing tools and writing guidance as a community to make that art practicable by individual practitioners and developers. Do you think that designing these security systems is still an art? Now, one of the things that I like to teach when I teach threat modeling is that you have to understand the dev frameworks, and you have to understand the cloud controls and the way that orchestration works.

Jim Manico: Well said.

John Steven: Like, I have a whole day of training on those aspects.

Jim Manico: I'm with you.

John Steven: Because I think if you don't, then you don't get down to the level of detail. Now, it's unsurprising that a common complaint with my training is that it's maybe too technical. I even had a developer say, "I have to understand code to do this," which I found fascinating. You are a developer, right? So I think that's just another, you know, a testament to it. This can be challenging. I think there are things, though, that make this easier. And whether it's Azure best practices, whether it's Amazon well-architected, or whether it's GCP security blueprints, all of them are taking the stance of, "Here's the security controls on I'm programming, and here's how I think you should use them, here's what they provide."

So, you know, I think that we are at that cusp that you described, where people can take the Cloud documentation and begin to understand their security properties in a way that they haven't before. And I think that's exciting. I agree with you that the scanning technology in IC is really interesting and can help you kind of see if people are on the paved road or not. And I think that's interesting and that's why I took the time to mention what GCP is doing because whether it's, again, as your best practices, or Amazon well-architected, if you as a developer can say, "I've read their documentation, and I understand what they want me to do, or what they've threat model so I can stand on the shoulder of a giant, now I can use my freely available IC scanning tool to see if I'm on the paved road, or if I'm bushwhacking. And if I'm bushwhacking, then I may have to call the security people. But if I stay on the paved road, then I'm probably going to be able to floor it and move as fast as I want. And so, I think that using the cloud service provider's security patterns and using the freely available tools to check your configuration to see if you're on the paved road is a powerful combination.

Jim Manico: Absolutely.

Recommended talk: A Beginner's Guide to eBPF Programming with Go • Liz Rice • GOTO 2021