An AI Primer for Application & Cloud Security

An introduction to how Artificial Intelligence (AI) can help Application & Cloud Security teams now and in the future.

By: James Chiappetta (with help and perspective from many other practitioners)

Important Disclaimers:

• The opinions stated here are the authors’ own, not necessarily those of past, current, or future employers. and the authors have not taken any incentives to mention them.
• The mentioning of books, tools, websites, and/or products/services are not formal endorsements and the authors have not taken any incentives to mention them.
• Recommendations are based on past experiences and not a reflection of future ones.
• Artificial Intelligence is a rapidly developing space and therefore any recommendations, opinions, or examples are based on present knowledge and subject to change.
• We don’t recommend putting any sensitive information into publicly accessible AI tools and you should consult your organization’s policy on the use of these tools.

Background

Today, it feels like we are in a drag race and artificial intelligence (AI) is the hot rod heading down the drag strip. The problem is that no one truly knows when the race will end, how fast we are going, or what happens if we do get to the finish line.

As we think about Cybersecurity and more specifically, Application Security (AppSec) and Cloud Security (CloudSec), there is an interesting parallel that can be drawn between these critical areas of security and AI powered level 5 fully autonomous driving. There are many variables with infrastructure and technology that need solving before we achieve safe and reliable level 5 autonomous driving for everyone. Despite the massive amount of money, effort, and technological advancements, we are still a long way away, potentially decades, from all of our cars driving us.

That said, AI is not absent and is helping us drive more safely and effectively today, than ever. It’s become a fundamental tool for driving, and therein lies the point. We are already seeing a similar impact of AI in the AppSec and CloudSec space. We aren’t sure AI can fully write and secure all code yet, but it certainly is a helpful tool for security engineers and development communities right now. So, come with us as we investigate how AI could impact AppSec and CloudSec.

First Things First

Artificial Intelligence is a rapidly developing space and therefore any recommendations, opinions, or examples are based on present knowledge and subject to change. The examples used in this post are purely illustrative and we don’t recommend putting business specific or sensitive data into public AI tools.

This post will provide examples and utility for those trying to figure out how to parse out the noise and find what would be most beneficial for your AppSec and CloudSec programs. While we briefly spoke about AI and what it means for secure code reviews, we really need to go well beyond that and take a look at providing answers to the following questions:

1. What are the here and now benefits and examples of how AI can help Application & Cloud Security teams? We will go over some practical examples:

• Continuous Attack surface Mapping, Threat Modeling, and Documentation
• Developer Security and Education
• Security Blast Radius Analysis
• Business logic review & Application Pentesting

2. What are some known limitations of AI for Application and Cloud Security? Hint: precision

3. How can your organization guardrail and control risk associated with using AI products?

• If you are a security professional, you may be most concerned with controlling, analyzing and maintaining appropriate usage and data scrubbing.

Let’s get to it!

Continuous Threat Modeling, Attack Surface Mapping, & Documentation

If you have read our post about secure design reviews, then you will know threat models are a key way to provide an objective security perspective to stakeholders while gaining a better understanding of the attack surface.

There are 3 main challenges with threat models:

1. Tedious to create, especially if the development team doesn’t have any architecture established.
2. Usually requires moderate security knowledge to perform.
3. Can go out of date as soon as the code starts to change.

With these 3 challenges defined, let’s see how AI can potentially help.

Addressing the first 2 challenges

Today, Generative AI tools, like ChatGPT or Bard, can help make sense of code repositories. Knowing this now, let’s try asking it to generate a threat model for an open source project, CFSSL.

Here is what we prompted: Can you create a threat model of the cfssl open source project?

It generated a decent list but it lacked context and priority. So, we prompted: Can you please prioritize the list of threats and mitigations in the previously provided threat model of cfssl. Please add prioritization context on threat and mitigation. Add level of effort to mitigate each threat and add the likelihood of the threat. Also add rationale to the level of effort and likelihood ratings. Make a table of the threat model above and add the most likely threat actor for each threat scenario.

The following is what we got back:

This is not bad but too generic so we prompted again: Would the threat model change if the use case for cfssl for building a certificate authority to support an organization’s public key infrastructure?

The answer was obviously yes, and it updated the threat model. Since it was still a wall of text, we then prompted to put it in a table format and add the most likely threat actors.

Here is what came back:

Important disclaimer: This is purely an illustrative example and we don’t recommend putting anything business specific into public ChatGPT.

After reading through the table, we felt that the threats were all valid and the table was a solid start to a threat model. ChatGPT definitely reduced the barrier of entry to creating a threat model, but after going through the exercise, we realized that we:

1. Had a rough understanding of CFSSL and how to use it
2. Knew what threats and mitigations to expect
3. Knew exactly what format we wanted

We don’t know how long this exercise would take someone without 15 years of security experience doing threat modeling, but we would expect it to take longer. However, anyone can now apply these same prompts for future threat models if they wanted, which is great for scaling things like threat modeling out.

Security and development teams should be able to do similar tasks like above as they build out their docs and code. While these artifacts are lacking further business context for the use of the CFSSL library, ChatGPT calls that out. This certainly is a good start for a basic threat model, but we wouldn’t recommend using the output as a final deliverable. You will still need an architectural diagram to marry these threats and associated mitigation(s). This is a vital step in increasing the utility of a threat model.

Addressing challenge 3

In the real world, threat models are often constructed at the whiteboard/design phase of the software development lifecycle (SDLC), meaning there may not be code or documentation ready for review. AI can convert the dialog of the whiteboard session into a summary and form the start of a threat model based on the discussion. The team should ultimately validate, supplement, and prioritize accordingly.

We also believe that AI tools will be able to keep threat models and documentation up to date as the code changes. Based on the use cases, we believe having AI tools integrated with source code management (SCM) tooling will allow organizations to threat model all code repos, just like that of static code analysis.

We hope that down the road a security engineer can use auto generated threat models on a repo. This way they can clone a developer’s repo, create a new branch, implement necessary changes using some mix of AI tools and their knowledge, and then open a PR/MR. By doing this the security engineer and developer can review both the threat model and a proposed code solution. This shifts the conversation from a report meeting to a solutions meeting and should earn the AppSec team massive brownie points as development enablers rather than blockers (more on this in a subsequent post).

Pro tips:

• If you are going to use a ChatGPT like solution that has been trained using the internet, be sure you review everything for accuracy. Remember: Often the first thing you get out of a solution like chat-gpt will be generic, but with iterative refinement of what you’re asking for you can get something higher-quality and customized for you.
• Again, please be advised that putting anything into public tools that is business sensitive can lead to inadvertent data leakage. Seek guidance from your internal Security and Compliance teams on how to best leverage AI tools. We also have a few points in this later in the post.

Developer Security & Education

Developers are often the primary customers of tooling provided by App & Cloud Security teams. Google, Apple, Amazon, and many other customer focused companies make it a point to ensure their customers are trained and have access to tools or people that can answer questions for them. Tools like Github Co-Pilot, AWS CodeWhisper, Sourcegraph Cody, and OpenAI ChatGPT are starting to make suggestions while developing code through local IDEs. These tools all vastly vary in ability to actually help a developer create more secure code, and the developer ultimately needs to know what drives the code creation. For now, means security engineers still need to find opportunities to work with developers on code security standards and libraries.

Meeting developers where they are and supplementing their work stream with AI can be helpful and is the best approach for now as it keeps them in the driver’s seat as they develop.

Since we heard a lot of buzz earlier in the year about AI creating usable code, we wanted to explore that concept. Let’s revisit our ChatGPT thread from above and start to prompt for code output. Because we needed to keep adding prompts as we went and found the code to be not completely functional, we will spare the full results.

Here is what we prompted at first: Please generate Python code that will be an API to create new certificates using CFSSL. It will be an AWS lambda function with an AWS API gateway in front of it, so also generate CloudFormation for these components. The API needs to be highly available and secure, so please take that into consideration

The initial output didn’t have any authentication or authorization taken into account. So, we prompted it to do that: Can you add authentication and authorization to the example code above that would use AWS Cognito? Can you please generate CloudFormation to reflect adding AWS Cognito authentication and authorization to an AWS API gateway?

Then, we realized that the CloudFormation it generated would leave the API Gateway exposed to the internet, so we needed to prompt it to fix that: How would you make sure that this API can only be called within a private network and not open to the internet? Could you update the CloudFormation, if needed to reflect private network access from the previous message?

At this point, things were not going as well as we hoped but improving. Since we already built a threat model, why can’t it just update the code to be in line with that. But since that chat had progressed quite a bit, we figured it would be best to check in, so we prompted: How could you make the code from previous messages more secure?

Here is what it spit out, which had some differences from the previous threat model:

The suggestions provided sounded good, so we added the following prompt to the chat: Please update the code to reflect all recommendations above?

The response was missing the CloudFormation so we asked it to regenerate that with the updated security requirements. Here are the ChatGPT instructions given for the CloudFormation generated:

Given there was a lot of back and forth so we figured it would be beneficial for ChatGPT to provide a How To style guide for all of this! Can you please generate a how-to guide to deploy the python code for the lambda, aws components, and cloudformation above?

I think you get the point and can see the full chat here, if interested.

Our quick takeaways from this experiment:

• The threat model, instructions, how to guide, really anything text based, was most useful but only after a few iterations. Once getting the right prompts down, this could save security teams a half hour to an hour or so per threat model.
• In order to get the most useful output, you need to know what you want to build and the components necessary. This is where having the threat model + attack surface will come in handy. But, the code generated requires extra work and validation with every prompt.

ChatGPT, at its best, divorces knowing what you want to do from knowing how to do it, which can make life easier for people who aren’t hands-on keyboards every day but have a lot of experience. This feels like it has the opportunity to make things more difficult for junior developers who might not know what to ask. But, this is why having a human driven code review process comes in.

Anything creative in nature, like a threat model or documentation, seems like where AI right now can help the most. For coding, we like the IDE based suggestion model vs the generate me all the code approach.

Pro Tip:

• If you don’t know what to ask, you could consider the method of asking ChatGPT to impersonate a particular expert persona, and then have it recommend what prompts you should ask it to flesh out the path towards achieving a particular outcome. Not foolproof but a tactic to overcome a bigger gap between what you want and knowing what steps need to happen to get there.
• AI can help developers much faster than reaching out to a security engineer, but AI can’t quite make the experience as interpersonal as working with a human. It was recently found that 45% of responses from chatgpt were considered “empathetic”. We encourage Security teams to keep engagement frequent and positive with their stakeholders.

Security Blast Radius Analysis

It’s not improbable to know true blast Radius, given a threat model, attack surface component map, historical code analysis (findings and fixes), and vulnerability analysis. The challenge lies with connecting all of this together.

We asked ChatGPT to give it a whirl with our CFSSL example: Can you describe the security blast Radius of the cfssl system we designed above?

Here is what it came back with:

Cool. But, you might be asking “what is the difference between this analysis and the threat model?”. Great question! The threat model is a prospective on the possible threat scenarios of an expected design, whereas a blast radius analysis is a retrospective threat analysis of a system as built. A threat model is what could be possible vs a blast radius analysis which is what is actually possible.

We thought it would be useful to have ChatGPT also describe the differences for the CFSSL use case we have been using.

The challenge, as with similar prompts, is knowing if the input and output are comprehensive and accurate. The blast radius analysis in our example wasn’t bad but we would need to double back on the threat model and code in order to know if the result was accurate. Not to belabor the point, but this is yet another example of AI helping you get to your destination more efficiently, but only if you know the destination and the best route because you grew up in the neighborhood.

Business Logic Review & Pentesting

If we have AI creating and protecting our data, then we will also have AI going to battle on the other side. This reminds us of when “Son of Anton” from Silicon Valley was used by Dinesh and Gilfoyle as AI chatbots for themselves. When AI Dinesh meets AI Gilfoyle, the office suddenly loses power, and computers start sparking. Regardless, we need to be ready for the fact that AI enhanced tools can be used against organizations in the very near future. In the meantime, check out tldrsec’s AI page for new tools and trends in this area.

For now, internal teams (of humans) have the best context regarding what an application can or should do. Therefore, manual pentesting will continue to be most effective when done by humans. In the future, automated pentesting will get better with AI, yet we still think humans will play an integral part, especially for reporting and prioritizing.

Pro Tips:

• There’s a bull case for AI enabling elements of the process rather than taking over the whole thing. Tasks like enumerating common test cases based on a threat model, requirements doc, or system spec could at least accelerate a pentester getting them farther, faster than without gen AI.
• We are optimistic about the future but for now we recommend leading with expertise when using AI tools for manual security work. That doesn’t mean: don’t use AI, but just make sure you leverage human knowledge to drive the outcome.

Controlling Risk associated with Using AI products

The one thing that isn’t a solved problem yet, is how do businesses control risks with using AI products or features. Thematically, it boils down to sensitive data leakage and lack of human validation of AI generated output for business decisions. Here are 6 things to consider:

1. Central Model Training: See if you can opt out of central model training.
2. Data Retention: See if the data can be deleted after a transaction is completed.
3. Register of Products: Create a central and approved set of AI products and tools.
4. Deployment Strategies: As you consider solutions, ask about if they offer full SaaS, a hybrid, or a full on premises deployment. Depending on data/risk, you may not want all components or data to leave your premises. At a minimum, you may want a private network channel (e.g. Private Link) and tenant for more sensitive data flows.
5. Employee Education: Employees need to know what tools are safe to use and how to use them without putting data at risk.
6. Employee Responsibility: Whether the information is from a google search or ChatGPT, someone has to be responsible for the queries and output. There is no way to hold these tools accountable for inaccurate answers, so the responsibility has to be pushed onto the employee. This means all answers ultimately have to be verified before using them for important business decisions or products.

None of these are silver bullets and frankly there is a similar data leakage risk akin to using online file conversion tools but this is the first time we are dealing with products generating custom and knowledge-based intellectual property at the scale of the internet.

Are you wondering what ChatGPT suggests for mitigating data leakage risks using generative AI? Us too, so here is a screen grab of the output:

Takeaways

1. AI is changing so fast and there are many human controlled or influenced variables to deal with still, but this doesn’t mean AI can’t immediately help security engineers and developers improve common tasks or create documentation.
2. Threat modeling has historically been difficult to scale due to knowledge gaps and manual effort. AI can help create a base set of threats and mitigations for teams right now and in hopefully not so distant future keep threat models up to date.
3. Developers can rely on AI tools available today that are able to help them learn security concepts and implement features as they develop in their IDE. There are significant accuracy and knowledge based limitations with this, but that’s where security teams and senior developers can help fill gaps.
4. One of the biggest security challenges has been correlating application context and infrastructure. We hope by piecing the threat model + code + the output from scan tools together with AI, that we can get the closest we have all come to knowing true blast radius and risk.
5. There are many limitations with AI today, but this is a rapidly evolving space. Our hope is that AI can be harnessed and tuned for precision rather than to be like humans for security use cases. This would address some of the gaps we experienced in writing this.
6. There are strong data leak implications with using public AI tools. Organizations will need to come up with their own stance and risk model for this to deal with.

Words of Wisdom

It’s hard to imagine a world where we couldn’t find useful resources with a simple query. We all need to adjust and embrace the adoption of AI in our lives. Cybersecurity is no different. It will take collaboration among us all to make it into the best shape we can.

“If you always do what you’ve always done, you’ll always get what you’ve always got.” — Henry Ford

Contributions and Thanks

A special thanks to those who helped peer review and make this post as useful as it is: Henry Stanley, Shane Caldwell, Austin Paradis, Brian Lozada, Dor Zusman, Robert Prast, William Speirs, John Nichols, and Vishal Jindal. As well as many others who helped provide perspective, knowledge, and feedback as we created this.

A special thanks to you, the reader. We hope you benefited from reading this in some way, and we want everyone to be successful at this. While these posts aren’t a silver bullet, we hope they get you started.

Please do follow our page if you enjoy this and our other posts. More to come!