Mastering the Art of Hiring Right in Application Security
A practical guide to interviewing successfully for your Application Security Engineering team so you can hire the right talent.
Written By: James Chiappetta
Disclaimer: The opinions stated here are my own, not necessarily those of my past, current, or future employer.
Background
If your profession is in or near the technology space then you probably have heard that it’s not “easy” to find the right talent for your team. No one is alone on this and there are countless posts on the topic. AppSec in particular has been an area where technical organizations have had a heightened level of difficulty finding talent. That coupled with a rising interest and demand for this subject matter expertise has made an already challenging situation even worse. One way to illustrate this is with the google search trend data below.
I can assure you, this isn’t a feat of impossibility. This post is to help you structure the interview process and ask the right questions when hiring Application Security Engineers for your needs.
First Things First
Broadly speaking, it’s difficult for technical organizations to find Software, DevOps, and Systems Engineers. The problems hiring in engineering are not unique to just AppSec. But as it relates to AppSec there are some nuances with hiring the right talent.
There are a few distinct areas we will need to cover in this post:
- Defining a team culture statement
- Defining role competencies for leadership, technical experience, and cultural fit
- Get recruiting support and point them in the right direction
- Structuring the interview process and get the team focused
- Asking the right behavioral and technical questions
- Indexing on the answers and getting calibrated feedback from the panel
- Determining if there is a need for staff augmentation
Let’s get to it!
Team Culture
Define a team culture statement and post it on the job description — I have written before about the importance of building out a set of values the team lives by. Having a clear understanding of what makes a successful team member is crucial. Hiring a cultural mismatch can have negative impacts on not just the team but the stakeholders it supports. This might lead to eroding trust, which is a sure fire way to find yourself in an AppSec Doom Loop. Write a paragraph at the top of your job description that clearly articulates what values the team operates with.
Role Competencies
Build out a set of role competencies — Competencies need to take into account what is most important for the here and now, then account for how an individual may grow in the long run. Don’t go to market without a clear expectation set on the needs of the business and team. Your needs for an AppSec Engineer should be predicated on how a candidate fits against the list of expected competencies for the role. This will help suss out the short term requirements versus long term growth for the candidate and team.
Leadership Competencies —
- Empowers others to make security decisions
- Negotiates trade offs between security requirements and usability
- Communicates technical security requirements and issues clearly, without instilling fear
- Knows when to take a technical challenge on themselves, gain support, and delegate aspects of the work
Technical Competencies —
- Fundamentals of Systems and Software Architecture
- Secure design and threat modeling
- Working knowledge of OWASP top 10 vulnerabilities
- Cloud infrastructure security fundamentals
- Security Tooling R&D, implementation, management, and scaling
- Programing and scripting experience
- Application, Network, or Hardware pentesting experience
- Stakeholder management and partnership
- Dealing with ambiguity and project ownership
- External contributions to the community
Cultural Competencies —
- Earns the trust of their peers and stakeholders
- Has empathy for those they work with closest (typically developers) and is cognizant of their time and effort
- Demonstrates that they can balance ideal state against obstacles that surface
- Thinks through problems or challenges aloud and doesn’t aim to bias with their opinion
- Wants to or has contributed beyond themselves, (e.g. open source, blog posts, or presentations).
Recruiting Support
Partner with your recruitment team — Some of the strongest working relationships I have formed in my career have been with technical recruiters, with both internal in organizations and with extetnal recruitment firms. You have to be in lock step with your recruitment team as they will be creating a high quality funnel for your roles. If you want to make sure you aren’t wasting time with all parties then you will need to set clear expectations of who the ideal candidate would be. This will require some prep work and would be best done together with your existing security team (if there is one) and recruiting. What is that work? Well, mostly what is covered in this post but your organization may want to train you on their specific guidelines for interviewing as well.
Pro tips:
- If you are a hiring manager, you aren’t just trying to sell the role and company but yourself as well. So having the recruitment team know your personal strengths as a leader is going to be key as an early sell. The rest will be on you to show through actions that you are the real deal.
- Use profiles of those you have worked with in the past to seed the recruitment team.
Structuring the Interview Process
Build an AppSec Interview Matrix — Someone (probably my subconscious) once told me that no great business process exists without a table or matrix. So here you go…
Interview Questions
The list of questions below are bucketed in three categories. You may want to adjust this as you see fit for the seniority of the role you are trying to fill. Additionally, you are going to also want to decide on the signal you are trying to get from the candidate’s answers. For instance, you may ask: “What is the difference between a vulnerability assessment and a pentest”, and the candidate could by definition distinguish between the two. However, you may want to also know if the candidate also has performed each of these before and to what level of depth.
Soft ball questions —
Q: Which area of Application Security interests you the most and why?
Q: How do you keep up with emerging technologies, patterns, and vulnerabilities?
Q: What is your favorite vulnerability you have found and can you walk me through how you found it?
Q: Walk me through how you perform a full security review? What steps or tasks are needed in order to ensure an application is ready for a production release?
Q: What is the difference between vulnerability assessment and a pentest?
Q: What is software composition analysis and why has it been more important to organizations over the past few years?
Q: Does building software and systems in a cloud infrastructure provider present more or less risk for an organization? Why?
Behavior and situational questions —
Q: What are you trying to avoid in your next role?
Q: Tell me a time you had to deal with a difficult customer or stakeholder? How did you handle it and what was the outcome? Follow up: looking back, what would you have done differently?
Q: Tell me about a time you needed to balance the needs of security against the needs of a product or engineering team? How did you find the right balance? Follow up: looking back, what would you have done differently?
Q: Tell me about a time when you needed to take full ownership of a technical area or project that had no owner? What was the project and how did you see it through to completion? Follow up: looking back, what would you have done differently?
Q: What was the hardest programming or automation project you have ever completed? What technologies did you use? Follow up: looking back, what would you have done differently?
Q: Why is it important to the success of an Application Security Team to build trust with engineers and product developers? How do you build trust and grow it over time?
Q: Tell me about a time you needed to get buy-in on an under-funded or unfunded area in security. How did you know the area needed to be funded and did you end up getting the funding needed? Follow up: looking back, what would you have done differently?
Q: Tell me about a time you inadvertently affected a production or critical system? How did you know you did this and what did you do about it? Follow up: looking back, what would you have done differently?
Q: Tell me about a time you needed to partner with another security team such as Incident Response or Compliance. What was the shared project and what was the outcome? What role did you play in the project? Follow up: looking back, what would you have done differently?
Q: Tell me about a time you had to balance the priorities of other teams (feature development/deliverables etc.) and your own priorities (security, secure development, best practices, etc.).
More technical questions —
Q: When is threat modeling a good exercise to perform and how do you measure the success of one?
Q: What is an Indirect Object Reference weakness? Have you ever personally found one and what is the mitigation? Follow up: assuming the issue is pervasive across an organization, what would be a phased approach to addressing the issue?
Q: What would be your approach to eliminating an entire vulnerability class across an organization? Have you ever come close to doing this before and how did you or would you measure success?
Q: Can you perform a CSRF attack on the login page and how would an attacker exploit that vulnerability?
Q: Can you walk us through your process to threat model a web page with a comment box and submit button?
Q: Given a web endpoint that takes a basic GET request and set of parameters for checking if a user account already exists, how would you approach attacking or fuzzing it? Note: The endpoint takes first name, last name, DOB, email, and last 4 of a social. What tools or scripts would you use? Follow up: how would you approach this if you could only use a programming language?
Q: What approach have you taken with security vulnerability identification tools: high signal vs low noise OR low signal vs high noise? Where and why would you want to use each of these techniques?
Q: An engineer needs to send a sensitive file to a 3rd party and asks you how they should encrypt it before sending. What do you tell them? Follow up: say you can’t use their first suggestion and need an alternative
Q: How do JWTs work and what is the importance of securing the generation, transport, and validation of them?
Q: What is your playbook for when a Security Researcher discloses a newly emergent vulnerability that applies to the organization? How do you handle this situation?
Q: How do you best handle a web borne application attack, especially if the attack is still active?
Q: Imagine a developer is building a brand new login web page for users on the internet and they ask you how they can make sure it’s secure. What are some things you would tell them?
Pro Tip: During an interview, you only have a limited amount of time to interview someone. Make each question count. This also promotes a non-static interview script because it lets the interview evolve based on the candidate.
Parsing out the Answers and Collecting Feedback
Listen and fully understand what the candidate is saying — I am sure someone has said this at some point before me but effective communication isn’t going to make a huge difference if the listener doesn’t understand what is said.
“The evaluating stage occurs most effectively once the listener fully understands what the speaker is trying to say.” — Effective Listening Skills.
What I am trying to say is, your interviewers need to listen and fully understand what the candidate is saying without filling in the blanks in their head. This is why I have listed follow ups to most behavioral questions.
Probing effectively is a highly underrated interview skill. A guideline I like to reference in interviews is: how does the information the candidate is conveying reflect their ability to make a decision? Are they structuring this decision based on facts, data, experience or some other heuristic? Important that the candidate is able to demonstrate the thought process, not just the answer.
Trigger phrases like “so then I” or “next, I” imply they made a decision on a course of action. If they don’t have good reasoning for these steps, it’s a yellow flag and I bring them back to that moment to help me understand why they did what they did. — Henry Stanley
Take good notes, process, and make your decision — Make sure your interview panel lets the candidate know in advance that they will be taking notes so that they don’t feel ignored. This sets the stage so your panel can write down answers to process later. These notes should help guide the decision making process for each panel member. Every company does their debriefing process differently but the big takeaway here is to make sure the interview panel members have a solid list of strengths, opportunities, follow ups to discuss, and a recommendation for hiring. This will help immensely when you come together to discuss as a team.
It is important to note that you want calibrated feedback from your panel. Calibrated means that everyone has the same idea of what a senior, staff, etc engineer is capable of. This will avoid the scenario where the interview panel has conflicting expectations of what meets the bar for the level.
Pro Tip: Make sure your panel knows what to expect to see in a good answer. The last thing you want is boilerplate questions provided by HR for other roles just to fill a void. These may be purposely ignored because people have no idea what a good answer would be.
Staff Augmentation
The “I need someone right now” option, staff augmentation — It takes time to find, interview, and get talent on the team. From my experience, it can take upwards of 9 months on average. So, if you are up against some tight deadlines on key business projects but have no one to do the work, then staff augmentation may be your best bet. This can be done a few ways:
- Find internal resources who can execute some security work and build a Security Champions program around it. This is going to be hard to achieve in practice but it is an option.
- Use a reputable security consultancy company. This is where I personally have had the strongest outcomes. It gives you immediate access to talent on staff who can come in and execute within a week or two.
- Find an independent contractor. This may be cheaper than using a company but could take just as long as finding a full time employee.
Making the justification — Not every company is going to have the budget for this but I believe it can easily be justified. Especially since AppSec doesn’t directly “make money” for the company but it prevents potential loss. Structure the engagement in a way where you have the resource for 6 to 12 months while the team searches for the full time employee. If you are looking for a full time employee anyway then you would instead be paying the staff augmentation, albeit at a premium.
This should be cost/risk decision. It may be worth it when you consider putting key deliverables at risk, delaying necessary security work, or missing security weaknesses that become surface area for attackers. Those weaknesses will surface at some point and will become costly to fix once in production. Obviously come up with your own spin on this, but these have always been some key thoughts on the need.
Pro Tip: If you are in a hurry for resources then you should consider conducting a retrospective of how you found yourself in this position. Interviewing shouldn’t be a rush job and certainly a less than ideal state for a team to be in. Even if the answer is “A bunch of people left at the same time” I still want to know why that happened.
Takeaways
- Set clear expectations around the culture of the team, what competencies are important for the role, and how you want to structure the overall interview process. This is going to allow for everyone involved, especially recruiting, to operate with high levels of efficiency.
- We live in the information age. If you don’t know the answer to a knowledge-based question, you can always do some research online. Interviewing to determine how well someone learns is more beneficial than interviewing to determine how much someone knows. Don’t simply ask knowledge based questions, such as “what is a remote code execution?”. Rather, ask a more open ended and situational question. This helps the interviewer understand the candidate’s experience more holistically and get to deeper areas of technical knowledge naturally.
- Don’t simply accept a yes or no answer from your interview panel. Talk about it as a team and work through the strengths and opportunities of the candidate. One person’s yes may actually be a no for someone else after hearing the reasoning.
- Time is not your side. If you need someone right away, then staff augmentation can be a key way to derisk the business in the shorter term.
Words of Wisdom
Finding the right people doesn’t have to be as hard as people make it out to be. You can find the right people but it may take a little pragmatism, vision for where you want to go, and effort to get everyone aligned. I will leave you all with two quotes I find particularly valuable:
Find ballplayers, not those who look good in baseball caps — Tom Monahan
Do not hire someone who does your work for money, but they who do it for the love of it — Henry David Thoreau
If you are interested in how to incorporate compliance into AppSec, then stay tuned for my next post!
Contributions and Thanks
A special thanks to those who helped peer review and make this post as useful as it is: Luke Matarazzo, John Nichols, Mike Antico, Kyle Suero, Jeremy Shulman, Henry Stanley, and Vishal Jindal
A special thanks to you, the reader. I hope you benefited from it in some way and I want everyone to be successful at this. While these posts aren’t a silver bullet, I hope they get you started.
Please do follow my page if you enjoy this and my other posts. More to come!
