Earlier this year, I wrote extensively about vulnerability disclosure policies and benefits as well as how trust impacts the disclosure process between researchers and vendors. While writing these posts, I looked for publicly available (free!) literature on product security incident response (PSIRT) processes to share. I thought I’d find vendors publishing their PSIRT best practices on operations or how to publish an advisory, but 99% of what I found was network incident response focused and not relevant for application or product security teams. I suddenly realized that despite all my years working in a PSIRT, I'd never published any operational guidance that would help other defenders learn from my experiences - and it was time to change that.
Since identifying this resource gap I've published a few resources to help companies build out their PSIRTs, spoke with Dark Reading Radio, and delivered a webcast ‘Building a Product Security Incident Response Team: Learnings from the Hivemind,’ an expanded version of a presentation I gave during Black Hat 2016. This post is a high-level review of that presentation and suggests several questions you should consider when building out your PSIRT functionality. Each organization has a different culture, problem set, technology base, set of dynamics... so solutions will vary. Feel free to cherry-pick what works for your organization.
The type of people you need on your incident response team is dependent on technology as well as resources. In general, you'll need technical resources that throughout this process will be triaging and reviewing incoming vulnerability reports, validating and fixing those vulnerabilities, and communicating the resolution of those incidents internally and externally.
From my experience as well as interviews with about a dozen PSIRTs, team structure varies primarily based on the type of technology being supported and if your customers need to take steps to install a software or firmware update.
The SDL emphasizes designing secure code, training for secure code practices, and mentions the need for planning and documenting your incident response process in the release phase, and implementing your incident response process in the release phase... But nowhere could I find guidance on what should be in that incident response process!
The PSIRT case management process can be broken down into five main steps:
- Identify Issue - private report? public disclosure? internally found?
- Assess Impact - Is this reproducible? How serious is it? What products does it affect?
- Development & Test Fix
- Release fix (potentially w/ CVE)
- Post Release Customer Support
Internal Policies:Once you've defined what your incident response process looks like and who will carry out those functions, you need to communicate internally, set expectations and define procedures both within the PSIRT team as well as with your development engineering team. This includes defining your vulnerability prioritization model, setting internal guidelines for remediation service level agreements, and defining necessary escalation paths.
Externally you'll need to do essentially the same thing: communicate and align expectations, and set the foundation for a healthy PSIRT. The first step to a robust incident response is setting up a Vulnerability Disclosure Policy or channel, such as a bug bounty program. It is also important to have a clear and concise ‘security advisory knowledge base’ for your customers to find your security guidance and a mechanism to recognize and incentivize positive behavior and build a relationship with security research community.
Tools and Infrastructure:
How do you want to receive vulnerability reports? Where will you document technical details of the vulnerability investigation? What do you need to document? Remember that what your developers need to know to fix a vulnerability is not all you need to document - your leadership team will want updates on vulnerability trends and security debt, and your customers will want to know about mitigations and workarounds... Here’s a handy checklist of data to capture during your vulnerability investigation that will keep you square on the technical and business risks that you'll need to know.
BONUS: Using third party or open source code? Incident response doesn't have to become much harder–get a source code scanning tool to hook up into a vulnerability intelligence source that tells you what libraries you use where, and what vulnerabilities they contain so you can patch efficiently. To learn more, register for our upcoming webcast with Jake Kouns of Risk Based Security and Christine Gadsby of Blackberry on the process to improve security when using open source software. Register here.
For more in-depth context on these recommendations, some common mistakes to avoid, and additional considerations, visit our PSIRT resource suite, watch my recent webcast, and feel free to reach out to me with any questions @Kym_Possible or email@example.com.