Bugcrowd Blog

All You Need to Know About Bug Bounty Testing Environments

Posted by Grant McCracken on Oct 12, 2016 1:05:00 PM

By way of a quick refresher, in regards to setting up a bug bounty program, we've already covered step zero, setting your scope, and the importance of focus areas, as well as some considerations to make around exclusions on your program.

Now that we’ve covered most of what goes into writing a bug bounty brief, including rewards and disclosure policies, let’s take a look at what environment you'll be providing for researchers to test against. Regardless of how you decide to set up your application(s), it's important to remember that our goal is to attract great talent from the crowd, sustain activity, and ultimately minimize the challenges of setting up and running a bug bounty for you and your internal teams.

Here are some general considerations to keep in mind when setting up your testing environment:

Initial Considerations

1. Be prepared

Make sure your environment can stand up to testing. We've seen more than a few applications - both production and staging - go down simply from the load created by researcher testing.

You should anticipate being able to support at the very least, several (depending on the size of your testing pool if private) to hundreds (if public) of researchers sending at least six requests per second for an extended period of time. You should also prepare for the use of scanners against your application, as many many researchers will run them as part of their methodology.

Keep in mind that the time of your program launch is typically going to also be the time when the most people will testing against it. If your application breaks or responds too slowly during this initial rush, researchers will simply focus their efforts elsewhere, and may not return.

In that same vein, we recommend that you don’t share credentials or environments. It may be tempting to create a single environment for researchers to test in, and then provide them all with unique admin credentials - but you can likely imagine how this could go wrong. Researchers could begin deleting other researchers, changing core settings (e.g. the environment subdomain), and dumping their polyglot payloads in every available form field - leaving other researchers tripping over endless prompt boxes. Understandably, this sort of environment doesn’t create the greatest of researcher experiences, and you run the risk of losing researcher testing to programs with more segregated accounts. Remember, just as researchers are competing for findings and rewards, you are similarly competing for researchers' use of time and resources.

2. Be mindful of testing impact and plan ahead

At this point, you should have already advised your internal stakeholders and departments how the bug bounty program will affect them, but in regards to your testing environment specifically, there are additional considerations to make.

Consider how any sensitive areas of your application might be affected during the course of testing, thus negatively impacting other departments within your organization. The most common culprit is the 'contact' form, which typically directs to internal sales or marketing teams. If a researcher starts aggressively testing on one of these forms, this can potentially result in hundreds or thousands of form submissions - which as you can imagine, could lead to your sales and marketing teams getting drowned in a deluge of test form submits.

If you have forms like this, it's worth designating them as out-of-scope, so as to make sure researchers don't test them. If you want to know about vulnerabilities on these forms (e.g. insufficient anti-automation), you may consider testing against a staging environment, as these forms could be set not to affect persons within your organization.

Additionally, be sure to file a penetration testing request with AWS/Heroku/whoever so that they know you're running a bug bounty program. Note: Most of these request forms ask for source IPs, but if you inform them that you're running a bug bounty program, they're usually understanding of the fact that there's no way to know all the IPs that researchers will be coming from.

 

Now that you know what to keep in mind regarding your testing environment, you have to decide what kind of environment you can test against. When it comes to running a bug bounty program, there are typically two options for your testing environment: (1) production; or (2) staging. There are benefits and use cases for each option depending on the business objectives, capabilities, and sensitivities of your organization, discussed below.

 

Testing Against a Staging Environment

For many of the reasons you might opt to run a private program, you may also decide to set up a staging environment for researchers to test against. Some of the benefits include:

  • If your environment happens to fall over, no production clients or users are affected.
  • Researchers won't come into contact with personally identifiable information or sensitive user data on a non-production instance.
  • You don't have to wonder if traffic is malicious, or if it’s just researchers performing testing.
  • Researcher traffic won't interfere with website metrics collected by other departments.
  • By testing on staging, you're often able to run the absolute latest version of your application, allowing you to identify issues before releasing them into production.
  • If you’re looking to test specific functionality such as payment processing; testing may be easier in a staging environment since researchers can use fake/test credit cards, banking info, SSNs, etc.

Testing Against a Production Environment

However, if none of the above points are particular areas of consideration for the applications you're looking to have tested, then running against production could also be the best route for your organization. In fact, nearly every public bounty program runs on production environments, and it’s common, understandable, and reasonable to do so, especially in the following scenarios:

  • If you have a large scope that extends to any property you own (see Tesla's and Spotify's programs), it may be simply infeasible to set up a staging version of every property.
  • If you're testing a non-web application, including open source applications, mobile applications or devices, it may be difficult to support and maintain a non-production instance to test against.
  • If you have limited resources, testing against a production environment eliminates the need to set up and maintain a separate instance for testing.

 

Want to learn more about the ins and outs of running a successful bug bounty program? Here is a list of additional resources referenced in this post:

At AppSecUSA 2016? Join myself and my colleague Daniel Trauner discuss these subjects and more in our talk 'If You Can't Beat 'Em, Join 'Em' Friday, October 14th at 2:15 PM ET. 

 

Running Your Own Program
Grant McCracken

Written by Grant McCracken

Solutions Architect at Bugcrowd