In a recent post on “Debunking Myths: Penetration Testing is a Waste of Time”, Rohit Sethi looks at some of the disadvantages of the passive and irresponsible way that application pen testing is generally done today: wait until the system is ready to go live, hire an outside firm or consultant, give them a short time to try to hack in, fix anything important that they find, maybe retest to get a passing grade, and now your system is ‘certified secure’.
A test like this “doesn’t tell you:
- What are the potential threats to your application?
- Which threats is your application “not vulnerable” to?
- Which threats did the testers not assess your application for? Which threats were not possible to test from a runtime perspective?
- How did time and other constraints on the test affect the reliability of results? For example, if the testers had 5 more days, what other security tests would they have executed?
- What was the skill level of the testers and would you get the same set of results from a different tester or another consultancy?”
Sethi stresses the importance of setting expectations and defining requirements for pen testing. An outside pen tester will not be able to understand your business requirements or the internals of the system well enough to do a comprehensive job – unless maybe if your app is yet another straightforward online portal or web store written in PHP or Ruby on Rails, something that they have seen many times before. You should assume that pen testers will miss something, possibly a lot, and there’s no way of knowing what they didn’t test or how good a job they actually did on what they did test. You could try defect seeding to get some idea of how careful and smart they were (and how many bugs they didn’t find), but this assumes that you know an awful lot about your system and about security and security testing (and if you’re this good, you probably don’t need their help). Turning on code coverage analysis during the test will tell you what parts of the code didn’t get touched – but it won’t help you identify the code that you didn’t write but should have, which is often a bigger problem when it comes to security.
You can’t expect a pen tester to find all of the security vulnerabilities in your system – even if you are willing to spend a lot of time and money on it. But pen tests are important because this is a way to find things that are hard for you to find on your own:
- Technology-specific and platform-specific vulnerabilities
- Configuration and deployment mistakes in the run-time environment
- Pointy-Hat problems in areas like authentication and session management that should have been taken care of by the framework that you are using, if it works and if you are using it properly
- Fussy problems in information leakage, object enumeration and error handling – problems that look small to you but can be exploited by an intelligent and motivated attacker with time on their side
- Mistakes in data validation or output encoding and filtering, that look small to you but…
And if you’re lucky, some other problems that you should have caught on your own but didn’t, like weaknesses in workflow or access control or password management or a race condition.
Pen testing is about information, not about vulnerabilities
The real point of pen testing, or any other kind of testing, is not to find all of the bugs in a system. It is to get information.
- Information on examples of bugs in the application that need to be reviewed and fixed, how they were found, and how serious they are.
- Information that you can use to calibrate your development practices and controls, to understand just how good (or not good) you are at building software.
Testing doesn’t provide all possible information, but it provides some. Good testing will provide lots of useful information.
James Bach (Satisfice)
This information leads to questions: How many other bugs like this could there be in the code? Where else should we look for bugs, and what other kinds of bugs or weaknesses could there be in the code or the design? Where did these bugs come from in the first place? Why did we make that mistake? What didn’t we know or what didn’t we understand? Why didn’t we catch the problems earlier? What do we need to do to prevent them or to catch them in the future? If the bugs are serious enough, or there are enough of them, this means going all the way through RCA and exercises like 5 Whys to understand and address Root Cause.
To get high-quality information, you need to share information with pen testers. Give the pen tester as much information as possible
- Walk through the app with pen testers, hilight the important functions, and provide documentation
- Take time to explain the architecture and platform
- Share results of previous pen tests
- Provide access behind proxies etc
Ask them for information in return: ask them to explain their findings as well as their approach, what they tried and what they covered in their tests and what they didn’t, where they spent most of their time, what problems they ran into and where they wasted time, what confused them and what surprised them. Information that you can use to improve your own testing, and to make pen testing more efficient and more effective in the future. When you’re hiring a pen tester, you’re paying for information. But it’s your responsibility to get as much good information as possible, to understand it and to use it properly.
