1. Introduction
This part of the tutorial, which is dedicated to the security testing, is going to wrap up the discussions around testing strategies proven to be invaluable in the world of software development (microservices included). Although the security aspects in the software projects become more and more important every single day, it is astonishing to consider how many companies neglect security practices altogether. At least once a month you hear about a new major vulnerability or data breach disclosure. Most of them could be prevented way before reaching the production!
The inspiration for this part of the tutorial mostly comes out of the Open Web Application Security Project (shortly, OWASP), a worldwide not-for-profit charitable organization focused on improving the security of software. It is one of the best and up-to-date resources on software security, available free of charge. You may recall that some of the OWASP tooling we have seen already along the tutorial.
Table Of Contents
2. Security Risks
Security is a very, very broad topic. So what kind of security risks attribute to the microservice architecture? One of the OWASP initiatives is to maintain the Top 10 Application Security Risks, a list of the most widely discovered and exploited security flaws in the applications, primarily web ones. Although the last version is dated 2017, most risks (if not all of them) are still relevant even these days.
For an average developer, it is very difficult to be aware of all possible security flaws the applications may exhibit. Even more difficult is to uncover and mitigate these flaws without expertise, dedicated tooling or/and automation. Having the security experts on the team is probably the best investment but it is surprisingly difficult to find good ones. With that, the tooling aspect is exactly what we are going to be focusing on, narrowing the discussion only to the open-sourced solutions.
3. From the Bottom
Security should be a comprehensive measure, not an afterthought. It is equally as important to follow the secure coding practices as to secure the infrastructure. Like in the construction industry, it is absolutely necessary to start from a solid foundation.
There are a couple of tools which perform the security audit of the Java code bases. The most widely known one is the Find Security Bugs, the SpotBugs plugin for security audits of Java web applications which relies on static code analysis. Besides the IDE integrations, there are dedicated plugins for Apache Maven and Gradle so the analysis could be baked right into the build process and automated.
Let us take a look on Find Security Bugs usage. Since most of the JCG Car Rentals microservices are built using Apache Maven, the SpotBugs and Find Security Bugs are among the mandatory set of plugins.
<plugin>
<groupId>com.github.spotbugs</groupId>
<artifactId>spotbugs-maven-plugin</artifactId>
<version>3.1.11</version>
<configuration>
<effort>Max</effort>
<threshold>Low</threshold>
<failOnError>true</failOnError>
<plugins>
<plugin>
<groupId>com.h3xstream.findsecbugs</groupId>
<artifactId>findsecbugs-plugin</artifactId>
<version>LATEST</version>
</plugin>
</plugins>
</configuration>
s<executions>
<execution>
<goals>
<goal>check</goal>
</goals>
<phase>verify</phase>
</execution>
</executions>
</plugin>
By default, the build is going to fail in case any issues have been discovered during the analysis (but the configuration is really flexible in this regard). Specifically for Find Security Bugs there is also an SBT integration (for Scala-based projects) although it looks abandoned.
To run a bit ahead, if you are employing a continuous code quality solution, like for example SonarQube (which we are going to talk about later in the tutorial), you will benefit from the code security audits as part of the quality checks pipeline.
4. Zed Attack Proxy
Leaving the static code analysis behind, the next tool we are going to look at is Zed Attack Proxy, widely known simply as ZAP.
The OWASP Zed Attack Proxy (ZAP) is one of the world’s most popular free security tools and is actively maintained by hundreds of international volunteers. It can help you automatically find security vulnerabilities in your web applications while you are developing and testing your applications. It’s also a great tool for experienced pentesters to use for manual security testing. – https://www.owasp.org/index.php/OWASP_Zed_Attack_Proxy_Project
There are several modes which ZAP could be exploited. The simplest one is just to run the active scan against the URL where the web frontend is hosted. But to get most out of ZAP, it is recommended to configure it as a man-in-the-middle proxy.
Besides that, what is interesting about ZAP is the fact it could be used to find the vulnerabilities by scanning web services and APIs, using their OpenAPI or SOAP contracts. Unfortunately, ZAP does not support OpenAPI v3.x yet but the issue is opened and hopefully is going to be fixed at some point.
Out of all JCG Car Rentals microservices only Reservation Service uses the older OpenAPI specification which ZAP understands and is able to perform the scan against. Assuming the valid access token is obtained from the Keycloak, let us run our first ZAP API scan.
$ docker run -t owasp/zap2docker-weekly zap-api-scan.py
-z "-config replacer.full_list(0).description=keycloak
-config replacer.full_list(0).enabled=true
-config replacer.full_list(0).matchtype=REQ_HEADER
-config replacer.full_list(0).matchstr=Authorization
-config replacer.full_list(0).regex=false
-config replacer.full_list(0).replacement=Bearer\ $TOKEN"
-t http://host.docker.internal:18900/v2/api-docs
-f openapi -a
...
Total of 15 URLs
PASS: Directory Browsing [0]
PASS: In Page Banner Information Leak [10009]
PASS: Cookie No HttpOnly Flag [10010]
PASS: Cookie Without Secure Flag [10011]
PASS: Incomplete or No Cache-control and Pragma HTTP Header Set [10015]
PASS: Web Browser XSS Protection Not Enabled [10016]
PASS: Cross-Domain JavaScript Source File Inclusion [10017]
PASS: Content-Type Header Missing [10019]
PASS: X-Frame-Options Header Scanner [10020]
PASS: X-Content-Type-Options Header Missing [10021]
PASS: Information Disclosure - Debug Error Messages [10023]
PASS: Information Disclosure - Sensitive Information in URL [10024]
PASS: Information Disclosure - Sensitive Information in HTTP Referrer Header [10025]
...
PASS: Cross Site Scripting (Persistent) [40014]
PASS: Cross Site Scripting (Persistent) - Prime [40016]
PASS: Cross Site Scripting (Persistent) - Spider [40017]
PASS: SQL Injection [40018]
PASS: SQL Injection - MySQL [40019]
PASS: SQL Injection - Hypersonic SQL [40020]
PASS: SQL Injection - Oracle [40021]
PASS: SQL Injection - PostgreSQL [40022]
PASS: Possible Username Enumeration [40023]
PASS: Source Code Disclosure - SVN [42]
PASS: Script Active Scan Rules [50000]
PASS: Script Passive Scan Rules [50001]
PASS: Path Traversal [6]
PASS: Remote File Inclusion [7]
...
FAIL-NEW: 0 FAIL-INPROG: 0 WARN-NEW: 1 WARN-INPROG: 0 INFO: 0 IGNORE: 0 PASS: 97
As the report says, no major issues have been discovered. It is worth to note that ZAP project is very automation-friendly and provides a convenient set of the scripts and Docker images along with dedicated Jenkins plugin.
5. Archery
Moving forward, let us spend some time and look at Archery, basically a suite of the different tools (including Zed Attack Proxy by the way) to perform the comprehensive security analysis.
Archery is an opensource vulnerability assessment and management tool which helps developers and pentesters to perform scans and manage vulnerabilities. Archery uses popular opensource tools to perform comprehensive scanning for web application and network. – https://github.com/archerysec/archerysec
The simplest way to get started with Archery is to use prebuilt Docker container image (but in this case the integrations with other tools would need to be done manually):
$ docker run -it -p 8000:8000 archerysec/archerysec:latest
Arguably the better way to have Archery up and running in Docker is to use the Docker Compose with the deployment blueprint provided. It bundles all the tooling and wires it with Archery.
Although the typical way to interface with Archery is through its web UI, it also has a RESTful web APIs for automation purposes and could be integrated into CI/CD pipelines. The management part of the Archery feature set includes integration with JIRA for ticket management.
Please notice nonetheless the project is still in development phase, it has been showing quite promising adoption, certainly worth keeping an eye on.
6. XSStrike
Cross-Site Scripting (XSS) is steadily one of the most exploited vulnerabilities in the modern web applications (and is the second most prevalent issue in the OWASP Top 10, found in around two thirds of the applications). Since the JCG Car Rentals platform has a public web frontend, the XSS is the real issue to take care of and the tools like XSStrike are enormously helpful in detecting it.
XSStrike is a Cross Site Scripting detection suite equipped with four hand written parsers, an intelligent payload generator, a powerful fuzzing engine and an incredibly fast crawler. – https://github.com/s0md3v/XSStrike
The XSStrike is written in Python so you would need the 3.7.x release to be installed in advance. Sadly, the XSStrike does not play well with the single-page web applications (like JCG Web Portal for example, which is based on Vue.js). But still, we could benefit from running it against JCG Admin Web Portal instead.
$ python3 xsstrike.py -u http://localhost:19900/portal?search=bmw
XSStrike v3.1.2
[~] Checking for DOM vulnerabilities
[+] WAF Status: Offline
[!] Testing parameter: search
[!] Reflections found: 1
[~] Analysing reflections
[~] Generating payloads
[-] No vectors were crafted.
It turned out to be not very helpful for JCG Car Rentals web frontends but let this fact not discourage you from giving XSStrike a try.
7. Vulas
Just a few weeks ago SAP had open-sourced the Vulnerability Assessment Tool (Vulas), composed from several independent microservices, that it has been used to perform 20K+ scans of more than 600+ Java development projects.
The open-source vulnerability assessment tool supports software development organizations in regards to the secure use of open-source components during application development. The tool analyzes Java and Python applications … – https://github.com/SAP/vulnerability-assessment-tool
The Vulas tool is targeting one of the OWASP Top 10 security threats, more specifically using components with known vulnerabilities. It is powered by vulnerability assessment knowledge base, also open-sourced by SAP, which basically aggregates public information about the security vulnerabilities in open source projects.
Once Vulas is deployed (using Docker is probably the easiest way to get up to speed) and vulnerabilities database is filled in, you may use Apache Maven plugin, Gradle plugin or just plain command line tooling to integrate the scanning into Java-based applications.
To illustrate how useful Vulas could be, let us take a look on the sample vulnerabilities discovered during the audit of the Customer Service microservice, one of key components of the JCG Car Rentals platform.
Vulas Workspace
Although the Vulas web UI is quite basic, the amount of the details presented along with each uncovered vulnerability is just amazing. Functionally, it is somewhat similar to the OWASP dependency-check we have talked about in the previous part of the tutorial.
8. Another Vulnerability Auditor
AVA, or Another Vulnerability Auditor in full, is a pretty recent open-source contribution from the Indeed security team.
AVA is a web scanner designed for use within automated systems. It accepts endpoints via HAR-formatted files and scans each request with a set of checks and auditors. The checks determine the vulnerabilities to check, such as Cross-Site Scripting or Open Redirect. The auditors determine the HTTP elements to audit, such as parameters or cookies. – https://github.com/indeedsecurity/ava
Similarly to the XSStrike, it is also Python-based and is quite easy to install. Let us use AVA to perform the XSS audit for JCG Admin Web Portal.
$ ava -a parameter -e xss vectors.har 2019-03-27 01:56:38Z : INFO : Loading vectors. 2019-03-27 01:56:38Z : INFO : Loading scanner. 2019-03-27 01:56:41Z : INFO : Found 0 issues in 0:00:02.
The results are promising, no issues have been discovered.
9. Orchestration
The tremendous popularity of the orchestration solutions and service meshes could give a false impression that you would get the secure infrastructure with zero efforts. In reality, there are a lot of things to take care of and the tools like kubeaudit from Shopify may be of great help here.
10. Cloud
Secure applications deployed into poorly secured environments may not get you too far. The things go even wilder by including the cloud computing into equation. How would you ensure that your configuration is hardened properly? How to catch the potential security flaws? And how to scale that across multiple cloud providers, when each one has own vision on cloud security?
Netflix has faced these challenges early on and made the contribution to the community by open-sourcing the Security Monkey project.
Security Monkey monitors your AWS and GCP accounts for policy changes and alerts on insecure configurations. Support is available for OpenStack public and private clouds. Security Monkey can also watch and monitor your GitHub organizations, teams, and repositories. – https://github.com/Netflix/security_monkey
There are also many other open-source projects for continuous auditing the cloud deployments, tailored for a specific cloud provider. Please make sure you are covered there.
11. Conclusions
In this section of the tutorial we have talked about security testing. The discussion revolved around three main subjects: static code analysis, auditing vulnerable components and scanning the instances of the web applications and APIs. This is great start but certainly not enough.
Complex distributed systems, like microservices, have a very wide surface area to attack. Hiring security experts and making them the part of your team could greatly reduce the risks of being hacked or unintentionally leak sensitive data.
One of the interesting initiatives with respect to Java ecosystem is the establishment of the Central Security Project to serve as one-stop place for the security community to report security issues found in open source Apache Maven components.
12. What’s next
This part wraps up the testing subject. In the next part of the tutorial we are going to switch over to continuous delivery and continuous integration.
Thanks for checking out our vulnerability-assessment-tool (aka Vulas)! Your are absolutely right about the GUI, we know it needs some rework, and we do hope that we get some help from the community! Let me just point out that while OwaspDC and Vulas do address the same problem, our approach uses the code itself as the ultimate source of truth (and not metadata, such as product names, CPEs, and the like) and therefore has much fewer false positives (and false negatives) than OwaspDC, whose approach is otherwise very cool (and possibly, complementary to ours). Also, Vulas offers reachability analysis, both… Read more »
Hi Antonino,
Thank you very much for the feedback and for the update. I truly believe Vulas has a great potential and it is certainly one of the tools to keep an eye on. I didn’t know about reachability analysis and mitigation support, thank you very much for highlighting that, one more reason to assess Vulas! Thank you!
Best Regards,
Andriy Redko