Writing and maintaining the test automation code is not always a piece of cake. As a matter of fact, we frequently face many scenarios where automated test cases don’t work as expected and might lead to false positive or false negative results, in such cases, debugging is the only way out. Debugging is the primary skill set that an automation tester must adopt. It increases the morale and confidence in automation testers to provide a better code solution to fix the problem permanently.
Debugging issues in the test automation framework becomes more complicated when the test automation framework has a huge number of test cases. With the expansion of features in the application, the number of test cases gradually increases. In such a scenario, fixing complex issues of hybrid frameworks might require enhanced techniques of debugging. In this article, we will deep dive into such essential debugging tecniques that will not only fix script issues easily but also save a good amount of debugging time.
What is Debugging?
In simple terms, debugging is a process of software engineering to identify and resolve an error in the source code by adopting various debugging techniques. Debugging is basically divided into four common stages:
- Identification: The first step of debugging is to discover the issue and try reproducing the same in the local system to know why the issue occurred. This is an important step as we need to identify the root cause of the issue to deploy a permanent solution.
- Isolation: Is the second step where the idea is to separate the buggy code from the healthier code. The unit testing of buggy code is required to identify the steps that need to be performed to fix the issue. Isolation of buggy code would further reduce time and not affect the other code.
- Resolution: This is the primary step towards fixing the buggy code. This stage is dependent on the above two mentioned stages, the resolution can be deployed as soon as the above two stages are completed. Few tips for fixing the code:
- Deep dive into the code and libraries being used to get an understanding about the working of the framework
- Refer the proper document and solutions on StackOverflow
- Execute the code in debug mode
- Perform code walkthrough and unit testing in the local system
- Refactor or re-design the framework architecture in the worst case scenario
- Review: This is the final stage of debugging that developers usually try to skip. Reviewing is done to ensure that the fix deployed is working fine and not hampering the other code. Ideally, the review should be done by both i.e the developer who actually fixed the buggy code and by the other developer who is responsible for reviewing and giving a green signal for code merge.
Various Techniques Of Debugging Test Automation Scripts
The number of automated test cases does not always work smoothly, as a matter of fact, the change in UI of an application or change in business logic of an application can ruin the execution of Selenium based test scripts. The reason for test script failure is not always a buggy code in the test automation framework, but can also fail due to the bug found in the application itself. Hence, the proper use of assertions is mandatory to identify the basic reason for test case failure before moving to an advanced level of debugging.
Whenever we are developing the Selenium Test Automation framework from scratch, it is important to have utilities imported to help improve the debugging of source code. At the time of failures, it becomes very challenging to debug the buggy code with the usual java console output. To overcome such challenges, let’s have a look at the debugging techniques that can make developers life easy:
1. Debugging With Logging Utilities
Logging utilities help developers to output a variety of logs as per the target. In case of issues in the test scripts, the logging can be enabled to identify the exact location of buggy code.
Let’s consider an example of Log4j logging framework. This is a Java based open source framework and can be used for logging purposes by just importing the maven dependency.
Now let’s have a quick look at the different component of Log4j:
- Loggers : It contains all the information about the logging level. Loggers offer different severity levels for logs. The initial step is to create an object of Logger class:
Logger logger = Logger.getLogger(“ClassName”);
There are 5 kinds of severity log levels that Logger provides:
- All
- Debug
- Info
- Warn
- Error
- Fatal
- Off
- Appenders : The logs that have been generated with the above mentioned types of severity log levels have to be pushed to the destination to view the logs, this role is performed by appenders. It sends the log events to the destination folder or prints on console as per the configuration done. There are three types of appenders available to output the logs:
- ConsoleAppender
- FileAppender
- Rolling File Appender
- Layouts : Layouts provide different methods for formatting of the logs. The logging information can be formatted in different formats with the below provided methods:
- Public static Logger getRootLogger()
- Public static Logger getLogger (String name)
2. Capturing Screenshots
Usually when we execute our regression or smoke test suite, we don’t observe the execution for long hours. Hence, we would always want to know where exactly the test case failed so that we can take the necessary steps towards fixing the issue. The idea that can be adopted to fulfill such a case is to capture a screenshot of the webpage at the moment where the test script failed. And later, as a part of debugging, looking at the screenshots we can easily identify where our test method failed.
Selenium provides an interface TakesScreenshot that can be used to capture screenshots of web pages. To get the execution result/status of the test method, we can use the ITestResult interface.
import java.io.File;
import org.apache.commons.io.FileUtils;
import org.openqa.selenium.By;
import org.openqa.selenium.OutputType;
import org.openqa.selenium.TakesScreenshot;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.chrome.ChromeDriver;
import org.testng.Assert;
import org.testng.ITestResult;
import org.testng.annotations.AfterClass;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Test;
import io.github.bonigarcia.wdm.WebDriverManager;
public class TestScript {
private WebDriver driver;
public String expectedTitle = "This is a wrong title";
@BeforeClass
public void setup() {
WebDriverManager.chromedriver().setup();
driver = new ChromeDriver();
}
@Test
public void verifyLoginPage() {
driver.get("https://www.pcloudy.com/");
WebElement loginButton =
driver.findElement(By.xpath("//a[text()='Login']"));
loginButton.click();
String actualTitle = driver.getTitle();
Assert.assertEquals(actualTitle, expectedTitle,
"Login Page Title didn't matched with the expected title");
}
@AfterMethod
public void screenShot(ITestResult result) {
if (ITestResult.FAILURE == result.getStatus()) {
try {
TakesScreenshot screenshot =
(TakesScreenshot) driver;
File src =
screenshot.getScreenshotAs(OutputType.FILE);
FileUtils.copyFile(src, new
File("/home/ramit/Pictures/" + result.getName() + ".png"));
System.out.println("Screenshot captured of
failed test case");
} catch (Exception e) {
System.out.println("Exception
occured while taking screenshot " + e.getMessage());
}
}
}
@AfterClass
public void tearDown() {
driver.quit();
}
}3. Session Recording
This is another advanced way of debugging. As the execution time of a regression suite is too long, it is difficult to sit tight and observe the entire execution of the constantly failing test cases. In such cases we can enable the session recording and save it for future debugging purposes. Many times, the test cases failing in the regression suite aren’t reproducible when executed as a single class, in such scenarios, test session recordings are the best way out. With this, we can easily visualize and validate the test actions and ensure that no unexpected alerts and elements are popping up.
Session recording is an enhanced level debugging comparative to logs or screenshot capturing but it is probably useful when the test suite contains a huge number of test cases. With the session recording, we can also get to know about the server performance and UI usability, and can also be shared with developers in case the bugs are found in the production/staging environment to replicate the bug easily.
Since there is no direct support from Selenium to record test sessions, you may use a third party tool like Monte Media Library. If you want to skip writing the code to record selenium sessions, you can also install third party software in your physical machine for screen recording like SimpleScreenRecorder, this also provides a feature to schedule the recording.
4. Adding Breakpoints
Breakpoints are a part of IDE that can temporarily halt the execution of the code. Once the execution gets paused at the breakpoint, we can acquire data of the essential elements in the source code. The debugging with breakpoints can be done easily with the below sequence:
- Set up the breakpoints where buggy code is observed
- Execute the source code in debug mode
- Validate the data returned in debugger
- Resume the debugging if multiple breakpoints are added
- Stop the debug mode execution
- Fix error and deploy the code
The above is a screenshot of the Intellij IDE in which 4 breakpoints are added. The checkmarks appear on the added breakpoints at run-time as soon as the breakpoint is recognized by the debugger. In case multiple breakpoints are added in the code, the execution can be resumed by pressing F9 to acquire data from different breakpoints.
Debugging with breakpoints leverages automation testers to fix the issues in interactive mode.
5. Debugging Selenium Tests On Already Opened Browser
Many times it so happens that only the last few steps of the test method are observed as constant failures. In such cases, we can fix the buggy code and execute the entire test again. This consumes a lot of execution time and till then we have to sit tight to observe whether the updated code at the end of the test method is working fine or not.
To overcome this situation, we can debug selenium based test scripts on an already opened browser. Firstly, we can launch the browser manually and perform the web actions manually that works fine with the test script and then execute only the buggy code or fixed/updated code. This would save time in executing healthier code and will only execute the code that needs to be verified.
To achieve this use case, we would be using Chrome DevTools Protocol that allows clients to inspect and debug chrome browsers. To launch chrome on remote debugging port, we need to run below command for Linux:
google-chrome –remote-debugging-port=9222 –user-data-dir=<some directory path>
- For remote debugging port, you can specify any open port.
- For the user data directory, you need to specify the directory where the new chrome profile will be created.
Once you run the above mentioned command on the terminal, the fresh chrome browser should get launched.
Example :
Considering a test of the pCloudy login page, let’s open the pCloudy website manually on this debugging chrome browser and then let’s execute the below script from the next required step.
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
import org.testng.Assert;
import org.testng.annotations.AfterClass;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Test;
import io.github.bonigarcia.wdm.WebDriverManager;
public class TestScript {
private WebDriver driver;
public String expectedTitle = "Remote Mobile Web &
Application Testing on Real Android Devices - pCloudy";
@BeforeClass
public void setup()
{
WebDriverManager.chromedriver().setup();
ChromeOptions opt = new ChromeOptions();
opt.setExperimentalOption("debuggerAddress",
"localhost:9222");
driver = new ChromeDriver(opt);
}
@Test
public void verifyLoginPage()
{
//driver.get("https://www.pcloudy.com/");
WebElement loginButton =
driver.findElement(By.xpath("//a[text()='Login']"));
loginButton.click();
driver.findElement(By.id("userId")).sendKeys("ramit.dhamija@gmail.com");
String actualTitle = driver.getTitle();
Assert.assertEquals(actualTitle,
expectedTitle,"Login Page Title didn't matched with the expected
title");
}
@AfterClass
public void tearDown()
{
driver.quit();
}
}Code Walkthrough:
Once we have opened the pCloudy website manually on debugging chrome browser, we have then executed the above test script to continue from the same opened browser.
To set up the test on an already opened browser, we have used ChromeOptions setExperimentalOption method to set the experimental option i.e. debuggerAddress. The debugger address would be the same address on which the debugging chrome is launched.
Conclusion
No matter how careful we are in writing the test cases, there is always a chance that it will fail by either producing false positives or false negatives. Using the right technique to debug these failed test cases becomes a crucial step to leverage automation test scripts. Here we have discussed in detail some of the techniques you can use when debugging Selenium based Test scripts. Learning to write test scripts is vital but knowing how to debug them when it fails is also a much sought after skill. We hope these few tips will help you debug test scripts better and effectively use selenium automation.
Published on Java Code Geeks with permission by Ramit Dhamija, partner at our JCG program. See the original article here: Different Techniques Of Debugging Selenium Based Test Scripts Opinions expressed by Java Code Geeks contributors are their own. |
