HTML5 introduces a great new element that can be used to draw arbitrary content on a pane: the canvas element. What has been a standard feature for fat client applications for decades is now introduced to the world of web applications. Web developers no longer need to use proprietary plugins to draw images or charts in their applications.
But when it comes for testing, this new feature imposes new challenges to the web development community. How to test that the canvas element is in an appropriate state at some point in time? Standard technologies like selenium focus on the markup that is generated by the web server and not on the pixels drawn on the canvas.
More promising in this field are technologies that use image processing to verify that an application renders its data correctly. One of these frameworks is sikuli. Sikuli is an open-source research project that was started at the MIT and is now maintained by Raimund Hocke.
To give a more practical introduction, let’s assume we have a simple web application that uses the HTML5 canvas element to implement some simple image processing functionality like a grayscale, a brighten and a threshold filter as well as an undo button (the code for this application can be found as usual on github):
The installation of sikuli is (of course) platform dependent. The installer, which can be downloaded from the sikuli download page, is a Java Swing application that asks you about your typical usage pattern. As we do not want to use the python IDE, we choose option 4 from the list of options. The actual jar file is then downloaded and prepared for our OS. After the installation process has finished, we find an OS dependent jar file within the installation directory. As our example project uses maven as build system, we have to introduce a system scope dependency after we have copied the library into the lib folder:
<dependency>
<groupId>org.sikuli</groupId>
<artifactId>sikuli</artifactId>
<version>1.0</version>
<scope>system</scope>
<systemPath>${basedir}/lib/sikuli-java.jar</systemPath>
</dependency>When sikuli is used for the first time, it extracts some native libraries into a new folder, here in our example into ${basedir}/lib/libs. This folder has to be added to the user’s path environment variable.
Now that we have installed sikuli, let’s setup arquillian so that we can write our first unit test. How to setup arquillian is described for example here. As I don’t want to repeat everything, in the following you will find only the unit test class:
@RunWith(Arquillian.class)
public class FilterTest {
public static final String WEBAPP_SRC = "src/main/webapp";
@ArquillianResource
URL deploymentURL;
private Screen screen;
@Before
public void before() throws URISyntaxException, IOException {
screen = new Screen();
if (Desktop.isDesktopSupported()) {
Desktop.getDesktop().browse(deploymentURL.toURI());
} else {
fail();
}
}
@Deployment
public static WebArchive createDeployment() {
return ShrinkWrap.create(WebArchive.class, "html5-sikuli-webapp.war")
.addClasses(HomeBackingBean.class)
.addAsWebResource(new File(WEBAPP_SRC, "home.xhtml"))
.addAsWebResource(new File(WEBAPP_SRC, "resources/css/style.css"), "resources/css/style.css")
.addAsWebResource(new File(WEBAPP_SRC, "resources/images/rom.jpg"), "resources/images/rom.jpg")
.addAsWebResource(new File(WEBAPP_SRC, "resources/js/html5Sikuli.js"), "resources/js/html5Sikuli.js")
.addAsWebResource(new File(WEBAPP_SRC, "resources/js/jquery-2.0.3.js"), "resources/js/jquery-2.0.3.js")
.addAsWebInfResource(EmptyAsset.INSTANCE, "beans.xml")
.setWebXML(new File(WEBAPP_SRC, "WEB-INF/web.xml"));
}The createDeployment() method sets up the war archive, which is deployed by arquillian to JBoss AS 7.1.1.Final (see arquillian.xml file). In our @Before method we use the SDK class Desktop to open the default browser and point it to the deployment URL. Here we also create an instance of sikuli’s class Screen. This class provides all methods needed to perform the interaction with our application. Let’s look at this in more detail:
@Test
@RunAsClient
public void testGrayScale() throws FindFailed {
screen.wait(getFullPath("originalImage.png"));
screen.find(getFullPath("btnUndo_disabled.png"));
screen.click(getFullPath("btnGrayscale.png"));
screen.find(getPattern("grayscaleImage.png", 0.9f));
screen.click(getFullPath("btnUndo_enabled.png"));
screen.click(getPattern("originalImage.png", 0.9f));
}
private Pattern getPattern(String path, float similarity) {
Pattern p = new Pattern(getFullPath(path));
return p.similar(similarity);
}
private String getFullPath(String path) {
return "src/test/resources/img/" + path;
}As sikuli is based on image processing, we can define where to click and what to verify with screenshots we have taken before. In this simple example I have stored all screenshots as png files within the src/test/resources/img folder of our project. More advanced projects may need a more sophisticated folder hierarchy. As you can see, we first wait for the application to show up. Once sikuli has found the first screenshot, we verify that the button “Undo” is disabled. This is done by calling the method find() with an image of the disabled button. Now we can click on the button “Grayscale” (again specified by an image of the button) and then verify that the grayscale version of the image is found on the screen.
Sikuli does not only compare both images pixel by pixel, but if you like it computes the similarity of the found screen region with the requested region. This helps when you need to be more tolerant (e.g. if you want to test the application in different browsers and these render the buttons slightly different). The default value for the similarity attribute is 0.7f, but if you increase it to 1.0f you have a simple pixel by pixel comparison.
But this is not all. With sikuli you can do nearly all things you could do as human interactor:
- Type characters using screen.type()
- Double click with screen.doubleClick()
- Perform drag and drop operations with screen.dragDrop()
- Use the mouse wheel
- …
Conclusion
Sikuli is a powerful and easy to use tool to perform integration tests for web applications that heavily rely on HTML5′s canvas object. The same is of course true for standard fat client applications (Swing, JavaFX). Together with arquillian you can setup comprehensive test suites that cover a lot of “real” use cases.
