Getting started with the Applitools SDK

This blog assumes you have some basic knowledge about automated visual testing with Applitools. If it isn’t clear to you why it is awesome and would like more info, go checkout their site applitools.com.

I first heard about Applitools through one of my colleagues at Surge in spring of 2017. I was explaining some of the challenges with my current client. Specifically, the amount of time developers spend testing each release (developers are the QA department) and how there was an initiative to focus on CD (continuous delivery). I mentioned that while we have several functional tests, they did not verify that our application’s pages look like they should. This is when my Surge colleague suggested I look into Applitools.

Over the course of several “Improvement Days” my client holds, I was able to work on a POC with Applitools. The project I work on is a web application that consists of the Java Selenium automation environment using JUnit. It was pretty straight forward getting things setup via Applitool’s getting started page as you can see here: https://applitools.com/resources/tutorial

This was the sample code provided for my environment at the time of this writing:

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import com.applitools.eyes.selenium.Eyes;
import com.applitools.eyes.RectangleSize;

public class HelloWorld {

   public static void main(String[] args) {

       // Open a Chrome browser.
       WebDriver driver = new ChromeDriver();

       // Initialize the eyes SDK and set your private API key.
       Eyes eyes = new Eyes();
       eyes.setApiKey("YOUR_API_KEY");

       try{

           // Start the test and set the browser's viewport size to 800x600.
           eyes.open(driver, "Hello World!", "My first Selenium Java test!",
                   new RectangleSize(800, 600));

           // Navigate the browser to the "hello world!" web-site.
           driver.get("https://applitools.com/helloworld");

           // Visual checkpoint #1.
           eyes.checkWindow("Hello!");

           // Click the "Click me!" button.
           driver.findElement(By.tagName("button")).click();

           // Visual checkpoint #2.
           eyes.checkWindow("Click!");

           // End the test.
           eyes.close();

       } finally {

           // Close the browser.
           driver.quit();

           // If the test was aborted before eyes.close was called, ends the test as aborted.
           eyes.abortIfNotClosed();
       }

   }

}

Regardless of what your automation environment is the getting started code is similar across all environments. I followed this code and did get things working for a simple test. I was able to generate my first validation without any problems. It was easy but it did make me wonder how to generically apply this to over 300 existing functional tests.

For my POC I needed something more than just a simple test so I took an Abstract class that had about 50 tests that extend from it. I simply took the above code and added an @After method to quickly take a validation at the end of each. I did add a JUnit TestName rule as a generic way to add a test name for each test.

@Rule
  public TestName testName = new TestName();

  ...

  @After
  public void shutdown() throws Exception {
    // let's take a validation at the end of each test

    // Initialize the eyes SDK and set your private API key.
    Eyes eyes = new Eyes();
    eyes.setApiKey("YOUR_API_KEY");

    try{
       // Start the test and set the browser's viewport size to 800x600.
       eyes.open(driver, "Applitools POC", this.getClass().getName() + "." + testName.getMethodName(), new RectangleSize(800, 600));

       eyes.checkWindow();

       // End the test.
       eyes.close();

     } finally {
       // Close the browser.
       driver.quit();

       // If the test was aborted before eyes.close was called, ends the test as aborted.
       eyes.abortIfNotClosed();
     }
}

With this in place I was able to reproduce some recent UI bugs we had and verify that those failures showed up in Applitools which they did. After presenting the POC we got the approval to go ahead with the purchase of Applitools.

Before the POC was production-worthy there were a number of things to figure out:

• • How to make the Applitools SDK integration code reusable?
  • What should we use for a batch strategy with CI (we use bamboo)?
  • How does this work on developers’ local environments?
  • How are batches handled in developer environments
  • Add a few validations to our tests (using the above changes)

Do developers maintain their own baselines? Or just let CI handle it? (Baselines are tied to environments – os + browser + view port)

• How do we get rid of phantomJS (no longer supported) and use real browsers. If we are testing the UI, let’s get cross-browser testing to work! (This was actually part of the POC but is a much larger topic and will be covered in part 2 of this publication)
• How does Applitools work with our pull requests (branching and merging)

We want to be able to run one or more validations for each of our tests, so it makes sense to create a Junit TestWatcher rule for this.

public class EyesWatcher extends TestWatcher {}

We’re going to need a couple of member variables. First we need an Eyes object. The eyes object is the main object in the Applitools SDK. Here we will also initialize it and make it public to make it available to clients that might need it for more advanced scenarios where the TestWatcher alone isn’t enough.

public Eyes eyes = new Eyes();
//For premise and private cloud clients using your own server
public Eyes eyes = new Eyes(new URI("https://privateeyes.applitools.com"));
/*
 * For more details on premise and private cloud setup see http://support.applitools.com/customer/en/portal/articles/2266273-how-to-set-your-test-to-run-with-dedicated-server
 */

Next, we will add a String to capture the test name. Each test in Applitools must be given a name. Using the TestWatcher we can capture the actual test name of the method being tested. This can really be anything you want but as you will see later we will derive the test name from the class and method that is being tested.


private String testName;


Next, we need a BatchInfo object. This is used for grouping the results of each test run.


private static BatchInfo batch;


Next, we define a constant for storing our applitools key which we pull in from a system property. Note: from version 3.32 of the Java SDK there is no need to set the api key via code if you have it set with the environment variable APPLITOOLS_API_KEY, but for our example we will use a system property.


private static final String APPLITOOLS_KEY = System.getProperty("applitoolsKey");


Lastly, we define a constant for storing our application name which we pull in from a system property with a sensible default.


private static final String APPLICATION_NAME = System.getProperty("applicationName", "Applitools Test App");


Now that we have our member variables defined we can get things ready for primetime. This includes setting up the batch info object and configuring the eyes object among other things:

private void initializeApplitools() {
    String localBranchName = System.getProperty("branchName", System.getenv("bamboo_planRepository_branchName"));
    eyes.setIsDisabled(APPLITOOLS_KEY == null);

    if (!eyes.getIsDisabled() && localBranchName != null) {
      if (batch == null) {
        String buildNumber = System.getenv("bamboo_buildNumber");
        batch = new BatchInfo(localBranchName + (buildNumber != null ? " #" + buildNumber : ""));

        // Aggregates tests under the same batch when tests are run in different processes (e.g. split tests in bamboo).
        if (buildNumber != null) {
          batch.setId(batch.getName());
        }
      }

      eyes.setApiKey(APPLITOOLS_KEY);
      eyes.setBatch(batch);

      eyes.setBranchName(localBranchName);

      // set the default parent branch to master if the parent branch is not specified
      eyes.setParentBranchName(System.getProperty("parentBranchName", "master"));

      eyes.setIgnoreCaret(true);
    }
  }

As you can see in line #2 I’m using bamboo for CI. This line will attempt to pull in the branchName system property if it is defined. If not, it will use the branch name from bamboo. The branchName system property is used for running Applitools in local development environments. While this example uses bamboo it would be very easy to substitute accordingly for other CI environments.

Line 3 simply uses the APPLITOOLS_KEY to determine whether or not visual testing is enabled. Visual testing is always enabled for CI but is optional for local development.

Line 5 is a simple check to determine if visual testing is enabled (both the applitools key and branch name are required).

Lines 6–13 setup the BatchInfo object. This is very important when you have several tests. In most cases simply creating a new BatchInfo object with a meaningful name and calling eyes.setBatch() with it is usually enough. However, we have over 300 functional tests which can take some time to run. So to speed things up we run our tests in parallel using separate processes/JVMs (per test cycle). In this case simply setting the batch is not enough. To link the batch for each process together we need to set the batchId to the same value which should be unique across all processes/JVMs. Our case use the branch name + build number which is sufficient.

Here’s an example of the Batch View Summary:

Line 16 where we set the api key.

Lines 19–22 setup the branch name and parent branch name for Applitools. Applitools needs to know about these in order properly merge baseline validations. This topic can be rather complicated and is out of scope for what I want to cover. I may followup on this topic with a separate blog but for now I recommend reading this:

http://support.applitools.com/customer/portal/articles/2142886

Line 24 tells Applitools to ignore the cursor when doing validations. (Random validation failures can occur when a cursor is blinking in a text box).

Now we can tap into the methods provided by TestWatcher. Specifically, the starting and finished methods.

@Override
  protected void starting(Description description) {
    if (!eyes.getIsDisabled() && eyes.getBatch() == null) {
      throw new IllegalArgumentException(
          "The branchName parameter or the Bamboo environment variables are required if visual testing is enabled " +
              "(the applitoolsKey property is provided).");
    }
    testName = description.getTestClass().getSimpleName() + "." + description.getMethodName();
  }

The starting method is called just before a test is about to start. Here we perform checks to ensure Applitools is configured properly. This is also where we can set the testName. This can be whatever you like but for my project it just made sense to derive the name from the class and method name.

@Override
  protected void finished(Description description) {
    try {
      // End visual testing. Validate visual correctness.
      if (eyes.getIsOpen()) {
        eyes.close(true);
      }
    }
    finally {
      testName = null;
      // Abort test in case of an unexpected error.
      eyes.abortIfNotClosed();
    }
  }

The finished method is started at the end of every test. Here we close the Applitools session (line 6) assuming one was created. Note, the true parameter (which is the default) will raise an exception if Eyes finds any mismatches (Same behavior test frameworks handle assertion failures). A false parameter passed to the close method will return a TestResults object with the result information. Passing false will also not throw an exception if there are a visual differences. For more details see:

http://support.applitools.com/customer/en/portal/articles/2191066-tests-results-advanced-information

We also set the testName to null and do some additional clean up in the even of an unexpected error.

All that is left now is to provide a few convenience methods for our tests to perform validations.

public void eyesCheck() {
    eyesCheck(null);
  }

  /**
   * Convenience method for performing the applitools validation.
   *
   * @param tag or step name of the validation
   */
  public void eyesCheck(String tag) {
    if (!eyes.getIsOpen()) {
      WebDriver remoteDriver = WebDriverRunner.getAndCheckWebDriver();

      if (remoteDriver instanceof WrapsDriver) {
        remoteDriver = ((WrapsDriver) remoteDriver).getWrappedDriver();
      }
      /*
       * Rather than call eyes.open with a specified viewport, just use the current viewport size.
       * In our example we will simply call Eyes.setViewportSize(driver, new RectangleSize(width, height));
       * in our @Before method. The problem with specifying it here is that it's possible (depending on the app)
       * for instabilities to occur as we are resizing the viewport right before our first validation is taken.
       * (ui elements can shift around, be reresized, etc).
       */
      eyes.open(remoteDriver, APPLICATION_NAME, testName);
      //eyes.open(remoteDriver, APPLICATION_NAME, testName, new RectangleSize(width, height));
    }
    eyes.check(tag, Target.window());
  }

Obviously, the important stuff is in the eyesCheck(String tag) method. Here initialize the eyes object by calling eyes.open(…). As you can see, the example uses Selenium/Selenide and we need to get the remote driver which is needed by Applitools. This block (11–26) will only be executed if the eyes object hasn’t been opened. Essentially this will be executed the first time you make a validation for every test. In Applitools you can have multiple validations (steps) for each test so for subsequent validations these lines will be skipped.

eyes.check(tag, Target.window()) does the actual validation. This is a great place to add ignore regions or other customizations for every validation. For example, in my project we had an ignore region on the version number which is on every page. e.g. eyes.check(tag, Target.window().ignore(By.cssSelector(“.version”));

That’s it! Now all you need to do is add this to your test.

@Rule
public EyesWatcher eyesWatcher = new EyesWatcher();

Now your tests just call eyesWatcher.eyesCheck() or eyesWatcher.eyesCheck(“Some Description”) and that’s it! Now go to your dashboard at eyes.applitools.com and see the results:

The above code along with an example can be found here:

https://github.com/surgeforward/applitools

Revisiting the list of things to figure out we got most of them done:

• • How to make the reusable?
  • What should we use for a batch strategy with CI (we use bamboo and run tests in parallel)?
  • How does this work on developers’ local environments?
  • How are batches handled in developer environments
  • Add a few validations to our tests (using the above changes)

Do developers maintain their own baselines? Or just let CI handle it? (Baselines are tied to environments — os + browser + view port)

• How do we get rid of phantomJS (no longer supported) and use real browsers. If we are testing the UI, let’s get cross-browser testing to work. (This was actually part of the POC but is a much larger topic and will be covered in part 2)
• How does Applitools work with our pull requests (branching and merging) — We did setup the pieces required for this to work