Skip to main content

FusionCharts Automated Testing Tool

While working at FusionCharts, every release we were faced with a daunting task of black box testing. Daunting mainly because of the breadth and the depth of the product. This post talks only about black box testing. FusionCharts Product as a whole contains about roughly 90 charts and each chart can be visually tweaked with a set of about 300 chart options/attributes. Ignoring any further permutation and combination, we have right there around 90*300 test cases. Apart from chart options, api testing which consisted of events and methods was needed. Clearly automation was required, as our manual testing would cover only a very small sample set that too based on smart assumptions.

With this problem at hand, I broke it down as follows:
  1. Headless Browser
    - Visual Regression
    - API Testing
  2. User Browser
    - Visual Regression
    - API Testing
We liked to call it the FusionCharts Automated Testing Suite.

Headless testing would be based of a headless browser and integrated in to our nightly builds. It was aimed mainly to be capable of running on individual developer machines as and when needed. This was targeted to be a quick smoke test giving an idea if something major had been broken.


User Browser based testing would require a more comprehensive server deployment. This part would spawn all the different major user facing browsers (chrome, firefox, ie) and emulate tests on them. Give a more in depth report on the state of the entire product. Meant to be run before tagging a customer facing release.


Visual Regression testing would aim at the visual changes that the product undergoes. This would be based of a version which would be assumed as a stable product release. Any variation from the baseline version would be a tagged as a failure and then manually triaged as an intended change or bug.


API Testing would listen to different events or trigger various methods exposed by the product. It would compare the output to the expected value and mark it as a fail or pass accordingly.

The Process and Stack

- https://nodejs.org/ and npm for building a cross platform CLI tool and pack management.

Visual Regression 


  1. Load the input spec for the test case.
  2. Generate the web page with the baseline version of the product.
  3. Host the web page on a local server.
  4. Load the page on the browser.
  5. Take a screenshot and save the image.
  6. Swap the baseline version of the product with the test version on the web page.
  7. Load the page on the browser.
  8. Take another screenshot and save the image.
  9. Compare the two images.
  10. Mark the test case as pass or fail depending on the image comparison.
  11. Generate a result report as json and html.
  12. The html report would contain the baseline image, test image and the diff image of the two.

Input spec / test case - Custom JSON Format

Server - Express http://expressjs.com/

Headless Browser -  PhantomJS http://phantomjs.org/

Launching and Controlling User Browsers - Selenium Webdriver Bindings http://webdriver.io/

Image Comparison - Resemble.js http://huddle.github.io/Resemble.js/

Result Reporting - ejs http://www.embeddedjs.com/



API Testing

  1. Load the input spec for the test case.
  2. Generate the web page with the test version of the product.
  3. Host the web page on a local server.
  4. Load the page on the browser.
  5. Trigger events or call the intended api method.
  6. Listen to the events or resulting change from the api method.
  7. Compare the received result with the expected result.
  8. Mark the test case as pass or fail depending on the comparison.
  9. Generate a result report json.
  10. Create an html report appended to the larger testing report which includes visual regression.
Input spec / test case - Custom JSON Format

Server - Express http://expressjs.com/

Headless Browser -  PhantomJS http://phantomjs.org/

Launching and Controlling User Browsers - Selenium Webdriver Bindings http://webdriver.io/

Listening to Events and API Methods on Browser - socket.io http://socket.io/

Result Reporting - ejs http://www.embeddedjs.com/










Comments

Post a Comment

Popular posts from this blog

Nouveau - Summer Project

Implementing a software scripting engine on Fermi to achieve safe memory re-clocking. Fermi stands for Nvidia GPUs based on Fermi architecture. NVidia cards have long had the possibility to reclock at least some of the engines of its GPUs. Up to the geforce 7 (included), reclocking used to happen at boot time and usually didn't involve memory reclocking at all. It changed with geforce 8 (nv50) where almost all laptops got the capability to reclock both the VRAM and the main engines. This was introduced in order to lower power consumption when the GPU was mostly idle. The default boot clocks were usually in some intermediate state between the slowest and the fastest clocks. The reclocking process for these cards is mostly understood and Nouveau is not far from being safely reclock on the fly, even while gaming. Geforce 200 (nva3) introduced load-based reclocking on all the cards. This started being a real problem because the default boot clocks are a third to a half of the ...
4 comments
Read more

Getting Started with Open Source Projects

Fellow Open-source Enthusiasts Going to conferences is great , getting inspired by attending them is even better but you know what is better? Getting to attend conferences [National/International] for free , getting cool stickers and t-shirts being delivered at your place , being one of the first few to get your hands on a developer device or developer release of an upcoming software/OS and all this for free! I will hopefully be pardoned by true open-source enthusiasts for making this look like an advertisement rather than a motivator but I somehow couldn't resist.  Now , ignoring the above perks because they are 'perks' , the real deal is getting to learn while you work. Being involved with any open-source project brings along a great deal of learning , experience and contribution to real-time projects which people around the globe use in their daily lives. Once you are into development and learning , conferenc...
3 comments
Read more

Beginner's Guide to KDE Development

  Google Summer of Code Doc Camp Sprint   October, 2011     The Doc Camp was held at the Google HQ in Mountain View, California ; it consisted of 2 major components - an unconference and 3-5 short form Book Sprints to produce 'Quick Start' guides for specific GSoC projects. The unconference explored topics proposed by the participants. Many topics on free documentation of free software were proposed for discussion during the event. Each Quick Start Sprint brought together 5-8 individuals to produce a book on a specific GSoC project. The Quick Start books were launched at the opening party for the GSoC Mentors summit immediately following the event.     Four free software projects (KDE, OpenStreetMap, OpenMRS, and Sahana Eden) had each sent three to five contributors to write books about the projects this week. The KDE team consisted of four members , namely Karan , Rohan , Valorie and yours truly. A book in a week was an amazing achievem...
Post a Comment
Read more