PCF Testing

Here is some information about our implementation of selenium core for automated testing of the miro guide.   I used selenium ide to record the tests with in the browser to make sure the steps were ordered properly and then edited as necessary.

I’d love your input and invite anyone would like to contribute test cases.

Selenium: The Selenium team at seleniumhq.org provides a great reference guide Selenium Reference, and some pretty good forums. However, there were still a few stumbling blocks that had to be overcome in getting everything up and running.

Setup Issues: The first was the cursed issue of cross-site scripting security.  Initially I took it to mean that selenium core and the page under test merely had to co-exist on the same server.  So I was trying to test testchannelguide2.participatoryculture.org with selenium.participatoryculture.org.  The first attempt at running a simple test failed,  and I spent a lot of time looking at errors in the forums that were similar to  what I was seeing, but lacked a resolution.  Do a search for selenium property Location.href and you’ll see I was not alone in my pain.  A closer read of the same origin policy cleared things up, and Matt the awesome admin solved the problem by soft-linking the selenium directory underneath the testchannelguide directory.

Test Issues: In setting up the test cases there are three main things I encountered, one is a bug within selenium, the other 2 are specific challenges presented by the guide.

1. Key Press 13 - is the command used simulate that the enter key has been pressed.  This is necessary when testing the miro guide, because we don’t have a submit button for the Search entry.  This works great on Firefox, however when running the Search in other browsers, I encountered and reported this selenium bug.
2. Feed Submission - must be unique.  When submitting a feed to the miro guide, we perform a check to verify that that feed has not already been submitted.  When repeatedly running a feed submission automated test, it’s challenging.  The solution was to use a feature for the blip.tv search api that allows you to specify the number of results to return.  By appending the pagelen argument to the feed with a randomly generated number, you can assure the uniqueness of the feed url.  Here’s the example in the submit feed step 1 test case.
3. Feed submission thumbnail - is an issue that isn’t yet solved.  The submit form requires that the user upload a thumbnail of the feed.  Since it isn’t possible to specify the files that exist on a machine, I think the solution will be to have a pause where the user will need to manually browse to the file.  It’s an ugly solution, and for now that is something that still needs to be run manually.

Test case management:  Currently we use litmus to manage our manual test cases for both miro and the guide.  And while there are selenium tools that will manage the results, it makes most sense to keep all the results in one location.  So the links to the test suites have been incorporated into litmus test cases.  When a user selects the test to run, it contains a link to open the suite in the users browser, the tester runs the test case and reports the result in litmus.  This has the additional benefit of keeping the test sequences small and makes it easier to track where the errors have occurred.

The tests: The tests are managed in our subversion repository.  If you would like to take a look at the actual test cases, you can browse them.  In order to run in the selenium test runner, test cases have to be included in a test suite, for example the Submit Feed Suite.  Instead of putting all the test cases into one giant suite, similar functions are grouped together, so for example, the Search Feed Suite will test a basic search as well as other various edge cases and input errors, for example: short search terms.