Quality Assurance Testing with TestSwarm and QUnit

From HTML5 Video
Jump to: navigation, search



Kaltura's HTML5 Media Library is continuously tested upon each commit to the git repository.

Testing is performed on Internet Explorer 7 through 9, Firefox 3.6 and the latest Firefox, the latest Google Chrome, Android 2.x, 3.x, and 4.x, the latest Safari, latest iPhone, and the latest iPad running on a private virtualization cluster.

Each revision to the library triggers a job submission to our TestSwarm server. Test results can be viewed at http://html5video.org/testswarm/user/html5video .

Running Tests

TestSwarm is a simple php application that anyone may visit with their browser and their browser will be added to the testing swarm, running any tests in the testing queue.

To add your browser to the testswarm, visit http://html5video.org/testswarm/run/html5video .

Once your browser loads the page, TestSwarm will continually run tests in an iFrame testrunner until the browser is closed.

Testswarm Running Tests.png

Interpreting Test Results

The Kaltura HTML5 Video Library's test results are stored at http://html5video.org/testswarm/user/html5video/ where testswarm provides a way to drill into test results and view errors and passes from test runs.

To open a specific test job, click on the Name of the job:

Testswarm Recent Jobs.png

And this will open the job report for this commit.

Testswarm Job Report.png

Clicking on the svn revision number will take you to the ChiliProject repository browser where you may view the difference to the library that this revision contains. Clicking on the Name of a test will run the test in your browser, and clicking on a green, red, or black square will take you to the result page for specific browser and test combinations:

Here we can see the result of a failed autoplay test on iOS 3.6

Writing Tests

Testing with QUnit

QUnit is a testing framework for JavaScript that jQuery uses to perform regression and unit testing on the jQuery library.

From the QUnit website:

QUnit is a powerful, easy-to-use, JavaScript test suite. It's used by the jQuery project to test its code and plugins but is capable of testing any generic JavaScript code (and even capable of testing JavaScript code on the server-side).

QUnit runs any JavaScript and adds assertions that will pass or fail according to weather the code being tested operates as expected.

The following is a simple example with two tests.

test("a basic test example", function() {
        ok( true, "this test is fine" );
        var value = "hello";
        equal( value, "hello", "We expect value to be hello" );

Here is the full source code for a web page that includes the previous tests and the appropriate javascript includes:

<!DOCTYPE html>
        <script src="http://code.jquery.com/jquery-latest.js"></script>
        <link rel="stylesheet" href="http://code.jquery.com/qunit/git/qunit.css" type="text/css" media="screen" />
        <script type="text/javascript" src="http://code.jquery.com/qunit/git/qunit.js"></script>
                        test("a basic test example", function() {
                                ok( true, "this test is fine" );
                                var value = "hello";
                                equal( value, "hello", "We expect value to be hello" );
        <h1 id="qunit-header">QUnit example</h1>
        <h2 id="qunit-banner"></h2>
        <div id="qunit-testrunner-toolbar"></div>
        <h2 id="qunit-userAgent"></h2>
        <ol id="qunit-tests"></ol>
        <div id="qunit-fixture">test markup, will be hidden</div>

Running this test will result in the following:

Testswarm Simple Test.png

Automatic Test Selection

Each plugin that is stored in the svn repository lives in its own folder within the "modules" folder like http://www.kaltura.org/kalorg/html5video/trunk/mwEmbed/modules/KalturaSupport

Within the plugin folder there is a folder called tests that contains html files. If the file ends with .qunit.html then that file will be added to the testing queue upon each update to the svn repository.

Kaltura QUnit Implementation

Kaltura has built upon QUnit to test its HTML5 video library and made the creation of tests very straightforward. Merely include the qunit-bootstrap.js file and the mwEmbedLoader.php in your demonstration or testing page, and then add your tests in a JavaScript <script> block in a function called jsKalturaPlayerTest. The final step is adding the qunit-kaltura-bootstrap.js below any mw.setConfig and jsCallbackReady directives necessary to setup players and plugins for testing.

<script type="text/javascript" src="../../../tests/qunit/qunit-bootstrap.js"></script>
<script type="text/javascript" src="../../../mwEmbedLoader.php?debug=true"></script>
<script type="text/javascript">	
function jsKalturaPlayerTest( videoId ){
	var $iframe = $('#' + videoId + '_ifp').contents();
	var iframeContext = window.frames['kaltura_player_ifp'];
	test( "Iframe in DOM", function() {
		ok( $('#' + videoId + '_ifp').length , 'iframe found' );
		equal( typeof iframeContext, 'object', "iframe JavaScript context available" );
<!-- qunit-kaltura must come after qunit-bootstrap.js and after mwEmbedLoader.php and after any jsCallbackReady stuff-->
<script type="text/javascript" src="resources/qunit-kaltura-bootstrap.js"></script>

Please view the source for the Basic Player Test and the tests folder for further examples.