UnitTestSuites

Unit Test Suites

Unit test suites are conformance tests written as part of a Test Driven Development cycle in a software project. Those tests make sure that solved bugs don't regress by checking the results of the application under test against an expected value. This way unit test software be used to do automatic checks of an implementation for conformance against its specification.

Unit testing is a valuable software quality assurance method. This page gives an overview of testing Wine with unit testing.

Wine's conformance test suite

Since 2003 or so, Wine has had a conformance test suite to ensure that bugs don't creep into Wine's source tree, and to make sure Wine's behaviour matches MS Windows's behaviour (in both 32-bit and 64-bit). (There are also Win16 conformance tests at http://win16test.googlecode.com that should get merged into WineHQ's source tree someday.)

The ConformanceTests check the behavior of not only the publicly available Windows API, but the real binary behavior of Windows, including bugs and undocumented things. The tests allow for black-box reverse engineering.

Wine's conformance test suite is great, but it's far from complete. We improve it by adding tests as we add new features or fix bugs.

Using the test suites of Windows software projects

Unit test suites from Windows-compatible software projects can be used to help checking Wine for Windows conformance: Simply let them run on the Wine API while they check their own functionality and then compare the results of running them again under Windows.

LanguageTestSuites for programming languages that already run on Wine are probably the easiest unit test suites to run. It might take a little effort to use their results, though, as there's usually a big fluffy interpreter between the test and Wine.

Unit tests written in C or C++ are even more likely to be easy to run, and when they fail, to point directly to the Wine bug than user problem reports or language test suite failures.

AppInstall: Using Windows software as binary blob test suites

Tracking down bugs from user problem reports is laborious, as the observed symptoms are often very far removed from the cause and/or hard to reproduce. Wouldn't it be great if there were a source of bug reports that pinpoints the exact problem in Wine for us? That would make fixing the bug and adding a conformance test much easier.

AppInstall is the result of Austin English's scripts for Google Summer of Code 2009 that download and test prebuilt applications using new AutoHotKey based tests.

AppBuildTest: Building Windows applications in Wine (and then using their unit tests)

Simply building open-source apps on Wine using, say, mingw32 or Visual C++ is a good automated test of Wine. You'd be surprised how many bugs this can turn up! Compiling the source on Wine gives even further possibilities for analysis of both the application under test as the Wine API. For example a dynamic analysis at runtime (e.g. with Valgrind) to find memory leaks.

AppBuildTest is the lame and maybe not final name for a set of debugged build scripts Dan Kegel is writing. The goal is to have a single shell script that can run in either Cygwin+Windows or Cygwin+Wine and successfully build an application from source and run the app's own automated test suite. The same script should succeed in both Windows and Wine.

Projects with known buildbots

Here's a list of open source projects with automated test suites that could be built and tested in Wine, along with a note on their AppBuildTest (and AppInstall) status. Projects with their own public buildbots or other continuous build-and-test rigs showing that their tests all pass on Windows are the best candidates for this list.

Apache

Cairo

Chromium

CMake

Firefox

Gyp

Inkscape

KDE

OpenOffice.org

Python

SCons

Subversion

Webkit

Projects without known buildbots

We should think twice about using these; if a project doesn't have a continuous build and test rig, chances are they might not have purged their test suite of flaky tests yet (since flakiness often only becomes apparent when you test continuously). If you find a live automated build and test results site for one of these, please add its URL and move it to the previous section.

Apache Portable Runtime

Cygwin

Parrot

Perl 5

CategoryQualityAssurance

UnitTestSuites (last edited 2010-04-14 20:14:59 by AustinEnglish)