Phase Two (AutoTest)

In the next phase of testing we use special programs designed for checking if specific kernel subsystems work correctly. We also carry out regression and performance tests of the kernel. The latter are particularly important for kernel developers (and for us), since they allow us to identify changes that hurt performance. For example, if the performance of one of our filesystems is 10% worse after we have upgraded the 2.6.x-rc1 kernel to the 2.6.x-rc2 one, it is definitely a good idea to find the patch that causes this to happen.

For automated kernel testing we recommend you to use the AutoTest suite (http://test.kernel.org/autotest/) consisting of many test applications and profiling tools combined with a fairly simple user interface.

To install AutoTest you can go into the /usr/local directory (as root) and run

# svn checkout svn://test.kernel.org/autotest/trunk autotest

Although it normally is not recommended to run such commands as root, this particular one should be safe, unless you cannot trust your DNS server, because it only downloads some files and saves them in /usr/local . Besides, you will need to run AutoTest as root, since some of its tests require superuser privileges to complete. For this reason you should not use AutoTest on a production system: in extreme cases the data stored in the system the privileged tests are run on can be damaged or even destroyed, and we believe that you would not like this to happen to your production data.

By design, AutoTest is noninteractive, so once started, it will not require your attention (of course, if something goes really wrong, you will have to recover the system, but this is a different kettle of fish). To start it you can go to /usr/local/autotest/client (we assume that AutoTest has been installed in /usr/local) and execute (as root)

# bin/autotest tests/test_name/control

where test_name is the name of the directory in /usr/local/autotest/client/tests that contains the test you want to run. The control file tests/test_name/control contains instructions for AutoTest. In the simplest cases only one such instruction is needed, namely

job.run_test('test_name')

where test_name is the name of the directory that contains the control file. The contents of more sophisticated control files can look like this:

job.run_test('pktgen', 'eth0', 50000, 0, tag='clone_skb_off')
job.run_test('pktgen', 'eth0', 50000, 1, tag='clone_skb_on')

where the strings after the test name represent arguments that should be passed to the test application. You can modify these arguments, but first you should read the documentation of the test application as well as the script tests/test_name/test_name.py (eg. tests/pktgen/pktgen.py) used by AutoTest to actually run the test (as you have probably noticed, the AutoTest scripts are written in Python). The results of the execution of the script tests/test_name/test_name.py are saved in the directory results/default/test_name/, where the status file contains the information indicating whether or not the test has been completed successfully. To cancel the test, press Ctrl+C while it is being executed.

If you want to run several tests in a row, it is best to prepare a single file containing multiple instructions for AutoTest. The instructions in this file should be similar to the ones contained in the above-mentioned control files. For example, the file samples/all_tests contains instructions for running all of the available tests and its first five lines are the following:

job.run_test('aiostress')
job.run_test('bonnie')
job.run_test('dbench')
job.run_test('fio')
job.run_test('fsx')

To run all of the tests requested by the instructions in this file, you can use the command

bin/autotest samples/all_tests

but you should remember that it will take a lot of time to complete. Analogously, to run a custom selection of tests, put the instructions for AutoTest into one file and provide its name as a command line argument to autotest.

To run several tests in parallel, you will need to prepare a special control file containing instructions like these:

def kernbench():
        job.run_test('kernbench', 2, 5)

def dbench():
        job.run_test('dbench')

job.parallel([kernbench], [dbench])

While the tests are being executed, you can stop them by pressing Ctrl+C at any time.

For people who do not like the command line and configuration files, ATCC (AutoTest Control Center) has been created. If you run it, for example by using the command ui/menu, you will be provided with a simple menu-driven user interface allowing you to select tests and profiling tools, view the results of their execution and, to a limited extent, configure them.

If you are bored with the selection of tools available in the AutoTest package, you can visit the web page http://ltp.sourceforge.net/tooltable.php containing a comprehensive list of tools that can be used for testing the Linux kernel.