As we have already mentioned, it is good to check the effects of the changes made to the kernel on the performance of the entire system (by the way, this may be an excellent task for beginner testers, who do not want to deal with development kernels yet, although they eagerly want to help develop the kernel). Still, to do this efficiently, you need to know how to do it and where to begin.
To start with, it is recommended to choose one subsystem that you will test
regularly, since in that case your reports will be more valuable to the kernel
developers. Namely, from time to time messages like "Hello, I've noticed that
the performance of my network adapter decreased substantially after I had
upgraded from 2.6.8 to 2.6.20. Can anyone help me?'' appear on
the LKML. Of course, in such cases usually no one has a slightest idea of what
could happen, because the kernel 2.6.20 was released two and a half years
(and gazillion random patches) after 2.6.8. Now, in turn, if you report
that the performance of your network adapter has dropped 50% between the
kernels 2.6.x-rc3 and 2.6.x-rc4, it will be relatively easy to
find out why. For this reason it is important to carry out the measurements of
performance regularly.
Another thing that you should pay attention to is how your tests actually work. Ideally, you should learn as much as possible about the benchmark that you want to use, so that you know how to obtain reliable results from it. For example, in some Internet or press publications you can find the opinion that running
$ time makein the kernel source directory is a good test of performance, as it allows you to measure how much time it takes to build the kernel on given system. While it is true that you can use this kind of tests to get some general idea of how ''fast'' (or how ''slow'') the system is, they generally should not be regarded as measurements of performance, since they are not sufficiently precise. In particular, the kernel compilation time depends not only on the ''speed'' of the CPU and memory, but also on the time needed to load the necessary data into memory from the hard disk, which in turn may depend on where exactly these data are physically located. In fact, the kernel compilation is quite I/O-intensive and the time needed to complete it may depend on some more or less random factors. Moreover, if you run it twice in a row, the first run usually takes more time to complete than the second one, since the kernel caches the necessary data in memory during the first run and afterwards they can simply be read from there. Thus in order to obtain reproducible results, it is necessary to suppress the impact of the I/O, for example by forcing the kernel to load the data into memory before running the test. Generally, if you want to carry out the ''
time make'' kind of benchmarks, it is best to use the
kernbench script (http://ck.kolivas.org/kernbench/), a newer
version of which is included in the AutoTest suite (see
Section 2.2). Several good benchmarks are also available from
http://ltp.sourceforge.net/tooltable.php (some of them are included in
AutoTest too). Still, if you are interested in testing the kernel
rather than in testing hardware, you should carefully read the documentation of
the chosen benchmark, because it usually contains some information that you may
need.
The next important thing that you should always remember about is the stability (ie. invariableness) of the environment in which the measurements are carried out. In particular, if you test the kernel, you should not change anything but the kernel in your system, since otherwise you would test two (or more) things at a time and it would not be easy to identify the influence of each of them on the results. For instance, if the measurement is based on building the kernel, it should always be run against the same kernel tree with exactly the same configuration file, using the same compiler and the other necessary tools (of course, once you have upgraded at least one of these tools, the results that you will obtain from this moment on should not be compared with the results obtained before the upgrade).
Generally, you should always do your best to compare apples to apples. For example, if you want to test the performance of three different file systems, you should not install them on three different partitions of the same disk, since the time needed to read (or write) data from (or to) the disk generally depends on where exactly the operation takes place. Instead, you should create one partition on which you will install each of the tested filesystems. Moreover, in such a case it is better to restart the system between consecutive measurements in order to suppress the effect of the caching of data.
Concluding, we can say that, as far as the measurements of performance are concerned, it is important to