|CacheChk | CheckIt! | Landmark | Snooper | Sysinfo
/ EIDE |
Notes on Benchmarks
Benchmarks are small programs aimed at measuring the speed of a computer system and/or some of its parts. Usually, the benchmark lets the measured part execute some task (the processor executes a certain number of instructions, the disk copies a certain amount of data, etc.) and takes note of the elapsed time, converting it into an absolute value. Thus, if your hard disk copied, say, 10 Mb of data in 20 secs., the benchmark will credit it of a transfer speed of 500 Kb/sec.
Should we trust benchmarks?
If we want
benchmarks to have some value, we must compare systems
using exactly the same conditions, that is the same
settings. Performance may change dramatically if we
modify some configuration switch, or if we simply change
video driver. This is especially true when we use some
graphic interface operating system, such as
Windows. In such case, we may notice that, for instance,
video speed depends more on the quality of the software
driver than on the real (i.e.
hardware) speed of the video card.
I used are 16-bit. This was necessary as I wanted to
compare older 16-bit processors (8086, 8088, 80286) with
newer ones. If we would recompile all of our programs
from 16-bit to 32-bit, we would get better performances
for 32-bit CPUs (386, 486, etc.). It is known that, when
running 16-bit code, a 386 CPU offers little advantages
over a 286 at the same clock (if we compare the
instruction timings of 286 and 386 CPUs, we would notice
that most of them run at the same number of cycles).
Most of these benchmarks, then, seem definitely unreliable when 686 class CPUs (Pentium Pro, Pentium II) are involved. For instance, CheckIt credits a Pentium-Pro-200 to be just as fast as a Pentium-100, and equals a Pentium-II-266 to a Pentium-166.
Disk benchmarks are even more objectionable: just browse my small database to see how certain systems have faster disks according to one benchmark, which become much slower if we turn to another. A certain hard disk may be very fast when copying large files, but may result much slower when dealing with very small files, or with empty directories. Note also that most DOS benchmarks underestimate the performance of SCSI devices, as DOS is a single-task o.s., while SCSI is designed for working in a multi-tasking environment.
Video benchmarks are in some cases completely unreliable: just have a look at my WinTune results and you will be able to find absurd values or gross inconsistencies.
So, do we need these benchmarks?
Probably not. My
benchmarks are definitely non-professional, or
non-scientific if you prefer, and you should not use them
to extract "absolute" classifications between
systems. I tested all machines at my disposal with five
different DOS-based benchmarking programs so to give a
wider degree of accuracy.
Choice of benchmarks is rather fortuitous, as I used either freeware programs, or commercial programs of which I have a regular using license. This may help somebody wanting to compare his system.
Cache Check v. 4 (CACHECHK.EXE dated 8/2/96)
(postcard-ware) by Ray Van Tassle.
CheckIt! v. 2.01 (CHECKIT.EXE dated 17/11/89)
Tool by Touchstone Software.
Landmark v. 2.0 (SPEED.EXE dated 30/5/90)
test by Landmark Research Intl. Corp. Download.
Snooper v. 3.44 (SNOOPER.EXE dated 19/7/96)
information utility by Vias and Associates. Download.
Norton Sysinfo ver. 7.0 (SYSINFO.EXE dated 8/6/93)
Part of the
Norton Utilities by Symantec corp. (but you can find it
in other packages as well).
Where can I read more about benchmarks?
Keep in touch with the comp.alt.benchmark newsgroup.
Read the comp.benchmarks Frequently Asked Questions.
A very large database can be found at PDS (Performance Database Server).
I will add some links here, as soon as I have a bit of time...