You are here

Comparing CPU times on different machines

In optimization, it happens very often that you want to compare your results with previous and competitive works. In some cases, it might end up with a running time comparison; for example because both approaches give the optimal solutions or because in addition to the objective value comparison, you want also to assess the speed of your method. At that stage, there are very few options: the computers are not the same, but you still want to make a fair comparison (you are a scientist, aren't you?). Do you need to re-write the other approach? Is all the necessary information in the paper so that your implementation will be exactly the same? Probably not. In that case, you need to read more here.

Let's say that, on the same instance, your algorithm runs in 1.02s on your brand new PC (a Core 2 Duo 64bits, 2.4GHz, OS Ubuntu), when you activated the optimize option in the compiler, and you want to compare it with a previous a competitive algorithm that was developped earlier on a Pentium 4 (3.6 Ghz, OS Windows), where the running time is 12s without the optimize option in the compiler. We assume that RAM is not a problem (no swaping on Disk).

  • The fist step is to ensure that you are running exactly the same instance (sometimes, name is identical, but instance has been changed in between :-( )
  • The second step is to identify your computer and the previous one in the list given by http://www.roylongbottom.org.uk/linpack%20results.htm

So you will get the two following lines from the tables:


Old PC

                              Opt    No opt
CPU                 MHz    MFLOPS    MFLOPS
Pentium 4          3678   1017.01    209.01

New PC

32 and 64 Bit Linux Results from Ubuntu GCC 
                                 Opt    No opt
CPU                    MHz    MFLOPS    MFLOPS
Core 2 Duo  64b  Ub   2400      1577       378




The index of the Old PC is 209.01 (No opt) and the index of the New PC is 1577 (Opt).
It means that your New PC is 1577/209.01 = 7.545 time faster than the Old PC.
Hence the running time of the previous algorithm on your New PC would have been 12/7.545 = 1.59s

So your new algorithm is faster :-) Of course, this is subject to errors, compiler specifications and so on, and I would conclude that both algorithms run in a similar time frame (both run in about 1-2s). What is important, is what you can write in an article or conference paper:

"To have a fair comparative study, the CPU times have been scaled according to the Linpack Benchmarks [1]."

and add this reference to the paper (don't forget to check the web page and update the reference date):

[1] R. Longbottom. (2011). Linpack Benchmark Results On PCs. Webpage.
      http://www.roylongbottom.org.uk/linpack%20results.htm

 

The OR-Group.