Assessing the perfomance of heuristics or metaheuristics is not an easy task. Comparing exact methods is a lot easier. All of them reach the optimal solution and prove it. So comparing CPU times is enough. If you have to do that, please refer to our news where you can compare CPU times on different machines. But, now what can we do when we want to compare non-exact methods?
One solution is to use some nonparametric tests. There are many of them and we encourage researchers to read the abundant literature on it. for those who are in a hurry, we advice you to refer to the Friedman's test and the excellent summary written by our colleague and friend Juan G. Villegas. The summary can be downloaded in the form of a pdf file "Friedman test".
Of course, there are many other tests and we refer the reader to the following papers:
On the advice of many other people, I also link this article to the following book.
I hope this will be helpful.