Problem trying to benchmark
Community Forums/General Help/Problem trying to benchmark
| ||
| So I'm having real difficulty timing some (any!) code... Rather than duplicate the very large amount of scary text, here is a link to where I've already asked about this on StackOverflow (and received no success so far): http://stackoverflow.com/questions/16765657/inconsistent-results-trying-to-microbenchmark Short version: programs refuse to take the same amount of time when run repeatedly, and I therefore can't benchmark code because the time numbers are meaningless. Test programs get slower and slower the more times they run, eventually reaching up to 300% slower than the first run despite no changes in the code. And if the order in which I run programs is going to affect how fast they are, I have no hope of ever knowing what effect my changes have on a piece of code! The problem kicks in after several seconds, which can require up to 20 repeated runs of a simple test program, but it always happens eventually. CPU frequency scaling is disabled; no user programs running in the background; Task Manager / top report 0% background CPU usage; same problem across Windows, Ubuntu, Lua, and C; across different process priorities (including realtime); tried timing with time (the program), time.h, QueryPerformanceCounter (on Windows) and rdtsc: all show the same slowdown curve. At this point I'm assuming it's something to do with my PC being old and crap, but I really want to know why it's happening and how to work around it if at all possible. "Try using another machine" is certainly in my plans if necessary though. |
| ||
| That is odd. Ive had the occasional random timings happen (always been either because of some kind of system i/o in the background or accidental overflows in the program itself), but i cant think of anything that would give you those results. Your c example ran stable on my amd machine, differing by a few ms between runs. What happens if you have the program itself repeat the timed code a few times? edit: factory settings, so cool'n'quiet and all that active. |
| ||
| What happens if you have the program itself repeat the timed code a few times? Basically the same thing... with this program: ...this happens: This is consistent with the behaviour of C programs I've been testing by hand; the effect isn't anywhere near as bad as it is for Lua, but still big enough to be a problem for comparisons. My computer is oooold (nearly six years? I can't remember...), and while it still serves me very well for regular use, I'm starting to wonder if this problem is perhaps something to do with e.g. thermal paste degrading on the inside and meaning it's incapable of operating at 100% performance any more...? I guess there's no easy way to check that. |
| ||
| Yeah i was thinking about that. It should really go slower then that though if it were the case. Also, afaik older chips should just shut themselves down. edit: well im tired, 600/1000ms is quite a difference. You could grab some software for temperature monitoring i guess. Its usually fairly obvious if things are running too hot. Another way would be to give it a few second pause between iterations so it can cool down a bit. Other then that, what kind of system are you running? |
| ||
| Because the programs should work, and because its failing across different languages, I would suggest that its a problem with the environment the programmes are running in. Something stupid like (for example) your IDE is debugging and the debug file is being appended each time by (say) copy and add, or run though to end of file and add. (Or the CLI window being stupid on similar grounds) SO rewrite so that the program runs x times but stores the results itself then prints then all at then end. Edit Oh. I see you just did this |
| ||
| Because the programs should work, and because its failing across different languages, I would suggest that its a problem with the environment the programmes are running in. Something stupid like (for example) your IDE is debugging and the debug file is being appended each time by (say) copy and add, or run though to end of file and add. (Or the CLI window being stupid on similar grounds) SO rewrite so that the program runs x times but stores the results itself then prints then all at then end. Edit Oh. I see you just did this Edit 2, ahh nearly did this, might still be a scroll problem in the CLI |
| ||
| Mmm, tried that too, didn't make any significant difference to the results. what kind of system are you running? The machine that's giving me trouble is an ancient laptop. It has this CPU: http://ark.intel.com/products/27236/Intel-Core-Duo-Processor-T2500-2M-Cache-2_00-GHz-667-MHz-FSB ...but according to that product sheet. the maximum permitted temperature is 100 degrees; it idles at 60 and runs at around 72 with the above test program. I won't pretend to know anything much about this field but that seems like it should be perfectly acceptable, and shouldn't be having any effect on performance? |
| ||
| I suspect memory management. The recursive calculation of Fib(38) will at some point have Fib(37) pending function calls. That's hundreds of megabytes of temporary storage. The repeated tests are happening within a single program, so you are not really starting with a clean slate. Maybe you should try a C program with your current test of Fib(38) as a standalone exe. Then a separate program would call that thirty times. That would, I guess, let each test have a fresh start. |
| ||
| The recursive calculation of Fib(38) will at some point have Fib(37) pending function calls. That's hundreds of megabytes of temporary storage. ? Maybe in a lazily-evaluated language, but C is strictly-evaluated so the "pending" calls don't have any kind of concrete existence until they actually activate, one at a time. These functions shouldn't be doing anything memory-wise other than repeatedly climbing up and down a comparatively-short bit of stack. You can run the program and see that it never allocates anything beyond the basic startup cost, less that 1MB. For what it's worth, I was actually calling a single test of fib(38) by hand 30 times until zzz pointed out that I could let the host program automate that, and it exhibited pretty much identical behaviour. |
| ||
| Final brainwave: I actually managed to make the problem go away (not solve it per se., but w/e) by turning frequency scaling off - but with the fixed speed set to "always low", so that the CPU is only working at half capacity and has no chance to overheat. Measurements are now slower (obviously), but uniformly within 1% as they are on everyone else's machines. I guess this confirms that it's a problem with my doddering old CPU. From this point on I will be going with the "use a different computer" solution |
| ||
| If the frequency adjustments are continueously scaling things down, then there's a chance that your CPU may be running (too) hot. Could be the fan starting to go bad? |
| ||
| Ha, entirely by accident I stumbled upon a temporary solution to this that's going to make you guys laugh. I angled my desk fan to blow past the back of my laptop (it's a laptop, that's why it was getting so hot in the first place) and ...the machine is suddenly as cold as if it were turned off! SpeedFan confirms that the machine runs substantially cooler internally when I have the desk fan on. So I went back to the test programs and... they run exactly as expected. No spikes except for when the system is busy, CPU never reaches the 70-degree "throttle point", and I can take meaningful performance measurements. Now I need to try and remember what it was I wanted to benchmark in the first place. And also to buy a new computer. |
| ||
| ... or open it up and clean out all the dust! |
| ||
| I'm glad you solved it! Interesting topic overall. |
| ||
| I would consider replacing the thermal paste. Arctic silver is really good stuff. I know it's a real pain opening up a laptop but it's worth protecting your machine if you plan on getting even more miles out of it. I have some crazy old machines still going real strong thanks to constant TLC. Dust is always my enemy though. |
   |