TITLE: timing and profiling gotchas

(Newsgroup: comp.lang.c++.moderated, 14 Jun 2000)


CLAMAGE: Steve Clamage <stephen.clamage@sun.com>

>> Constructing a new stream object is pretty elaborate. It involves
>> memory allocation for buffers and possibly other things.
>> If you are going to read several files sequentially, it's more
>> efficient to construct once and re-use the same stream object.

BROUGHAM: brougham3@yahoo.com

> I was willing to grant you this, but out of curiosity, I wanted to know
> how much more inefficient my way would be.  I was surprised.

[ (original code shown below)

ifstream f;
 
f.open( "fileA" );
// work with fileA ...
f.close();
 
f.open( "fileB" );
// work with fileB ...
f.close();
]

> Now, this is where it gets strange.  If I don't read any data in,
> constructing a new ifstream object each time is roughly twice as long
> for 100,000 operations (1 sec vs .5 sec).

CLAMAGE: 

That is believable.

BROUGHAM:

> With the code as shown, constructing a new ifstream object each time 
> is *faster* on my system.  7.63 secs vs. 7.70 seconds.  I ran it three 
> times, and got similar results each time.

Surely you don't expect the time to be repeatable to 3 significant
figures. What you are seeing is that the time to construct the object
is negligible compared to the time to open and read the file, and the
difference is lost in the noise. That was one of my comments that you
didn't quote.

BROUGHAM:

> This just goes to show that when it matters, profile the code in
> question.

CLAMAGE:

Yes, but you didn't do any profiling; you just timed different
programs, and you have to understand what your timing results mean.

Timing file operations is tricky, since you can't predict the results
of file caching or other system dependencies. It's possible for the
entire file to be in a memory cache, so that only the results of the
first iteration are meaningful. It's also possible for the time to
open a file to vary during one program run, and take much longer than
the time to construct and destroy a stream object, something I also
said before.

On some systems you can take steps to eliminate these outside
variations, but then you are not getting results that mean anything
for a real program on the real system.  ("System A when configured
this way runs the Sieve of Eratothsenes faster than system B." "Yes,
but I can't use a system configured that way, and I don't want to run
the Sieve; I want to run a database application.")

I was recently working with someone trying to evaluate the performance
of a large and complex program built with two competing compilers. It
turned out that the benchmark results were dominated by the time it
took to open and initialize the shared libraries the program used. The
benchmark times didn't mean anything.



_______________________________________________
cpptips mailing list
http://cpptips.hyperformix.com