Wednesday, April 21, 2010

Is my machine really faster for my work? Performance Analysis of MacBook Pro Core i7 2.66 GHz

Have you ever wondered, when considering a new computer: Exactly how much faster is this machine than what I'm using now?

I studied Computer Architecture -- and analysis thereof -- with Montek Singh, at UNC Chapel Hill in 2003. If Montek taught me any one thing, it was that you have to take measurements to determine system performance. And if he taught me two things, the second was that you can only evaluate a system's performance under your own workloads.

So, of course, I need to determine whether my new machine is genuinely faster than my existing machine doing the sort of work that I, Mark Lindsey typically perform.

My NEW laptop is the April 14, 2010 model. It's a MacBook Pro, 15" display, Core i7 2.66 GHz CPU, with 500 GB hard drive and 8 GB RAM. I'm thankful to be part of a firm that appreciates that value of quality tools.

My OLD laptop is the Early 2006 model. It was a MacBook Pro, 17" display, Core Duo 2.1 GHz, 320 GB 5400 RPM drive and 2 GB of RAM.

I compared these two laptops.

Wireshark Packet Capture Processing Performance



I often have to wait on Wireshark to process packets.

Load 249 MB Packet Capture -- 0.8x faster



Unfortunately, this particular test is estimated. The old Core Duo machine couldn't handle the memory required for this file, and Wireshark died. But if it had loaded at the predicted speed, the the old Core Duo would have taken 72 seconds, while the Core i7 completed loading in 39 seconds.




Load 87 MB Packet Capture -- 1.58x faster



Both machines could load the 87 MB packet capture. Core Duo: 31 seconds. Core i7: 12 seconds.




Display all VoIP Calls in 87 MB Packet Capture -- 0.8x faster



Core i7: 11 seconds. Core Duo: 20 seconds.



Calculate Statistics > Conversations in 87 MB Packet Capture -- 120x faster



Core i7: 5 seconds. Core Duo: 65 seconds. (In fairness, the Core i7 is running a slightly newer version of Wireshark, so they probably made some efficiency gains.)




Shell Scripting



I use shell scripting a lot. I've noticed Mac OS X is pretty slow; I believe there's some security stuff going on making the creation of each process an expensive task. Therefore, shell scripts that run millions of individual processes to accomplish the task are kinda slow.

10,000 instances of random file and text processing: 0.8x faster



My shell script was:

date
x=10000
while [ $x -gt 0 ]
do
ls -al "/etc" |
cut -c12- |
fmt -1 |
fmt -80 |
sort |
uniq -c > /tmp/x ;
let x=$x-1
done
date


This shell script does typical text and file processing stuff. It execute a total of 60,000 processes, opens an individual file 10,000 times, and does some simple in-the-shell processing.



grep through 300 MB of log files: 50% faster




time egrep '^INVITE' XSLog* | wc -l


This searched through 10 log files, each of which was 30 MB. They happened to be BroadSoft BroadWorks log files. It was a total of 7680738 lines of logs.



Generate 32 MB of random data to a disk file: 1.33x faster



time dd if=/dev/urandom bs=16384 count=2000 of=/tmp/random_bytes




Calculate pi to 2000 digits: 0.9x faster



Since Dr Boyd told me how to do it in the hallway of Nevins Hall at Valdosta State University ("Well, I guess you could take the arctangent of one, and multiply that by four") I've always liked using this to measure a system's performance.

"But Mark, I thought you said you were doing realistic workloads!" Who says taylor-series polynomial expansion isn't realistic? What ELSE are you doing to do with 731 million transistors?

echo "scale=2000 ; a(1)*4" | time bc -l




Microsoft Word 2008



I have to use Microsoft Word frequently because YOU USE MICROSOFT WORD. I have to write documents that other people edit; and sometimes I have to edit documents that other people wrote. And, unfortunately, not enough of YOU use LaTeX.

The Microsoft Word implementation for Mac OS X is pretty terrible. It seems much slower than the MS Windows version. Nevertheless, I end up using it.

Open a 1.5 MB docx in Microsoft Word: 0.6x faster



This is a 68 page document with several embedded graphics. The file size is 1.5 MB.



Render Preview PDF View from MS Word: 0.8x faster



From within Word, I opened the 1.5 MB, 68-page document mentioned above. Then I Chose File > Print and pressed the Preview button to open a PDF copy in the Apple Preview application.



Open Word to create new document: 69% faster






Search with Spotlight



I tested Spotlight searching in three separate levels: (a) my "BW" directory (7,824 files, 143 MB); (b) my "ECGDC" directory (8,031 files, 591 MB); (c) "My Mac". At this point, the contents of both machines is practically identical.

Search 7824-file / 143 MB directory for the word "INVITE": 1.2x faster





Search 8031-file / 591 MB directory for the word "date": 3.2x faster





Search 8031-file / 591 MB directory for the word "quality": 6.5x faster





Search entire machine for "Authentication": 1.2x faster





Conclusions



Excluding a few outliers, I should expect my new machine (Core i7) to be 80% (0.8x) to 120% (1.2x) faster than my old machine; by this I mean that a 30 second task on my old machine should take between 13 and 16 seconds on the new machine.

Methods



To time many processes, I used a stopwatch to measure the speed. On others, such as Wireshark or shell scripts, the system can calculate duration for me.

On opening MS Word files, I always opened the file once, then closed Word, then opened it again and timed the second or third opening. I ran some tests multiple times, and reported on the lowest of the times for each of the machines.

I know the right way to do this is measure elapsed duration, but I wanted to use the term "faster". I wasn't sure whether everybody used that term in a consistent way when comparing equivalent workloads. To calculate "fasterness", I divided the long duration by the short duration, and subtracted one. E.g., if a process took 9.8 seconds on the Core Duo, and 4.4 seconds on the Core i7, I'd calculate

( 9.8 / 4.4 ) - 1 = 1.227

i.e., the Core i7 is 1.2x, or 123% faster.

Thursday, April 08, 2010

Are Exceptions so Sacred? My Disagreement with the Pragmatic Programmers.

For the past decade I've tried to grapple with the best way to use Exceptions in programming. I've personally been moving away from using exceptions, based on Hunt and Thomas's advice in "The Pragmatic Programmer".


One of the problems with exceptions is knowing when to use them. We believe that exceptions should rarely be used as part of a program's normal flow; exceptions should be reserved for unexpected events. Assume that an uncaught exception will terminate your program and ask yourself, "Will this code still run if I remove all the exception handling?" If the answer is "no," then maybe exceptions are being used in nonexceptional circumstances.

or example, if your code tries to open a file for reading and that file does not exist, should an exception be raised?

Our answer is, "It depends." If the file should have been there, then an exception is warranted. Something unexpected happened--a file you were expecting to exist seems to have disappeared. On the other hand, if you have no idea whether the file should exist or not, then it doesn't seem exceptional if you can't find it, and an error return is appropriate.


They go on to distinguish between opening /etc/passwd on a Unix system, versus opening a file specified by a user:

public boolean open_user_file(String name) throws FileNotFoundException {
File f = new File(name);
if (!f.exists()) {
return false;
}
ipstream = new FileInputStream(f);
return true;
}


Andrew Hunt and Dave Thomas: You're just wrong! Here's the problem: a method can really only return one value, or it can throw one exception. If the return value has to be used to return an indication of success, then it can't be used to return something functional.

Look at their "good" example in open_user_file -- where does ipstream come from? It just comes out of nowhere -- out of global scope somewhere. This was their trick to let the function affect two values on its return -- both the return value (boolean) to show whether success had occurred, and also the ipstream.

If they hadn't used the global scope like this, they'd have had to store one of those two in a parameterized value, which would have been awkward.

All this to avoid using Exceptions!

It's just silliness, and it works against the intention of the Java creators. They wanted to keep all the silly checking for oddball stuff out of the main stream of code:


readFile {
try {
open the file;
determine its size;
allocate that much memory;
read the file into memory;
close the file;
} catch (fileOpenFailed) {
doSomething;
} catch (sizeDeterminationFailed) {
doSomething;
} catch (memoryAllocationFailed) {
doSomething;
} catch (readFailed) {
doSomething;
} catch (fileCloseFailed) {
doSomething;
}
}


Did you catch that? Inside the try block you'll see the Sunny-Day Scenario. Let's just assume everything goes right -- this is how it should work. It's the main course of logic.

Friday, April 02, 2010

Fixed: Wireshark crash on 10.6

I've had problems for a while with Wireshark crashes on Mac OS X 10.6 ("Snow Leopard"). I finally tracked it down today.

1. Install XCode

2. Use this dtrace one-liner to watch all the files being opened on the machine:
dtrace -n 'syscall::open*:entry { printf("%s %s",execname,copyinstr(arg0)); }'

3. Found where "CrashReport" ran in the output, and then looked back from there to see what files had been opened. I identified these files, and removed them:
~/.fonts.cache-1
~/.fonts
~/.wireshark
~/.wireshark-etc

(For you Finder users, these are all hidden files in my home directory.)

One of those must have had something bogus in it, because wireshark started up the next time.