MARK benchmark program

[egc]

Greetings -

I would say about once every other week or so I get a query from someone asking for 'hardware advice' - they're interested in determining a good platform for running MARK analyses, and are wondering where to put their $$$ (e.g., more RAM, faster processor) etc. The simple, if not overly helpful answer is always 'more is always better' (more RAM, more processor speed), so sometimes the discussion is more about 'how bad is my current machine, and is it worth upgrading?'.

One way to gauge machine performance (i.e., assess 'how bad' your current machine is) is via what are known as 'benchmark suites'. To this end, I've grabbed one of the data sets Gary distributes with MARK (for a multi-state analysis), and tweaked it a bit so it will provide a decent test of your machine. The actual code (which you need to cut and paste into an ASCII file I'll call benchmark.inp) is at the bottom of this note.

To run the benchmark, do the following (instructions assume you're running Windows - if you're running MARK under emulation, then email me, and I'll send you modified directions, if I can).

1. cut and past the benchmark code (at bottom) into an ASCII file called benchmark.txt. Don't do this in Word, or some othe nonsence program - use a simple ASCII or text editor (even notepad will suffice). Put the benchmark.inp file on your desktop.

2. open a command window using Start -> Run -> cmd

3. cd to the Desktop. For most people, that means

cd c:\Documents and Settings\<user>\Desktop

where <user> is the name of the account you're using on the machine.

4. run the benchmark in MARK. You do this using

c:\progra~1\mark\mark i=benchmark.inp o=benchxx.lst

where benchmark.inp is the file you're inputting into the MARK executable, and benchxx.lst is the output (listing) file (where xx is the number of the run you're going to use). For example, if you make at least 3 runs of the benchmark (3 is the minimum), then

c:\progra~1\mark\mark i=benchmark.inp o=bench1.lst

for the first run, o=bench2.lst for the second, and so on (you can put all this in a batch file, if you know how).

Once you've run the benchmark at least 3 times, you'll want to report (py posting here) several things:

1. your machine (Mb RAM, CPU type and speed, desktop or laptop)

2. your operating system (as specifically as possible)

3. number of runs of the benchmark you're going to average over (the benchmark uses the simulated annealing algorithm to maximize the likelihood, and as such, each run might take slightly more or less time than any other run).

4. average time for

a. time for numerical optimization
b. time to computer number of parameters
c. time to estimate profile CI's

For purposes of a baseline, I'll use one of the machines I routinely use - its pretty 'middle-of-the road' as far as computers goes these days (the following is a typical posting of your benchmark results).

1. Dell Dimension 4600 - 2 Gb RAM, 3.00 GHz CPU - threading turned on
2. Windows XP Professional SP 2
3. 10 runs of the benchmark

a. avg time for numerical optimization 182.1 seconds
b. time to estimate number of parameters 4.2 seconds
c. time to estimate profile CI's 615.2 seconds

Notice that the really 'slow' step on the benchmark' is the time required to estimate profile CI's for all of the parameters (which of course is a stark example of why profile CI's are not the default in MARK).


----<code below here>----

proc title Multi-strata Example;
proc chmatrix occasions=4 groups=1 etype=Multistrata strata=3 NoHist hist=300;
glabel(1)=Group 1;
time interval 1 1 1;
strata A B C;
A000 491;
A00A 15;
A00B 12;
A00C 11;
A0A0 37;
A0AA 11;
A0AB 3;
A0AC 3;
A0B0 24;
A0BA 2;
A0BB 7;
A0BC 2;
A0C0 25;
A0CA 2;
A0CB 2;
A0CC 6;
AA00 112;
AA0A 11;
AA0B 7;
AA0C 7;
AAA0 29;
AAAA 9;
AAAB 3;
AAAC 3;
AAB0 10;
AABA 1;
AABB 3;
AABC 1;
AAC0 10;
AACA 1;
AACB 1;
AACC 3;
AB00 40;
AB0A 2;
AB0B 3;
AB0C 1;
ABA0 4;
ABAA 1;
ABAB 0;
ABAC 0;
ABB0 10;
ABBA 1;
ABBB 3;
ABBC 1;
ABC0 5;
ABCA 0;
ABCB 0;
ABCC 0;
AC00 39;
AC0A 2;
AC0B 2;
AC0C 3;
ACA0 4;
ACAA 1;
ACAB 0;
ACAC 0;
ACB0 4;
ACBA 0;
ACBB 1;
ACBC 0;
ACC0 10;
ACCA 1;
ACCB 1;
ACCC 3;
0A00 529;
0A0A 54;
0A0B 33;
0A0C 34;
0AA0 137;
0AAA 44;
0AAB 14;
0AAC 15;
0AB0 46;
0ABA 5;
0ABB 14;
0ABC 5;
0AC0 46;
0ACA 5;
0ACB 5;
0ACC 14;
00A0 650;
00AA 210;
00AB 70;
00AC 70;
000A 1000;
B000 491;
B00A 12;
B00B 15;
B00C 11;
B0A0 24;
B0AA 7;
B0AB 2;
B0AC 2;
B0B0 37;
B0BA 3;
B0BB 11;
B0BC 3;
B0C0 25;
B0CA 2;
B0CB 2;
B0CC 6;
BA00 40;
BA0A 3;
BA0B 2;
BA0C 1;
BAA0 10;
BAAA 3;
BAAB 1;
BAAC 1;
BAB0 4;
BABA 0;
BABB 1;
BABC 0;
BAC0 5;
BACA 0;
BACB 0;
BACC 0;
BB00 112;
BB0A 7;
BB0B 11;
BB0C 7;
BBA0 10;
BBAA 3;
BBAB 1;
BBAC 1;
BBB0 29;
BBBA 3;
BBBB 9;
BBBC 3;
BBC0 10;
BBCA 1;
BBCB 1;
BBCC 3;
BC00 39;
BC0A 2;
BC0B 2;
BC0C 3;
BCA0 4;
BCAA 1;
BCAB 0;
BCAC 0;
BCB0 4;
BCBA 0;
BCBB 1;
BCBC 0;
BCC0 10;
BCCA 1;
BCCB 1;
BCCC 3;
0B00 529;
0B0A 33;
0B0B 54;
0B0C 34;
0BA0 46;
0BAA 14;
0BAB 5;
0BAC 5;
0BB0 137;
0BBA 14;
0BBB 44;
0BBC 15;
0BC0 46;
0BCA 5;
0BCB 5;
0BCC 14;
00B0 650;
00BA 70;
00BB 210;
00BC 70;
000B 1000;
C000 493;
C00A 12;
C00B 12;
C00C 13;
C0A0 24;
C0AA 7;
C0AB 2;
C0AC 2;
C0B0 24;
C0BA 2;
C0BB 7;
C0BC 2;
C0C0 40;
C0CA 3;
C0CB 3;
C0CC 9;
CA00 40;
CA0A 3;
CA0B 2;
CA0C 1;
CAA0 10;
CAAA 3;
CAAB 1;
CAAC 1;
CAB0 4;
CABA 0;
CABB 1;
CABC 0;
CAC0 5;
CACA 0;
CACB 0;
CACC 0;
CB00 40;
CB0A 2;
CB0B 3;
CB0C 1;
CBA0 4;
CBAA 1;
CBAB 0;
CBAC 0;
CBB0 10;
CBBA 1;
CBBB 3;
CBBC 1;
CBC0 5;
CBCA 0;
CBCB 0;
CBCC 0;
CC00 113;
CC0A 7;
CC0B 7;
CC0C 11;
CCA0 10;
CCAA 3;
CCAB 1;
CCAC 1;
CCB0 10;
CCBA 1;
CCBB 3;
CCBC 1;
CCC0 30;
CCCA 3;
CCCB 3;
CCCC 9;
0C00 530;
0C0A 33;
0C0B 33;
0C0C 54;
0CA0 46;
0CAA 14;
0CAB 5;
0CAC 5;
0CB0 46;
0CBA 5;
0CBB 14;
0CBC 5;
0CC0 138;
0CCA 14;
0CCB 14;
0CCC 44;
00C0 650;
00CA 70;
00CB 70;
00CC 210;
000C 1000;
proc estimate link=Sin varest=2ndPart SIMANNEAL;
model={S(s*t) p(s*t) psi(s*t) Sin link};
fixed=3;
parm(12)=1;
parm(15)=1;
parm(18)=1;
group=1 S A:Strata 1 rows=3 cols=3 Triang;
1 2 3;
2 3;
3;
group=1 S B:Strata 2 rows=3 cols=3 Triang;
4 5 6;
5 6;
6;
group=1 S C:Strata 3 rows=3 cols=3 Triang;
7 8 9;
8 9;
9;
group=1 p A:Strata 1 rows=3 cols=3 Triang;
10 11 12;
11 12;
12;
group=1 p B:Strata 2 rows=3 cols=3 Triang;
13 14 15;
14 15;
15;
group=1 p C:Strata 3 rows=3 cols=3 Triang;
16 17 18;
17 18;
18;
group=1 Psi A to B rows=3 cols=3 Triang;
19 20 21;
20 21;
21;
group=1 Psi A to C rows=3 cols=3 Triang;
22 23 24;
23 24;
24;
group=1 Psi B to A rows=3 cols=3 Triang;
25 26 27;
26 27;
27;
group=1 Psi B to C rows=3 cols=3 Triang;
28 29 30;
29 30;
30;
group=1 Psi C to A rows=3 cols=3 Triang;
31 32 33;
32 33;
33;
group=1 Psi C to B rows=3 cols=3 Triang;
34 35 36;
35 36;
36;
profile intervals chat=1.0000 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36;
design matrix constraints=36 covariates=36 identity;
blabel(1)=S A:Strata 1;
blabel(2)=S A:Strata 1;
blabel(3)=S A:Strata 1;
blabel(4)=S B:Strata 2;
blabel(5)=S B:Strata 2;
blabel(6)=S B:Strata 2;
blabel(7)=S C:Strata 3;
blabel(8)=S C:Strata 3;
blabel(9)=S C:Strata 3;
blabel(10)=p A:Strata 1;
blabel(11)=p A:Strata 1;
blabel(12)=p A:Strata 1;
blabel(13)=p B:Strata 2;
blabel(14)=p B:Strata 2;
blabel(15)=p B:Strata 2;
blabel(16)=p C:Strata 3;
blabel(17)=p C:Strata 3;
blabel(18)=p C:Strata 3;
blabel(19)=Psi A to B;
blabel(20)=Psi A to B;
blabel(21)=Psi A to B;
blabel(22)=Psi A to C;
blabel(23)=Psi A to C;
blabel(24)=Psi A to C;
blabel(25)=Psi B to A;
blabel(26)=Psi B to A;
blabel(27)=Psi B to A;
blabel(28)=Psi B to C;
blabel(29)=Psi B to C;
blabel(30)=Psi B to C;
blabel(31)=Psi C to A;
blabel(32)=Psi C to A;
blabel(33)=Psi C to A;
blabel(34)=Psi C to B;
blabel(35)=Psi C to B;
blabel(36)=Psi C to B;
rlabel(1)=S A:Strata 1;
rlabel(2)=S A:Strata 1;
rlabel(3)=S A:Strata 1;
rlabel(4)=S B:Strata 2;
rlabel(5)=S B:Strata 2;
rlabel(6)=S B:Strata 2;
rlabel(7)=S C:Strata 3;
rlabel(8)=S C:Strata 3;
rlabel(9)=S C:Strata 3;
rlabel(10)=p A:Strata 1;
rlabel(11)=p A:Strata 1;
rlabel(12)=p A:Strata 1;
rlabel(13)=p B:Strata 2;
rlabel(14)=p B:Strata 2;
rlabel(15)=p B:Strata 2;
rlabel(16)=p C:Strata 3;
rlabel(17)=p C:Strata 3;
rlabel(18)=p C:Strata 3;
rlabel(19)=Psi A to B;
rlabel(20)=Psi A to B;
rlabel(21)=Psi A to B;
rlabel(22)=Psi A to C;
rlabel(23)=Psi A to C;
rlabel(24)=Psi A to C;
rlabel(25)=Psi B to A;
rlabel(26)=Psi B to A;
rlabel(27)=Psi B to A;
rlabel(28)=Psi B to C;
rlabel(29)=Psi B to C;
rlabel(30)=Psi B to C;
rlabel(31)=Psi C to A;
rlabel(32)=Psi C to A;
rlabel(33)=Psi C to A;
rlabel(34)=Psi C to B;
rlabel(35)=Psi C to B;
rlabel(36)=Psi C to B;
proc stop;

[cooch]

1. Quad Opteron 885 (2.6 GHz) running GNU/Linux (Fedora Core 5) - 32 GB RAM
2. Windows under emulation (wine)
3. 10 runs of benchmark

a. avg time for numerical optimization 92.1 seconds
b. time to estimate number of parameters 1.3 seconds
c. time to estimate profile CI's 295 seconds


Notes:

This is standard 'MARK for Windows' running under the wine environment (emulation of sorts) on a high-end GNU/Linux workstation. Certain parts of the benchmark are immensely faster (specifically, estimation of the number of parameters) - the advantage is somewhat less for the other parts (although still impressive at about a two-fold gain in speed).

The next step will be testing a version of MARK compiled specifically for this machine.

[egc]

The latest version of mark.exe isn't compatible with the previously posted benchmark code. I've updated it (below) - you run it as per the instructions for the first benchmark.

Code: Select all

proc title benchmark program;
proc chmatrix occasions=4 groups=1 etype=Multistrata mixtures=2 strata=3 hist=300;
   glabel(1)=Group 1;
   time interval  1 1 1;
   strata  A B C;
A000 491;
A00A 15;
A00B 12;
A00C 11;
A0A0 37;
A0AA 11;
A0AB 3;
A0AC 3;
A0B0 24;
A0BA 2;
A0BB 7;
A0BC 2;
A0C0 25;
A0CA 2;
A0CB 2;
A0CC 6;
AA00 112;
AA0A 11;
AA0B 7;
AA0C 7;
AAA0 29;
AAAA 9;
AAAB 3;
AAAC 3;
AAB0 10;
AABA 1;
AABB 3;
AABC 1;
AAC0 10;
AACA 1;
AACB 1;
AACC 3;
AB00 40;
AB0A 2;
AB0B 3;
AB0C 1;
ABA0 4;
ABAA 1;
ABAB 0;
ABAC 0;
ABB0 10;
ABBA 1;
ABBB 3;
ABBC 1;
ABC0 5;
ABCA 0;
ABCB 0;
ABCC 0;
AC00 39;
AC0A 2;
AC0B 2;
AC0C 3;
ACA0 4;
ACAA 1;
ACAB 0;
ACAC 0;
ACB0 4;
ACBA 0;
ACBB 1;
ACBC 0;
ACC0 10;
ACCA 1;
ACCB 1;
ACCC 3;
0A00 529;
0A0A 54;
0A0B 33;
0A0C 34;
0AA0 137;
0AAA 44;
0AAB 14;
0AAC 15;
0AB0 46;
0ABA 5;
0ABB 14;
0ABC 5;
0AC0 46;
0ACA 5;
0ACB 5;
0ACC 14;
00A0 650;
00AA 210;
00AB 70;
00AC 70;
000A 1000;
B000 491;
B00A 12;
B00B 15;
B00C 11;
B0A0 24;
B0AA 7;
B0AB 2;
B0AC 2;
B0B0 37;
B0BA 3;
B0BB 11;
B0BC 3;
B0C0 25;
B0CA 2;
B0CB 2;
B0CC 6;
BA00 40;
BA0A 3;
BA0B 2;
BA0C 1;
BAA0 10;
BAAA 3;
BAAB 1;
BAAC 1;
BAB0 4;
BABA 0;
BABB 1;
BABC 0;
BAC0 5;
BACA 0;
BACB 0;
BACC 0;
BB00 112;
BB0A 7;
BB0B 11;
BB0C 7;
BBA0 10;
BBAA 3;
BBAB 1;
BBAC 1;
BBB0 29;
BBBA 3;
BBBB 9;
BBBC 3;
BBC0 10;
BBCA 1;
BBCB 1;
BBCC 3;
BC00 39;
BC0A 2;
BC0B 2;
BC0C 3;
BCA0 4;
BCAA 1;
BCAB 0;
BCAC 0;
BCB0 4;
BCBA 0;
BCBB 1;
BCBC 0;
BCC0 10;
BCCA 1;
BCCB 1;
BCCC 3;
0B00 529;
0B0A 33;
0B0B 54;
0B0C 34;
0BA0 46;
0BAA 14;
0BAB 5;
0BAC 5;
0BB0 137;
0BBA 14;
0BBB 44;
0BBC 15;
0BC0 46;
0BCA 5;
0BCB 5;
0BCC 14;
00B0 650;
00BA 70;
00BB 210;
00BC 70;
000B 1000;
C000 493;
C00A 12;
C00B 12;
C00C 13;
C0A0 24;
C0AA 7;
C0AB 2;
C0AC 2;
C0B0 24;
C0BA 2;
C0BB 7;
C0BC 2;
C0C0 40;
C0CA 3;
C0CB 3;
C0CC 9;
CA00 40;
CA0A 3;
CA0B 2;
CA0C 1;
CAA0 10;
CAAA 3;
CAAB 1;
CAAC 1;
CAB0 4;
CABA 0;
CABB 1;
CABC 0;
CAC0 5;
CACA 0;
CACB 0;
CACC 0;
CB00 40;
CB0A 2;
CB0B 3;
CB0C 1;
CBA0 4;
CBAA 1;
CBAB 0;
CBAC 0;
CBB0 10;
CBBA 1;
CBBB 3;
CBBC 1;
CBC0 5;
CBCA 0;
CBCB 0;
CBCC 0;
CC00 113;
CC0A 7;
CC0B 7;
CC0C 11;
CCA0 10;
CCAA 3;
CCAB 1;
CCAC 1;
CCB0 10;
CCBA 1;
CCBB 3;
CCBC 1;
CCC0 30;
CCCA 3;
CCCB 3;
CCCC 9;
0C00 530;
0C0A 33;
0C0B 33;
0C0C 54;
0CA0 46;
0CAA 14;
0CAB 5;
0CAC 5;
0CB0 46;
0CBA 5;
0CBB 14;
0CBC 5;
0CC0 138;
0CCA 14;
0CCB 14;
0CCC 44;
00C0 650;
00CA 70;
00CB 70;
00CC 210;
000C 1000;
proc estimate link=Sin varest=2ndPart SIMANNEAL;
     model={S(s*t)p(s*t)psi(s*t)};
     group=1 S A:A rows=3 cols=3 Triang Time=1;
     group=1 S B:B rows=3 cols=3 Triang Time=4;
     group=1 S C:C rows=3 cols=3 Triang Time=7;
     group=1 p A:A rows=3 cols=3 Triang Time=10;
     group=1 p B:B rows=3 cols=3 Triang Time=13;
     group=1 p C:C rows=3 cols=3 Triang Time=16;
     group=1 Psi A to B rows=3 cols=3 Triang Time=19;
     group=1 Psi A to C rows=3 cols=3 Triang Time=22;
     group=1 Psi B to A rows=3 cols=3 Triang Time=25;
     group=1 Psi B to C rows=3 cols=3 Triang Time=28;
     group=1 Psi C to A rows=3 cols=3 Triang Time=31;
     group=1 Psi C to B rows=3 cols=3 Triang Time=34;
   profile intervals chat=1.0000 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
           17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36;
     design matrix constraints=36 covariates=36 identity;
   blabel(1)=S A:A;
   blabel(2)=S A:A;
   blabel(3)=S A:A;
   blabel(4)=S B:B;
   blabel(5)=S B:B;
   blabel(6)=S B:B;
   blabel(7)=S C:C;
   blabel(8)=S C:C;
   blabel(9)=S C:C;
   blabel(10)=p A:A;
   blabel(11)=p A:A;
   blabel(12)=p A:A;
   blabel(13)=p B:B;
   blabel(14)=p B:B;
   blabel(15)=p B:B;
   blabel(16)=p C:C;
   blabel(17)=p C:C;
   blabel(18)=p C:C;
   blabel(19)=Psi A to B;
   blabel(20)=Psi A to B;
   blabel(21)=Psi A to B;
   blabel(22)=Psi A to C;
   blabel(23)=Psi A to C;
   blabel(24)=Psi A to C;
   blabel(25)=Psi B to A;
   blabel(26)=Psi B to A;
   blabel(27)=Psi B to A;
   blabel(28)=Psi B to C;
   blabel(29)=Psi B to C;
   blabel(30)=Psi B to C;
   blabel(31)=Psi C to A;
   blabel(32)=Psi C to A;
   blabel(33)=Psi C to A;
   blabel(34)=Psi C to B;
   blabel(35)=Psi C to B;
   blabel(36)=Psi C to B;
   rlabel(1)=S A:A;
   rlabel(2)=S A:A;
   rlabel(3)=S A:A;
   rlabel(4)=S B:B;
   rlabel(5)=S B:B;
   rlabel(6)=S B:B;
   rlabel(7)=S C:C;
   rlabel(8)=S C:C;
   rlabel(9)=S C:C;
   rlabel(10)=p A:A;
   rlabel(11)=p A:A;
   rlabel(12)=p A:A;
   rlabel(13)=p B:B;
   rlabel(14)=p B:B;
   rlabel(15)=p B:B;
   rlabel(16)=p C:C;
   rlabel(17)=p C:C;
   rlabel(18)=p C:C;
   rlabel(19)=Psi A to B;
   rlabel(20)=Psi A to B;
   rlabel(21)=Psi A to B;
   rlabel(22)=Psi A to C;
   rlabel(23)=Psi A to C;
   rlabel(24)=Psi A to C;
   rlabel(25)=Psi B to A;
   rlabel(26)=Psi B to A;
   rlabel(27)=Psi B to A;
   rlabel(28)=Psi B to C;
   rlabel(29)=Psi B to C;
   rlabel(30)=Psi B to C;
   rlabel(31)=Psi C to A;
   rlabel(32)=Psi C to A;
   rlabel(33)=Psi C to A;
   rlabel(34)=Psi C to B;
   rlabel(35)=Psi C to B;
   rlabel(36)=Psi C to B;
proc stop;