Simmanneal under new Mark

[Eldar]

It is great that new Mark uses many cores. But it means that there have to be some strategies of optimal core use. Does anybody know:
1. Is there any logic how many threads can be effectively used by one Mark, or is it really scalable without limits?
2. Simulated annealing uses as many cores as you provide but does it really need them? So inside fortran is simanneal really parallelized?
As a consequence of this is a question about hyperthreading. With normal optimisation I've got a feeling that it is much better to turn it on, but with simanneal it probably should be switched off.

I would appreciate any ideas on this topic especially experience from complicated models that run for long time and do not depend on read/write time.
Eldar

[gwhite]

MARK does parallel processing of the inner loop over the encounter histories. Each encounter history has a probability that contributes to the likelihood. If there are, say, 1000 unique encounter histories, then 4 processors would break the loop up into 250 encounter histories computed on each processor. So all optimization methods used in MARK benefit from the parallel processing, including simulated annealing. There is some overhead in this, which you can estimate by running a job using only one processor, and then repeating with 2 processors. The cpu time reported is the sum across all processors. But the actual wall clock time will be a little more than half. Note that most of the computation for computing the variance-covariance matrix is also in computing the likelihood function, so this calculation is also benefiting from parallel processing.

Gary

[Eldar]

Thanks Garry!
So for simulated annealing multiple cores will help too. Great!
If there is no limit for parallelisation then I don't understand why with 24 cores and complicated MS model I am getting load around 70-75% (maximum 78%)? Is it happening because of the read-write limits (it is a workstation, not a cluster, so there should not be any)?
Eldar

[gwhite]

You will generally not see a load of 100% because once the log-likelihood has been computed, there is a series of computations to decide what is the best set of parameter estimates to consider next. This calculation is done in series, not parallel. So, parallel processing only occurs during the loop over the encounter histories to compute the log-likelihood value, and not in the actual optimization code.

Generally it is not necessary to save one processor back for other jobs because MARK is not able to completely tie up your machine with only one MARK job running. However, if you start multiple MARK jobs....

Gary