parallel computing

[caspar]

Hi,

I noticed a potential problem with using mark.wrapper.parallel().
Running a list of 70 models in sequence (simply mark.wrapper) was successful, but for the same list of models R went out of memory (The full 39G of RAM!) when using mark.wrapper.parallel. I do not know whether this is a general phenomenon, or whether it only applies to my notebook.
Has anyone observed similar issues?

Thanks in advance

Caspar

[jlaake]

Eldar should probably chime in here but if you are using Windows, with MARK now being multi-threaded I think you are better off using mark.wrapper and let MARK.exe use the multiple CPUs. If you are on Linux, I'm not sure that mark.exe has kept up with the changes and the parallel wrapper may be useful. The parallel wrapper was written before Mark.exe was multi-threaded.

In any situation where you are spawning multiple procesess, memory is fixed so you have to be careful. It will depend on the size of each individual run whether it will all fit.

--jeff

[Eldar]

How many cpus you were using? If you have more than 4 then from my experience combination of cpus=.. and threads=8 is optimal.
Making external=TRUE can also help, as you may just have not enough RAM to keep output of all the runs.
Adding to Jeff's comment the only case when mark.wrapper.parallel is much more effective is profile likelihood as Mark is not paralellizing that part.
Eldar

[gwhite]

Eldar:

MARK does parallel process the profile likelihood computation. The same function that is used to generate the ML estimates is called. The inner loop that computes the -2log likelihood from the encounter histories is what is processed in parallel in MARK. So, anytime the likelihood is computed, the process is in parallel.

Gary