ML-SECR huge running time

[Benoit Pisanu]

ML-SECR huge running time with lot of data and learned response from multiple sessions

Hi all

I’m running density analyses with secr 2.3.1, using a dataset including 3950 capture-recapture events of Siberian chipmunks (Tamias sibiricus), collected on a 14 ha area using 104 traps (spaced on average 30m), during monthly CMR trapping sessions of 5-days occasions over 3 years during the period of activity of squirrels (i.e. March-October, hence 24 sessions).
I know there is heterogeneity in both capture probabilities and movement scale according to age and sex. Rather than using finite mixture (h2 or h3), I separated my dataset according to age (juveniles and adults) and to sex (f & m), and included these levels as different “sessions”.
I also know that there is local trap-happiness for squirrels (site-fidelity), and I strongly suspect within session trap-happiness – with some individuals selecting some traps within their home range within a session.

I have no trap saturation, hence have been driven to use ML SECR rather than IP-S method.

I firstly ran the following models on a single year, including thus 28 sessions (8 sessions with adult females, 8 with adult males, 6 with young females and 6 with young males – as no juveniles are found in the population in March-April).

secr0<- secr.fit(chiptot, buffer=150, model = list(D~session, g0~1, sigma~1), method='BFGS')
secr0gbk<- secr.fit(chiptot, buffer=150, model = list(D~session, g0~bk, sigma~1), method='BFGS')
secr0sigbk<- secr.fit(chiptot, buffer=150, model = list(D~session, g0~1, sigma~bk), method='BFGS')


The model secr0 achieved calculations after ~20 hours !! Results were reliable, giving very similar density estimates compared to a previous analyse I made using S-IP method in DENSITY 4.4.
The model secr0bk achieved calculations after 6 DAYS….again giving reliable (and better estimates according to delta AICc….).
The model secr0sigbk is still running (seems to iterate well, probably will end tomorrow….)

Does anyone know how to reduce running time for such models – more specifically if I want to run models including all 3 years of my dataset???

We are using Dell Precision 390 computers – with Intel® Core™2 CPU 6320, 1.86 Ghz and 2.0 Go RAM

Thanks!
benoit

[howeer]

Hello,
Some of the SECR models we fit to ourdata take > 1 week to fit. To reduce computation time we tried:
1) reducing the resolution of the integration mesh (fewer points spaced further apart). We used mask.check to make sure this would not cause bias.
2) including gradtol and stepmax arguments in the call to secr.fit. This only works with method = "Newton-Raphson".
E.g. fitted.model = secr.fit(seeh,model=list(g0~1,sigma~1), mask=mask,verify = T,gradtol=1e-04,stepmax=10). We checked for differences in parameter estimates when we included these arguments vs. leaving them at default values.

Good luck!
Eric

[murray.efford]

Ouch - that does sound intolerable. My first step would be to precompute the mask (rather than specifying buffer in secr.fit) and to look very carefully at the mask to ensure it has no more points than needed. The default resolution (64 x 64) is often excessive try e.g. 32 x 32. Some marginal points on the default rectangular mask may actually be far from a trap (use method = 'trapbuffer' in make.mask with an appropriate buffer to trim these out). Also, method = 'BFGS' is probably slower than the default.

With the models you show it would make sense to maximize the conditional likelihood (CL = TRUE in secr.fit), and follow up with derived(); that avoids fitting a separate D for each session, but you still end up with session-specific densities because n varies among sessions.

Beyond that, you may need to make some compromises on model structure. The number of sessions is quite large, perhaps unnecessarily so, and inflating that by using separate sessions for age and sex classes is possibly not the way to go. With CL models you have the option of fitting age and sex as individual covariates (although I suspect that will be even slower). If you don't need to pool data between sexes then it will be faster to split the dataset into males and females. I can't remember whether the fairly new 'bk' models are slower to fit than simple 'b' models.

Try some of these suggestions and see how you go. I would like to see if we can get it going in minutes rather than days - if you send me an example of the data offline I may be able to help more.
Murray

[murray.efford]

Of course Eric's first point and mine are the same... I should read before I write!
Murray

[Benoit Pisanu]

Hi Eric and Murray

I realize that i didn’t read enough the manual’s FAQ!!
Thank’s for your help and for applying the golden rule “to repeat is to teach” (although not too often )

I ran the following:
tempmask32<-make.mask(mytraps, buffer = 100, spacing = 32, nx = 32, type = "trapbuffer")
secr05omask32<- secr.fit(chiptot05, mask=tempmask32, model = list(D~session, g0~1, sigma~1), CL=T, method='BFGS')

Reducing the buffer width to 100 (rather than 150) and the integration mesh down to 32x32, using ‘trapbuffer’, and maximizing CL, greatly improved running time, with models now achieving a solution within <1-30 mn !! (28 sessions model: 2 mn for null, 15 mn for "g0~bk", 30 mn for “sigma~bk”)
For information: with mask set to default: 20 hrs for null, 7 days for ”g0~bk”, 13 days for “sigma~bk”
The parameters estimates do not depart from previous models set to default, and ‘mask.check’ indicated that -LL did not greatly varied with buffer and spacing width variation within session. Note that models with “g0~bk” unambiguously best fit the data in all cases.

Using the method ‘BFGS’ produced successful computations in all cases (i.e. separating by age and sex [28 sessions], pooling age levels [16 sessions], pooling all individuals [8 sessions], even keeping session with very low number of capture events). Running time were similar to models ran successfully with the 'N-R' default method. However, the default (N-R) only works with "g0~bk" or "sigma~bk" when i) reducing the model structure by pooling all age and sex levels, ii) eliminating a session with very little capture events (session8), and iii) setting “gradtol=1e-04, stepmax=10” as Eric proposed. Differences in density estimates between the 2 methods are nevertheless small, bias on average about 1% (-1.5%; 0.4%), whatever the (same) situation I tried.

Although my problems seem to be solved (until now), method ‘BFGS’ appear very useful in the sense that it can compute density estimates for sessions with sparse data. Are there any arguments to prefer N-R to BFGS in my case? – Below some details on the data set.

Let me know if you’re interested in having a look at the dataset

Cheers

Ben

> summary(mytraps)
Object class traps
Detector type multi
Detector number 104
Average spacing 30.59961 m
x-range 611500.1 611896 m
y-range 2406230 2406755 m

adf1 adf2 adf3 adf4 adf5 adf6 adf7 adf8 adm1 adm2 adm3 adm4 adm5 adm6
Occasions 5 5 5 5 5 5 4 5 5 5 5 5 5 3
Detections 33 72 48 59 50 38 21 5 15 24 11 12 14 5
Animals 18 34 26 32 22 22 13 1 8 14 10 8 8 4
Detectors 104 104 104 104 104 104 104 104 104 104 104 104 104 104

adm7 adm8 juf3 juf4 juf5 juf6 juf7 juf8 jum3 jum4 jum5 jum6 jum7 jum8
Occasions 5 5 5 5 5 5 5 5 5 5 5 5 5 5
Detections 11 5 9 68 45 17 44 5 13 80 72 16 41 4
Animals 4 1 8 30 23 11 27 2 8 33 24 13 22 3
Detectors 104 104 104 104 104 104 104 104 104 104 104 104 104 104