Issues with model-averaging option in PRESENCE

[Alfricho]

Dear all,
I am currently on the model averaging part of occupancy estimates of a mammal in Bolivia. I see that PRESENCE 3.1 has an option to do this automatically (in Tools -> Model averaged estimates), and I guess this is the most time-efficient way to do it. However, after running this option I get the estimates for some of sites that are higher than 1 and all the standard errors are just huge (although the SE values are reasonable for individual models). What could be the problem? Is there another way to take the estimates in a time-efficient way?

Probably the fact that I have built many models, (>370, for which 190 account for 90% of the model weight), may be a factor for at least the high standard errors? Also, a few of the models present the ‘Numerical convergence warning’ (however they represent around 1% of all models); although the number of significant digits are general higher than 5.0. I did not get any variance-covariance warnings.
Many thanks for any answers,
Alfredo

[darryl]

Jim Hines could confirm, but I understand the model averaging option is very much in the 'beta' stage so it's likely there's a few bugs in there at the moment.

I'd be more concerned by the fact that you've fit >370 models. I hope you have a lot of data! The fact that you need 190 models to account for 90% of the AIC weight is also a bit concerning. Either you're fitting a lot of very similar models, have highly correlated predictor variables, little signal in the data to explain, or all of the above.

[Alfricho]

Many thanks Darryl for your helpful reply.

I was also concerned about the large number of models I have ended up building. I think I am commiting a couple of the mistakes you are suggesting, so I will re-run the models after checking for these issues. Another aspect that is inflating the number of models is the fact that I selected 4 detection (p) models that had a dAIC of <2 (no covariates used for psi), so that later I combined each psi model with each of the 4 p models, although I think this is not as important as the above problems.

Thanks,
Alfredo

[BetsieR]

Similar to Alfredo, I am using the model averaging function in PRESENCE 3.1 and am getting very large SEs and some estimates of psi >1.0, even though none of my individual models had convergence warnings or var-covar problems.

For example, I ran a set of 32 candidate occupancy models for a species of salamander; after correcting for small sample size, the 95% confidence set includes 18 models. The estimate of psi from the top model is 0.967 (SE 0.049) whereas the model-averaged estimate is 0.959 (SE 1.524).

Thus, I’m wondering about Darryl’s comment that there may be bugs in the model averaging tool. Can anyone confirm whether this is the case?

Thanks for any clarifications.

--Betsie

[darryl]

The model averaging tool in PRESENCE it still a work in-progress, and is only being done for the first site in your data set so is specific to the covariates values for that site. Sorry, but you still need to be doing in all 'by hand' at the moment. You need to go through the output for each model, copy the psi estimates and SE's into a spreadsheet, then get your model averaged estimates from there.
Darryl

[BetsieR]

Hi Darryl,

Thanks, somehow I missed the memo on that...

Ironically, if the only problem is that PRESENCE is only doing the model averaging for the first site in the dataset, that would actually be okay for what I am currently doing. My first site is a dummy "mean" site (with no detection history and average values entered for all the covariates) because I wanted an overall (not site-specific) occupancy estimate. But I suspect there might be other issues since I am getting those big SEs, so I'll take your advice and work this out by hand.

Appreciate the quick reply!

--Betsie

[darryl]

Note that a 'mean site' doesn't necessarily give you a good 'overall' estimate. Really to get an accurate 'overall' value you should be using the model(s) fit to the sample data to predict occupancy for all sampling units in your populations of interested (including those not surveyed), then combine the predictions to get the overall value. It does require you to have the covariates values for all sampling unit however. If you don't have that, but have used used a robust sampling methodology (ie how you decide which sampling units, ie sites, you'd collect data from and which ones you wouldn't), then a 'mean site' might be ok as an overall estimate. If you're sampling method was a 'dodgy' then the 2 could be very different.

[jhines]

Darryl's right that the model-averaging is a work-in-progress. I put the code in PRESENCE when someone asked about it, but soon realized that I didn't know what most people would want from it. When you don't have covariates, is seems pretty obvious to me that people would probably want model-averaged estimates of the parameters psi, gamma, epsilon, p,... Most people seem to have covariates in occupancy analysis, so I wasn't sure what to compute. I didn't think people would want a model-averaged estimate of every parameter for each site and survey. As Darryl said, computing a model-averaged estimate from means of the covariates doesn't seem that attractive either.

So, any feedback on what you'd like from model-averaging would help me improve this tool. Also, if you're getting unreasonable output, I'd like to see your data/results (presence_backup5.zip).

Jim

[BetsieR]

I appreciate Darryl’s cautionary note about obtaining an “overall” occupancy estimate. I’ll have to think about the implications for my particular situation; unfortunately we didn't have the resources to obtain measures of covariates for sites other than those actually surveyed (we would have had to physically visit an unknown number of other suitable sites to get data on stream widths, etc.).

Jim, I see the dilemma now re: what to do when there are covariates. I tend to agree the site and survey-specific estimates would not be very useful. Note that when I run the model averaging tool, I sometimes get estimates of psi and p for just the first site (as Darryl indicated) but not always – sometimes I get estimates for all sites and/or surveys. I’d be happy to have you look at the results for one or two of the species I’m working on, so I’ll send you those backup files.

Thanks!
Betsie

[jhines]

I've re-written the model-averaging routine in PRESENCE so that it computes model-averaged estimates of psi (or the first estimated parameter) for each site. The old routine assumed that each model had an estimate of psi printed for each site, where the current routine does not. I think this is better than what PRESENCE was doing, but I need to work on having it work with the other parameters (ie., detection, colonization,...).

I imagine most people will want the model-averaged psi's, colonization and extinction, but I can't think of a reason for wanting model-averaged p's. Any feedback on this would be appreciated.

By default, this routine will produce a lot of output... estimates of psi for each site and model, followed by the model-averaged estimate. Once you're convinced it works, you can ask it not to print the individual estimates from each model.

Jim