What to do when your best model has a couple of dodgy SE's

[louise.fairless]

Dear all,

I would like some advice on the best way forward when after setting up all your candidate models, the highest ranking model with the lowest AIC has a couple of problem parameters with crazy SE's. Lets consider a situation where your best model is one where survivorship varies over time and you want to report the mean survivoship over a certain period, or indeed the whole time period. If all parameters (except confounded ones) were calculated well, we would easily be able to report a mean value of survivorship with a sensible SE. But, what if a small fraction of these parameters are poorly estimated with SE's in their tens or even hundreds. With these problematic estimations, we would not get a sensible mean survivorship with a sensible SE, for example we would get a phi value of something like 0.79 with an SE of 36.7.

Would it be satisfactory to simply state where problem parameter occur and calculate a mean value over a reduced number of time intervals by omitting these problem parameters?

Obviously, the first things to do would be to look at the m-array to try to understand why these problem parameters occur and if they are fixable, but if nothing is clear in the m-array and the problem is more to do with sparse data in a few places, is the approach above the only option?

Thanks in advance for your thoughts,

Best wishes,


Louise

[bacollier]

Louise,
First question I would ask is whether or not those parameter are actually being 'counted' in the AIC value. if the parameters with dodgy SE's are not being counted (because they are inestimable) then your AIC values may not actually be representing the true AIC (remembering its 2K, so if you are missing 3 K, the AIC is 6 lower than it should be) thus your highest ranking model may not really be the best given the data. I don't remember offhand if MARK counts them or not when they are inestimable, I don't think it does if I remember right unless that was changed recently.

Next, I don't know what model you are running or how it is structured and could probably better advise with that knowledge, but I suspect since you are talking m-array you have some sort of a temporal model that has very few 'events' within some sort of time frame? You could either fix those inestimable parameters, or you could merge time frames (thinking like a PIM) top ensure that you have enough events for estimating each parameter.

Bret

[louise.fairless]

Bret,

Thank you for taking the time to reply to my message. The inestimable parameters are not counted in MARK, but I have adjusted the number of parameters for the models where parameter counting is an issue. Am I right to say that as long as I adjust the model to have the right number of parameters, its ranking as the best model (with the lowest AIC) is valid because the parameters have been adjusted to the correct number of parameters that should be estimated by the model? Please can you say if I have this correct? For example, if my best model should have 100 identifiable parameters, but only 98 were estimable due to some sparsity in the data, as long as I adjust the number of parameters from 98 to 100 (which will adjust the AIC) the model ranking is valid, right???. There are few unestimable parameters in the model, 98% of parameters are estimable in a model with 260 parameters.

Best wishes,

Louise

[bacollier]

Bret,

Thank you for taking the time to reply to my message. The inestimable parameters are not counted in MARK, but I have adjusted the number of parameters for the models where parameter counting is an issue. Am I right to say that as long as I adjust the model to have the right number of parameters, its ranking as the best model (with the lowest AIC) is valid because the parameters have been adjusted to the correct number of parameters that should be estimated by the model? Please can you say if I have this correct? For example, if my best model should have 100 identifiable parameters, but only 98 were estimable due to some sparsity in the data, as long as I adjust the number of parameters from 98 to 100 (which will adjust the AIC) the model ranking is valid, right???. There are few unestimable parameters in the model, 98% of parameters are estimable in a model with 260 parameters.

Best wishes,

Louise

Louise,
Forgoing any vagaries in parameter estimability and its affect on the log-likelihood for whatever model you are working with, then your correction to the AIC values should be ok (anyone have a different opinion?).

As an option, if your 'dodgy' parameters are at the boundary an option might be to fix them for your analysis, that might help solve your problem. However, if you don't fix them then you should do whatever you can to ensure that your model fully estimates those parameters using one of the approaches I previously suggested.

Bret

[louise.fairless]

Thank you for your help Bret, I will investigate further into what is causing these parameters and will look at either fixing the problem parameters or merging time frames as you suggested to give these problem parameters the best chance of being estimated.

Best wishes,

Louise

[dhewitt]

Sorry to be so late to this thread. This is a common topic of discussion here for our capture-recapture program with a long-lived fish. The issue doesn't HAVE to be related to sparse data (although it usually is).

We get survival estimates on the boundary at 1.0 (either tiny or huge SEs), and they are probably reasonable. We have lots of data and good re-encounter probabilities and we often see estimates in other years > 0.95. So we run into the same problem Louise describes and have never fully resolved what to do about it.

It concerns me for a couple reasons, one of which is that I worry that the maximum of the likelihood is not truly found and the SEs (and perhaps even MLEs) for other parameters might be affected. Sometimes we have 10-20% of the parameters like this in a given model, and given the great re-encounter probs we have (> 0.8) and the sometimes poor years of survival, model selection always pushes time-dependent models to the top.

We have looked at dropping those parameters, but that seems like a bad idea because survival was probably very close to 1.0. If we drop it, we are "leaning" more toward estimates of lower survival and biasing our overall assessment. If we fix the parameter at 1.0, does that affect other parameters in the likelihood in a worrisome way? How do you deal with the SE for that value when model-averaging, or calculating a overall measure of central tendency for survival over the period (with Delta method)?

Oh, and Bret is of course correct that you have to be very careful about parameter counts from MARK in these situations -- always know the number and adjust accordingly, rather than trust what MARK says. But I don't know that those corrections necessarily mean that model selection is "clean" in the face of boundary estimates. I do agree with Bret that if you have 98% parameters estimable, you're probably OK on the model selection issue at least.

[ganghis]

Hi Dave,
This seems like a situation that screams out for random effects and/or Bayesian estimation. If parameters are hitting the boundary because of lack of data, doing this should pull the estimates back towards the mean. On the other hand, if there really is signal that S is close to 1.0, estimates should remain close to 1.0. If you use MCMC, you should be able to calculate a posterior predictive distribution for things like cumulative survival in a fairly straightforward manner (model averaging would be a bit more involved).

Cheers, Paul

[dhewitt]

We sorta gathered that's the way we'd end up heading. As for model-averaging being "a bit more involved," I certainly agree!