negative AIC values

[jules]

Hi

I am a huge MARK fan and have just found this forum which looks most helpful.

I found a message concerning the ordering of AIC values by MARK when there are negative values and this mentioned that negative AICs had been discussed before. I haven't been able to find this discussion so I am sorry if I am being repetative.

My question is: I would assume AIC should be positive and a negative value implies that MARK is unable to calculate the likelihood due to a problem with the data not fitting the model structure. However, I seem to end up with negative AIC values even for data sets which I think are pretty good. Also there seems to be no correlation between data sets for which AIC for all models are negative and weather or not sensible parameters can be estimated. Could anyone who knows let me know if negative AICs are a serious problem and what I may be able to do with the data to get around the problem (if there is anything?)

Thank you so much in advance.

Julia Jones

[gwhite]

Julia:
Negative AICc values are not a problem -- they occur because a constant that should be in the likelihood has been left out to improve the speed of numerical computations. For example, in the closed captures likelihood, their is a factorial term in the denominator of the multinomial coefficient for each of the unique encounter histories. Because these terms are identical regardless of the parameter estimates, it is best to leave out the term and save the computer time required to compute them.

But, as a result, the constant being left out causes the AICc values to be negative. Interpretation of negative AICc values is still the same as if they were positive -- you're trying to minimize the quantity. Other than being negative, I don't think you'll see any differences in how MARK handles them.

Gary

[jules]

Dear gary

Thank you so much for getting back to me (and so fast!). This is brillient news, so now I understand the message (pasted below) which I found when I did a web search on neagtive AICs. With some data sets I get AICs varying from negative to positive values. I am guessing I remove the zero and order them by their magnitude and select the nearest to zero?

Thank again for your help

Julia

''Now, the fact that the AIC is negative has been discussed elsewhere in this forum. But, given the way MARK orders the models based on AIC, the best model will come LAST, not first, if the AIC values are negative. Basically, the best model is the model with the AIC closest to 0 (since AIC is just f(L) + np), which normally just means the smallest AIC. But not when the AIC values are negative. Unless I'm mistaken..."

[gwhite]

Julia:
I don't believe that the quote you appended is correct. The best model will still be the model at the top of the AICc list, as ordered by MARK. Remember, all the likelihoods have had a constant left off, and hence, all the AICc values have this same constant times -2 left off. Subtracting a constant from the values does not change the sort order, even if the values are negative. You are still looking for the model with the minimum AICc value.

Gary

[cooch]

Julia:
I don't believe that the quote you appended is correct.

The quote came from me, actually.

Bascally, it was something I posted a long time ago on the original forum. The background details are perhaps relevant - here is the original posting:

I have been fooling with some of the closed capture models in MARK, when I realized you need to be a bit careful with intepreting the AIC values.

Here's and example. I simulated a data set with N=5000 individuals, and a constant sampling fraction (i.e., true p) in each year of 25% (i.e., true p=0.25). Here are the simulated encounter histories:

0001 525;
0010 505;
0011 172;
0100 522;
0101 182;
0110 185;
0111 64;
1000 544;
1001 158;
1010 181;
1011 70;
1100 162;
1101 62;
1110 56;
1111 17;

Now, the parameter space is defined by p (probability of initial capture), c (probability of capture given you have already been captured at least once before), and N. So, based on these simulations, the true model is one where p and c are fixed to the same value) (i.e., model (p=c)N.

But, just for grins, I ran models p(.)c(.)N, p(t)c(t)N, and p(t)c(.)N (in the case of model p(.)c(.)N, p and c are constant, but allowed to differ).

Here are the AIC values, ranked in ascending order (the MARK default) - I give the estimates of N from each model to the right:

p(t)c(.)N -32344.90 N-hat=3405
p(t)c(t)N -32343.05 N-hat=3405
(p=c)N -32341.03 N-hat=4949
p(.)c(.)N -32339.12 N-hat=4910

Now, the fact that the AIC is negative has been discussed elsewhere in this forum. But, given the way MARK orders the models based on AIC, the best model will come LAST, not first, if the AIC values are negative. Basically, the best model is the model with the AIC closest to 0 (since AIC is just f(L) + np), which normally just means the smallest AIC. But not when the AIC values are negative. Unless I'm mistaken...

which (apparently) I was.

The best model will still be the model at the top of the AICc list, as ordered by MARK. Remember, all the likelihoods have had a constant left off, and hence, all the AICc values have this same constant times -2 left off. Subtracting a constant from the values does not change the sort order, even if the values are negative. You are still looking for the model with the minimum AICc value.

Gary

So, then, the question is more a function of why (in the example I was working with orignally) the 'true model' under which the data were generated ended up at the bottom of the list (implying it has the worst level of support). That is what led me to my original statement.

[gwhite]

Evan:
The reason why your models came out ranking "funny" is because the model {p(t) c(.)} does not estimate N. That is, you do not have a constraint in this model to estimate the last p. Likewise, {p(t) c(t)} also is not a valid model to estimate N. The following are the correct results for the simple CAPTURE models:

Simulated Closed Captures, N=5000, p=c=0.25

-------------------------------------------------------------------------------------------
Delta AICc Model
Model AICc AICc Weight Likelihood #Par Deviance
-------------------------------------------------------------------------------------------
{M0 p(.)=c(.)} -32341.029 0.00 0.63176 1.0000 2.0000 17.180
{Mb p(.) c(.)} -32339.120 1.91 0.24324 0.3850 3.0000 17.088
{Mh p2(.)=c2(.)} -32337.210 3.82 0.09360 0.1482 4.0000 16.997
{Mt p(t)=c(t)} -32335.026 6.00 0.03140 0.0497 5.0000 17.180
-------------------------------------------------------------------------------------------

As you can see, the model weights and ordering are all consistent with the M0 model used to generate the data. That is, the negative AICc value is not causing a problem.

Even the model averaging turns out pretty well:

Simulated Closed Captures, N=5000, p=c=0.25

Population Size (N) Group 1 Parameter 4
Model Weight Estimate Standard Error
---------------------------------------- ------- -------------- --------------
{M0 p(.)=c(.)} 0.63176 4949.5470000 76.2155220
{Mb p(.) c(.)} 0.24324 4910.7825000 145.8021500
{Mh p2(.)=c2(.)} 0.09360 5170.7600000 973.3527500
{Mt p(t)=c(t)} 0.03140 4949.5473000 76.2155310
---------------------------------------- ------- -------------- --------------
Weighted Average 4960.8229440 177.1123370
Unconditional SE 181.7881082
95% CI for Weighted Average Estimate is 4604.5182519 to 5317.1276361
Percent of Variation Attributable to Model Variation is 5.08%

Of course, these are EXCELLENT data, so the procedures had better perform well.

Gary

[cooch]

Evan:
The reason why your models came out ranking "funny" is because the model {p(t) c(.)} does not estimate N. That is, you do not have a constraint in this model to estimate the last p. Likewise, {p(t) c(t)} also is not a valid model to estimate N.

Yup, so I realize. I was fiddling with this stuff back when I hadn't much experience with the closed population estimators. Its all coming back to me now.

Of course, thats the nice thing about the web - you past triumphs, and your moments of humiliation - are recorded for all to see.