Deviance in POPAN (and some error messages)

[Onoufrios]

Hi all,

I've read on the forum and in the Book that a deviance cannot be computed in POPAN models, i.e. you get a deviance = 0.
Running some models with the latest stable MARK version (8.0) I get a deviance for all my models, which is always somewhere around the negative thousands (e.g. -4956.4576). Running the exact same analyses with the last stable MARK version (7.2), i get a deviance of zero, which as I understand it is what's expected.
Anybody has any idea why this happens? I don't really need to know the deviance, I just want to confirm that I'm not doing anything wrong.

Also, I quite often get an "error opening file xxxxxxx.tmp" message, followed by a "could not parse output properly" error message. This happens when I try to view the full output, the beta and the real estimates, and when I try to retrieve a model (also when i try to run a model, but only the first error message appears in this case). These messages are somewhat random, since I don't get them all the time and, when I get them, I just keep pressing the same buttons until the errors give up (until the next time they decide to re-appear)! These errors appear in both 8.0 and 7.2 versions of MARK regardless of the type of model (CJS, POPAN etc), so maybe it has something to do with my computer? Any help would be appreciated.

Thanks!

PS I wasn't sure if this was an "analysis help" or "software problems" kind of post, sorry if I got it wrong.

[gwhite]

I changed the saturated model calculation for POPAN to be consistent with the Pradel and Link-Barker models, hence the change. No different in the -2logL or the results, just the deviance value.

Gary

[Onoufrios]

Thanks a lot for the answer. Having the deviance calculated directly is sure better than having to run a POPAN model "under cover" as a Link-Barker model!

I have two more questions though:

Using the latest version of MARK (8.1 and not 8.0 as I say in my previous post) when I run a model, in the "append this model's output to database" window, very often I get this: "WARNING Numerical underflow occurred during final processing of this model" and sometimes also this: "WARNING Numerical underflow occurred during variance-covariance calculation of this model" (the latter usually in models with lots of parameters). These errors don't seem to affect parameter estimates or model AIC. Is this warning message something I should be worried about? Running the exact same analyses with MARK 7.2 I don't get any warnings. Maybe I should just use the previous version of MARK?

Second question (again, using POPAN models): I input my datafile, set number of groups and sampling occasions and proceed to run the general fully time-dependent model: Phi(g*t)P(g*t)Pent(g*t)N(g). Running the model using i) the PIMs (with the default PIM structure for group and time dependence) or ii) the Design Matrix (with the Design -> Full menu command), and setting the correct number of parameters, I should get the same model AIC right? It is the same model structure, with the same link functions (set through Parm-Specific function) and the same number of parameters, so why do I get a ΔAIC = 19? I also get a difference in AIC when fitting reduced models, although in these models the difference in AIC is smaller. In fact, as the model structure gets simpler, the AIC difference of models run with PIMs or with the Design Matrix tends to decrease. I sense that the answer is in front of me, but I fail to see it..

Thanks for helping me!

[cooch]

Thanks a lot for the answer. Having the deviance calculated directly is sure better than having to run a POPAN model "under cover" as a Link-Barker model!

I have two more questions though:

Using the latest version of MARK (8.1 and not 8.0 as I say in my previous post) when I run a model, in the "append this model's output to database" window, very often I get this: "WARNING Numerical underflow occurred during final processing of this model" and sometimes also this: "WARNING Numerical underflow occurred during variance-covariance calculation of this model" (the latter usually in models with lots of parameters). These errors don't seem to affect parameter estimates or model AIC. Is this warning message something I should be worried about? Running the exact same analyses with MARK 7.2 I don't get any warnings. Maybe I should just use the previous version of MARK?

The newest version of MARK is compiled with a different (newer) version of the gfortran compiler. Said new version of the compiler introduces more warning flags than the older compiler that was used toi compile version 7.2. The 'problems' are inherent in the challenges of optimizing the likelihood for those models. The 7.2 version simply doesn't tell you about them, where the 8.1 version does. Your choice. You can run 7.2, and pretend that the challenges aren't there, or run 8.1, and be made aware that they are.

Second question (again, using POPAN models): I input my datafile, set number of groups and sampling occasions and proceed to run the general fully time-dependent model: Phi(g*t)P(g*t)Pent(g*t)N(g). Running the model using i) the PIMs (with the default PIM structure for group and time dependence) or ii) the Design Matrix (with the Design -> Full menu command), and setting the correct number of parameters, I should get the same model AIC right? It is the same model structure, with the same link functions (set through Parm-Specific function) and the same number of parameters, so why do I get a ΔAIC = 19? I also get a difference in AIC when fitting reduced models, although in these models the difference in AIC is smaller. In fact, as the model structure gets simpler, the AIC difference of models run with PIMs or with the Design Matrix tends to decrease. I sense that the answer is in front of me, but I fail to see it..

Thanks for helping me!

PIM-based and DM-based default to different link functions. Look at the model deviances, *not* the AIC values (i.e., learn to read right-to-left in the browser) -- if the deviances are the same, then they're the same models.

[Onoufrios]

The 7.2 version simply doesn't tell you about them, where the 8.1 version does. Your choice. You can run 7.2, and pretend that the challenges aren't there, or run 8.1, and be made aware that they are.

Understood, thanks.

PIM-based and DM-based default to different link functions. Look at the model deviances, *not* the AIC values (i.e., learn to read right-to-left in the browser) -- if the deviances are the same, then they're the same models.

The deviances are not the same. However, they should be, since I am fitting the same models!
Yes, PIM- and DM- based default to different link functions, but I specified the same link functions (through the Parm-Specific dialog window) for these models: Logit for Phi and P, MLogit(1 through 4) for the Pent (I have 4 groups) and Log for N.
I might have done something wrong in the huge Design Matrix for the reduced models (13 occasions, 4 groups..), which could explain the different model deviances. But this cannot explain the differences in the fully time-dependent model, in which I did not change anything from the defaults (apart of course from the link functions).
So, to put it another way: Let's say that we begin a new POPAN analyses (input our data, name the groups etc) and A) fit the general time-dependent model with the default PIMS (i.e. g*t in all parameters) or B) go to Design -> Full and then fit this model (again, not changing anything). In both cases we ourselves specify the link functions (as in the previous paragraph). Aren't A and B the same models, since their structure is the same?

Maybe the fact that MARK uses an identity design matrix by default when working with PIMs has anything to do with it?

Thank you very much for taking the time to answer my questions!

Also, if anybody else had the same problem with "error opening file xxxxxxx.tmp" and/or "could not parse output properly" error messages, I found that a simple update of my text editor (notepad++) fixed the problem.

[cooch]

The 7.2 version simply doesn't tell you about them, where the 8.1 version does. Your choice. You can run 7.2, and pretend that the challenges aren't there, or run 8.1, and be made aware that they are.

Understood, thanks.

PIM-based and DM-based default to different link functions. Look at the model deviances, *not* the AIC values (i.e., learn to read right-to-left in the browser) -- if the deviances are the same, then they're the same models.

The deviances are not the same. However, they should be, since I am fitting the same models!
Yes, PIM- and DM- based default to different link functions, but I specified the same link functions (through the Parm-Specific dialog window) for these models: Logit for Phi and P, MLogit(1 through 4) for the Pent (I have 4 groups) and Log for N.
I might have done something wrong in the huge Design Matrix for the reduced models (13 occasions, 4 groups..), which could explain the different model deviances. But this cannot explain the differences in the fully time-dependent model, in which I did not change anything from the defaults (apart of course from the link functions).
So, to put it another way: Let's say that we begin a new POPAN analyses (input our data, name the groups etc) and A) fit the general time-dependent model with the default PIMS (i.e. g*t in all parameters) or B) go to Design -> Full and then fit this model (again, not changing anything). In both cases we ourselves specify the link functions (as in the previous paragraph). Aren't A and B the same models, since their structure is the same?

Maybe the fact that MARK uses an identity design matrix by default when working with PIMs has anything to do with it?

Thank you very much for taking the time to answer my questions!

Also, if anybody else had the same problem with "error opening file xxxxxxx.tmp" and/or "could not parse output properly" error messages, I found that a simple update of my text editor (notepad++) fixed the problem.

POPAN models are 'twitchy' at best. With large dimension problems, likely somewhat sparse data from some axes of the problem, and full time-dependence, then the chances are very high that there will be significant challenges for the optimization routines to work well, or consistently over what might seem equivalent structures and/or link functions. I have any number of situations where what I know to be identical models (PIM vs DM) don't yield the same deviances for the fully general model, but do for simpler, reduced-parameter models. I also have any number of cases where simulated annealing doesn't work, while the default optimization does (which may not be the same as, giving the right answer). I suspect that there are all sorts of weird nuances to the likelihood for POPAN models (perhaps Carl or Gary have looked at this), likely exacerbated for big models with lots of time-dependence (as an aside, the fully time-dependent model is probably a silly model in practice anyway, given the number of confounded parameters).

[Onoufrios]

I have any number of situations where what I know to be identical models (PIM vs DM) don't yield the same deviances for the fully general model, but do for simpler, reduced-parameter models.

This is what seems to happen in my case. I'm just not so confident in my skills to know that the models are identical.

(as an aside, the fully time-dependent model is probably a silly model in practice anyway, given the number of confounded parameters).

My most general model is not the fully time-dependent anyway, i just fitted it to perform a Release GOF test. But, seing the discrepancies in AIC and deviance I got woried that I'm doing something wrong in the Design Matrix (it is quite big! ).

Thank you very much for clarifying these things for me!

Cheers,
Onoufrios