MCMC in Multi-state models

[StefanVriend]

Hello,

I am a student and I am doing a mark-resighting study on Black-headed gulls.
The data consists of 19 occasions, 967 individuals and five individual covariates (i.e. ring location, body mass, head-bill length, wing length and a condition index). Moreover, the framework is Multi-state (recaptures only), with three states.

The initial analysis with MLE seemed to have inaccurate estimates and problems with local minima. My supervisor and I therefore decided to switch to a Bayesian approach, via MCMC's. This analysis appeared to give a more accurate estimate than with MLE's, until I noticed something peculiar. The MCMC models were run in default settings (i.e. 4000 'tuning' samples, 1000 'burn in' samples, 10000 to store and 1 chain) and looked just fine. However, when I ran the exact same model twice, by accident, I noticed some differences in their outputs.

In the outputs of these two runs of the same model, the -2logLikelihood was equal (i.e. 4872 and 4812, respectively), but the "-2logLikelihood for means of beta estimates" were very different (i.e. 4879 and 5025, respectively), as well as the DIC (i.e. 4864 and 4598, respectively). Additionally, the standard deviations of both the beta estimates and the real estimates were bigger in the model with the highest "-2logLikelihood for means of beta estimates" and the lowest DIC.

I have no clue what to do right now. Do you have any suggestions on how to proceed?

Kind regards,
Stefan

[cooch]

The initial analysis with MLE seemed to have inaccurate estimates and problems with local minima.

Based on what?

My supervisor and I therefore decided to switch to a Bayesian approach, via MCMC's.

Two points:

Bayesians often use MCMC to evaluate moments from posterior distributions. However, simply using MCMC as a 'numerical tool' does not make you a Bayesian. So, unless you're doing interesting things with priors, or using MCMC to estimate multivariate hyperdistributions in MARK, you're not really using a 'Bayesian approach.'

Second, MCMC is not a solution for multi-modal likelihoods -- use simulated annealing. MCMC can be used to confirm the presence of multiple modes in the likelihood (as discussed in Chapter 10), but there isn't strong evidence that it will actually solve much of anything (in fact, you often simply end up with multiple modes in the posterior, forcing you to decide if standard moments -- like mean, or median -- are meaningful.)

This analysis appeared to give a more accurate estimate than with MLE's,

How are you defining 'accurate'?

until I noticed something peculiar. The MCMC models were run in default settings (i.e. 4000 'tuning' samples, 1000 'burn in' samples, 10000 to store and 1 chain) and looked just fine.

Based on what?

However, when I ran the exact same model twice, by accident, I noticed some differences in their outputs.

In the outputs of these two runs of the same model, the -2logLikelihood was equal (i.e. 4872 and 4812, respectively),

How are 4872 and 4812 'equal'?

but the "-2logLikelihood for means of beta estimates" were very different (i.e. 4879 and 5025, respectively), as well as the DIC (i.e. 4864 and 4598, respectively). Additionally, the standard deviations of both the beta estimates and the real estimates were bigger in the model with the highest "-2logLikelihood for means of beta estimates" and the lowest DIC.

Totally expected -- MCMC is based on 'random draws', and those, there will be small Monte Carlo differences between any two runs.

I have no clue what to do right now. Do you have any suggestions on how to proceed?

Simulated annealing.

[StefanVriend]

The initial analysis with MLE seemed to have inaccurate estimates and problems with local minima.

Based on what?

Based on large standard deviations and many non-estimable parameters, especially in the psi parameters.

Before I did this analysis, I used the CJS framework to dig into survival and resighting. Since survival estimates were low when compared with earlier studies, we thought a multi-state framework would give us estimates of true rather than apparent survival.

My supervisor and I therefore decided to switch to a Bayesian approach, via MCMC's.

Two points:

Bayesians often use MCMC to evaluate moments from posterior distributions. However, simply using MCMC as a 'numerical tool' does not make you a Bayesian. So, unless you're doing interesting things with priors, or using MCMC to estimate multivariate hyperdistributions in MARK, you're not really using a 'Bayesian approach.'

Second, MCMC is not a solution for multi-modal likelihoods -- use simulated annealing. MCMC can be used to confirm the presence of multiple modes in the likelihood (as discussed in Chapter 10), but there isn't strong evidence that it will actually solve much of anything (in fact, you often simply end up with multiple modes in the posterior, forcing you to decide if standard moments -- like mean, or median -- are meaningful.)

Okay, thanks for these insights.

This analysis appeared to give a more accurate estimate than with MLE's,

How are you defining 'accurate'?

Smaller standard deviations.

until I noticed something peculiar. The MCMC models were run in default settings (i.e. 4000 'tuning' samples, 1000 'burn in' samples, 10000 to store and 1 chain) and looked just fine.

Based on what?

These are the default settings. Since I am fairly new to MARK and new to MCMC I chose to keep those settings default.

However, when I ran the exact same model twice, by accident, I noticed some differences in their outputs.

In the outputs of these two runs of the same model, the -2logLikelihood was equal (i.e. 4872 and 4812, respectively),

How are 4872 and 4812 'equal'?

My apologies, I meant 'nearly equal'.

but the "-2logLikelihood for means of beta estimates" were very different (i.e. 4879 and 5025, respectively), as well as the DIC (i.e. 4864 and 4598, respectively). Additionally, the standard deviations of both the beta estimates and the real estimates were bigger in the model with the highest "-2logLikelihood for means of beta estimates" and the lowest DIC.

Totally expected -- MCMC is based on 'random draws', and those, there will be small Monte Carlo differences between any two runs.

I have no clue what to do right now. Do you have any suggestions on how to proceed?

Simulated annealing.

Thanks for your insights and advice. Is this possible within MARK? And is this combined with the MLE-approach or the MCMC-approach?

Sorry for all the questions; as I said, I am just a student and new to these analyses, so I am still finding a way to fully comprehend these matters.

King regards,
Stefan