Meaning of different c-hats

[Luc te Marvelde]

When i read the 'Markbook' the 'c-hat topic' is not really clear to me (probably because of my lack on understanding). Here are my questions:


What the differrence between the c-hat generated by my model (which i can find in my model output) and the c-hat which is calculated by dividing the observed c-hat by the mean (bootstrapped) c-hat.

And... which one tells me the amounth of overdispersion i got in my data?

And...which c-hat shouldnt be more than 3 (=rule of thumb in GOF)?



Thanks in advance!

Luc te Marvelde
University of Groningen (RUG)

[cooch]

What the differrence between the c-hat generated by my model (which i can find in my model output) and the c-hat which is calculated by dividing the observed c-hat by the mean (bootstrapped) c-hat.

And... which one tells me the amounth of overdispersion i got in my data?

Pretty obvious from the book - pages 23-25. The c-hat you use (if using bootstrapping) is some measure of the observed deviance (from the model) divided by some measure of the distribution of deviances (from the bootstrapping, for example).

And...which c-hat shouldnt be more than 3 (=rule of thumb in GOF)?

The c-hat in question is your estimate of c-hat - either from the boostrap, the median regression, or RELEASE.



Thanks in advance!

Luc te Marvelde
University of Groningen (RUG)[/quote]

[Luc te Marvelde]

After reading the 'Goodness of Fit'-chapter agian and again and discussing it with other MARK-users it still is not clear to me. Almost everyone i talk to gives me a different answer. Therefor ill put some results up here, and i hope you can tell me whether my model fits or not. If you know, can you explain to me what the different methods i discribed tells me?



====================================================
General model output:

Number of Estimated Parameters {phi(sex*plot),p(plot)} = 12
DEVIANCE {phi(sex*plot),p(plot)} = 4276.1013
DEVIANCE Degrees of Freedom {phi(sex*plot),p(plot)} = 300
c-hat {phi(sex*plot),p(plot)} = 14.253671



====================================================
Using a bootstrap (100x):

I did a bootstrap analysis on model {phi(sex*plot),p(plot)}, these are the summed results:


Statistical Summary of Numerical Variables
(Number of Observations = 100)

95% Confidence Interval
Variable Mean Standard Error Lower Upper
---------- --------------- -------------- --------------- ---------------
DEVIANCE 4122.271 10.1302 4102.415 4142.126




so.. c-hat= 4276.1013 / 4122.271 = 1.0373



=====================================================
Ordened deviance method:

When i rank all 100 bootstrapped deviances, my observed deviance falls between the 93th and the 94th bootstrapped deviance.

Should my observered devaince fall in the tail (say the last 5%) of the expected deviances or just the other way around?

======================================================





Thanks in advance,
Luc te Marvelde

[cooch]

I did a bootstrap analysis on model {phi(sex*plot),p(plot)}, these are the summed results:


Statistical Summary of Numerical Variables
(Number of Observations = 100)

95% Confidence Interval
Variable Mean Standard Error Lower Upper
---------- --------------- -------------- --------------- ---------------
DEVIANCE 4122.271 10.1302 4102.415 4142.126


so.. c-hat= 4276.1013 / 4122.271 = 1.0373

Correct

=====================================================
Ordened deviance method:

When i rank all 100 bootstrapped deviances, my observed deviance falls between the 93th and the 94th bootstrapped deviance.

Should my observered devaince fall in the tail (say the last 5%) of the expected deviances or just the other way around?

So, you have some evidence for lack of fit. If the model fit perfectly, you'd anticipate that 50% of the simulated deviances would fall above the observed value, and 50% would fall below.

So, looking at the rank ordering of the deviances gives you some evidence of whether or not there is lack of fit (and even gives you a P-value if you're so inclined), while the c-hat is the 'adjustment' you need to acccount for this lack of fit when deriving AIC vaues or other metrics for model selection.

In fact, this is discussed on p. 23-25 of the GOF chapter. Specifically, on p. 24, the use of the ranked decviances, and the estimation of c-hat, are both specifically accounted for.

[Luc te Marvelde]

thanks for the fast reply!

One additional question.. does the c-hat from the general output mean nothing? (=model specific c-hat)

[Luc te Marvelde]

One additional question.. does the c-hat from the general output mean nothing? (=model specific c-hat)

[jeffmoore]

G White recently clarified this topic for me as well.

I'm surprised (should I be?) that your rank-order deviance approach suggests some potential lack of fit (P = 0.065), but your estimate of c-hat is so close to 1 after bootstrapping. I guess I would have expected a larger adjustment required (closer to 1.5?) following your rank-order results.

Suggestion: since rank order of your statistic is between 93 and 94, you should probably consider running more bootstraps (N = 1000?). The Cooch and White manual suggest stopping at 100 bootstraps only if your rank-order is closer the 50:50 mark.

To answer your last question: no, the observed c-hat from your general output does not "mean nothing". Recall you divided this observed c-hat by expected c-hat from bootstrapping to come up with your best estimate of c-hat (1.04 in your case).

[cooch]

To answer your last question: no, the observed c-hat from your general output does not "mean nothing". Recall you divided this observed c-hat by expected c-hat from bootstrapping to come up with your best estimate of c-hat (1.04 in your case).

True - it doesn't mean nothing 'in context' - its the value you compare against the distribution of generated c-hats. But, taken alone, it is not particularly interpretable. (utility -> 0).