Sums of Akaike weights and inference

Forum for discussion of general questions related to study design and/or analysis of existing data - software neutral.

Sums of Akaike weights and inference

Postby howeer » Fri Sep 01, 2006 11:18 am

In a set of 24 candidate models with 5 predictors of survival in a known fate analysis, I have 1 factor that shows up in all 4 models with AICc weights < 2, and has a Sum of Akaike weights = 0.76 (factor was included in 1/2 of the candidate models), so I can infer that this factor affects survival. However, I also have 2 other factors which each appear in half the models with AICc weights < 2.0, and have Sums of Akaike weights close to 0.5 (they were each included in 1/2 of the models in the candidate set). What inferences can I make about the importance of these factors based on these values? What other evidence can I use to decide whether these are significant/relevant predictors of survival?
howeer
 
Posts: 39
Joined: Wed Jun 21, 2006 10:49 am

Re: Sums of Akaike weights and inference

Postby bmitchel » Mon Sep 04, 2006 12:04 pm

Hello Eric,

I would argue that the Sum of Akaike Weights, on its own, tells you nothing about whether a given factor significantly affects survival, either in a statistical or biological sense. It would be fairly easy to construct a model set with spurious factors, one of which (by chance) will have a high Sum of Akaike Weights but which (in reality) is meaningless. The Sum of Weights only tells you how important a factor is relative to the other factors in the model set. So be very careful saying that a factor affects survival based on the Sum of Akaike Weights - this is very shaky inference.

If you want to know how important a factor is, you will probably need to look at the model-averaged estimate and standard error / confidence interval. Does the confidence interval for the parameter overlap zero (a decent measure of statistical significance)? Does the size of the estimated parameter represent a biologically meaningful amount?

It may also help to look at the effect of the factor on survival, across a range of reasonable factor values. If changing the factor does not change your model-averaged survival estimate, it is probably not biologically important.

Brian
bmitchel
 
Posts: 28
Joined: Thu Dec 09, 2004 9:57 am

Re: Sums of Akaike weights and inference

Postby cooch » Mon Sep 04, 2006 4:45 pm

howeer wrote:In a set of 24 candidate models with 5 predictors of survival in a known fate analysis, I have 1 factor that shows up in all 4 models with AICc weights < 2, and has a Sum of Akaike weights = 0.76 (factor was included in 1/2 of the candidate models), so I can infer that this factor affects survival. However, I also have 2 other factors which each appear in half the models with AICc weights < 2.0, and have Sums of Akaike weights close to 0.5 (they were each included in 1/2 of the models in the candidate set). What inferences can I make about the importance of these factors based on these values? What other evidence can I use to decide whether these are significant/relevant predictors of survival?


Assessment of the relative importance of variables has often been based only on the best model (e.g., often selected using a stepwise testing procedure). Variables in that best model are considered “important,” while excluded variables are considered not important. This is too simplistic. Importance of a variable can be refined by making inference from all the models in the candidate set (see Burnham and Anderson 2002, chaps. 4–6). Akaike weights are summed for all models containing predictor variable x(j), j = 1, . . . , R; denote these sums as w_+(j ). The predictor variable with the largest predictor weight, w_+(j ), is estimated to be the most important; the variable with the smallest sum is estimated to be the least important predictor.

This procedure is superior to making inferences concerning the relative importance of variables based only on the best model. This is particularly important when the second or third best model is nearly as well supported as the best model or when all models have nearly equal support. (There are “design” considerations about the set ofmodels to consider when a goal is assessing variable importance. See Burnahm and Anderson)
cooch
 
Posts: 1628
Joined: Thu May 15, 2003 4:11 pm
Location: Cornell University


Return to analysis & design questions

Who is online

Users browsing this forum: Google [Bot] and 11 guests

cron