model selection philosophy

[AndyT]

I am wondering if some of you can suggest whether the following approach is valid, and any problems concerns you can see:

As I'm sure most of us are, I am much more interested in survival rates than recaputre. I have a dataset where I think sex, habitat (four types), time, and a covariate for body condition all should be important for survival. Read: many parameters.

In order to simplify my model selection procedure, I was thinking of first finding out the best combination of parameters describing p by setting phi to be fully parameterized, and running models where only the parameters for p vary. I would then use the p parameters from the top model and hold p constant in all of the models that I run to test phi.

Does this make sense? Any input would be a great help.

Thanks very much,
Andy

[rlong]

Hi Andy,

I was in a similar situation within an occupancy modeling framework, where I was interested in estimating both detection (p) and occupancy (psi) as functions of site- and visit-specific covariates. I was actually very interested in p because I needed to compare the performance of a number of different detection devices, but I also wanted to build predictive occurrence models using the estimates for psi.

After receiving some input from folks with extensive occupancy modeling experience, I decided to take an approach similar to yours. I first performed model selection for p, while keeping a "general" set of potential psi covariates in every model. However, when I went to model psi, instead of using only the p parameters for the top model, I again used a general set of p covariates for all models. In my case, my "top" p model was not clearly better than the others, so including all p covariates when modeling psi was more appropriate as it allowed this model selection uncertainty to be incorporated in the psi modeling. I suspect that if you had a clear "top" model for p, the two methods may end up giving the same results.

Best,

Robert

[cooch]

I am wondering if some of you can suggest whether the following approach is valid, and any problems concerns you can see:

As I'm sure most of us are, I am much more interested in survival rates than recaputre.

Actually, this is an unwise generalization - there is often much interesting biological information in encounter rate. Simply treating it as a statistical 'nuisance parameter' is invariably a mistake. You should ask yourself 'why' encounter rate varies - no simply 'account for it'. The answer might be uninteresting, but then - it might turn out (in some cases) to be more interesting than survival (entire papers have been written on the interesting elements of why some individuals are more detectable than others).

[AndyT]

Thanks very much for the input Robert, that's a big help.

Thank you as well for replying Gary. I agree that recapture is interesting and biologically relevant (I have put as much thought into the parameters I am using to model recapture as I have survival). I am merely trying to minimize the number of models I propose at once and separate survival from recapture by varying them one at a time while holding the other constant.

[cooch]

Thanks very much for the input Robert, that's a big help.

Thank you as well for replying Gary.

Hmmm...if I'm Gary, that would make me (i) very smart, but with a weakness for (ii) over-rated football teams playing out of Denver.

I agree that recapture is interesting and biologically relevant (I have put as much thought into the parameters I am using to model recapture as I have survival). I am merely trying to minimize the number of models I propose at once and separate survival from recapture by varying them one at a time while holding the other constant.

Actually, I disagree with this approach to model selection, although it is sometimes used out of necessity. Your model set should be specified a priori - including appropriate structures for all the parameters. What you've described is only slightly elevated above 'data dredging', in my opinion - in effect, you're ignoring model selection uncertainty on a parameter-specific basis - by selecting the parameter structure only for the most parsiomnious model, before modeling other parameters. I can easily generate exampes where doing what you describe will lead you completely down the wrong path - especially if your sample size is small, and if there is fair support among competing models.

Moreoever, it begs the question of what model you will use to estimate c-hat? Do you start with a very general model, estimate c-hat, then discard the general model once you've 'winnowed down the models' for the encounter rate?

[AndyT]

Hmmm...if I'm Gary, that would make me (i) very smart, but with a weakness for (ii) over-rated football teams playing out of Denver.

Whoops!! Sorry about that, I typed too quickly.

You've given me much to think about, particularly about estimating c-hat...thanks for taking the time to comment!

Andy

[darryl]

Actually, I disagree with this approach to model selection, although it is sometimes used out of necessity. Your model set should be specified a priori - including appropriate structures for all the parameters. What you've described is only slightly elevated above 'data dredging', in my opinion - in effect, you're ignoring model selection uncertainty on a parameter-specific basis - by selecting the parameter structure only for the most parsiomnious model, before modeling other parameters. I can easily generate exampes where doing what you describe will lead you completely down the wrong path - especially if your sample size is small, and if there is fair support among competing models.

I agree with Evan that having an a-priori candidate set would be an ideal, and something we should strive for, but I've had situations where you have multiple parameter types, each with a few different realistic structures/potential covariates and so you can quickly end up with 1000's of possible models. How to whittle that down to practically-sized candidate set is ususally easier said than done, in which case I've often resorted to something along similar lines to what Andy (use a 'best' model for your 'other' parameters) or Robert (use the 'general' model for your other paramters) have suggested. And while not ideal, I think it's a reasonable approach provided that; a) we're honest about what we've done when we write things up; and b) we treat the results as exploritory rather than definitive. If anyone out there has some practical suggestions on what to do when you have 100's or 1000's of possible models (even after thinking hard about what's biological relevent/plausible, etc), especially when you have multiple data-types, I'd be interested to hear them.

Of course, if we were able to implement a formal experiment and control/manipulate that factors we thought were important, that would simplify things greatly.

Cheers
Darryl

[cooch]

I agree with Evan that having an a-priori candidate set would be an ideal, and something we should strive for, but I've had situations where you have multiple parameter types, each with a few different realistic structures/potential covariates and so you can quickly end up with 1000's of possible models. How to whittle that down to practically-sized candidate set is ususally easier said than done, in which case I've often resorted to something along similar lines to what Andy (use a 'best' model for your 'other' parameters) or Robert (use the 'general' model for your other paramters) have suggested. And while not ideal, I think it's a reasonable approach provided that; a) we're honest about what we've done when we write things up; and b) we treat the results as exploritory rather than definitive. If anyone out there has some practical suggestions on what to do when you have 100's or 1000's of possible models (even after thinking hard about what's biological relevent/plausible, etc), especially when you have multiple data-types, I'd be interested to hear them.

Of course, if we were able to implement a formal experiment and control/manipulate that factors we thought were important, that would simplify things greatly.

Cheers
Darryl

I accept that on occasion, there are some practical limits - but in my experience, if the plausible candidate model set has 1000's of models, then the analyst probably hasn't done a particularly good job thinking through what truly constitutes plausible (distinguishing between 'biological' and 'statistical' plausibility). Of course I don't mean Darryl here - statisticians like Darryl can always be excused for narrowness of biological insight. (kidding, kidding..). Darryl makes several very good points (no surprise there), but thought I'd add my two cents worth in quick followup.

More often than not, careful consideration of prior results, biological insight (the 'warm and fuzzy' version of an informative prior), will winnow down most model sets. Its also important to consider whether the point of the exercise is to 'get a parameter estimate', or 'test a biological hypothesis'. If the latter, then some selectivity is no doubt in order. A model set that is a large collection of (say) variants on the 'time-dependence' theme doesn't show much deep consideration (in many cases). Time-dependence is, in an of itself, boring - and irrelevant. Its strictly analogous to the null heterogeneity hypothesis in ANOVA - you do an ANOVA, and find group means differ. Whoopee! What is more important is - what do the differences relate to? Similarly, finding that a time-dependent CJS model (for example) is a better model (pick your criterion) than (say) a dot model is not particularly interesting biologically. (and, of course, this must logically be true, since a dot model is conceptually impossible, since no parameter is truly constant - even if your data are lousy enough to give more support to the dot model than the time-dependent model). What is of interest is - what are the underlying drivers for the temporal variation? If you believe that certain extrinsic factor drive variation in some parameter, then your model set contains those models where the parameter of interest is constrained to be a function of the covariate(s) you think important. I would say the majority of the time (in the work I've done), model structures for individual parameters run from dot models -> 2-4 constrained models -> time-dependent models - so, typically, 6-8 model structures per parameter. With two parameters, and (say) 6 model structures, thats (6-8)xn models (where n is the number of basal parameters), which is a lot more manageable than 'thousands'.

I usually tell my students that if your candidate model set has >80 models, you probably need to do more work thinking about what models should be in the model set. I don't dispute this is not always easy, but which is the preferable argument: ' here is my model set, and perhaps I did leave out some important, plausible models' versus ' I tried every model I can think of'. In the former case, the debate becomes one of the decisions - biologically-motivated - about what models to consider. In the case of the latter, you end up 'telling stories' to justify the models you found to be more parsimonious. While I admit that this can be fun, its not particularly comforting. And, as I noted earlier, I can construct data sets where this 'parameter stepwise' approach will quickly lead you to the wrong conclusion.

There is no doubt you occasionally need to balance approaches. What Darryl (and previous) posts describes is not the most eggregious thing you can do (heck, I've done it myself in some of my own work), but one that should be approached with considerable caution, and complete upfront honesty.

There are still some technical issues (as noted in my earlier posts in this thread), but that's another debate.

[Morten Frederiksen]

Evan, I think you're being too dogmatic here - and partly overlooking Darryl's point about multiple parameter types. I regularly use Burnham's combined recovery/recapture model, which has four parameter types. With 6-8 model structures for each parameter type (which I agree is a reasonable number), this adds up to 1296-4096 possible models ... In these situations, narrowing down the options for the 'nuisance' parameters before concentrating on (e.g.) survival is really the only reasonable way forwards, I think. So sometimes the dreaded 'data dredging' is unavoidable ...

Morten

I accept that on occasion, there are some practical limits - but in my experience, if the plausible candidate model set has 1000's of models, then the analyst probably hasn't done a particularly good job thinking through what truly constitutes plausible (distinguishing between 'biological' and 'statistical' plausibility). Of course I don't mean Darryl here - statisticians like Darryl can always be excused for narrowness of biological insight. (kidding, kidding..). Darryl makes several very good points (no surprise there), but thought I'd add my two cents worth in quick followup.

More often than not, careful consideration of prior results, biological insight (the 'warm and fuzzy' version of an informative prior), will winnow down most model sets. Its also important to consider whether the point of the exercise is to 'get a parameter estimate', or 'test a biological hypothesis'. If the latter, then some selectivity is no doubt in order. A model set that is a large collection of (say) variants on the 'time-dependence' theme doesn't show much deep consideration (in many cases). Time-dependence is, in an of itself, boring - and irrelevant. Its strictly analogous to the null heterogeneity hypothesis in ANOVA - you do an ANOVA, and find group means differ. Whoopee! What is more important is - what do the differences relate to? Similarly, finding that a time-dependent CJS model (for example) is a better model (pick your criterion) than (say) a dot model is not particularly interesting biologically. (and, of course, this must logically be true, since a dot model is conceptually impossible, since no parameter is truly constant - even if your data are lousy enough to give more support to the dot model than the time-dependent model). What is of interest is - what are the underlying drivers for the temporal variation? If you believe that certain extrinsic factor drive variation in some parameter, then your model set contains those models where the parameter of interest is constrained to be a function of the covariate(s) you think important. I would say the majority of the time (in the work I've done), model structures for individual parameters run from dot models -> 2-4 constrained models -> time-dependent models - so, typically, 6-8 model structures per parameter. With two parameters, and (say) 6 model structures, thats (6-8)xn models (where n is the number of basal parameters), which is a lot more manageable than 'thousands'.

I usually tell my students that if your candidate model set has >80 models, you probably need to do more work thinking about what models should be in the model set. I don't dispute this is not always easy, but which is the preferable argument: ' here is my model set, and perhaps I did leave out some important, plausible models' versus ' I tried every model I can think of'. In the former case, the debate becomes one of the decisions - biologically-motivated - about what models to consider. In the case of the latter, you end up 'telling stories' to justify the models you found to be more parsimonious. While I admit that this can be fun, its not particularly comforting. And, as I noted earlier, I can construct data sets where this 'parameter stepwise' approach will quickly lead you to the wrong conclusion.

There is no doubt you occasionally need to balance approaches. What Darryl (and previous) posts describes is not the most eggregious thing you can do (heck, I've done it myself in some of my own work), but one that should be approached with considerable caution, and complete upfront honesty.

There are still some technical issues (as noted in my earlier posts in this thread), but that's another debate.

[cooch]

Evan, I think you're being too dogmatic here -

Me?

and partly overlooking Darryl's point about multiple parameter types. I regularly use Burnham's combined recovery/recapture model, which has four parameter types. With 6-8 model structures for each parameter type (which I agree is a reasonable number), this adds up to 1296-4096 possible models ... In these situations, narrowing down the options for the 'nuisance' parameters before concentrating on (e.g.) survival is really the only reasonable way forwards, I think. So sometimes the dreaded 'data dredging' is unavoidable ...

I use the same models, and still argue that there is no way you should ever end up with 1000+ plausible models (I've never ended up with more than 40-50 or so). Based on what we know about the species we work with (in my case, typically a lot), it is usually not overly difficult to reduce the model space considerably (i.e., some models just aren't plausible, given what we know). (quick aside: I've done some tests with some collegaues - give them a data set they're naive about - and one for a system they know a lot about - the model set they come up with in the latter case is invariably much smaller - often by orders of magnitude - than the one they came up with in the former case). Since most big data sets come from heavily studied species, I'd be really surprised if folks working with those data sets really need that many models).

But, more to the point, I don't think you can 'narrow down one parameter', and then start 'modeling another' in the piece-wise, sequential fashion you (and other suggest), not without extreme care. Doing so more or less violates the basic concepts of model selection uncertainty.

Consider the following example: suppose you have a CJS model, and start by making the parameter structure for survival very general, and then start doing a stepwise selection for reduced struture for encounter rate. Suppose in so doing you find a 'dot' model for p has the lowest AIC, but some other models with other studtures have some weight. What do you do? If you pick only the 'dot' model, and then start modeling phi using a 'dot' model for p, you ignore uncertainty for other parameteizations for p. Also, your model 'selection' (as it were) for p is conditional on the structure you've set for survival. Simply making said survival structure general does not mean you're necessarily going to pick the 'right structure' for encounter rate. I've run into numerous examples of this - where I know the true model (because the encounter histories are simulated), but if I try to model each parameter at a time, I end up often quite some 'distance' from 'truth' (as it were). We look to achieve parsimony over the entire model, and I'm suggesting only great caution in assuming you can get there from here by simplifying 'nuisance parameters', and then modeling the parameters of interest.

Moreoever, my point about c-hat still stands - if you use this stepwise approach to model building, what model do you use to estimate c-hat?

At the minimum, then, I'd want to consider several 'nuisance paramater' structures, especially if there was any evidence of uncertainty among models (i.e., if multiple models had some AIC weight). If this leaves you with hundreds of models, then run hundreds of models. If it takes a few days to set them up, so be it.