Astronomical chat estimates. Sample size? Missing obs?

[benjamin]

I am analyzing a relatively large data set of ~4,000 study sites. Sampling occasions for data collection ranges from 3-21 sampling occasions (mean =12 visits). Consequently, although this is a large data set, there are a number of missing observations. Occurrence data is collected for a number of species ranging in occurrence from relatively rare to ubiquitous. I have 5 site-level covariates and 5 sampling covariates I am interested in testing. At the moment, I am running single-season occupancy models for multiple species. For one such ubiquitous species, I found a naïve estimate of 0.97 and detection probability of 0.60. I am interesting in assessing model fit and received the following results using 500 bootstraps for the chat estimate:

Test Statistic (data) = 2784554169.5975

From 500 parametric bootstraps...
Probability of test statistic >= observed = 0.0020
Average simulated Test Stat = 7125.4114
Median simulated Test Stat = 4626.2077
Estimate of c-hat = 390792.0533 (=TestStat/AvgTestStat)
Estimate of c-hat = 601908.5872 (=TestStat/MedianTestStat)

What could lead to such an astronomical estimate of chat? Sample size? Missing observations? Model complexity? I came across a past post by Darryl suggesting a minimum of 10,000 bootstraps. Am I using too few bootstraps? Any suggestions and/or thoughts would be very much appreciated. Thank you.

[darryl]

Is it possible that the closure assumption is being violated during the time it takes to complete 21 surveys? Are the 3 surveys conducted over a similar, or much shorter time frame?

When you have a large number of repeat surveys, it's going to have a very small expected value, hence will probably have a large contribution to the test statistic (you should be able to check that on the output). This might help to diagnose which sites are causing the 'problem'

Darryl

[benjamin]

Thank you for the reply, Darryl. Well, the closure assumption is certainly being tested for some sites (although I have read number of studies that have used a very flexible definition of closure in populations, and so I am somewhat unclear regarding how to define closure). The surveying is undoubtedly opportunistic, but the surveys are site-specific and are all conducted within the same "season". I could potentially collapse or pool the weekly visits to reduce the number of sampling sessions although I think some information could be lost. The 3 surveys are not necessarily conducted over a shorter time frame, i.e., the 3 surveys could be spaced out across the sampling period.

When you mention a large number of repeat surveys leading to a large contribution to the test statistic, do you mean that at a site-specific level? That is interesting, because I have been more worried about the sites providing a sparse number of observations as opposed to those sites providing more counts.

Specifically, what should I be looking for on the output to identify the sites causing the inflation of the test statistic?

Thank you!

Is it possible that the closure assumption is being violated during the time it takes to complete 21 surveys? Are the 3 surveys conducted over a similar, or much shorter time frame?

When you have a large number of repeat surveys, it's going to have a very small expected value, hence will probably have a large contribution to the test statistic (you should be able to check that on the output). This might help to diagnose which sites are causing the 'problem'

Darryl

[benjamin]

Darryl, just a quick follow-up on my last post. The number of bootstraps is not the problems. I ran 10,000 and the results remained a large chat value.

In your experience, extremely large chats have more to do with the sampling scheme or assunptions (e.g., large number of repeated visits) as opposed to complex models (i.e., large number of covariates) or a large number of sites?

Is it possible that the closure assumption is being violated during the time it takes to complete 21 surveys? Are the 3 surveys conducted over a similar, or much shorter time frame?

When you have a large number of repeat surveys, it's going to have a very small expected value, hence will probably have a large contribution to the test statistic (you should be able to check that on the output). This might help to diagnose which sites are causing the 'problem'

Darryl

[darryl]

I haven't looked into this explicitly myself, but this type of behaviour seems to be more of a problem when there are a few sites with a large number of surveys (perhaps mixed with other sites with fewer surveys).

Lets think about some extremes here. With 3 surveys, there's only 8 possible detection histories. The probability of each will depend on the values for psi and p, but for argument sake lets say the probability = 0.125. If you have 10 sites with 3 surveys, then expected number of sites with each history is 1.25. If a particular history was only observed once, the contribution to the test statistic for that history is (1-1.25)^2/1.25=0.05.

With 21 surveys, there's 2,097,152 possible histories and for arguments sake lets say they have equal probability. With only 10 sites, the expected number is 4.8E-06 and with 1 observation, contribution is 209713.2 !!

Now I would have expected the bootstrap to be able to produce similar values, and it doesn't seem to be for you example for some reason. Not sure why as (like I said) I haven't had a chance to look into details. Anyone looking for a project??

[benjamin]

Hmm. I admit I have never quite thought about the nuances or implications of many repeated visits (in combination with sites with "few" visits), but your example highlights the problem. I wonder if this is an issue for any study or scheme that has a diversity of surveys. In addition, I am curious about the larger ramifications on the modeling in general when covariates are introduced. For example, what if the sites that are characterized by more surveys are highly correlated with a site-level covariate? It does make me consider splitting the sampling season into 3 separate "seasons" of 7 sampling occasions (total =21) and modeling occupancy across those "seasons". In this case, there would still be range in the number of surveys across sites, but the range would be much reduced.

[darryl]

A few additional pertinent points

1. This test has not been shown to have any power to detect systematic lack of fit in occupancy (at least not so far)
2. Repeat surveys basically allow us to assess how much effort has gone into searching a location before confirming the species is present there, enabling us to separate (probabilistically) false and true absences at places with no detections.
3. If main inferences are about presence/absence, provided detection probabilities are reasonably high (overall), occupancy results may be relatively robust even if test is indicating problem with detection component.
4. Is occupancy is approaching 1 (or 0), there's very little variation in terms of presence/absence hence trying to model it with covariates may not be very successful.
5. 'Seasons' should be defined biologically in terms of what aspect of the real world your trying to quantify in terms of presence/absence. First define what you want it to mean, then design your study (ideally; or select which bits of data to use) accordingly.

[jhines]

I've seen a similar result with another dataset with a large number of surveys. I agree with Darryl that this causes the extremely small expected values, resulting in large chi-square values. It seems that the simulations don't generate data very often in the extremely small expected histories, but a slight deviation from the model assumptions can result in some detection histories which are extremely unlikely. I'm not sure what the best course of action would be, but my first inclination would be to pool surveys.

[benjamin]

Thank you Darryl and Jim for your responses and suggestions. It gives me more to think about.