The File Drawer Problem And Tolerance For Null Results
The purpose of the present paper is to assess the efficacy of confidence intervals for Rosenthal's fail-safe number. Although Rosenthal's estimator is highly used by researchers, its statistical properties are largely unexplored. First of all, we developed statistical theory which allowed us to produce confidence intervals for Rosenthal's fail-safe number. This was produced by discerning whether the number of studies analysed in a meta-analysis is fixed or random. Each case produces different variance estimators. For a given number of studies and a given distribution, we provided five variance estimators. Confidence intervals are examined with a normal approximation and a nonparametric bootstrap. The accuracy of the different confidence interval estimates was then tested by methods of simulation under different distributional assumptions. The half normal distribution variance estimator has the best probability coverage. Finally, we provide a table of lower confidence intervals for Rosenthal's estimator.
1. Introduction
Meta-analysis refers to methods focused on contrasting and combining results from different studies, in the hope of identifying patterns among study results, sources of disagreement among those results, or other interesting relationships that may come to light in the context of multiple studies [1]. In its simplest form, this is normally done by identification of a common measure of effect size, of which a weighted average might be the output of a meta-analysis. The weighting might be related to sample sizes within the individual studies [2, 3]. More generally there are other differences between the studies that need to be allowed for, but the general aim of a meta-analysis is to more powerfully estimate the true effect size as opposed to a less precise effect size derived in a single study under a given single set of assumptions and conditions [4]. For reviews on meta-analysis models, see [2, 5, 6]. Meta-analysis can be applied to various effect sizes collected from individual studies. These include odds ratios and relative risks; standardized mean difference, Cohen's , Hedges' , and Glass's ; correlation coefficient and relative metrics; sensitivity and specificity from diagnostic accuracy studies; and -values. For more comprehensive reviews see Rosenthal [7], Hedges and Olkin [8], and Cooper et al. [9].
2. Publication Bias
Publication bias is a threat to any research that attempts to use the published literature, and its potential presence is perhaps the greatest threat to the validity of a meta-analysis [10]. Publication bias exists because research with statistically significant results is more likely to be submitted and published than work with null or nonsignificant results. This issue was memorably termed as the file-drawer problem by Rosenthal [11]; nonsignificant results are stored in file drawers without ever being published. In addition to publication bias, other related types of bias exist including pipeline bias, subjective reporting bias, duplicate reporting bias, or language bias (see [12–15] for definitions and examples).
The implication of these various types of bias is that combining only the identified published studies uncritically may lead to an incorrect, usually over optimistic, conclusion [10, 16]. The ability to detect publication bias in a given field is a key strength of meta-analysis because identification of publication bias will challenge the validity of common views in that area and will spur further investigations [17]. There are two types of statistical procedures for dealing with publication bias in meta-analysis: methods for identifying the existence of publication bias and methods for assessing the impact of publications bias [16]. The first includes the funnel plot (and other visualisation methods such as the normal quantile plot) and regression/correlation-based tests, while the second includes the fail-safe (also called file-drawer) number, the trim and fill method, and selection model approaches [10, 14, 18]. Recent approaches include the test for excess significance [19] and the -curve [20].
The most commonly used method is the visual inspection of a funnel plot. This assumes that the results from smaller studies will be more widely spread around the mean effect because of larger random error. The next most frequent method used to assess publication bias is Rosenthal's fail-safe number [21]. Two recent reviews examining the assessment of publication bias in psychology and ecology reported that funnel plots were the most frequently used (24% and 40% resp.), followed by Rosenthal's fail-safe number (22% and 30%, resp.).
2.1. Assessing Publication Bias by Computing the Number of Unpublished Studies
Assessing publication bias can be performed by trying to estimate the number of unpublished studies in the given area a meta-analysis is studying. The fail-safe number represents the number of studies required to refute significant meta-analytic means. Although apparently intuitive, it is in reality difficult to interpret not only because the number of data points (i.e., sample size) for each of studies is not defined, but also because no benchmarks regarding the fail-safe number exist, unlike Cohen's benchmarks for effect size statistics [22]. However, these versions have been heavily criticised, mainly because such numbers are often misused and misinterpreted [23]. The main reason for the criticism is that, depending on which method is used to estimate the fail-safe , the number of studies can greatly vary.
Although Rosenthal's fail-safe number of publication bias was proposed as early as 1979 and is frequently cited in the literature [11] (over 2000 citations), little attention has been given to the statistical properties of this estimator. This is the aim of the present paper, which is discussed in further detail in Section 3.
Rosenthal [11] introduced what he called the file drawer problem. His concern was that some statistically nonsignificant studies may be missing from an analysis (i.e., placed in a file drawer) and that these studies, if included, would nullify the observed effect. By nullify, he meant to reduce the effect to a level not statistically significantly different from zero. Rosenthal suggested that rather than speculate on whether the file drawer problem existed, the actual number of studies that would be required to nullify the effect could be calculated [26]. This method calculates the significance of multiple studies by calculating the significance of the mean of the standard normal deviates of each study. Rosenthal's method calculates the number of additional studies , with the mean null result necessary to reduce the combined significance to a desired α level (usually 0.05).
The necessary prerequisite is that each study examines a directional null hypothesis such that the effect sizes from each study are examined under or (). Then the null hypothesis of Stouffer [27] test is The test statistic for this is with , where are the standard errors of . Under the null hypothesis we have [7].
So we get that the number of additional studies , with mean null result necessary to reduce the combined significance to a desired level (usually 0.05 [7, 11]), is found after solving So, is calculated as where is the number of studies and is the one-tailed score associated with the desired level of significance. Rosenthal further suggested that if , the likelihood of publication bias would be minimal.
Cooper [28] and Cooper and Rosenthal [29] called this number the fail-safe sample size or fail-safe number. If this number is relatively small, then there is cause for concern. If this number is large, one might be more confident that the effect, although possibly inflated by the exclusion of some studies, is, nevertheless, not zero [30]. This approach is limited in two important ways [26, 31]. First, it assumes that the association in the hidden studies is zero, rather than considering the possibility that some of the studies could have an effect in the reverse direction or an effect that is small but not zero. Therefore, the number of studies required to nullify the effect may be different than the fail-safe number, either larger or smaller. Second, this approach focuses on statistical significance rather than practical or substantive significance (effect sizes). That is, it may allow one to assert that the mean correlation is not zero, but it does not provide an estimate of what the correlation might be (how it has changed in size) after the missing studies are included [23, 32–34]. However, for many fields it remains the gold standard to assess publication bias, since its presentation is conceptually simple and eloquent. In addition, it is computationally easy to perform.
Iyengar and Greenhouse [12] proposed an alternative formula for Rosenthal's fail-safe number, in which the sum of the unpublished studies' standard variates is not zero. In this case the number of unpublished studies is approached through the following equation: where (this results immediately from the definition of truncated normal distribution) and is the desired level of significance. This is justified by the author that the unpublished studies follow a truncated normal distribution with . and denote the cumulative distribution function (CDF) and probability distribution function (PDF), respectively, of a standard normal distribution.
There are certain other fail-safe numbers which have been described, but their explanation goes beyond the scope of the present article [35]. Duval and Tweedie [36, 37] present three different estimators for the number of missing studies and the method to calculate this has been named Trim and Fill Method. Orwin's [38] approach is very similar to Rosenthal's [11] approach without considering the normal variates but taking Cohen's [22] to compute a fail-safe number. Rosenberg's fail-safe number is very similar to Rosenthal's and Orwin's fail-safe number [39]. Its difference is that it takes into account the meta-analytic estimate under investigation by incorporating individual weights per study. Gleser and Olkin [40] proposed a model under which the number of unpublished studies in a field where a meta-analysis is undertaken could be estimated. The maximum likelihood estimator of their fail-safe number only needs the number of studies and the maximum value of the studies. Finally, the Eberly and Casella fail-safe number assumes a Bayesian methodology which aims to estimate the number of unpublished studies in a certain field where a meta-analysis is undertaken [41].
The aim of the present paper is to study the statistical properties of Rosenthal's [11] fail-safe number. In the next section we introduce the statistical theory for computing confidence intervals for Rosenthal's [11] fail-safe number. We initially compute the probability distribution function of , which gives formulas for variance and expectation; next, we suggest distributional assumptions for the standard normal variates used in Rosenthal's fail-safe number and finally suggest confidence intervals.
3. Statistical Theory
The estimator of unpublished studies is approached through Rosenthal's formula: Let , , be i.i.d. random variables with and . We discern two cases: (a) is fixed or (b) is random, assuming additionally that . This is reasonable since the number of studies included in a meta-analysis is like observing counts. Other distributions might be assumed, such as the Gamma distribution, but this would require more information or assumptions to compute the parameters of the distribution. In both cases, estimators of , , and can be calculated without distributional assumptions for the with the method of moments or with distributional assumptions regarding the .
3.1. Probability Distribution Function of
3.1.1. Fixed
We compute the PDF of by following the next steps.
Step 1. in the formula of the estimator (6) are i.i.d. distributed with and . Let and according to the Lindeberg-Lévy Central Limit Theorem [42], we have So the PDF of is
Step 2. The PDF of Rosenthal's can be retrieved from a truncated version of (8). From (3), we get that We advocate that Rosenthal's equations (3) and (9) implicitly impose two conditions which must be taken into account when we seek to estimate the distribution of : Expression (10) is justified by the fact that the right hand side of (9) is positive, so then . Expression (11) is justified by the fact that expresses the number of studies, so it must be at least . Hence, expressions (10) and (11) are satisfied when is a truncated normal random variable.
The truncated normal distribution is a probability distribution related to the normal distribution. Given a normally distributed random variable with mean and variance , let it be that . Then conditional on has a truncated normal distribution with PDF: −, for and otherwise [43].
Let it be , such that . So the PDF of then becomes where .
Then, we have The characteristic function is where , , .
From (14) we get where .
Also, where
Proofs for expressions (14), (15), and (16) are given in the Appendix.
Comments. Consider the following: (1) For a significantly large we have that . So (13) becomes Also we get (2) A limiting element of this computation is that takes discrete values because it describes number of studies, but it has been described by a continuous distribution.
3.1.2. Random
It is assumed that . So taking into account the result from the distribution of for a fixed we get that the joint distribution of and is
3.2. Expectation and Variance for Rosenthal's Estimator
(a) When is fixed, expressions (19) and (20) denote the expectation and variance, respectively, for . This is derived from the PDF of ; an additional proof without reference to the PDF is given in the Appendix. (b) When is random with , the expectation and variance of are Proofs are given in the Appendix.
3.3. Estimators for , , and
Having now computed a formula for the variance which is necessary for a confidence interval, we need to estimate , , and . In both cases, estimators of , , and can be calculated without distributional assumptions for the with the method of moments or with distributional assumptions regarding the .
3.3.1. Method of Moments [44]
When is fixed, we have When is random, we have
3.3.2. Distributional Assumptions for the
If we suppose that the follow a distribution, we would replace the values of and with their distributional values. Below we consider special cases.
Standard Normal Distribution. The follow a standard normal distribution; that is, . This is the original assumption for the [11]. In this case we have Although the origin of the is from the standard normal distribution, the studies in a meta-analysis are a selected sample of published studies. For this reason, the next distribution is suggested as better.
Half Normal Distribution. Here we propose that the follow a half normal distribution HN, which is a special case of folded normal distribution. Before we explain the rational of this distribution, a definition of this type of distribution is provided. A half normal distribution is also a special case of a truncated normal distribution.
Definition 1. The folded normal distribution is a probability distribution related to the normal distribution. Given a normally distributed random variable with mean and variance , the random variable has a folded normal distribution [43, 45, 46].
Remark 2. The folded normal distribution has the following properties: (a) probability density function (PDF): (b)
Remark 3. When , the distribution of is a half normal distribution. This distribution is identical to the truncated normal distribution, with left truncation point 0 and no right truncation point. For this distribution we have the following. (a) , for . (b) , .
Assumption 4. The in Rosenthal's estimator are derived from a half normal distribution, based on a normal distribution .
Support. When a researcher begins to perform a meta-analysis, the sample of studies is drawn from those studies that are already published. So his sample is most likely biased by some sort of selection bias, produced via a specific selection process [47]. Thus, although when we study Rosenthal's assuming that all are drawn from the normal distribution, they are in essence drawn from a truncated normal distribution. This has been commented on by Iyengar and Greenhouse [12] and Schonemann and Scargle [48]. But at which point is this distribution truncated? We would like to advocate that the half normal distribution, based on a normal distribution , is the one best representing the Rosenthal uses to compute his fail-safe . The reasons for this are as follows. (1) Firstly, assuming that all are of the same sign does not impede the significance of the results from each study. That is, the test is significant when either or occurs. (2) However, when a researcher begins to perform a meta-analysis of studies, many times can be either positive or negative. Although this is true, when the researcher is interested in doing a meta-analysis, usually the that have been published are indicative of a significant effect of the same direction (thus have the same sign) or are at least indicative of such an association without being statistically significant, thus producing of the same sign but not producing significance (e.g., the confidence interval of the effect might include the null value). (3) There will definitely be studies that produce a totally opposite effect, thus producing an effect of opposite direction, but these will definitely be a minority of the studies. Also there is the case that these other signed are not significant. Hence, in this case
Skew Normal Distribution. Here we propose that the follow a skew normal distribution; that is, ~ SN.
Definition 5. The skew normal distribution is a continuous probability distribution that generalises the normal distribution to allow for nonzero skewness. A random variable follows a univariate skew normal distribution with location parameter , scale parameter , and skewness parameter [49], if it has the density Note that if , the density of reduces to the .
Remark 1. The expectation and variance of are [49]
Remark 2. The methods of moments estimators for , and are [50, 51] where , , , , and . The sign of is taken to be the sign of .
Explanation. The skew normal distribution allows for a dynamic way to fit the available -scores. The fact that there is ambiguity on the derivation of the standard deviates from each study from a normal or a truncated normal distribution creates the possibility that the distribution could be a skew normal, with the skewness being attributed that we are including only the published -scores in the estimation of Rosenthal's [11] estimator.
Hence, in this case and taking the method of moments estimators of , , and , we get where , , , , and .
3.4. Methods for Confidence Intervals
3.4.1. Normal Approximation
In the previous section, formulas for computing the variance of were derived. We compute asymptotic % confidence intervals for as where is the th quantile of the standard normal distribution.
The variance of for a given set of values depends firstly on whether the number of studies is fixed or random and secondly on whether the estimators of , , and are derived from the method of moments or from the distributional assumptions.
3.4.2. Nonparametric Bootstrap
Bootstrap is a well-known resampling methodology for obtaining nonparametric confidence intervals of a parameter [52, 53]. In most statistical problems one needs an estimator of a parameter of interest as well as some assessment of its variability. In many such problems the estimators are complicated functionals of the empirical distribution function and it is difficult to derive trustworthy analytical variance estimates for them. The primary objective of this technique is to estimate the sampling distribution of a statistic. Essentially, bootstrap is a method that mimics the process of sampling from a population, like one does in Monte Carlo simulations, but instead drawing samples from the observed sampling data. The tool of this mimic process is the Monte Carlo algorithm of Efron [54]. This process is explained properly by Efron and Tibshirani [55] and Davison and Hinkley [56], who also noted that bootstrap confidence intervals are approximate, yet better than the standard ones. Nevertheless, they do not try to replace the theoretical ones and bootstrap is not a substitute for precise parametric results, but rather a way to reasonably proceed when such results are unavailable.
Nonparametric resampling makes no assumptions concerning the distribution of, or model for, the data [57]. Our data is assumed to be a vector of independent observations, and we are interested in a confidence interval for . The general algorithm for a nonparametric bootstrap is as follows. (1) Sample observations randomly with replacement from to obtain a bootstrap data set, denoted by . (2) Calculate the bootstrap version of the statistic of interest . (3) Repeat steps (1) and (2) several times, say , to obtain an estimate of the bootstrap distribution. In our case (1) compute a random sample from the initial sample of , size , (2) compute from this sample, (3) repeat these processes times. Then the bootstrap estimator of is From this we can compute also confidence intervals for .
In the next section, we investigate these theoretical aspects with simulations and examples.
4. Simulations and Results
The method for simulations is as follows. (1) Initially we draw random numbers from the following distributions: (a) standard normal distribution, (b) half normal distribution , (c) skew normal distribution with negative skewness SN(), (d) skew normal distribution with positive skewness SN(). (2) The numbers drawn from each distribution represent the number of studies in a meta-analysis and we have chosen = 5, 15, 30, and 50. When is assumed to be random, then the parameter is equal to the values chosen for the simulation, that is, 5, 15, 30, and 50, respectively. (3) We compute the normal approximation confidence interval with the formulas described in Section 3 and the bootstrap confidence interval. We also discern whether the number of studies is fixed or random. For the computation of the bootstrap confidence interval, we generate 1,000 bootstrap samples each time. We also study the performance of the different distributional estimators in cases where the distributional assumption is not met, thus comparing each of the six confidence interval estimators under all four distributions. (4) We compute the coverage probability comparing with the true value of Rosenthal's fail-safe number. When the number of studies is fixed the true value of Rosenthal's number is When the number of studies is random [from a Poisson () distribution] the true value of Rosenthal's number is We execute the above procedure 10,000 times each time. Our alpha-level is considered 5%.
This process is shown schematically in Table 1. All simulations were performed in and the code is shown in the Supplementary Materials (see Supplementary Materials available online at http://dx.doi.org/10.1155/2014/825383).
| | ||||
| Variance formula for normal approximation confidence intervals | Bootstrap confidence interval | Real value of | ||
| Distributional | Moments | |||
| | ||||
| studies Draw from , , SN, SN | Fixed (standard normal distribution values , ; half normal distribution , ; skew normal with negative skewness , ; skew normal with positive skewness , ) | Fixed | Using the we compute the and respectively the standard error needed to compute the confidence interval | Fixed (standard normal distribution values , ; half normal distribution , ; skew normal with negative skewness , ; skew normal with positive skewness , ) |
| Random (standard normal distribution values , ; half normal distribution , ; skew normal with negative skewness , ; skew normal with positive skewness , ; ) | | Using the we compute the and respectively the standard error needed to compute the confidence interval | Random (standard normal distribution values , ; half normal distribution , ; skew normal with negative skewness , ; skew normal with positive skewness , ; ) | |
| | ||||
We observe from Table 2 and Figure 1 that the bootstrap confidence intervals perform the poorest both when the number of studies is considered fixed or random. The only case in which they perform acceptably is when the distribution is half normal and the number of studies is fixed. The moment estimators of variance perform either poorly or too efficiently in all cases, with coverages being under 90% or near 100%. The most acceptable confidence intervals for Rosenthal's estimator appear to be in the distribution-based method and to be much better for a fixed number of studies than for random number of studies. We also observe that, for the distribution-based confidence intervals in the fixed category, the half normal distribution HN(0,1) produces coverages which are all 95%. This is also stable for all number of studies in a meta-analysis. When the distributional assumption is not met the coverage is poor except for the cases of the positive and negative skewness skew normal distributions which perform similarly, possibly due to symmetry.
| | ||||||||||||||||||
| Draw from | Values of and from the standard normal distribution | Values of and from the half normal distribution HN(0, 1) | Values of and from the skew normal distribution with negative skewness SN() | Values of and from the skew normal distribution with positive skewness SN() | ||||||||||||||
| | ||||||||||||||||||
| Standard normal distribution | Fixed | Distribution Based CI | 0.948 | 0.950 | 0.948 | 0.952 | 0.994 | 0.110 | 0.002 | 0.000 | 0.985 | 0.999 | 1.000 | 1.000 | 0.982 | 0.998 | 1.000 | 1.000 |
| Moments Based CI | 0.933 | 0.996 | 1.000 | 1.000 | 0.529 | 0.088 | 0.005 | 0.000 | 0.842 | 0.686 | 0.337 | 0.120 | 0.842 | 0.686 | 0.337 | 0.120 | ||
| Bootstrap CI | 0.929 | 0.996 | 1.000 | 1.000 | 0.514 | 0.089 | 0.005 | 0.000 | 0.830 | 0.680 | 0.337 | 0.120 | 0.830 | 0.680 | 0.337 | 0.120 | ||
| Random | Distribution Based CI | 0.966 | 0.956 | 0.951 | 0.955 | 0.999 | 1.000 | 0.084 | 0.001 | 0.998 | 1.000 | 1.000 | 1.000 | 0.990 | 0.999 | 1.000 | 1.000 | |
| Moments Based CI | 1.000 | 1.000 | 1.000 | 1.000 | 0.535 | 0.094 | 0.006 | 0.000 | 1.000 | 0.702 | 0.338 | 0.122 | 1.000 | 0.702 | 0.338 | 0.122 | ||
| Bootstrap CI | 0.929 | 0.996 | 1.000 | 1.000 | 0.429 | 0.074 | 0.004 | 0.000 | 0.804 | 0.649 | 0.322 | 0.115 | 0.804 | 0.649 | 0.322 | 0.115 | ||
| | ||||||||||||||||||
| Half normal distribution HN(0, 1) | Fixed | Distribution Based CI | 0.635 | 0.021 | 0.000 | 0.000 | 0.945 | 0.952 | 0.951 | 0.948 | 0.864 | 0.624 | 0.279 | 0.053 | 0.841 | 0.483 | 0.142 | 0.014 |
| Moments Based CI | 0.861 | 0.187 | 0.000 | 0.000 | 0.771 | 0.880 | 0.911 | 0.927 | 0.885 | 0.657 | 0.126 | 0.003 | 0.885 | 0.657 | 0.126 | 0.003 | ||
| Bootstrap CI | 0.858 | 0.217 | 0.000 | 0.000 | 0.775 | 0.884 | 0.913 | 0.929 | 0.887 | 0.672 | 0.138 | 0.003 | 0.887 | 0.672 | 0.138 | 0.003 | ||
| Random | Distribution Based CI | 0.720 | 0.027 | 0.000 | 0.000 | 0.989 | 0.995 | 0.996 | 0.997 | 0.966 | 0.915 | 0.762 | 0.459 | 0.901 | 0.578 | 0.198 | 0.027 | |
| Moments Based CI | 1.000 | 1.000 | 0.130 | 0.000 | 0.806 | 0.937 | 0.971 | 0.984 | 1.000 | 1.000 | 0.995 | 0.358 | 1.000 | 1.000 | 0.995 | 0.358 | ||
| Bootstrap CI | 0.858 | 0.217 | 0.000 | 0.000 | 0.715 | 0.859 | 0.899 | 0.920 | 0.885 | 0.698 | 0.152 | 0.004 | 0.885 | 0.698 | 0.152 | 0.004 | ||
| | ||||||||||||||||||
| Skew normal distribution with negative skewness SN() | Fixed | Distribution Based CI | 0.872 | 0.666 | 0.399 | 0.174 | 0.980 | 0.472 | 0.184 | 0.048 | 0.953 | 0.970 | 0.977 | 0.981 | 0.944 | 0.948 | 0.949 | 0.957 |
| Moments Based CI | 0.917 | 0.979 | 0.944 | 0.858 | 0.597 | 0.375 | 0.200 | 0.074 | 0.845 | 0.860 | 0.882 | 0.895 | 0.845 | 0.860 | 0.882 | 0.895 | ||
| Bootstrap CI | 0.912 | 0.978 | 0.945 | 0.857 | 0.586 | 0.377 | 0.199 | 0.074 | 0.840 | 0.857 | 0.881 | 0.894 | 0.840 | 0.857 | 0.881 | 0.894 | ||
| Random | Distribution Based CI | 0.903 | 0.688 | 0.409 | 0.178 | 0.996 | 1.000 | 0.687 | 0.306 | 0.987 | 0.997 | 0.998 | 0.999 | 0.965 | 0.964 | 0.965 | 0.968 | |
| Moments Based CI | 1.000 | 1.000 | 0.999 | 0.968 | 0.609 | 0.399 | 0.237 | 0.103 | 1.000 | 0.872 | 0.886 | 0.902 | 1.000 | 0.872 | 0.886 | 0.902 | ||
| Bootstrap CI | 0.912 | 0.978 | 0.945 | 0.857 | 0.514 | 0.342 | 0.181 | 0.066 | 0.818 | 0.845 | 0.874 | 0.889 | 0.818 | 0.845 | 0.874 | 0.889 | ||
| | ||||||||||||||||||
| Skew normal distribution with positive skewness SN() | Fixed | Distribution Based CI | 0.880 | 0.673 | 0.402 | 0.164 | 0.982 | 0.471 | 0.186 | 0.050 | 0.956 | 0.972 | 0.976 | 0.979 | 0.948 | 0.952 | 0.951 | 0.955 |
| Moments Based CI | 0.923 | 0.980 | 0.947 | 0.852 | 0.596 | 0.372 | 0.201 | 0.076 | 0.850 | 0.865 | 0.874 | 0.896 | 0.850 | 0.865 | 0.874 | 0.896 | ||
| Bootstrap CI | 0.918 | 0.978 | 0.946 | 0.846 | 0.583 | 0.372 | 0.200 | 0.077 | 0.841 | 0.862 | 0.873 | 0.896 | 0.841 | 0.862 | 0.873 | 0.896 | ||
| Random | Distribution Based CI | 0.911 | 0.696 | 0.415 | 0.169 | 0.996 | 1.000 | 0.683 | 0.314 | 0.989 | 0.996 | 0.998 | 0.999 | 0.967 | 0.967 | 0.964 | 0.966 | |
| Moments Based CI | 1.000 | 1.000 | 0.999 | 0.964 | 0.606 | 0.399 | 0.236 | 0.105 | 1.000 | 0.875 | 0.880 | 0.905 | 1.000 | 0.875 | 0.880 | 0.905 | ||
| Bootstrap CI | 0.918 | 0.978 | 0.946 | 0.846 | 0.514 | 0.335 | 0.180 | 0.068 | 0.819 | 0.850 | 0.868 | 0.893 | 0.819 | 0.850 | 0.868 | 0.893 | ||
| | ||||||||||||||||||
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)
(j)
(k)
(l)
(m)
(n)
(o)
(p)
In the next sections, we give certain examples and we present the lower limits of confidence intervals for testing whether , according to the suggested rule of thumb by [11]. We choose only the variance from a fixed number of studies when the are drawn from a half normal distribution HN(0,1).
5. Examples
In this section, we present two examples of meta-analyses from the literature. The first study is a meta-analysis of the effect of probiotics for preventing antibiotic-associated diarrhoea and included 63 studies [24]. The second meta-analysis comes from the psychological literature and is a meta-analysis examining reward, cooperation, and punishment, including analysis of 148 effect sizes [25]. For each meta-analysis, we computed Rosenthal's fail-safe number and the respective confidence interval with the methods described above (Table 3).
| | |||||
| Fixed number of studies | Random number of studies | Bootstrap based CI | |||
| Distribution based CI | Moment based CI | Distribution based CI | Moment based CI | ||
| | |||||
| Study 1 [24] Rosenthal's | (2060, 2188) | (788, 3460) | (2059, 2189) | (369, 3879) | (740, 3508) |
| | |||||
| Study 2 [25] Rosenthal's | (73709, 74012) | (51618, 96102) | (73707, 74013) | (40976, 106745) | (51662, 96059) |
| | |||||
We observe that both fail-safe numbers exceed Rosenthal's rule of thumb, but some lower confidence intervals, especially in the first example, go as low as 369, which only slightly surpasses the rule of thumb ( in this case). This is not the case with the second example. Hence the confidence interval, especially the lower confidence interval value, is important to establish whether the fail-safe number surpasses the rule of thumb.
In the next section, we present a table with values according to which future researchers can get advice on whether their value truly supersedes the rule of thumb.
6. Suggested Confidence Limits for
We wish to answer the question whether for a given level of significance and the estimator , which is the rule of thumb suggested by Rosenthal. We formulate a hypothesis test according to which An asymptotic test statistic for this is under the null hypothesis.
So we reject the null hypothesis if.
In Table 4 we give the limits of above which we are 95% confident that . For example if a researcher performs a meta-analysis of 25 studies, the rule of thumb suggests that over studies there is no publication bias. The present approach and the values of Table 4 suggest that we are 95% confident for this when exceeds 209 studies. So this approach allows for inferences about Rosenthal's and is also slightly more conservative especially when Rosenthal's fail-safe number is characterised from overestimating the number of published studies.
| | |||||||
| Cutoff point | Cutoff point | Cutoff point | Cutoff point | ||||
| | |||||||
| 1 | 17 | 41 | 369 | 81 | 842 | 121 | 1394 |
| 2 | 26 | 42 | 380 | 82 | 855 | 122 | 1409 |
| 3 | 35 | 43 | 390 | 83 | 868 | 123 | 1424 |
| 4 | 45 | 44 | 401 | 84 | 881 | 124 | 1438 |
| 5 | 54 | 45 | 412 | 85 | 894 | 125 | 1453 |
| 6 | 63 | 46 | 423 | 86 | 907 | 126 | 1468 |
| 7 | 71 | 47 | 434 | 87 | 920 | 127 | 1483 |
| 8 | 79 | 48 | 445 | 88 | 934 | 128 | 1498 |
| 9 | 86 | 49 | 456 | 89 | 947 | 129 | 1513 |
| 10 | 93 | 50 | 467 | 90 | 960 | 130 | 1528 |
| 11 | 99 | 51 | 479 | 91 | 973 | 131 | 1543 |
| 12 | 106 | 52 | 490 | 92 | 987 | 132 | 1558 |
| 13 | 112 | 53 | 501 | 93 | 1000 | 133 | 1573 |
| 14 | 118 | 54 | 513 | 94 | 1014 | 134 | 1588 |
| 15 | 125 | 55 | 524 | 95 | 1027 | 135 | 1603 |
| 16 | 132 | 56 | 536 | 96 | 1041 | 136 | 1619 |
| 17 | 140 | 57 | 547 | 97 | 1055 | 137 | 1634 |
| 18 | 147 | 58 | 559 | 98 | 1068 | 138 | 1649 |
| 19 | 155 | 59 | 571 | 99 | 1082 | 139 | 1664 |
| 20 | 164 | 60 | 582 | 100 | 1096 | 140 | 1680 |
| 21 | 172 | 61 | 594 | 101 | 1109 | 141 | 1695 |
| 22 | 181 | 62 | 606 | 102 | 1123 | 142 | 1711 |
| 23 | 190 | 63 | 618 | 103 | 1137 | 143 | 1726 |
| 24 | 199 | 64 | 630 | 104 | 1151 | 144 | 1742 |
| 25 | 209 | 65 | 642 | 105 | 1165 | 145 | 1757 |
| 26 | 218 | 66 | 654 | 106 | 1179 | 146 | 1773 |
| 27 | 228 | 67 | 666 | 107 | 1193 | 147 | 1788 |
| 28 | 237 | 68 | 679 | 108 | 1207 | 148 | 1804 |
| 29 | 247 | 69 | 691 | 109 | 1221 | 149 | 1820 |
| 30 | 257 | 70 | 703 | 110 | 1236 | 150 | 1835 |
| 31 | 266 | 71 | 716 | 111 | 1250 | 151 | 1851 |
| 32 | 276 | 72 | 728 | 112 | 1264 | 152 | 1867 |
| 33 | 286 | 73 | 740 | 113 | 1278 | 153 | 1883 |
| 34 | 296 | 74 | 753 | 114 | 1293 | 154 | 1899 |
| 35 | 307 | 75 | 766 | 115 | 1307 | 155 | 1915 |
| 36 | 317 | 76 | 778 | 116 | 1322 | 156 | 1931 |
| 37 | 327 | 77 | 791 | 117 | 1336 | 157 | 1947 |
| 38 | 338 | 78 | 804 | 118 | 1351 | 158 | 1963 |
| 39 | 348 | 79 | 816 | 119 | 1365 | 159 | 1979 |
| 40 | 358 | 80 | 829 | 120 | 1380 | 160 | 1995 |
| | |||||||
7. Discussion and Conclusion
The purpose of the present paper was to assess the efficacy of confidence intervals for Rosenthal's fail-safe number. We initially defined publication bias and described an overview of the available literature on fail-safe calculations in meta-analysis. Although Rosenthal's estimator is highly used by researchers, its properties and usefulness have been questioned [48, 58].
The original contributions of the present paper are its theoretical and empirical results. First, we developed statistical theory allowing us to produce confidence intervals for Rosenthal's fail-safe number. This was produced by discerning whether the number of studies analysed in a meta-analysis is fixed or random. Each case produces different variance estimators. For a given number of studies and a given distribution, we provided five variance estimators: moment- and distribution-based estimators based on whether the number of studies is fixed or random and on bootstrap confidence intervals. Secondly, we examined four distributions by which we can simulate and test our hypotheses of variance, namely, standard normal distribution, half normal distribution, a positive skew normal distribution, and a negative skew normal distribution. These four distributions were chosen as closest to the nature of the . The half normal distribution variance estimator appears to present the best coverage for the confidence intervals. Hence, this might support the hypothesis that are derived from a half normal distribution. Thirdly, we provide a table of lower confidence intervals for Rosenthal's estimator.
The limitations of the study initially stem from the flaws associated with Rosenthal's estimator. This usually means that the number of negative studies needed to disprove the result is highly overestimated. However, its magnitude can give an indication for no publication bias. Another possible flaw could come from the simulation planning. We could try more values for the skew normal distribution, for which we tried only two values in present paper.
The implications of this research for applied researchers in psychology, medicine, and social sciences, which are the fields that predominantly use Rosenthal's fail-safe number, are immediate. Table 4 provides an accessible reference for researchers to consult and apply this more conservative rule for Rosenthal's number. Secondly, the formulas for the variance estimator are all available to researchers, so they can compute normal approximation confidence intervals on their own. The future step that needs to be attempted is to develop an -package program or a Stata program to execute this quickly and efficiently and make it available to the public domain. This will allow widespread use of these techniques.
In conclusion, the present study is the first in the literature to study the statistical properties of Rosenthal's fail-safe number. Statistical theory and simulations were presented and tables for applied researchers were also provided. Despite the limitations of Rosenthal's fail-safe number, it can be a trustworthy way to assess publication bias, especially under the more conservative nature of the present paper.
Appendices
A. Proofs for Expressions (19), (20), and (22)
(a) Fixed . in the formula of the estimator (9) are i.i.d. distributed with and . Let ; then, according to the Lindeberg-Lévy Central Limit Theorem [42], we have So we have Then, from (6) we get Now we seek to compute , . For this we need the moments of the normal distribution, which are given in Table 5 [59].
| | ||
| Order | Noncentral moment | Central moment |
| | ||
| 1 | ||
| 2 | ||
| 3 | ||
| 4 | ||
| 5 | ||
| | ||
So
(b) Random . In this approach, we additionally assume that . So is a compound Poisson distributed variable [60]. Hence, from the law of total expectation and the law of total variance [44], we get Thus, from (6) we get To compute the final variance, it is more convenient to compute each component separately.
We will need the moments of a Poisson distribution [60], which are given in Table 6.
| | ||
| Order | Noncentral moment | Central moment |
| | ||
| 1 | ||
| 2 | ||
| 3 | ||
| 4 | ||
| 5 | ||
| | ||
We then have So Also Hence, we finally have
B. Proof of Expression (14): The Characteristic Function
From (13) we have that because = .
C. Proof of Expressions (15) and (16)
The cumulant generating function is For simplicity, let . Then with .
Then, leads to (15).
Next Then leads to (16).
Conflict of Interests
The authors declare that there is no conflict of interests regarding the publication of this paper.
Supplementary Materials
The Supplementary Materials contain the simulation code used in R and the Z-values from the meta-analyses used as examples in Section 5.
- Supplementary Materials
Copyright
Copyright © 2014 Konstantinos C. Fragkos et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
The File Drawer Problem And Tolerance For Null Results
Source: https://www.hindawi.com/journals/isrn/2014/825383/
Posted by: holcombwhopribed.blogspot.com
0 Response to "The File Drawer Problem And Tolerance For Null Results"
Post a Comment