Preliminary test selection

We made a selection of different aggregation tests, all freely available in R packages, and able to handle dichotomous traits [5–7, 12, 15, 20, 22, 25, 31, 35–37, 41, 42, 44–57, 59–62, 64, 66–68, 77, 78]. After literature review, we identified 10 R packages implementing 154 aggregation tests (see S1 Table). For the sake of scalability, we performed a preliminary selection, divided into 4 steps, to select the most computationally efficient methods. The first step uses the data publicly available within the SKAT package [35, 41, 42, 46, 54] to filter out the tests that did not work or were computationally too demanding. The SKAT data consists of a matrix Z, i.e., a numeric genotype matrix of 2000 individuals and 67 SNPs. Each row and each column represent a different individual and a different SNP, respectively. We use y. b, a numeric vector of binary phenotypes. Based on these data, we performed 5 analyses with different cohort sizes (100, 500, 1000, 1500, and 2000), ran each test, and retrieved the p-values and computational time for each. We removed tests with mean computational time above 10 seconds (step 1.A: 19 tests removed), maximum computational time above 10 seconds (step 1.B: 22 tests removed) and tests that did not work (step 1.C: 6 tests removed). The size of the genotype matrix did not correlate with computational time explosion for DoEstRare [22]. We ran a separate analysis to investigate that test (step 1.D) resulting in removing this test as a potential candidate due to scalability concerns.

The second step consisted in computing the difference and redundancy (with redundancy defined as the number of times two tests had the same p-value for the same analysis) for the 106 remaining tests. Tests being redundant with others were removed based on the maximum evolution of computational time to keep only the most computationally efficient ones. This was computed as

In total 18 tests were removed in this second step.

The third analytical step was based on the same data as in step 1 and 2, but extending the genotype matrix Z and phenotype vector y. b to 4000 and 8000 individuals (conserving a balanced cohort). For the new individuals, SNPs in the genotype matrix were randomly generated. We ran the remaining 88 tests on these two datasets to assess the computational time, removing again tests having both a maximum time above 10 sec and evolution above 10 (step 3: 11 tests removed).

Through these three steps, computationally prohibitive aggregation tests were removed. To ensure that there were no redundant tests left, we performed a final step (step 4 including 77 tests). We ran a power simulation using our state-of-the-art simulation framework (see Table 1 and S1 Fig). Based on 3000 repeats, this preliminary power simulation provided sufficient data to compute the differences and redundancies between each of them. We used the same criteria from step 2 to decide whether to keep or remove a test. We obtained 59 computationally scalable and non-redundant tests that together constituted the potential candidates for the ensemble approach.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Description of experiments.

https://doi.org/10.1371/journal.pcbi.1011488.t001

Detailed values of each parameter for the 9 type I error experiments grouped into 4 scenarios and the 18 empirical power experiments grouped into 7 scenarios, and the power analysis performed for preliminary test selection.

Overcoming limitations of aggregation tests with our ensemble methods

Based on the 59 aggregation tests, an ensemble method was created with the ability to overcome the main (classical) limitations of individual methods. Our two criteria to evaluate these methods were the ability to control type I error and achieving the best average power across all simulations. We investigated several possible ensemble methods, and present four of them. The first method, called Excalibur_baseline, is a baseline combination of all 59 state-of-the-art methods (S2 Table). The second method, called GoodTypeI_Excalibur, was constructed as follows:

1. From Set A = {59 state-of-the-art methods}, we selected the ones with a proportion of inflated type I error equal to zero (i.e. 31 methods), which will be referred to as Set B
2. From Set B, across all empirical power simulations,
   1. We computed a score defined as the number of times a test was the only one having a significant p-value (out of the tests in Set B)
   2. All tests that had a score equal to zero (e.g. CAST fisher and chisq) were removed, producing a new Set C
3. From Set C, we defined the reliability of a test as the sum of significant and non-significant p-values divided by the number of replicates. Methods with median reliability above 0.9 were kept, thus removing cmat and VT, for instance.

The GoodTypeI_Excalibur ensemble method thus consists of 24 tests (S2 Table).

A third ensemble, which we called Excalibur, was constructed by expanding the second ensemble method, using the following selection steps:

1. We started with tests present in GoodTypeI_Excalibur as candidates, other tests = Set A
2. From Set A, we selected test(s) with the minimum proportion of inflated type I error, Set B
3. From Set B, across all empirical power simulations,
   1. computed a score defined as the number of times a test was the only one having a significant p-value (out of the tests in Set B)
   2. From Set B, selected test(s) with maximum score and added it/them to candidates, and updated Set A by removing candidates from Set A
4. Recomputed type I error for all simulations for the updated candidates
   1. If the proportion of inflated type I error equaled to zero, we restarted the procedure at step 2
   2. Otherwise, we stopped the procedure, and removed the last candidates from Set A. This Set A constituted the new ensemble method to be tested

As the algorithm successfully looped 12 times, Excalibur ensemble method consists of 36 tests (S2 Table).

The 4^st ensemble method, is another attempt to expand the second ensemble (i.e. GoodTypeI_Excalibur) in an orthogonal manner:

1. We started with tests present in GoodTypeI_Excalibur as candidates, other tests = Set A
2. From Set A, across all empirical power simulations,
   1. computed a score defined as the number of times a test was the only one having a significant p-value (out of the tests in Set A).
   2. Computed their rank, i.e. in order of decreasing score.
   3. Computed a rank type I, ranging from the smallest proportion of inflated type I error to the biggest.
   4. Established a combined rank as the sum of rank and rank type I.
3. From Set A, selected test(s) with minimum combined rank and added it (them) to candidates, and updated Set A by removing candidates from Set A
4. Same as Step 4 (see above)

This implementation successfully looped 4 times, generating the new ensemble method called V2_Excalibur, based on 28 tests (see S2 Table).

Additional ensemble creation methods were explored but did not result in better ensembles than GoddTypeI_Exaclibur and therefore are not presented here. P-values for our ensemble methods are computed as the minimum p-value from the set of tests included in the ensemble method after multiple testing correction using Bonferroni.

Design of numerical experiments and simulations

In this section we describe the design of numerical experiments and simulations. To construct and validate our Excalibur ensemble method in terms of protecting type I error and to assess its power compared to the 59 tests, we used our simulation pipeline (see S1 Fig) to carry out simulation studies under a wide range of experiments (see Table 1). For each simulation, we determined sequence genotypes using COSI 2 [80, 81]. We simulated 10,000 COSI haplotypes, each resembling a 1 million base pair region based on COSI’s coalescent model that mimics the local recombination rate based on the linkage disequilibrium pattern and the population history for Europeans. For each simulation, we set disease prevalence β₀ to 1%, as previously done in [22, 41, 42, 46, 59, 82, 83], and α = (0.05, 0.01, 0.001).

Type I error simulations

To investigate whether our ensemble method and the aggregation tests under study preserved the desired type I error rate under various scenarios, we ran 9 experiments (see Table 1). We tested 4 input parameters potentially having an impact on type I error:

1. Case n_cases and control n_controls proportion within the cohort, using proportions of cases being set to 0.2, 0.5 or 0.8
2. Cohort size [5, 36, 41, 42, 46, 60–62, 83–86]: varying the total number of individuals (n) from 100, 500, 1000 to 5000.
3. Including or excluding common variants (i.e., MAF > causal MAF cutoff) [59, 60, 62]: taking into account only rare variants (MAF<causal MAF cutoff) or including also common variants (MAF>causal MAF cutoff)
4. Size of the regions to be analyzed [41, 42, 49, 55, 83]: testing with 1K, 3K and 5K base pair regions

We performed 10000 simulations for each experiment. To evaluate type I error, we estimated the empirical type I error rate as the proportion of p-values less than α, as proposed in [22, 41, 42, 46, 83].

For each simulation, we randomly selected regions of given size while ensuring a minimum number of variants (m≥2), minimum number of causal variants (m_c), and different regions for each simulation performed within the same experiment [22, 41, 42, 46, 59].

To evaluate type I error, datasets under the null model and dichotomous phenotypes (given the desired cohort size, and proportion of cases (n_cases) and controls (n_controls)) were generated from the null logistic regression model where X_i1 was a continuous covariate from N(0,1), and X_i2 was a binary covariate from Bernoulli(0.5), as proposed [22, 41, 42, 46, 54, 56, 59, 60, 62, 82, 83].

Empirical power simulations

To investigate the empirical power of our ensemble method and the aggregation tests under various scenarios, we ran 18 experiments (see Table 1). We tested seven input parameters potentially having an impact on the empirical power:

1. Case n_cases and control n_controls proportion within the cohort, using proportions of cases being set to 0.2, 0.5 or 0.8
2. Cohort size [5, 36, 41, 42, 46, 60–62, 83–86]: varying the number of individuals (n) from 100, 500, 1000 to 5000.
3. Percentage of protective variants [24, 36, 41, 42, 46, 49, 54, 59, 83]: percentage of causal variants with a protective effect set to 0%, 5%, 10% or 20%
4. Percentage of causal variants [22, 24, 41, 42, 46–49, 54, 58, 83, 87]: percentage of variants (m), with MAF<causal MAF cutoff, being assigned as causal, set to 1%, 5%, 10% or 20%
5. Including or excluding common variants [59, 60, 62]: taking into account only rare variants (MAF<causal MAF cutoff) or including also common variants (MAF>causal MAF cutoff)
6. Causal MAF cutoff [6, 24, 36, 41, 42, 46, 48, 54, 55, 59, 62, 83, 87]: threshold separating rare and common variants, set to 0.01 or 0.03
7. Size of the genetic region [41, 42, 49, 55, 83]: testing with 3K and 5K base pairs

We performed 1000 simulations per experiment. We estimated the empirical power as the proportion of p-values less than α, as proposed in [22, 41, 42, 46, 83]. For each simulation, we randomly selected regions of given size while ensuring a minimum number of variants (m≥2), minimum number of causal variants (m_c), and different regions for each simulation perform within the same experiment. We randomly assigned causal variants among the ones with MAF<causal MAF cutoff. Datasets under the alternative modeland dichotomous phenotypes (given the desired cohort size, proportions of cases (n_cases) and controls (n_controls)) were generated from the logistic regression model where X_i1 was a continuous covariate from N(0,1), and X_i2 was a binary covariate from Bernoulli(0.5), as proposed [22, 41, 42, 46, 54, 56, 59, 60, 62, 82, 83]. G_ij are the genotypes of the i causal rare variants (randomly selected subset of the simulated rare variants). For all aggregation tests under study (and able to integrate a weighting scheme of the variant), equation was used to set β as the effect size for causal variants and up weight rarer variants, as proposed [41, 42, 46, 83]. We scaled down c with larger percentage of causal variants (e.g c = , and when the percentage of causal variants was 20%, 10% or 5% respectively, as proposed [41, 42, 46, 82].

Type I error

We assessed type I error for all the tests over nine experiments, each based on 10 000 replications, accounting for 90 000 simulations in total (see Table 1). For each simulation and each test, type I error was evaluated given three α levels (0.05, 0.01, 0.001). The detailed results are in S3 Table. Several terms to analyze type I error results need to be noted:

• Significant p-value was a p-value below or equal to the α level under consideration, otherwise the p-value was considered not significant.
• NA if a test failed to return a p-value
• Type I error was equal to the number of significant p-values divided by the number of returned p-values (for a particular experiment and a given α level). Returned p-values equal the number of replicates (10 000) times reliability.

Type I error is thus not defined as the number of significant p-values divided by 10 000 (number of replicates) as this would result in an overestimation. NA is used to establish the reliability (see section material and methods) of the test for a particular experiment. The goal was to find the aggregation test that can controls type I error across all experiments and α levels (Fig 1 and supporting information in S3 and S4 Tables). In total, 27 type I error results were generated per test (i.e. 9 experiments, each evaluated at three different alpha levels). We defined:

• Proportion good as the number of times a test managed to control type I error divided by 27, i.e. the number of type I error results generated per test. “Good” or “well-controlled” were used as synonyms
• Proportion inflated is the proportion, out of the 27 results, in which a test had an inflated type I error
• Proportion conservative is the proportion, out of the 27 results, in which a test had a conservative type I error
• Proportion of NA is defined as the proportion a test fails to return a p-value
• The minimum, maximum and median reliability across all results are shown as well.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Summary of type I error for nine experiments.

X-axis is the experiment ID (see description in Table 1) evaluated at three α levels (e.g. 0.05, 0.01 and 0.001) for our 4 ensemble methods and 59 state-of-the-art aggregation methods (see Y-axis). Green: type I error below the α level threshold; therefore labelled as well controlled. Red: type I error above the α level threshold; therefore labelled as inflated. NA: test failed to return p-values.

https://doi.org/10.1371/journal.pcbi.1011488.g001

For each α level, we defined a confidence interval assuming that the type I error followed a binomial distribution with parameter 10000 (i.e., number of replicates per experiment) and α level. We defined an α level threshold as the upper limit of this interval, namely, 0.054, 0.0121 and 0.0018 for α levels 0.05, 0.01 and 0.001, respectively (see red cases in Fig 1 and upper blue lines in S2 and S3 Figs). Therefore, a test had an inflated type I error for a particular experiment if its type I error was above that α level threshold. We defined an α level threshold as the lower limit of this interval, namely, 0.046, 0.0079 and 0.0002 for α levels 0.05, 0.01 and 0.001, respectively (see bottom blue lines in S2 Fig and S3). Therefore, a test had a conservative type I error for a particular experiment if its type I error was below that α level threshold.

The results in Fig 1 show that 20 state-of-the-art methods and 3 versions of our ensemble methods could control type I error across all experiments and α-values. In addition, these tests exhibited a median reliability above 99%. An additional eight methods did not have inflated type I error but failed to retrieve a p-value in one experiment. Among them, five tests (i.e. p_ascore, p_cast_fisher, p_cmc, p_score, p_wss and p_t1p) had a reliability of 99% and can be considered to control well type I error, except in small cohorts. The other two methods (i.e. p_cmat and p_vt) suffered from a 2% reliability median. In total, 22 tests did not work each time (see Fig 1) and were unable to return a p-value for some simulations. This discovery we encountered during our investigation is both intriguing and merits further inquiry. However, we acknowledge that such an exploration lies outside the scope of the present manuscript. For example, type I error experiment ID n°9 presents up to 12 tests that were not successful. Because of our definition of type I error, tests that had a poor reliability had an increased type I error, in some cases leading even to a type I error of 1 (e.g. p_carv_variable and p_carv_hard, which both present the lowest median reliability, S4 Table). These results reveal the inherent difficulty of choosing the right test given a particular analysis and that a naïve combination of all tests (named Excalibur_baseline) leads to an inflated type I error.

Empirical power

We assessed empirical power for all tests over 18 experiments, following a standard procedure with each based on 1000 replications, producing 18,000 simulations in total. For each simulation and each test, the empirical power was evaluated at three α levels (i.e. 0.05, 0.01, 0.001). The detailed results are provided in S5 Table. We defined:

• Significant p-value as a p-value below or equal to the α level under consideration, otherwise the p-value was considered not significant.
• NA if a test failed to return a p-value
• Power is the number of significant p-values divided by 1000 (i.e. the number of replicates for a particular experiment)

Our goal was to find tests achieving the greatest power while controlling type I error across all experiments and α levels. We separated empirical power results:

• Good type I error (see S4 Fig): a test having the proportion of inflated type I error equal to zero (see S4 Table)
• Badly controlled type I error (see S5 Fig): tests having the proportion of inflated type I error above zero (see S4 Table). These tests might, in some experiments, have an inflated type I error.

We evaluated the power of each test for each experiment [18] given each α level [3] and giving 54 empirical power results per test. As shown in S5 Table, at α = 0.05, all methods had a high standard deviation, except for methods that perform poorly on all experiments. This demonstrates the impact of the parameters on each experiment (see Table 1), and highlights the difficulty of choosing the right statistical test. Note that experiments with ID n°6 and 10 exhibited the worst average power across all tests, with 0.109, while experiments with ID n°5, 11 and 16 had the best average power across all tests, 0.475, at nominal α = 0.05. This indicates a dramatic impact when including non-causal variants in the analysis, and additionally underlines the importance of an efficient variant filtering step (or variants selection methods) before running any aggregation test.

For each empirical power experiment, each test was ranked according to its power (S6 Table). Based on these rankings, we computed an average, best and worst ranking achieved by each test across all empirical power results (S7 Table). We found that the best average ranking method with good type I Error are our three other ensemble methods (i.e. Excalibur, V2_Excalibur, GoodTypeI_Excalibur, with average rank of 3.9, 4.4 and 5.6, respectively).

Comparing S4 and S7 Tables reveals the importance of performing a preliminary type I error analysis. For example, in the top 8 average ranked tests in S7 Table, p_wgscan_region, pBin_ _weighted_IBS_SKAT_MA, p_linear_weighted_davies_skat and pBin_linear_weighted_SKATO_MA exhibit a good type I error in only 18 out of 27 experiments at best, while the Excalibur_baseline method only achieves that in 1 experiment (Fig 1). One should thus be careful when comparing tests based on their empirical power without considering their type I error.

Without going into the details of the extensive literature [88–90] on the relationship between type I and type II errors (type II error = (1−Power)), increasing the power of a test generally leads to an increase of type I error. It is preferable to deal with a smaller power than an inflated type I error. In other terms, it is better to have a small number of reliable results, than numerous results with poor confidence, as aggregation tests aim to explore genomic data to guide researchers towards further in-vitro experiments. Fig 2 shows the proportion of experiments where type I error was well controlled along with the average power across all experiments for our four ensemble methods and the 59 state-of-the-art methods. Except for Excalibur_baseline, all three ensemble methods control type I error while improving the power, with Excalibur being the best. In summary, this analysis allowed us to establish a test performing on average with the best rank in power across all experiments and alpha levels, while controlling type I error in each experiment. In the 18 empirical power experiments, the best three methods are always our three ensemble methods, except for experiments ID n° 10 where robust_SKAT and robust_SKATO are better.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Proportion of experiments where type I error is well controlled and the average power.

Proportion of experiments where type I error was well controlled (see x-axis) and the average power (see y-axis) computed across all experiments [18] and α levels [3] (54 results per tests) for our 4 ensemble methods (in red) and 59 state-of-the-art methods (in turquoise). To improve visual interpretability, some methods were grouped (see Methods for the number being grouped). Legend ID 1) Excalibur_baseline 2) GoodTypeI_Excalibur 3) Excalibur 4) V2_Excalibur 5) robust_burden, p_cast_chisq, p_rvt1 6) robust_SKAT, robust_SKATO 7) p_linear_liu_burden 8) p_linear_weighted_liumod_burden 9) p_linear_davies_skat 10) p_linear_weighted_davies_skat 11) pBin_linear_IBS_SKAT_MA, pBin_linear_SKATO_MA 12) pBin_weighted_IBS_SKAT_MA 13) pBin_2wayIX_SKAT_MA 14) pBin_weighted_quadratic_SKAT_ERA 15) pBin_linear_weighted_SKATO_MA 16) p_ascore, p_cmc, p_score, p_wss 17) p_ascore_ord, p_assu, p_asum_ord, p_calpha, p_ssuw, p_wst 18) p_assu_ord, p_asum, p_bst, p_ssu, 19) p_calpha_asymptopic 20) p_carv_hard 21) p_carv_variable 22) p_carv_stepup 23) p_cast_fisher 24) p_cmat, p_vt, p_catt 25) p_rbt 26) p_rvt2 27) p_rwas 28) p_score_asymptopic 29) p_ssu_asymptopic 30) p_ssuw_asymptopic 31) p_ttest_asymptopic 32) p_wst_asymptopic, p_rebet 33) p_KAT 34) p_SKATplus 35) p_wgscan_region 36) p_pcr 37) p_rr 38) p_spls 39) p_t1p 40) p_t5p, p_wep 41) p_score_vtp 42) p_wod01 43) p_wod05 44) p_ada.

https://doi.org/10.1371/journal.pcbi.1011488.g002

In S6 Fig, we conducted a comprehensive analysis of 59 state-of-the-art methods using principal component analysis (PCA) based on 18,000 empirical power simulation results. Different implementations of the SKAT methods exhibit clustering patterns (e.g. pBin_weighted_quadratic_SKAT_ERA, pBin_2wayIX_SKAT_MA and pBin_linear_IBS_SKAT_MA). In order to assess the proportion of similarity among two tests, we compared the decision (p-value significant or not) evaluated at α = 0.05 for 18,000 empirical power simulation. S7 Fig shows the proportion of similarity above 0.5 of our 4 ensemble methods and 59 state-of-the-art tests. S8 Fig is similar to S7 Fig, while performing a hierarchical clustering and focusing on 36 state-of-the-art methods included in Excalibur. Most methods have a proportion of similarity bellow 0.5, with the highest similarity (i.e. 0.96) achieved by p_cast_chisq and p_rvt2.

Exploring some limitations of aggregation tests

To investigate the impact of some of the limitations on the performance of the aggregation tests, exhaustive simulations covering four type I error scenarios and seven empirical power scenarios were performed. S8 Table shows the different experiments in order to compare the impact of a single parameter value on the behavior of the tests (type I error and empirical power). Because all other parameters were set equal in all experiments within a scenario (Table 1), parameter by parameter conclusions can be drawn. The detailed results are provided in S9 and S10 Tables. We defined:

• Evolution as the type of change (e.g. increase, decrease, no change, …) in type I error or empirical power when increasing a parameter value
• Total evolution as the sum of changes in type I error or empirical power when changing a parameter value (e.g. when increasing the proportion of cases within the cohort)

For each scenario, we evaluated our 4 ensemble methods and the 59 state-of-the-art tests at three α levels. For example, S9 and S10 Figs show type I errors and empirical power results at nominal level α = 0.05 for the scenario in which cohort size was increased. Data in S11 Table confirms that increasing a parameter value (in this case the cohort size) can have a heterogeneous effect on the test behavior (even within the same test). The total power evolution gave a global overview of the impact of changing a parameter value on a test behavior and led to the following observations (Fig 3, S12 Table and S11 Fig). Here bellow we only discuss tests with a proportion of inflated type I error equal to zero, giving each time the 5 best state-of-the-art methods (based on their power):

1. Increasing the proportion of cases within the cohort led to an increase in empirical power for the 4 ensemble methods and 26 of the state-of-the-art methods (up to 0.4 increase in power for p_cast_chisq), while decreasing the power of the other tests (up to -0.57 decrease in power for p_ssuw_asymptopic). We identified that robust_SKATO and robust_SKAT perform best regardless of the proportion of cases to be low (0.2) or high (0.8). For low number of cases, the next best methods were p_ssuw_asymptopic, p_ttest_asymptopic and p_ssu_asymptopic, while for high number of cases, the next best methods were p_cast_fisher, p_cast_chisq and p_wep.
2. One could expect that increasing cohort size should lead to an increase in power, but our results show that this does not hold for nine of the methods. For example, p_rbt suffered from a 0.056 decrease in power, while the average gain of power across all tests was 0.16. Note that the GoodTypeI_Excalibur power was increased up to 0.84 when increasing the cohort size. For a small cohort (100 individuals), we identified as the best methods p_ssu_asymptopic, p_rbt, p_score, robust_SKATO, and robust_SKAT. For a large cohort (5000 individuals), the best methods were robust_SKAT, robust_SKATO, p_ttest_asymptopic, p_ssuw_asymptopic, p_cast_fisher and p_cast_chisq.
3. In the presence of an increasing number of protective variants, the average total power evolution across methods remained equal, except for 8 methods (all belonging to the Burden test class), with the worst decrease in power of 0.15 for p_wep. There was nearly no gain of power attributed to protective variants (i.e. best total evolution of power was only 0.04 for robust_SKAT). Regardless of the proportion of protective variants, the best methods were robust_SKAT, robust_SKATO, p_cast_fisher, p_cast_chisq and p_rvt1.
4. On average, the most impactful parameter regarding the total evolution of power was the percentage of causal variants, ranging from a decrease of 0.01 for p_asum_ord to an increase of 0.74 for robust_SKATO, stressing the importance of a decent method for variant filtering and selection prior to running an aggregation test. With a small percentage of causal variants (1%), the best methods where robust_SKAT, robust_SKATO, p_ssuw_asymptopic, p_ttest_asymptopic, and p_ssu_asymptopic. On the other hand, with a high percentage of causal variants (20%), the best methods were robust_SKATO, robust_SKAT, p_cast_fisher, p_cast_chisq and p_rvt1
5. All methods lost power when introducing common variants, up to 0.5 for V2_Excalibur. There was no clear gain of power linked to the introduction of common variants. The best methods to handle only rare variant were robust_SKAT, robust_SKATO, p_ttest_asymptopic, p_ssuw_asymptopic and p_ssu_asymptopic. The best methods to handle a mixture of common and rare variants were robust_SKATO, robust_SKAT, p_wep, p_t5p and p_rvt1.
6. On average, relaxing the MAF cutoff from 0.01 to 0.03 led to very small increase in total evolution of power (0.04) and very limited decrease (worst decrease was attributed to p_ssu_asymptopic with -0.127), while both CAST methods (p_cast_fisher and p_cast_chisq) presented a total evolution of 0.32. When using a small cutoff (0.01) for the MAF, the best methods were robust_SKATO, robust_SKAT, p_wep, p_t5p and p_ssu_asymptopic.
7. The size of the genetic region was, on average the less impactful parameter (gain of 0.02), ranging from a limited loss of 0.097 (for p_ssuw_asymptopic) to a maximum total power evolution of 0.16 (for p_ssu_asymptopic). For regions of 3,000 base pairs, the best methods were robust_SKAT, robust_SKATO, p_ttest_asymptopic, p_ssuw_asymptopic, p_ssu_asymptopic. With wider regions (5,000 base pairs), the ranking of the best state-of-the-art methods became robust_SKATO, robust_SKAT, p_ttest_asymptopic, p_ssu_asymptopic and p_wep

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Summary of power evolution across scenarios.

Total evolution of empirical power for seven scenarios (see X axis) for our 4 ensemble methods and 59 state-of-the-art methods (see Y axis) at nominal level α = 0.05 (data in S12 Table). Green: total power evolution above zero. Red: total power evolution below zero. Grey: total power evolution equal to zero. NA: no information for total power evolution. Prop case: evolution of power given the evolution of proportion of case in the cohort, based on empirical power ID n°14, n°11 and n°15 (Tables 1 and S5). Cohort size: evolution of power given the evolution of cohort size, based on empirical power ID n°6, n°7, n°8 and n°9 (Tables 1 and S5). % protective: evolution of power given the evolution of proportion of protective variants, based on empirical power ID n°11, n°12 and n°13 (Tables 1 and S5). % causal: evolution of power given the evolution of proportion of causal variants, based on empirical power ID n°2, n°3, n°4 and n°5 (Tables 1 and S5). Kind of variant: evolution of power given the inclusion of only rare variants versus rare and common variants, based on empirical power ID n°18 and n°10 (Tables 1 and S5). MAF cutoff: evolution of power given the evolution of causal MAF cutoff, based on empirical power ID n°1 and n°11 (Tables 1 and S5). Region size: evolution of power given the evolution of region size, based on empirical power ID n°17 and n°16 (Tables 1 and S5).

https://doi.org/10.1371/journal.pcbi.1011488.g003

As stated above, because of the heterogeneity of the results, these observations are global and on average, and cannot be extended to the two other α levels (S10 Table). These results underscore the inherent difficulty in choosing the right statistical test for a given set of data and analysis, demonstrating once more the usefulness of our ensemble method to guide the user towards the preferred methods.

Computational time analysis

The computational time of our 4 ensemble methods and 59 state-of-the-art methods depends on the scenario and parameters (see Table 1 and S13 and S12 Figs). For example, the average time for Excalibur on genetic regions of 3 kb and 5 kb in size, is 29 and 52 sec. respectively. The average time to run Excalibur on a cohort of 100 and 5000 individuals is 14 and 31 sec., respectively. Based on the minimum and maximum computational times across all 18,000 empirical power simulation, we established the best and worst computational time required to analyze 20,000 genetic regions (the entire exome) for each test (see S13 Table). Excalibur, as a combination of 36 tests, can take up to 458 hours (see S13 Table) to run the entire genome (up to 82 sec. per gene). A standard solution to this limitation is to use parallel computing. For example, using 10 threads of 8 Gb memory each could lower the computational time of such an analysis to less than two days.