The Permutation Test is Used to Reduce the Probability of Finding False Marker-QTL Associations

Citations:
Churchill, G.A., and R.W. Doerge. 1994. Empirical threshold values for quantitative trait mapping. Genetics 138:963-971.
Knott, S.A. and C.S. Haley. 1992. Aspects of maximium likelihood methods for the mapping of quantitative trait loci in line crosses. Genet. Res. 60:139-151.
Lander, E.S. and D. Botstein. 1989. Mapping Mendelian factors underlying quantitative traits using RFLP linkage maps. Genetics 121:185-199.
Liu, B.H. 1998. Statistical Genomics: Linkage, mapping and QTL analysis. Pg. 570. CRC Press, Boca Raton.

Synopsis:
1)The probability of Type I error is determined for the entire experiment by using a random sample of permutations.
2)The experiment wise error rate is different for each experiment. Factors that affect the experiment wise error include: the sample size, the genome size of that specie, the number of markers evaluated, the number of QTL that influence the trait, and the magnitude of the effects of the QTL's (Churchill and Doerge, 1994).
3)If the probability of Type I error is set at 5% for each marker-QTL association test, the probability of identifying marker-QTL associations that do not exist is unacceptably high.
4)If the point-wise Type I error rate is set at 5%, and 100 marker intervals are evaluated, we would expect to find 5 marker-QTL associations that are spurious. This means that we would expect to report 5 markers that are associated with QTL's and none of the QTL's would be real. Assuming each marker-QTL test is independent, as in the sparse marker map.
5)If the experiment wise Type I error rate is set at 5%, and 100 marker intervals are evaluated, we would expect that at least one marker-QTL association is spurious in 5% of such experiments.

Experiment Type I Error:
Let a = the probability of making a Type I error. 1 - a = the probability of making a correct decision, that is to accept Ho when Ho is true. If a = 0.05 then 1 - a = 0.95. If we evaluate one marker-QTL association then the probability of making the correct decision is 0.95. Suppose we test for a second marker-QTL association and this test is conducted on a different chromosome than that of the first test, now we have made two independent tests of Ho. The probability of making two correct decisions is (0.95)(0.95) = (0.95)² = 0.9025. The probability of making at least one wrong decision (Type I error) is
1 - (0.95)² = 0.0975 (Knot and Haley, 1992).

Suppose we have n independent tests of marker-QTL associations, then the probability of making at least one Type I error is P = 1 - (1 - a)ⁿ. Where P is the experiment wise Type I error rate. We can set a at some other value than a = 0.05 for each individual test. The individual tests for marker-QTL associations are sometimes referred to as point-wise tests.

When we use a LOD score to test for the presence of a marker-QTL association, we are testing Ho¹ :no QTL is present or Ho²: a QTL is present but is not linked to the marker versus H_A: a QTL is present and linked to the marker (Knott and Haley, 1992). If Ho is true, we would expect to falsely reject Ho 5% of the time when a = 0.05. This means that for each marker interval - QTL test we would expect to falsely identify a QTL 5 times out of 100 repetitions of the experiment. The point-wise Type I error rate is the probability of falsely identifying a marker interval - QTL association within that single interval between the two markers.

The Experiment wise Type I error rate is the probability of falsely identifying a QTL for the entire experiment. Each test for a marker interval - QTL association is a separate test. When we have many markers and many intervals, we are conducting a test of Ho at each interval. The more marker intervals we evaluate for the presence of a QTL, the more likely we are to falsely identify a QTL somewhere in the genome. The experiment wise error rate is much higher than the point wise error rate, because the point wise error rate is only for a single test of Ho.

Example 1:
The first case is for a sparse map where the markers are far enough apart that they are not linked. Let m= number of markers and a = 0.05 for each individual test of Ho. Then the probability of identifying at least one false QTL is given by
P = 1 - (1 - a)^m. For our example, let the point-wise error rate be a = 0.05, then P = 1 - (0.95)¹⁰⁰ = 0.994. This means that there is a 99% chance that at least one marker-QTL association will not be real. If we set a = 0.0001, then P = 1 - (0.9999)¹⁰⁰ = 0.01. The experiment-wise error rate is the probability of making at least one Type I error across all marker-QTL associations that are tested in the whole experiment. The experiment wise error rate is now 0.01.

Example 2:
For the case of dense markers, the markers are close enough to be linked. In this case we use the formula provided by Lander and Botstein (1989): P = 1 - (1 - a)^X. Where X = genome length in Morgans/length between markers. In soybean the genome length is about 30 M and suppose we evaluate a marker space every 20 cM. Then X = 30/0.2 = 150. When we set the point-wise error rate at a = 0.001, then P = 1 - (1 - 0.001)¹⁵⁰ = 0.14, which is still an unacceptably high probability of identifying at least one QTL that does not exist. If we set
the point-wise error rate at a = 0.0001, then P = 1 - (1 - 0.0001)¹⁵⁰ = 0.02. The experiment-wise error rate is 0.02.

Determining the appropriate Type I error rate for individual marker-QTL tests:
For the case of the sparse map, where markers are not closely linked, the point-wise error rate should be a = P/M. Where a is the point-wise Type I error rate, P is the experiment wise error rate and M is the number of markers evaluated.

Example 1:
In this example we assume that we have a sparse map where markers are not linked and each point-wise test is independent. We set the experiment-wise error rate at P = 0.05 and evaluate 100 markers. The point-wise error rate will be 0.05/100 = 0.0005 = a. This corresponds to a LOD score of ½(log₁₀e)
(Z_P/M)².

In this example, the LOD = ½(0.434)(3.28)² = 2.33. Where the log₁₀e = 0.434; P/M = 0.0005; Z_P/M = 3.28. With a LOD of 2.33 we expect to reject Ho when it is true 5% of the 100 tests. This means we expect that five of the marker-QTL associations we identify will be false QTL's. If we use the equation:
                   P = 1 - (1 - 0.0005)¹⁰⁰ = 0.05.
The experiment wise error rate is the probability of making at least one Type I error, which equals 0.05 when the point-wise error rate is set at a = 0.0005. Even when the Type I error rate is set at 0.0005 for each individual marker interval-QTL association test, the probability of Type I error for the 100 markers is still P = 0.05. This is because the more tests we make for marker-QTL associations, the greater the chances that we will falsely reject Ho when it is actually true.

Example 2:
For the case of the dense map, the critical LOD score is best determined using the permutation test of Churchill and Doerge (1994). Each experiment has a different sample size, genome size, map density, and proportion of missing data. In a mapping experiment, some markers are linked, while other markers are not linked. This makes it difficult to determine the critical value (LOD) to declare a marker-QTL association to be significant. Each mapping experiment is unique in the number of markers and linkage relationships between markers. The permutation test is used to develop an empirical distribution to determine the critical value of the LOD score. The linkage relationships between markers are maintained, but the linkage relationships between markers and QTL's are eliminated (Liu, 1998).
The permutation test works by randomly assigning the phenotypic data to each genotype. If Ho is true, then a LOD score that results in a 5% rejection rate of Ho, will provide an P = 0.05 experiment wise error rate. Randomly assigning phenotypic data to each data set maintains the linkage relationships between markers that is unique to that experiment. A sample of say, 1000 different random permutations of the phenotypic data is used. If Ho is true and a = 0.05, then we will falsely find marker-QTL associations in 50 of these random permutations due to chance associations between markers and phenotypic scores.
A LOD score is calculated for each marker interval for each permutation of the data. The LOD scores are then ranked from lowest to highest. The 950^th largest LOD score out of 1000 permutations is then the critical value for rejecting Ho for that point-wise marker interval - QTL association. The highest LOD score for each marker is then compiled across all marker intervals. The 950^th highest LOD score represents the critical value that provides an P = 0.05 experiment wise Type I error. The idea is that if there is no relationship between markers and QTLs, 5% of the tests of Ho will be rejected when Ho is actually true. We can find this critical LOD score value by developing a data set where Ho is true and then conducting repeated permutations of the data.
The formula P = 1 - (1 - a)ⁿ is based on the idea that each of the n tests for marker-QTL associations are independent. When markers are linked, the point-wise tests are not independent. This is because making a Type I error for one marker will increase the probability of making a Type I error at a linked marker. The structure of which markers are linked and which markers are not linked is different for each specific experiment. For this reason the correct a level for point-wise tests that will give a P = 0.05 experiment wise error rate cannot be determined using a formula. The best way to determine the a level for point-wise tests for a given experiment is to use the permutation test (Churchill and Doerge, 1994)