Division 1.3.2.2 Statistical analysis

Analysis of data shall be presented in a clear format, using standardised scientific units. The raw data and the programming code used for the statistical analysis shall be given in an editable form.

Data transformation may be necessary to ensure normality and to provide an appropriate scale on which statistical effects are additive. For many endpoint response variables, a logarithmic transformation is expected to be appropriate. In such cases, any difference between the genetically modified material and any other test material shall be interpreted as a ratio on the natural scale. However, when logarithmic transformation does not provide appropriate results, the natural scale or another scale shall be considered.

The total variability of each endpoint observed in the field trials shall be estimated and partitioned using appropriate statistical models in order to derive two sets of confidence limits and to set a lower and upper equivalence limit based on the variability observed among the reference varieties. One set of confidence limits shall be used in the test of difference; the other set and the equivalence limits shall be used in the test of equivalence.

A linear mixed statistical model shall be used for calculation of the confidence limits for both tests (that is to say, the difference and equivalence tests); a slightly different model shall be used to estimate the equivalence limits to be used in the equivalence test.

Denote by I an indicator variable (uncentered in the mixed model) such that I = 1 for a field plot having any of the non-GM reference varieties, and I = 0 otherwise. Then the random factors for model 1 should be, but not necessarily be restricted to, those representing the variation: (i) between the test materials (a set that includes the GM plant, its conventional counterpart, each of the non-GM reference varieties and any additional comparators); (ii) in the interaction between the test materials and I; (iii) between sites; and (iv) between blocks within sites. Model 2 should be identical to model 1 except that the random factor representing the interaction between the test materials and I is omitted.

The fixed factor for both models should have as many levels as there are test materials and represent the contrasts between the means of the test materials. The test materials are as defined above: the GM plant; its conventional counterpart; the set of non-GM reference varieties; and any additional test materials. The set of non-GM reference varieties is considered as a single level of the fixed factor. For the difference test, the component of the fixed factor of interest is the single degree-of-freedom contrast between the GM plant and its conventional counterpart. For the equivalence test, the component of the fixed factor of interest is the single degree-of-freedom contrast between the GM plant and the set of non-GM reference varieties.

Both the difference test and the equivalence test shall be implemented using the correspondence between hypothesis testing and the construction of confidence limits. In the case of equivalence testing, the approach used shall follow the two one-sided tests (TOST) methodology by rejecting the null hypothesis of non-equivalence when the both confidence limits fall between the equivalence limits. The choice of 90 % confidence limits corresponds to the customary 95 % level for statistical testing of equivalence.

The results of the difference and equivalence tests shall be represented visually for all the endpoints simultaneously, on a single graph or a few graphs.

The graph(s) shall show the line of zero difference between the genetically modified material and its conventional counterpart, and for each endpoint: the lower and upper adjusted equivalence limits; the mean difference between the genetically modified material and its conventional counterpart; and the confidence limits for this difference (see the set of possible outcomes for a single endpoint in the graph in Figure 1).

When, in addition to the conventional counterpart, another test material is used as comparator, the mean difference between the genetically modified material and that comparator, its confidence limits and its adjusted equivalence limits shall be displayed on the graph(s), for all such additional comparators, by referring this to the same zero baseline as defined by the conventional counterpart. The line of zero difference on the logarithmic scale corresponds to a multiplicative factor of unity on the natural scale. The horizontal axis shall be labelled with values that specify the change on the natural scale. In the case of logarithmic transformation, changes of 2x and ½x will appear equally spaced on either side of the line of zero difference.

Despite the expected proportion of spurious significant differences, the applicant shall report and discuss all significant differences observed between the genetically modified crop, its conventional counterpart and, where applicable, any other test material, focusing on their biological relevance (see Section 3 on Risk Characterisation).

For reporting, full details shall be given for each endpoint analysed, listing:

1. (a)

   the assumptions underlying the analysis;

2. (b)

   full specification of the mixed models chosen, including fixed and random effects;

3. (c)

   results of any test of interaction between the test materials and sites;

4. (d)

   fixed effects, together with the appropriate estimated residual variation with which they are compared, and variance components for the random factors;

5. (e)

   estimated degrees of freedom;

6. (f)

   any other relevant statistics.

A discussion on the likely impact of other growing conditions not tested in the field trial shall be provided.

Figure 1: simplified version of a graph for comparative assessment showing the seven outcome types possible for each single endpoint. After adjustment of the equivalence limits, a single confidence limit (for the difference) serves visually for assessing the outcome of both tests (difference and equivalence). Here, only the upper adjusted equivalence limit is considered. Shown are: the mean of the genetically modified crop on an appropriate scale (square); the confidence limits (whiskers) for the difference between the genetically modified crop and its conventional counterpart (bar shows confidence interval); a vertical line indicating no difference (for test of difference); and vertical lines indicating adjusted equivalence limits (for test of equivalence). For outcome types 1, 3 and 5, the null hypothesis of no difference cannot be rejected: for outcomes 2, 4, 6 and 7, the genetically modified crop is different from its conventional counterpart. Regarding interpretation of equivalence, four categories (i) to (iv) are identified: in category (i), the null hypothesis of non-equivalence is rejected in favour of equivalence; in categories (ii), (iii) and (iv), non-equivalence cannot be rejected.

1. A..

   Regarding test of difference, each outcome from the graph shall be categorised as follows and the respective appropriate conclusion shall be drawn.

   1. (i)

      Outcome types 1, 3 and 5: the confidence interval bar overlaps with the line of no-difference. The null hypothesis of no difference cannot be rejected and the appropriate conclusion is that there is insufficient evidence that the genetically modified crop and its conventional counterpart differ.

   2. (ii)

      Outcome types 2, 4, 6 and 7: the confidence interval bar does not overlap with the line of no-difference. The null hypothesis of no difference must be rejected and the appropriate conclusion is that the genetically modified crop is significantly different from its conventional counterpart.

2. B..

   Regarding test of equivalence, each outcome from the graph shall be categorised as follows, and the respective appropriate conclusion shall be drawn.

   1. (i)

      Outcome types 1 and 2 (category (i), Figure 1): both confidence limits lie between the adjusted equivalence limits and the null hypothesis of non-equivalence is rejected. The appropriate conclusion is that the genetically modified crop is equivalent to the set of non-genetically modified reference varieties.

   2. (ii)

      Outcome types 3 and 4 (category (ii), Figure 1): the mean of the genetically modified crop lies between the adjusted equivalence limits, but the confidence interval bar overlaps at least one of the adjusted equivalence limits on the graph. Non-equivalence cannot be rejected and the appropriate conclusion is that equivalence between the genetically modified crop and the set of non-genetically modified reference varieties is more likely to be the case than lack of equivalence.

   3. (iii)

      Outcome types 5 and 6 (category (iii), Figure 1): the mean of the genetically modified crop lies outside the adjusted equivalence limits, but the confidence interval bar overlaps with at least one of the adjusted equivalence limits. Non-equivalence cannot be rejected and the appropriate conclusion is that lack of equivalence between the genetically modified crop and the set of non-genetically modified reference varieties is more likely to be the case than equivalence.

   4. (iv)

      Outcome type 7 (category (iv), Figure 1): both confidence limits lie outside the adjusted equivalence limits. The appropriate conclusion is that there is lack of equivalence between the genetically modified crop and the set of non-genetically modified reference varieties.

   In the case of significant difference and/or lack of equivalence for any particular endpoint, further statistical analysis shall be carried out to assess whether there are interactions between any of the test materials and site, possibly using a simple standard ANOVA approach. Whatever approach is adopted, details shall be given, for each endpoint analysed, listing: (a) the assumptions underlying the analysis, and, when appropriate: (b) degrees of freedom; (c) the estimated residual variation for each source of variation, and variance components; (d) any other relevant statistics. These additional analyses are intended to aid the interpretation of any significant differences found and to study potential interactions between test materials and other factors.

   More detailed guidance for the application of the requirements of this Section is available F1from the Food Safety Authority.