Statistical methods
Item 27a: Statistical methods used to compare groups for primary and secondary outcomes, including harms
Example
“Primary outcome analysis: The primary outcome (days alive and out of hospital within 90 days of randomisation) will be analysed using a mixed-effects negative binomial regression model, with a random-intercept for centre [reference]. The model will be adjusted for the minimisation factors of patient age and ASA [American Society of Anesthesiologists] class (I, II, III, IV, and V) [reference], as well as the following prognostic baseline covariates: urgency of surgery (immediate, urgent, and expedited), Glasgow Coma Score (GCS), systolic blood pressure, and pulse rate [reference]. Urgency of surgery and ASA class will be included as categorical variables, while patient age, GCS, systolic blood pressure, and pulse rate will be included as continuous variables. Patient age and GCS will be included assuming a linear association with the outcome, and systolic blood pressure and pulse rate will be included using restricted cubic splines with 3 knots (knots will be placed based on Harrell’s recommended percentiles) [references]. Missing baseline data will be handled using mean imputation for continuous variables, and a missing indicator variable for categorical variables [reference].
Secondary outcome analysis: Mortality within 90 days and 1 year of randomisation will be analysed using an analogous mixed-effects logistic regression model (same random effects and covariate strategy as primary outcome). Duration of hospital stay and hospital re-admission will be analysed using a competing-risk time-to-event model, which includes mortality as a competing risk [reference]. Both models will adjust for the set of covariates specified above. Duration of stay in a level 2 or level 3 critical care bed will be analysed using a mixed-effects negative binomial regression model, with a random intercept for centre. The model will adjust for the set of covariates specified above ” [407].
“All primary comparisons between treatment arms will be on an intention-to-treat basis, that is, according to the group to which participants were randomised and without reference to their actual compliance with assigned treatment. Each of the co-primary endpoints will be analysed separately in time-to-event analyses. Event rates (time to first event within each endpoint definition) will be compared between groups using an HR and 95% CI from a Cox proportional hazards regression model fitted to the endpoint, with censoring for individuals not experiencing an endpoint event at their most recent study visit, and a single covariate being an indicator of the group to which the individual was randomised, statin or placebo. The proportional hazards assumption will be tested for each model. Loss to follow-up will be considered a censoring event. This equates to an assumption that data is missing at random given the participant’s treatment group and the timing of their loss to follow-up. The adequacy of this assumption will be checked in sensitivity analyses that will include both imputation approaches and adjustment for baseline covariates predictive of propensity for dropout.
A closed testing procedure will be used to allow for the multiple testing arising from two co-primary endpoints. This approach is based on the expectation that cardiovascular benefit will be the main contributor to improved disability-free survival and that a substantial effect of statins on the latter is unlikely in the absence of an effect on the former. First, major cardiovascular events will be tested at alpha=0.05 and, if the major cardiovascular events p value is <0.05 then second, disability-free survival will be tested at alpha=0.05. If the major cardiovascular events p value is not <0.05 than a p value for disability-free survival will not be presented ” [408].
Explanation
A clear and comprehensive account of the planned statistical methods for a trial facilitates implementation, replication, and critical assessment. Details of all statistical analyses are often reported in a full statistical analysis plan (SAP), a document that accompanies a trial protocol [71]. Similar to a protocol, a statistical analysis plan should be date stamped and have any revisions documented (Item 2).
The results for the primary and secondary outcomes can be substantially impacted by the choice of the analysis methods, which should align with the trial objectives and, if used, the estimands framework (Item 10). When more than one analysis method is applied to an outcome, there is the potential for inappropriately selecting the approach which leads to the most “interesting” finding. Pre-specifying the analysis plans in the protocol reduces the risk of selective reporting of outcomes and results [63, 64, 81, 82].
Adjusting for baseline covariates, including those used in any stratified randomisation, is often advised in the analysis, particularly when a baseline covariate is prognostic of the outcome, as it can lead to improved power to detect an intervention effect [409-411].
Trials are often affected by multiplicity issues [412, 413]. When multiple comparisons are performed (e.g., multiple outcomes, time points, subpopulations, interim analyses), there is an inflated risk of false positives (type 1 error). While there are no standardised rules for dealing with multiplicity, guidance is available on key issues to consider [412, 413].
Analysis plans for harms are challenging (Item 17). Data will often consist of a mix of systematically and non-systematically assessed harms, making classic hypothesis testing difficult. Also, harms are often measured differently (e.g., as events, rates, changes on a scale, or time to event) and in general, the numbers involved are low. Randomised trials are thus often underpowered to detect typical harms of interest, such as a rare but clinically serious events that impact trial participants’ quality of life. Also, the relevant risk difference between the compared groups that is interesting to detect may be modest. However, the objective of analysing harms in randomised trials is not only to detect a statistically significant difference, but also to identify preliminary evidence of possible harms (i.e., a signal detection approach). It may be helpful to consider the analyses of harms in three scenarios: systematically assessed adverse events for hypothesis testing; signal detection in emerging common events; and descriptive analysis for less frequent events [414].
Formal analytical approaches to harms data exist [414]. If trial investigators decide to use descriptive approaches, this could involve reporting all adverse events in an appendix. Furthermore, planning a prospective meta-analysis can in some situations help achieve sufficient power by pooling the results from multiple trials [415].
Among protocols for randomised trials approved in 2016, 87% detailed the main analysis plan for the primary outcome [9, 10]. A systematic review found that analysis plans described in protocols for up to half of studies (mostly clinical trials and systematic reviews) did not match those reported in publications [64].
The protocol should describe in sufficient detail the key considerations of the planned statistical analyses of the primary and secondary outcomes, regardless of whether a separate statistical analysis plan exists. It is important to specify the main analysis (often referred to as the primary analysis) of the primary outcome (Item 16), including the analysis methods to be used for statistical comparisons; precisely which trial participants will be included (Item 27b); and how missing data will be handled (Item 27c). The protocol should also indicate the effect measure (e.g., absolute risk) and the statistical significance level that will be used, as well as the intended use of confidence intervals when presenting the results. If applicable, any plans to perform an adjustment for multiple testing should be explained, including the rationale and the method of adjustment.
The protocol should also describe whether an adjusted analysis is to be performed, the covariates included in the adjusted analysis (or any criteria to select the covariates), and how any continuous covariates will be handled (e.g., modelled assuming linearity or nonlinearity) [416]. The potential for missing values in any of the covariates being adjusted for should be anticipated and plans to handle any missing values should be described (Item 27c). If both unadjusted and adjusted analyses are planned, then it is important to state which will be the primary analysis.
Some trials are designed based on Bayesian methods [417, 418]. In this case, the choices of priors, computational choices, and any modelling used should be described.
The same considerations apply to detailing the analysis of secondary outcomes in the protocol.
Summary of key elements to address
-
Statistical methods for each analysis
-
Main analysis method for statistical comparison
-
Effect measure (e.g., absolute risk) with confidence intervals
-
Statistical significance level
-
For Bayesian analysis: choices of priors, computational choices, details of any modelling, and effect measure with credible intervals
-
-
For adjusted analyses (if applicable):
-
Rationale for adjusted analyses
-
List of covariates for adjustment
-
Statistical methods
-
If both adjusted and non-adjusted analyses are planned, which will be the main analysis
-
-
Methods to account for multiplicity, if applicable
-
Reference to the full statistical analysis plan, if a separate document exists