How to conduct an ANOVA test and interpret the result: One-way, Welch’s, Kruskal-Wallis, Post hoc

The session's objectives are to apply appropriate Jamovi procedures for analyzing group differences, interpret output (effect size, p-values), and effectively communicate findings. Various ANOVA tests are introduced, including one-way ANOVA, Welch F-test, Kruskal-Wallis, repeated measures ANOVA, and factorial ANOVA. The focus for this session is on one-way ANOVA, Welch F-test, and Kruskal-Wallis test. ANOVA (Analysis of Variance) analyzes variation between and within groups, serving as an extension of the t-test for comparing three or more groups. The type of ANOVA used depends on the independent variable's nature (between-subjects, within-subjects, or a mix).

One-way ANOVA, Welch F-test, and Kruskal-Wallis tests are used to compare a continuous dependent variable across three or more categorical independent variable groups, where each participant belongs to only one category (between-subjects design). An example illustrates this: a school district examining math test scores across traditional in-class, online, and blended learning methods. The independent variable is learning methods, and the dependent variable is math test scores. Another example, a clinical trial evaluating 'Joyzepam' (new antidepressant), 'Anxifree' (established), and 'Placebo' on mood improvement, is set up as a demonstration. The mood gain is the continuous dependent variable, and the type of drug is the categorical independent variable with three levels.

Before running ANOVA, the data is inspected. Mood gain is a continuous variable, and 'Drug' is a categorical variable with three distinct groups (Anxifree, Joyzepam, Placebo), each containing different participants. Descriptive statistics are generated in Jamovi (Exploration > Descriptives) to understand the data. By splitting the results by 'Drug', the mean mood gain for Anxifree is 0.72, Joyzepam is 1.48, and Placebo is 0.45. Joyzepam shows the highest mean, suggesting a promising result, but statistical significance needs to be tested.

The null hypothesis for one-way ANOVA states no difference in mood gain between the three drug types. The alternative hypothesis suggests a difference. A significance level (alpha) of 0.05 is set. Assumptions for one-way ANOVA include: 1) dependent variable is normally distributed, 2) variances between groups are roughly equal (homogeneity of variance), 3) dependent variable is continuous, and 4) scores are independent between groups. Assumptions 3 and 4 are met by design. Assumptions 1 and 2 are tested using Jamovi's ANOVA function (ANOVA > Assumption Checks), specifically the Shapiro-Wilk test for normality and Levene's test for homogeneity of variance.

Based on assumption checks: if normality is not satisfied, use Kruskal-Wallis. If normality is satisfied but homogeneity of variance is not, use Welch F-test. If both are satisfied, use one-way ANOVA. In the example, both normality (p=0.605) and homogeneity of variance (p=0.266) are satisfied (p > 0.05), so standard one-way ANOVA is used. In Jamovi, under ANOVA, select 'Omega Squared' for effect size and move the 'Drug' variable to 'Estimated Marginal Means', checking 'Marginal Means Plots', 'Marginal Means Table', and 'Equal Cell Weights'. The ANOVA output shows a p-value < 0.001 and an omega squared of 0.66 (large effect size), indicating a significant overall difference between groups. Since there's a significant difference, post-hoc comparison is needed to identify specific group differences, using Tukey HSD for equal variances. The post-hoc results show significant differences between Anxifree and Joyzepam, and Joyzepam and Placebo, but no significant difference between Anxifree and Placebo.

A statistical report includes four components: research question/hypothesis, data description (descriptive stats, assumption checks, chosen test), inferential test results (test value, degrees of freedom, p-value, effect size), and interpretation (hypothesis support, additional information). Formatting guidelines like italicizing statistical terms and specific decimal places should be followed. A sample report demonstrates how to present the findings, including a table of mean and standard deviation, F-statistics, p-value, and omega squared, followed by a narrative detailing the significant differences found through ANOVA and post-hoc Tukey HSD comparisons. Examples of how published articles present similar results are also shown, emphasizing flexibility in reporting as long as key information is included.

The Welch F-test is used when the normality assumption is met, but the homogeneity of variance assumption is violated. In Jamovi, navigate to 'ANOVA' > 'One-Way ANOVA', move the dependent and grouping variables, then under 'Variances', select 'Don't assume equal variances' (which enables Welch's test). Also, check 'Descriptive table' and 'Descriptive plots'. If the p-value is significant, post-hoc tests are performed. For Welch F-test, the Games-Howell post-hoc test is appropriate due to unequal variances. The output matrix compares groups, and significant differences are identified by p-values less than 0.05. The reporting for Welch F-test is similar to one-way ANOVA, emphasizing that Welch F-test was used and noting any specific findings from the Games-Howell post-hoc test.

The Kruskal-Wallis test is a non-parametric alternative used when the normality assumption is not satisfied. In Jamovi, go to 'ANOVA' > 'Nonparametric' > 'One-Way ANOVA (Kruskal-Wallis)'. Move the dependent and grouping variables, then check 'Effect size' and 'Post hoc tests (DSCF Pair-wise Comparison)'. The test output includes a chi-square (K) statistic, degrees of freedom, p-value, and effect size. If the p-value is significant, the DSCF (Dunn's) pairwise comparison post-hoc test reveals which specific group differences are significant. Reporting involves presenting median values (as it's non-parametric) along with chi-square, degrees of freedom, p-value, and effect size, while clearly stating that Kruskal-Wallis and DSCF tests were utilized.

A practice activity involves analyzing student performance (math, English, science grades) across different year levels (freshman, sophomore, junior, senior) using a provided dataset in Jamovi. Participants are tasked with determining which ANOVA test is appropriate for each subject based on normality and homogeneity of variance assumptions. For each subject, they identify if students' performance varies by year level and the results of post-hoc comparisons. The results for English involve Welch F-test, Math requires Kruskal-Wallis, and Science uses one-way ANOVA. English shows significant differences with the Welch F-test, particularly between junior and freshman, while Math and Science do not show significant differences across year levels.