Statistics - A Full University Course on Data Science Basics

This section introduces statistics as the science of collecting, organizing, analyzing, and interpreting numerical data. It emphasizes that statistics aids in decision-making, particularly in fields like healthcare and public health, using examples such as the CDC's flu vaccine development. The distinction between 'individuals' (people or objects in a study) and 'variables' (characteristics measured) is clarified.

This part defines population as the entire group of interest and a sample as a portion of that population. It explains that a parameter describes a population (e.g., mean age of all Medicare enrollees), while a statistic describes a sample (e.g., mean age from a Medicare beneficiary survey). The discussion highlights the use of inferential statistics to draw conclusions about a population from a sample.

Variables are categorized into quantitative (numerical measurements, like temperature or blood pressure) and qualitative (characteristics, like type of insurance or country of origin). Quantitative variables are further classified as interval (meaningful differences, no true zero point, e.g., year of diagnosis) or ratio (meaningful differences, true zero point, e.g., platelet count). Qualitative variables are divided into nominal (unordered categories, e.g., sex or race) and ordinal (ordered categories, e.g., cancer stage).

This section defines a sampling frame as the list from which a sample is selected and discusses sampling errors (natural differences between sample and population) versus non-sampling errors (mistakes in design or data collection). Various sampling methods are explained, including simple random sampling (equal chance for every sample), stratified sampling (dividing into subgroups and sampling from each), systematic sampling (selecting every Kth individual), cluster sampling (sampling entire geographic clusters), convenience sampling (using readily available data), and multistage sampling (combining methods).

The process of conducting a statistical study begins with stating a hypothesis, identifying individuals and variables, and choosing a sampling method. Ethical concerns, data collection, statistical analysis, and reporting findings are crucial steps. The discussion differentiates between experimental studies (researcher assigns treatment) and observational studies (researcher observes without intervention). The importance of replication in scientific research is also highlighted.

This segment focuses on minimizing bias in surveys. It addresses non-response bias (when many refuse to participate), where respondents might have skewed opinions. It also warns about response bias, where participants might lie (intentionally or inadvertently, like recall bias) or be influenced by question wording or interviewer characteristics. Vague wording and the choice of measurement scales are also discussed as potential sources of bias.

Randomization is crucial in experimental design to prevent bias by evenly distributing lurking variables among treatment groups. The process involves recruiting participants, measuring baseline confounders and outcomes, and then randomly assigning individuals to groups (e.g., treatment vs. placebo). The concept of blinding—where participants and/or study staff are unaware of treatment assignments—is explained to further reduce bias, accounting for the placebo effect.

Frequency histograms are a specific type of bar chart used to visualize the distribution of quantitative data. The process of creating a histogram from a frequency table is detailed. Different types of distributions are introduced: normal (bell-shaped and symmetrical), uniform (bars of equal height), skewed left (tail on the left), skewed right (tail on the right), and bimodal (two peaks). The section also covers identifying outliers in distributions and explaining cumulative frequency through an ogive plot.

This part explores other types of graphs. Time series graphs display data over time, useful for observing trends (e.g., influenza rates over years). Bar graphs can represent both quantitative and qualitative data, allowing comparisons between categories; the impact of scale on visual interpretation is demonstrated. Pie charts are used for mutually exclusive categories, showing proportions of a whole (e.g., types of health insurance). Guidelines for effective graph creation, such as clear labeling and appropriate scales, are provided.

This section explains how to organize quantitative data using frequency tables and stem-and-leaf displays. A frequency table categorizes data into classes (intervals) and shows the count (frequency) and proportion (relative frequency) of data points in each. The concept of class width and limits is clarified. Stem-and-leaf displays offer an alternative, visually retaining individual data points while organizing them to reveal the data's distribution. Both methods are foundational for understanding data patterns.

Measures of central tendency describe the typical or central value of a dataset. The mode is the most frequently occurring number, while the median is the middle value when data is ordered. The mean is the average of all values. The lecture explains how to calculate each, noting that the median is more resistant to outliers than the mean. It also introduces specialized means like the trimmed mean (to reduce outlier impact) and weighted average (where some values contribute more to the mean). The relationship between these measures and data distribution (normal, skewed) is also explored.

This section focuses on quantifying data spread. It introduces the range (difference between maximum and minimum values), variance, and standard deviation (which represents the average distance of data points from the mean). Formulas for calculating variance and standard deviation for both samples and populations are provided. The coefficient of variation (CV) is presented as a dimensionless measure useful for comparing variability between different datasets. Finally, Chebyshev's Theorem, which provides a minimum percentage of data within a certain number of standard deviations from the mean for any distribution, is discussed.

The concept of percentiles is introduced, indicating the percentage of data points falling below a certain value (e.g., 77th percentile means better than 77% of data). Quartiles (Q1, Q2, Q3) divide data into four equal parts, corresponding to the 25th, 50th (median), and 75th percentiles. The interquartile range (IQR) is the difference between Q3 and Q1. The lecture then demonstrates how to construct a box and whisker plot using these quartiles and the minimum/maximum values, explaining how this plot visually represents data distribution and identifies potential outliers.

This segment focuses on the relationship between two quantitative variables (X and Y). A scattergram (or scatterplot) visually displays these (X, Y) pairs, helping assess if a linear relationship exists. X is the explanatory/independent variable, and Y is the response/dependent variable. Linear correlation describes the strength and direction (positive, negative, or no correlation) of this relationship. The correlation coefficient (R) quantifies this, ranging from -1.0 (perfect negative) to +1.0 (perfect positive). The crucial distinction that 'correlation does not imply causation' is heavily emphasized, highlighting the role of lurking variables.

Linear regression involves fitting a 'least squares line' to the data in a scatterplot, which minimizes the sum of squared vertical distances from data points to the line. The lecture details the components of the linear equation (Y_hat = a + bX), where 'b' is the slope (marginal change in Y per unit change in X) and 'a' is the Y-intercept. It explains how to calculate residuals (the difference between observed Y and predicted Y_hat). The coefficient of determination (R-squared) is introduced (R times R), indicating the proportion of variation in Y explained by X, serving as a measure of the model's predictive power.

This section revisits the normal distribution, highlighting its key properties: bell-shaped, symmetrical, and asymptotically approaching the horizontal axis. The Empirical Rule (68-95-99.7 rule) is introduced, stating the approximate percentages of data that fall within one, two, and three standard deviations from the mean in a normal distribution. This rule is more precise than Chebyshev's Theorem but only applies to normally distributed data. Practical application of the Empirical Rule to answer questions about data ranges and probabilities within a normal distribution is demonstrated.

Z-scores standardize data points from any normal distribution to a standard normal distribution (mean=0, standard deviation=1). The lecture explains how to convert an X value to a Z-score and use a Z-table to find the probability (or proportion of area under the curve) corresponding to a given Z-score. It covers finding probabilities for areas to the left or right of a Z-score, as well as between two Z-scores. Conversely, it demonstrates how to use the Z-table to find a Z-score corresponding to a given probability and then convert that Z-score back to an X value. Tips for correctly interpreting and applying Z-scores and probabilities are provided.

This final part distinguishes between frequency distribution (of an individual variable) and sampling distribution (of a sample statistic, like the mean, from all possible samples from a population). It introduces the Central Limit Theorem (CLT), a cornerstone of statistical inference: for a sufficiently large sample size (typically N>30), the sampling distribution of the sample mean (X_bar) will be approximately normal, regardless of the population's distribution. The CLT is crucial because it allows us to calculate probabilities related to sample means and estimate population parameters using sample statistics, leveraging the concept of standard error.