Summary
Highlights
Regression analysis is a statistical method for modeling relationships between variables, allowing for inference or prediction. It can be used to measure the influence of one or more variables on another, or to predict a variable based on others. Examples include predicting a person's salary based on education, working hours, and age, or predicting hospital patient stay durations. The variable to be predicted is the dependent variable, while those used for prediction are independent variables.
There are three main types discussed: simple linear, multiple linear, and logistic regression. Simple linear regression uses one independent variable, while multiple linear regression uses several, both for metric dependent variables. Logistic regression is used for categorical dependent variables, particularly binary outcomes like 'yes/no' or 'success/failure'.
Independent variables can be nominal, ordinal, or metric. For categorical independent variables with more than two categories, dummy variables are created. Each dummy variable represents one category and is coded as 0 or 1, with one category serving as a reference. The number of dummy variables needed is typically one less than the number of categories.
Simple linear regression models the relationship between two variables: one dependent (e.g., house price) and one independent (e.g., house size). The goal is to find a straight line that minimizes the error between actual data points and the line. This line is defined by an intercept (a) and a slope (b), which can be calculated by hand or using statistical software. The p-value indicates the statistical significance of the relationship.
Key assumptions include a linear relationship between variables, independence of errors, homoscedasticity (equal variance of errors), and normally distributed errors. These assumptions can be checked using scatter plots, QQ plots, Durbin-Watson test, and residual plots. Violations can lead to unreliable results.
Multiple linear regression extends simple linear regression by using several independent variables to predict a single metric dependent variable. The regression equation includes coefficients for each independent variable, which are interpreted similarly to simple linear regression. The concept of Y-hat (predicted values) is introduced to distinguish from observed actual values.
The first four assumptions (linear relationship, independence of errors, homoscedasticity, normally distributed errors) are similar to simple linear regression. An additional fifth assumption is no multicollinearity, meaning independent variables should not be highly correlated with each other. Multicollinearity can be detected using tolerance and Variance Inflation Factor (VIF) values. Solutions include removing correlated variables or combining them.
The unstandardized coefficients indicate the change in the dependent variable for a one-unit increase in the independent variable. Standardized coefficients allow for direct comparison of the relative importance of independent variables. P-values determine the significance of each coefficient. The model summary provides the multiple correlation coefficient (R), R-squared (coefficient of determination), adjusted R-squared, and the standard error of the estimate, which assess the overall model fit and predictive accuracy.
Logistic regression is used when the dependent variable is binary (e.g., diseased/not diseased). Unlike linear regression, which can produce values outside the 0-1 range, logistic regression uses a logistic function to estimate probabilities between 0 and 1. Coefficients are determined using the maximum likelihood method, with statistical software handling the calculations. The output includes classification tables (showing correct and incorrect predictions), Chi-square test (for overall model significance), and R-squared values for model fit.
The model coefficients are presented alongside p-values and odds ratios. Odds are defined as the ratio of an event happening to it not happening. The odds ratio compares the odds of an event occurring in two different groups. For example, an odds ratio of 0.64 for medication indicates that the odds of being diseased are 0.64 times lower for those who took the medication compared to those who did not. For continuous variables, the odds ratio indicates the change in odds for a one-unit increase in the variable, and can be calculated by exponentiating each coefficient.