ch2 in hands-on machine learning (project code)—part 1

The project aims to predict housing prices in California for a real estate company. The model will take specific region characteristics and predict the average house price, classifying it as a regression task due to the continuous nature of price prediction. Key features include location (latitude and longitude), proximity to the ocean, population density, median income, number of rooms, and bathrooms.

The problem is categorized as supervised learning because the model learns from historical house prices. It's a regression task as it predicts continuous values. It can be multi-variate regression if predicting multiple continuous values, but in this case, it's uni-variate as it predicts only one value (house price). The learning is batch learning, meaning data is static and the model learns once, rather than online learning.

Several data problems are identified: missing data in columns, categorical data needing conversion to numerical, and data truncation where house prices above $500,000 are capped, leading to model bias. Other issues include overfitting and underfitting. The primary evaluation metric for the model's performance will be the Root Mean Squared Error (RMSE).

Data engineering involves creating a pipeline to clean and transform data for model training. Feature engineering is introduced as a method to create new, more descriptive features (e.g., rooms per person) from existing ones to improve model accuracy. The importance of preparing messy real-world data is highlighted.

The video starts with importing necessary libraries like `pandas` for data manipulation, `matplotlib` for plotting, and `numpy`. It covers setting up the Python version and defining a folder for saving plots. The `load_housing_data` function is explained for loading a CSV file, and `housing.head()` is used to display the first few rows of the dataset to understand its structure and features.

Using `housing.info()`, metadata about the dataset, such as column names, non-null counts, and data types, is displayed. This reveals missing data in 'total_bedrooms' and the 'ocean_proximity' column being an object (string) and needing conversion. `value_counts()` is used to understand the distribution of categorical values, particularly for 'ocean_proximity', highlighting categories with very few entries like 'ISLAND'.

The `housing.describe()` method provides summary statistics (count, mean, min, max, quartiles) for numerical columns. Histograms are then generated for all numerical features using `housing.hist()`, allowing for visual inspection of data distributions and potential issues like outliers or skewed data.

The process of splitting the data into training and testing sets is crucial. The initial method involves a custom function that randomly splits the data based on a defined ratio (e.g., 20% for testing). The importance of using a `random_state` for reproducibility of splits is emphasized, though it cannot preserve the test set when new data is added.

To address the issue of test set instability when new data is added, a more robust splitting method using hashing is introduced. This method uses a unique ID (based on combined latitude and longitude) for each data point and applies a hashing function (MD5) to create a fingerprint. A portion of these fingerprints determines the test set, ensuring that the same data points always end up in the test set, even with new additions.

Addressing the problem of an unrepresentative random split, stratified sampling is presented. This technique ensures that the proportions of important features (like median income categories) are maintained across both the training and test sets. The `stratified_split` function from `sklearn.model_selection` is used for this purpose, leading to a more consistent and reliable evaluation of the model.

The video moves to visualizing the geographical distribution of houses. A scatter plot of `longitude` and `latitude` is created. Initially, the plot appears as dense clusters, making it difficult to discern patterns. The alpha parameter is adjusted to introduce transparency, allowing denser areas to appear darker and revealing points that were previously hidden, providing a clearer view of population density.

Further refinement of the geographical plot involves incorporating population (size of points) and median income (color gradient from blue to red for low to high income) into the scatter plot. This creates a more informative visualization, showing areas with high population and high house prices. The process includes downloading a map of California and overlaying the data points for better context.

The video explains how to calculate the correlation coefficient between different features using `housing.corr()`. It focuses on identifying features highly correlated with the 'median_house_value'. The 'median_income' is found to have a strong positive correlation, while other features show weaker or negative correlations. This helps in understanding which features are most impactful on house prices.

A scatter matrix is generated to visualize relationships between key features (median_house_value, median_income, total_rooms, housing_median_age). This reveals a clear upward trend between median income and house value. However, it also highlights three significant issues: a price cap at $500,000, and several horizontal lines at various price points, indicating data inaccuracies or truncation.

New, more informative features are engineered from existing ones. Examples include: 'rooms_per_household' (total rooms divided by households), 'bedrooms_per_room' (total bedrooms divided by total rooms), and 'population_per_household'. These new features aim to capture more nuanced information, such as the spaciousness of a house or population density, which can better predict house prices.

The correlation matrix is recalculated to include the newly engineered features. 'bedrooms_per_room' shows a negative correlation, indicating that more bedrooms per room might lead to lower prices (less spacious). 'rooms_per_household' has a positive correlation. The video then re-evaluates the `housing.describe()` output to identify outliers and inconsistencies in the data, particularly in 'total_bedrooms' (missing values) and identifying features with vastly different scales (e.g., median income vs. median house value), necessitating feature scaling.