Summary
Highlights
Data science is introduced as a creative discipline, not merely a technical one, focusing on using tools from coding, statistics, and math to work creatively with data. The ultimate goal is to gain insight from diverse data, even if it doesn't fit standard approaches, because everything carries meaning.
Data science is defined as a blend of coding, math, and statistics applied to diverse data for inclusive analysis. The field is highlighted as highly sought after, with articles like 'Data Scientist: the Sexiest Job of the 21st Century' from Harvard Business Review. This high demand stems from the rare qualities of data scientists—finding order and value in unstructured data—and its ability to provide competitive advantages and excellent pay.
The Data Science Venn Diagram by Drew Conway illustrates data science as the intersection of 'hacking' (coding), 'stats' (math/statistics), and 'domain expertise'. Coding skills are crucial for data gathering and preparation, while math helps select appropriate procedures and diagnose problems. Domain expertise ensures effective implementation of results. The intersections also reveal related fields like machine learning, traditional research, and the 'danger zone' (coding + domain expertise without statistics).
The data science pathway is broken down into four main stages: planning, data preparation, modeling, and follow-up. Planning involves defining goals, organizing resources, coordinating people, and scheduling. Data preparation focuses on gathering, cleaning, exploring, and refining data. Modeling includes creating, validating, evaluating, and refining statistical models. Follow-up entails presenting, deploying, revisiting, and archiving the model and its assets.
Data science is a collaborative field involving various roles: engineers (hardware/software foundation), big data specialists (machine learning algorithms for large datasets), researchers (domain-specific insights), analysts (day-to-day business tasks), business people (framing questions and managing projects), and entrepreneurs (creative problem-solvers). The ideal 'full stack unicorn' who can do everything at an expert level is rare, emphasizing the need for diverse teams.
Data science and Big Data are often confused but are distinct. Big Data is characterized by volume, velocity, and variety. While there's overlap (Big Data Science), one can exist without the other. Examples include machine learning and word counts without extensive data science skills, or genetics and streaming sensor data requiring data science without necessarily being 'Big Data'.
Data science is not synonymous with coding. Coding focuses on giving task instructions to machines. While sharing some tools like Python, R, and SQL, a key differentiator for data science is the crucial role of statistical ability for making valid inferences from variable and uncertain data. Tools like Excel, often overlooked in pure programming, are also vital in data science.
Although statistics is a core component of data science (as per the Venn diagram), the two fields are ecologically distinct. Data scientists often come from diverse backgrounds beyond formal statistics training and work in different contexts, particularly commercial settings. Machine learning and Big Data, central to data science, are not universally shared with traditional statistics.
Business intelligence (BI) focuses on applying data to real-life business operations, often using existing apps and simple statistics (counts, percentages). Data science supports BI by setting up frameworks and extending analysis to tackle complex questions with difficult data. Data science practitioners can, in turn, learn valuable design and usability principles from good BI applications like dashboards.
Ethical considerations are paramount in data science, rooted in the principle of 'do no harm'. Key issues include privacy (confidentiality of personal information), anonymity (difficulty in de-identifying data), copyright (respecting data usage rights), and data security (protecting valuable datasets from theft). Additionally, potential bias in algorithms and overconfidence in analyses require careful attention and humility.
This section introduces the technical methods used in data science, cautioning against technical preoccupation and emphasizing insight as the ultimate goal. The main categories include data sourcing (getting raw materials), coding (manipulating data), math (foundational principles), statistics (summarizing and analyzing data), and machine learning (finding patterns and making predictions).
Data sourcing methods involve obtaining the raw materials for analysis. Options include using existing data (in-house, open, or third-party), utilizing data APIs (Application Programming Interfaces) for direct data communication, scraping web data without APIs, or making new data through interviews, surveys, or experiments. The importance of data quality, adhering to the 'Garbage In, Garbage Out' principle, and understanding metrics is stressed.
Coding in data science involves using various technologies to manipulate data. Categories include specialized applications (spreadsheets, Tableau, SPSS, JASP), understanding web data formats (HTML, XML, JSON), and programming languages (R, Python, SQL, C/C++/Java, Bash, Regex). The importance of choosing tools wisely to match project goals and the Pareto Principle (80/20 rule) are highlighted.
Mathematics forms the foundation of data science methods. Essential mathematical concepts include elementary algebra (for manipulating equations), linear/matrix algebra (for efficient data handling by computers), and systems of linear equations (for solving multiple unknowns). Calculus is vital for understanding rates of change, optimizing functions, and forming the basis of many statistical procedures. Concepts like 'Big O' notation help understand the speed and complexity of operations, and probability theory, including Bayes' theorem, is fundamental for understanding likelihood and making informed decisions under uncertainty.
Statistics in data science helps to find patterns in data that may not be immediately obvious. It's used for summarizing and generalizing data, especially when dealing with incomplete information. Key areas include descriptive statistics (mode, median, mean, range, variance, standard deviation) for understanding the center, spread, and shape of data, and inferential statistics (hypothesis testing, estimation) for making conclusions about larger populations from sample data.
Machine learning (ML) is a core component of data science, residing at the intersection of coding and statistics. Its goal is to work in data space to reduce dimensionality, find clusters (e.g., k-means), identify anomalies, categorize cases (e.g., logistic regression, decision trees, neural nets), and make predictions (e.g., linear regression, ensemble models). ML is a tool to achieve useful insight from data, not an end in itself.
Effective communication is essential in data science. Interpretability involves creating a data-driven story, recognizing that 'analysis times story equals value'. Analysts must overcome egocentrism and false consensus to explain findings clearly to clients, providing specific, actionable next steps justified by data. Understanding the difference between correlation and causation is critical, relying on experimental studies, quasi-experiments, and domain expertise. Social understanding of client missions, identity, and context is also vital for successful implementation.
Presentation graphics are a powerful tool for communicating data science results. They differ from exploratory graphics, prioritizing clarity and narrative flow for clients. Effective graphics are simple, avoid distractions (like excessive color or 3D), and tell a story, as exemplified by a case study on Berkeley admissions and Simpson's Paradox. Reproducible research is crucial for data science projects, enabling revision, collaboration, and accountability. This includes showing all work, archiving data and code, explaining processes, and future-proofing analyses using tools like Jupyter Notebooks and RMarkdown.
The course concludes with practical next steps for data science enthusiasts, such as learning R or Python, practicing data visualization, brushing up on statistics and math, and exploring machine learning. It encourages engagement with the data science community through conferences and competitions (Kaggle), and highlights opportunities for service (DataKind). The overarching message is the democratic nature of data science: it's accessible and vital for everyone to learn to work with data intelligently and sensitively, as 'data science needs you'.