History of statistical computing

The speaker begins by showing a 'millionaire calculating machine' from over 50 years ago, highlighting that statisticians used to have expensive, state-of-the-art tools. Despite these early limitations, sophisticated analyses were conducted, such as two-to-the-end designs and probit lines, largely through manual calculations with aids like electric calculators. Fisher's quote, 'Most of my statistics has been learned on the machine,' emphasizes the importance of hands-on data interaction, a principle that should not be lost in the computer age. However, these methods were time-consuming, limiting innovation for many, as reflected in a 1952 Rothamsted report noting significant computing effort.

The computer age for agricultural research and statistics began with the Rothamsted 401 in 1954, considered the first computer primarily associated with this field. This machine was a massive, single-user system with limited data space, operating on paper tape for input and an electric typewriter for output. Programming involved intricate machine code, and 'optimal programming' was developed to speed up disk rotations. Despite its limitations, it allowed statisticians to begin writing sophisticated programs, moving from replicating manual calculations to creating general programs like Yates' algorithm and programs for multi-way tables and multivariate analysis.

By 1961, Rothamsted acquired a second 401 computer, leading to a massive increase in data analysis, but still resulting in machines running 'flat out'. The next significant leap in 1963 was the move to the Ferranti Orion, which supported high-level languages like extended Mercury Autocode and could run multiple programs simultaneously. This facilitated the development of general programs and algorithms. A major advancement occurred when John Nelder joined Rothamsted, leading to the development of Genstat, initially on an IBM 360, and later on an ICL 470. Early Genstat versions were batch-processed, requiring careful job stacking; however, the syntax was eventually modernized for interactive use, like in Genstat 5, incorporating lessons from interactive programs like GLIM. The development of general algorithms such as General Balance, Generalized Linear Models, and Residual Maximum Likelihood became key in providing research environments within statistical packages like Genstat.

A pivotal meeting in 1966, organized by John Nelder and Brian Cooper, led to the formation of the working party on statistical computing, which oversaw the applied statistics algorithms and the development of GLIM. Key themes from this era included concerns about the misuse of multiple regression and the potential for interactive computing. The speaker raises questions about the future of statistics, particularly concerning the increasing number of non-statisticians performing data analysis. There's a debate on whether statistical software should be free or commercial, with Genstat offering both. The future computing environment is also discussed: cloud computing versus powerful, portable personal devices. The speaker emphasizes the challenge for statisticians to remain relevant and proactive, particularly as the demand for their expertise increases in a data-rich world, urging the profession to engage with a wider audience.

The speaker shares a personal perspective from 30 years in the private sector, noting a low representation of private sector attendees at statistical conferences. He outlines three broad areas of computer use: manufacturing/operational databases, expert statisticians, and the growing number of people in business analyzing data without formal statistical training. The dominance of Microsoft products like Excel and PowerPoint in business poses a challenge, as these are not ideal for sophisticated statistical analysis. The nature of 'opportunistic' and large datasets in business often means standard statistical techniques are inadequate, as they were not designed for such scale or characteristics (e.g., highly skewed distributions). Errors in large datasets are inevitable, and automatic correction can obscure interesting features. The speaker highlights patterns in credit card transaction data to illustrate how real-world behavior and pricing policies create complex data structures that advanced statistical models must account for.

The core drivers in business are profit, not publication or developing new methods, which creates a disconnect between academia and industry. Models based on historical data can become invalid quickly, as seen in the credit crunch. Statisticians need to develop models that can evolve in real-time and integrate disparate data sources. While skeptical of expert systems, the speaker acknowledges the need for statisticians to find ways to support the vast number of people performing data analysis without formal training. He stresses the importance of effective communication and avoiding a 'scathing' attitude towards non-statisticians. Drawing on John Tukey's emphasis on exploratory data analysis, the speaker advocates for helping people analyze data better, particularly given the widespread use of tools like Excel and PowerPoint, which are ill-suited for this. He champions the idea that every business decision should be data-driven and that statisticians must actively engage with other disciplines to remain central to this process.