Smooth models for future data science

Regression models are used to describe the relationship between a response variable and variables that may predict it. In particular they allow the data analyst to specify the structure of the relationship, while estimating its exact form statistically. This interpretable structure is critical in many scientific applications where the models are used to test and refine understanding of mechanisms, and in industrial applications, such as energy forecasting, where it is critical to understand why a model makes the predictions it does. Originally the data analyst had to specify the structure of the relationship between the modelled variables with a rather restrictive degree of detail, making regression unwieldy or wholly unsuitable in many situations of moderate data complexity. In recent decades this restriction has been largely lifted, by the extensive development of methods and associated software for regression models to be specified much more flexibly. Models can be specified in the form 'the response variable depends on sums of smooth functions of one or more predictor variables'. This increased flexibility also increased the challenge of estimating models from data in a manner that is sufficiently computationally efficient, and sufficiently numerically stable, for routine applied use. It also introduced methodological challenges around the best way to represent the smooth functions from which models are built, and how to adapt standard statistical model selection and interval estimation techniques to the more flexible setting. The key issue is that once models are built in terms of smooth functions of variables, it is necessary to estimate 'how smooth' those function are. Doing this in a data driven way then has non-negligible consequences for other aspects of statistical inference. The applicant's work has played a substantial part in the development of a computationally general and feasible framework for this class of models. In particular the mgcv software package is a standard part of the R statistical computing language, and is widely used in science, medicine, government and industry as evidenced by over 50000 citations of the work and two REF case studies based on it. However further impact of this key UK based statistical infrastructure now requires three new developments. Firstly there is a need for better scalability across the full range of smooth regression models with substantial applied use. Currently excellent scalability has been achieved for univariate mean regression, but not for more general models in which response distribution parameters other than the mean are also important. Secondly, Python has become the language of choice for many modern data science applications in industry, for which smooth regression models would be key infrastructure. However the existing state of the art smooth regression methods are written in R and C (SAS have also implemented the methods), while the python support is very limited. A state of the art native python implementation is clearly required, leveraging significant methodological streamlining relative to the existing software and maximizing future impact. Thirdly recent developments in smoothness selection methods and numerical linear algebra offer a serious possibility of developing massively more efficient computational methods able to target dataset and model sizes far exceeding what is practical with current methods. All three of these developments would have immediate and long lasting industrial and scientific impact.