The data.table R package, used by nearly 2000 other software tools, is being overhauled to keep it working as the core R programming language changes its internal wiring. This matters because data.table is the engine behind much of the UK’s data-intensive research—from epidemiology tracking infection rates to financial analysts processing market data. The package relies on specific entry points into R’s C code that the R Core Team is now removing. Without this project, those tools would slow down or break entirely. The team is also adding new rolling statistics functions—minimum, maximum, median, product—critical for analysing time series data in finance, public health, and climate science. If successful, researchers and analysts across academia and industry will continue to manipulate large datasets quickly without their code breaking after routine R updates. The project also contributes fixes back to base R itself, strengthening the entire R ecosystem. This is infrastructure maintenance, not a scientific breakthrough, but it keeps the machinery of data science running for thousands of users who depend on it daily.
View original technical description
The data.table R package relies on a complex suite of C functionality that interacts with the R API to create fast and efficient tools for managing and reshaping data. It provides state-of-the-art performance for data manipulation, making it a cornerstone of high-performance R programming, particularly in data science and statistical computing. It currently supports nearly 2000 other packages and tools within the R ecosystem. This project aims to enhance and future-proof data.table by addressing three major areas of improvement and expansion: Adapting to C API Changes in Base R – data.table is an extension of base R, the software distribution that provides the R programming language and a set of R packages providing core functionality, which is maintained by the R Core Team (R Core). Base R continues to evolve, with R Core introducing modifications to its C API that impact extension packages relying on low-level functionality. Ensuring data.table remains fast and reliable requires proactively adapting to these changes, preventing performance regressions, and maintaining seamless integration with future R releases. In particular, data.table can no longer depend on several entry points into base R that have previously provided valuable tools for efficient handling of data, particularly large data. Bringing data.table into compliance with updates from R Core requires time and attention to each entry point that is now unavailable, including core parts of the growable vector idiom and fast string matching (which currently uses base R’s internal string cache). Rolling Statistics – Efficient computation of rolling (moving) statistics is essential for time series analysis, finance, epidemiology, and many other fields. This project will introduce and improve optimized rolling statistics functions directly within data.table. This includes new rolling functions: minimum, maximum, product, median, user-defined function (rewrite for efficiency, multi threading, support of adaptive index, support of rolling over list of columns rather than columns one by one); as well as new features like rolling over uneven time series (i.e. days/hours rather than records). General Maintenance and Issue Resolution – As base R updates, new issues arise that affect data.table’s performance and behaviour. This project will focus on addressing reported bugs, improving documentation, optimizing existing functionality, and ensuring compatibility with evolving R standards, safeguarding data.table as a robust and dependable tool for the R community. As a core principle of data.table, it will avoid breaking changes where possible. An important part of this process is engaging with the CRAN Team, who maintain the Comprehensive R Archive Network where data.table is published, to ensure the package remains available on that central platform as well as continuing our engagement with R Core to continue our collaborative relationship with them. The sustainability of data.table relies on the sustainability of base R, which depends on contributions from the wider community beyond R Core. Therefore this project will also contribute to maintenance of base R, participating in the R Contribution Working Group and their initiatives, such as in-person and online R Dev Days, where participants collaborate on bug fixes and improvements to base R. By tackling these critical areas, this project will reinforce data.table’s position as a leading data manipulation package in R, ensuring it remains performant, reliable, and well-maintained for users across academia, industry, and beyond.
Plain English summaries and category classifications on this site are generated by AI and may not perfectly reflect the original research.
Is something wrong? Let us know