Sustainable LLMs by Reinforcement Learning with an Intrinsic MDP formulation

Artificial Intelligence (AI) is pivotal in addressing global challenges—from forecasting extreme weather events to personalizing healthcare. Large Language Models (LLMs) are especially noteworthy for their ability to enhance a wide range of applications, attracting significant interest for their transformative potential. Despite their utility, the operation of these LLMs involves significant computation and energy costs. It was estimated that training a large AI model could cost around $4.6 million. Its energy consumption has been compared to that of powering hundreds of homes over a similar period. A life cycle assessment for such models can emit more than 626,000 pounds of carbon dioxide equivalent—nearly five times the lifetime emissions of the average American car (and that includes manufacture of the car itself). This project aims to revolutionize the way costly AI models are trained and maintained under data distribution shift. This research will: 1. develop an approach to enable LLMs to self-tune to reduce computational costs effectively under continually arrived data. 2. allow the models to autonomously modify their structures in response to new data, meanwhile preventing the loss of previously learned information over the long term. These strategies aim to drastically reduce the costs associated with LLMs. Empirical evidence will be provided to validate whether the research objectives have been met. We aim to pioneer a new paradigm that equips LLMs with the ability to self-optimize their computation consumption and architectures, thus enhancing their adaptability and efficiency. This is achieved through an innovative reinforcement learning (RL) method, where AI learns optimal behaviors through trial and error while considering long-term outcomes. By utilizing this approach, we can significantly streamline the process. Our research will develop an intrinsic Markov Decision Process (MDP) framework, enabling the AI agent to autonomously interact internally and evaluate and adjust its internal structure to reduce computation costs, improve adaptive capabilities, and avoid catastrophic forgetting. The rationale for adopting the RL methodology is intuitive. As data environments continue to evolve, LLMs must be updated regularly to remain effective. The need for frequent updates, coupled with the goal of managing long-term costs effectively, makes RL an ideal choice. RL is specifically designed to tackle problems that require a series of decisions, meanwhile considering the future consequences of decisions made today, helping to control rising costs over time. While existing research has employed RL for fine-tuning LLMs, it has predominantly focused on enhancing specific performance metrics such as accuracy or relevance. Our project uniquely targets the sustainable computational efficiency of LLMs. This approach stands out by fostering a self-regulating, continuously learning AI system that not only meets current needs but also anticipates and adapts to the increasing future challenges of rising computation costs. Moreover, this project introduces a novel intrinsic MDP framework and diverges from traditional approaches, which typically rely on a standard MDP where an agent interacts with an external environment. If successful, this research will reduce the computation and financial costs and environmental impacts of AI applications, enhancing AI’s effectiveness and accessibility across sectors like healthcare and environment. The project facilitates broader access to advanced AI tools, addressing high costs and energy demands that currently limit smaller entities. This supports a more equitable distribution of technology, potentially reducing the computational load of LLMs to allow for greater investments in healthcare and education, fostering balanced societal development.