European Large Open Multi-Modal Foundation Models For Robust Generalization On Arbitrary Data Streams

At the heart of ELLIOT is the development of open Multimodal Generalist Foundation Models (MGFMs): AI systems designed to learn general knowledge and patterns from massive amounts of data of various types — from videos, images, and text to sensor signals, industrial time series, and satellite feeds — and efficiently transfer the generic knowledge learned in generalist manner to a wide variety of downstream tasks. Unlike current foundation models that face significant challenges in terms of generalisation capabilities and support for multimodal data, ELLIOT’s models will be capable of robust generalisation across conditions not seen during the training, coping well with dynamic, noisy, and temporally-evolving multimodal data streams. Real and synthetic data, to mitigate data scarcity, will be leveraged for training MGFMs and for further adapting them for specific downstream tasks in domains like media, earth observation, robot perception, mobility, computer engineering and workflow automation. Real data used for training will include data directly provided by members of the consortium as well as data from relevant European Data Spaces, while complementary synthetic data will be generated by exploiting existing generative AI capabilities as well as new ones developed in the project. European HPC infrastructure is directly included in the consortium to ensure the availability of the necessary computing resources.