How Ai2 Democratizes LLM Research with OLMo (Open Language Model)
How Ai2 Democratizes LLM Research with OLMo (Open Language Model): The Allen Institute for Artificial Intelligence (Ai2), a non-profit organization founded in 2014 by Microsoft co-founder Paul Allen, has taken a significant step in actively contributing to the development of artificial intelligence for the common good. This move comes with the release of the OLMo LLM (Large Language Model). While some open-source models include codes and weights, Ai2 makes OLMo truly open by providing not only codes and weights but also training code, training data, and associated toolkits, all under the Apache 2.0 license.
With the unveiling of the cutting-edge OLMo model and its accompanying framework, Ai2 aims to foster innovation and collaboration in language models while raising awareness of the ethical and societal issues they pose.
Hanna Hajishirzi, OLMo Project Lead, Senior NLP Research Director at AI2, and professor at the Allen School of the University of Washington, explains:
Many language models today are published with limited transparency. Without access to training data, researchers cannot scientifically understand how a model operates. It's akin to discovering drugs without clinical trials or studying the solar system without a telescope. With our new framework, researchers will finally be able to delve into the science of LLMs, crucial for building the next generation of safe and trustworthy AI.
OLMo is the result of collaboration with the Kempner Institute for the Study of Natural and Artificial Intelligence at Harvard University and partners such as AMD, CSC, the Paul G. Allen School of Computer Science & Engineering at the University of Washington, and Databricks.
The OLMo 7B and 1B models were developed on the LUMI supercomputer at the CSC (Center for Science Information Technology), powered by AMD EPYC™ processors and AMD Instinct™ accelerators, trained using the MosaicML platform from Databricks.
The framework includes a suite of fully open AI development tools, including:
Complete pre-training data: The model is built on AI2's Dolma dataset, consisting of an open corpus of three trillion tokens for language model pre-training, including the code that produces the training data. The OLMo framework includes complete model weights for four 7B-scale model variants, each trained on at least 2 trillion tokens. Inference code, training metrics, and training logs are all provided. Evaluation: Ai2 has released the evaluation suite used in development, with over 500 checkpoints per model, every 1000 steps in the training process, and evaluation code under the auspices of the Catwalk project.
Eric Horvitz, Microsoft's Chief Scientist and founding member of AI2's Scientific Advisory Council, remarks:
I am excited about putting OLMo into the hands of AI researchers. This new offering follows Allen AI's tradition of providing valuable open models, tools, and data that have driven numerous advancements in the global AI community.
With OLMo, AI researchers and developers will experience:
More precision: With comprehensive knowledge of the training data underlying the model, researchers can work more efficiently, eliminating the need to rely on qualitative assumptions about how the model functions and allowing for scientific testing. Reduced carbon footprint: Currently, one training session is equivalent to the emissions of nine American households for a year, according to the EPA. By offering full access to the training and evaluation ecosystem, Ai2 significantly reduces redundancies in the development process, crucial for carbon emission reduction in AI. Sustainable results: Keeping models and datasets open and not behind APIs allows researchers to learn from and build upon previous models and work. Ai2 plans to add different model sizes, modalities, datasets, and capabilities to the OLMo family in the near future.
Noah Smith, OLMo Project Leader, Senior NLP Research Director at AI2, and professor at the Allen School of the University of Washington, concludes:
With OLMo, 'open' truly means 'open,' and all members of the AI research community will have access to every aspect of model creation, including training code, evaluation methods, data, and more... AI was once an open field centered around an active research community, but as models have evolved, become more expensive, and started to turn into commercial products, work on AI began to unfold behind closed doors. With OLMo, we hope to counteract this trend and empower the research community to come together to better understand and engage with language models scientifically, leading to more responsible AI technology that benefits everyone.