What is Falcon 180B, and why is it generating so much hype in the AI community?
In the evolving world of artificial intelligence, large language models and generative AI tools are paving the way for endless innovation. The demand for versatile and powerful language models is increasing at an incredible rate as companies strive to embed intelligence into everything from their communication and collaboration tools to contact center processes.
We’ve already seen LLM technology making waves in the workplace with new innovations like Zoom AI Companion, Microsoft Copilot, and Google Bard. Now, a new open LLM solution from the Technology Innovation Institute (TII) is poised to disrupt the industry yet again.
Falcon 180B, the advanced iteration of the TII’s flagship LLM, was introduced on September 6th, 2023, and it’s already breaking performance records. Here’s everything you need to know about the solution and what it can do for businesses.
What is Falcon 180B? The Basics
Falcon 180B is an open-access large language model that builds on the previous releases in the “Falcon” family. It’s a scaled-up version of the Falcon 40B model, an AI solution that ascended to the top of the Hugging Face LLM Leaderboard in May 2023.
Falcon 40B was one of the first open-source LLM solutions designed for researchers and commercial users, and 180B takes the functionality of that model to the next level.
Large Language models are currently forming the backbone of numerous AI-driven applications, from virtual assistants and chatbots to machine translation and sentiment analysis tools. They’re also a core component of many collaborative apps companies use today, such as Google Duet AI.
Unfortunately, many developers still struggle to build models that can excel in various language tasks. Researchers and innovators often encounter model size, versatility, and training data limitations. As a result, the LLM landscape is somewhat fragmented, with very few one-size-fits-all solutions.
Falcon 180B aims to deliver a quantum leap in language model generation. It boasts exceptional performance, thanks to 180 billion parameters, and distinguishes itself from the competition with greater accessibility and versatility. Unlike closed-source models, like GPT-4, Falcon 180B is specifically designed for research and commercial use.
How Does Falcon 180B Work?
As mentioned above, the Falcon 180B is an upgraded version of TII's previous Falcon 40B model. It’s an auto-regressive language model that uses an optimized transformer architecture. According to the TII team, the solution was trained on 3.5 trillion data tokens, including web data from RefinedWeb and Amazon SageMaker.
The LLM features a custom distributed training codebase (Gigatron) that leverages 3D parallelism with ZeRO and custom Trion kernels. The technology took a lot of work to develop, using up to 4096 GPUs simultaneously for 7 million GPU hours. This makes Falcon 180B around 2.5 times larger than competing models like Llama 2.
Currently, two versions of the model are available: 180B and 180b-Chat. The standard version is a raw, pre-trained model, which companies can fine-tune to suit their use cases. Alternatively, the chat version is ideal for managing generic instructions. TII says the Chat model is already fine-tuned on instruction, chat data sets, and several large-scale conversational datasets.
If all that sounds incredibly confusing, Falcon 180B is an ultra-powerful language model that can adapt to various tasks such as coding or knowledge testing.
What is Falcon 180B? The Performance
Strengthening the UAE’s position in the burgeoning AI market, Falcon 180B promises state-of-the-art results that transcend many of the solutions already in the current market. The tech has topped the Hugging Face leaderboard for pre-trained open-access models.
It scores better than proprietary solutions like Google’s PaLM-2 (the model powering Bard). Compared to the top closed-source LLMs, 180B falls only slightly behind GPT-4 from OpenAI. Falcon 180B’s incredible performance is a direct result of its extensive training.
The vast corpus of text fed into the model gives it an unparalleled ability to understand language and context. It can excel in language tasks, such as proficiency assessments and reasoning. It could even become a powerful tool for training the next generation of Gen-AI bots.
