French Start-up Mistral AI releases first ever OpenSource model
Mistral AI's Mixtral 8x7B, a scaled-down MoE model akin to GPT-4, surprises with a silent 87 GB torrent release, signaling a major leap in open-source AI.
French startup Mistral AI surprised everyone by quietly dropping its new language model, Mixtral 8x7B, as an 87 GB torrent. This release, which happened on X without a press release or further details, seems to be a scaled-down version of GPT-4.
While the tech world buzzes about Google's overly rehearsed AI demo video, now facing criticism, Mistral AI has made waves in the open-source community by releasing a Mixture of Experts (MoE) model. This model, Mixtral, consists of 8 7B-parameter experts.
What's a Mixture of Experts (MoE)?
It's a technique enhancing the efficiency and accuracy of large language models (LLMs). It divides complex tasks into smaller ones, each handled by a specialized 'expert.' These experts are mini-models trained in specific areas. A 'gating network' then decides which expert handles which part of a task, based on compatibility scores. This ensures the right expert works on the right task, making the overall model more capable and versatile.
Details from Reddit reveal more about Mistral's MoE. For each token's inference, only 2 out of 8 experts are used. The model's metadata gives a glimpse into its structure, with a total of 42 billion parameters, which is significantly smaller compared to GPT-4's estimated 1.8 trillion. Despite having a similar architecture to GPT-4, Mistral 8x7B's scale is reduced: it has 8 experts instead of 16, 7B parameters per expert (not 166B), and it maintains the same 32K context size as the original GPT-4.
Subscribe to our newsletter and receive a selection of cool articles every two weeks.
By clicking “Sign Up”, you accept our Terms of Service and Privacy Policy. You can opt-out at any time.
No spam. Twice a month.
Unsubscribe anytime.
Sign up to our newsletter and receive a selection of cool articles weekly.
By clicking “Sign Up”, you accept our Terms of Service and Privacy Policy. You can opt-out at any time.