Menu
Home
People
Places
Arts
History
Plants & Animals
Science
Life & Culture
Technology
Reference.org
1.58-bit large language model
open-in-new
<h2 id="bitnet">BitNet</h2> <p>In 2024, Ma et al., researchers at <a href="/facts/Microsoft/nGIDDXdx">Microsoft</a>, declared that their 1.58-bit model, <i>BitNet b1.58</i> is comparable in performance to the 16-bit <a href="/facts/Llama_2/rmrWj35n">Llama 2</a> and opens the era of 1-bit LLM.<a class="footnote-ref" id="fnref:6" href="#fn:6"><sup>6</sup></a> BitNet creators did not use the post-training quantization of weights but instead relied on the new <i>BitLinear</i> transform that replaced the <i>nn.Linear</i> layer of the traditional transformer design.<a class="footnote-ref" id="fnref:7" href="#fn:7"><sup>7</sup></a> </p><p>In 2025, Microsoft researchers had released an open-weights and open inference code model <i>BitNet b1.58 2B4T</i> demonstrating performance competitive to the full precision models at 2B parameters and 4T training tokens.<a class="footnote-ref" id="fnref:8" href="#fn:8"><sup>8</sup></a> </p> <h2 id="critique">Critique</h2> <p>Some researchers<a class="footnote-ref" id="fnref:9" href="#fn:9"><sup>9</sup></a> point out that the scaling laws<a class="footnote-ref" id="fnref:10" href="#fn:10"><sup>10</sup></a> of large language models favor the low-bit weights only in case of undertrained models. As the number of training tokens increases, the deficiencies of low-bit quantization surface. </p> <h2 id="sources">Sources</h2> <ul><li>Ma, Shuming; Wang, Hongyu; Ma, Lingxiao; Wang, Lei; Wang, Wenhui; Huang, Shaohan; Dong, Li; Wang, Ruiping; Xue, Jilong; Wei, Furu (2024-02-27). "The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits". <a href="/facts/ArXiv_(identifier)/H6EtgnBe">arXiv</a>:<a href="https://arxiv.org/abs/2402.17764">2402.17764</a> [<a href="https://arxiv.org/archive/cs.CL">cs.CL</a>].</li> <li>Ma, Shuming; Wang, Hongyu; Huang, Shaohan; Zhang, Xingxing; Hu, Ying; Song, Ting; Xia, Yan; Wei, Furu (2025). "BitNet b1.58 2B4T Technical Report". <a href="/facts/ArXiv_(identifier)/H6EtgnBe">arXiv</a>:<a href="https://arxiv.org/abs/2504.12285">2504.12285</a> [<a href="https://arxiv.org/archive/cs.CL">cs.CL</a>].</li> <li>Friha, Othmane; Amine Ferrag, Mohamed; Kantarci, Burak; Cakmak, Burak; Ozgun, Arda; Ghoualmi-Zine, Nassira (2024). <a href="https://doi.org/10.1109%2FOJCOMS.2024.3456549">"LLM-Based Edge Intelligence: A Comprehensive Survey on Architectures, Applications, Security and Trustworthiness"</a>. <i>IEEE Open Journal of the Communications Society</i>. 5: 5799–5856. <a href="/facts/Doi_(identifier)/muM9Etpq">doi</a>:<a href="https://doi.org/10.1109%2FOJCOMS.2024.3456549">10.1109/OJCOMS.2024.3456549</a>. <a href="/facts/ISSN_(identifier)/DPAflDvU">ISSN</a> <a href="https://search.worldcat.org/issn/2644-125X">2644-125X</a>.</li> <li>Hutson, Matthew (2024-05-30). <a href="https://spectrum.ieee.org/1-bit-llm">"1-bit LLMs Could Solve AI's Energy Demands"</a>. <i>IEEE Spectrum</i>. Retrieved 2025-04-22.</li> <li>Huyen, Chip (2024-12-04). <a href="https://books.google.com/books?id=S7M1EQAAQBAJ&pg=PA330"><i>AI Engineering</i></a>. "O'Reilly Media, Inc.". <a href="/facts/ISBN_(identifier)/15AdSPa9">ISBN</a> 978-1-0981-6627-4. Retrieved 2025-04-22.</li> <li>Kumar, Tanishq; Ankner, Zachary; Spector, Benjamin F.; Bordelon, Blake; Muennighoff, Niklas; Paul, Mansheej; Pehlevan, Cengiz; Ré, Christopher; Raghunathan, Aditi (2024). "Scaling Laws for Precision". <a href="/facts/ArXiv_(identifier)/H6EtgnBe">arXiv</a>:<a href="https://arxiv.org/abs/2411.04330">2411.04330</a> [<a href="https://arxiv.org/archive/cs.LG">cs.LG</a>].</li> <li>Morales, Jowi (2025-04-17). <a href="https://www.tomshardware.com/tech-industry/artificial-intelligence/microsoft-researchers-build-1-bit-ai-llm-with-2b-parameters-model-small-enough-to-run-on-some-cpus">"Microsoft researchers build 1-bit AI LLM with 2B parameters"</a>. <i>Tom's Hardware</i>. Retrieved 2025-04-21.</li> <li>Ouyang, Xu; Ge, Tao; Hartvigsen, Thomas; Zhang, Zhisong; Mi, Haitao; Yu, Dong (2024). "Low-Bit Quantization Favors Undertrained LLMS: Scaling Laws for Quantized LLMS with 100T Training Tokens". <a href="/facts/ArXiv_(identifier)/H6EtgnBe">arXiv</a>:<a href="https://arxiv.org/abs/2411.17691">2411.17691</a> [<a href="https://arxiv.org/archive/cs.LG">cs.LG</a>].</li> <li>Wang, Hongyu; Ma, Shuming; Dong, Li; Huang, Shaohan; Wang, Huaijie; Ma, Lingxiao; Yang, Fan; Wang, Ruiping; Wu, Yi; Wei, Furu (2023). "BitNet: Scaling 1-bit Transformers for Large Language Models". <a href="/facts/ArXiv_(identifier)/H6EtgnBe">arXiv</a>:<a href="https://arxiv.org/abs/2310.11453">2310.11453</a> [<a href="https://arxiv.org/archive/cs.CL">cs.CL</a>].</li></ul> <h2 id="references">References</h2> <ol> <li id="fn:1"><p>Ma et al. 2024, p. 1. - Ma, Shuming; Wang, Hongyu; Ma, Lingxiao; Wang, Lei; Wang, Wenhui; Huang, Shaohan; Dong, Li; Wang, Ruiping; Xue, Jilong; Wei, Furu (2024-02-27). "The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits". arXiv:2402.17764 [cs.CL]. <a href="https://arxiv.org/abs/2402.17764" target="_blank">https://arxiv.org/abs/2402.17764</a> <a href="#fnref:1" class="footnote-back-ref">↩</a></p></li> <li id="fn:2"><p>Friha et al. 2024, p. 5822. - Friha, Othmane; Amine Ferrag, Mohamed; Kantarci, Burak; Cakmak, Burak; Ozgun, Arda; Ghoualmi-Zine, Nassira (2024). "LLM-Based Edge Intelligence: A Comprehensive Survey on Architectures, Applications, Security and Trustworthiness". IEEE Open Journal of the Communications Society. 5: 5799–5856. doi:10.1109/OJCOMS.2024.3456549. ISSN 2644-125X. <a href="https://doi.org/10.1109%2FOJCOMS.2024.3456549" target="_blank">https://doi.org/10.1109%2FOJCOMS.2024.3456549</a> <a href="#fnref:2" class="footnote-back-ref">↩</a></p></li> <li id="fn:3"><p>Hutson 2024. - Hutson, Matthew (2024-05-30). "1-bit LLMs Could Solve AI's Energy Demands". IEEE Spectrum. Retrieved 2025-04-22. <a href="https://spectrum.ieee.org/1-bit-llm" target="_blank">https://spectrum.ieee.org/1-bit-llm</a> <a href="#fnref:3" class="footnote-back-ref">↩</a></p></li> <li id="fn:4"><p>Ma et al. 2024, p. 1. - Ma, Shuming; Wang, Hongyu; Ma, Lingxiao; Wang, Lei; Wang, Wenhui; Huang, Shaohan; Dong, Li; Wang, Ruiping; Xue, Jilong; Wei, Furu (2024-02-27). "The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits". arXiv:2402.17764 [cs.CL]. <a href="https://arxiv.org/abs/2402.17764" target="_blank">https://arxiv.org/abs/2402.17764</a> <a href="#fnref:4" class="footnote-back-ref">↩</a></p></li> <li id="fn:5"><p>Morales 2025. - Morales, Jowi (2025-04-17). "Microsoft researchers build 1-bit AI LLM with 2B parameters". Tom's Hardware. Retrieved 2025-04-21. <a href="https://www.tomshardware.com/tech-industry/artificial-intelligence/microsoft-researchers-build-1-bit-ai-llm-with-2b-parameters-model-small-enough-to-run-on-some-cpus" target="_blank">https://www.tomshardware.com/tech-industry/artificial-intelligence/microsoft-researchers-build-1-bit-ai-llm-with-2b-parameters-model-small-enough-to-run-on-some-cpus</a> <a href="#fnref:5" class="footnote-back-ref">↩</a></p></li> <li id="fn:6"><p>Huyen 2024, p. 330. - Huyen, Chip (2024-12-04). AI Engineering. "O'Reilly Media, Inc.". ISBN 978-1-0981-6627-4. Retrieved 2025-04-22. <a href="https://books.google.com/books?id=S7M1EQAAQBAJ&pg=PA330" target="_blank">https://books.google.com/books?id=S7M1EQAAQBAJ&pg=PA330</a> <a href="#fnref:6" class="footnote-back-ref">↩</a></p></li> <li id="fn:7"><p>Wang et al. 2023, p. 1. - Wang, Hongyu; Ma, Shuming; Dong, Li; Huang, Shaohan; Wang, Huaijie; Ma, Lingxiao; Yang, Fan; Wang, Ruiping; Wu, Yi; Wei, Furu (2023). "BitNet: Scaling 1-bit Transformers for Large Language Models". arXiv:2310.11453 [cs.CL]. <a href="https://arxiv.org/abs/2310.11453" target="_blank">https://arxiv.org/abs/2310.11453</a> <a href="#fnref:7" class="footnote-back-ref">↩</a></p></li> <li id="fn:8"><p>Ma et al. 2025. - Ma, Shuming; Wang, Hongyu; Huang, Shaohan; Zhang, Xingxing; Hu, Ying; Song, Ting; Xia, Yan; Wei, Furu (2025). "BitNet b1.58 2B4T Technical Report". arXiv:2504.12285 [cs.CL]. <a href="https://arxiv.org/abs/2504.12285" target="_blank">https://arxiv.org/abs/2504.12285</a> <a href="#fnref:8" class="footnote-back-ref">↩</a></p></li> <li id="fn:9"><p>Ouyang et al. 2024. - Ouyang, Xu; Ge, Tao; Hartvigsen, Thomas; Zhang, Zhisong; Mi, Haitao; Yu, Dong (2024). "Low-Bit Quantization Favors Undertrained LLMS: Scaling Laws for Quantized LLMS with 100T Training Tokens". arXiv:2411.17691 [cs.LG]. <a href="https://arxiv.org/abs/2411.17691" target="_blank">https://arxiv.org/abs/2411.17691</a> <a href="#fnref:9" class="footnote-back-ref">↩</a></p></li> <li id="fn:10"><p>Kumar et al. 2024. - Kumar, Tanishq; Ankner, Zachary; Spector, Benjamin F.; Bordelon, Blake; Muennighoff, Niklas; Paul, Mansheej; Pehlevan, Cengiz; Ré, Christopher; Raghunathan, Aditi (2024). "Scaling Laws for Precision". arXiv:2411.04330 [cs.LG]. <a href="https://arxiv.org/abs/2411.04330" target="_blank">https://arxiv.org/abs/2411.04330</a> <a href="#fnref:10" class="footnote-back-ref">↩</a></p></li> </ol>