[{"data":1,"prerenderedAt":4},["ShallowReactive",2],{"readme:airllm":3},"\u003Cp>\u003Cimg src=\"https:\u002F\u002Fgithub.com\u002Flyogavin\u002Fairllm\u002Fblob\u002Fmain\u002Fassets\u002Fairllm_logo_sm.png?v=3&raw=true\" alt=\"airllm_logo\" \u002F>\u003C\u002Fp>\n\u003Cp>\u003Ca href=\"#quickstart\" rel=\"nofollow ugc noopener\">\u003Cstrong>Quickstart\u003C\u002Fstrong>\u003C\u002Fa> | \n\u003Ca href=\"#configurations\" rel=\"nofollow ugc noopener\">\u003Cstrong>Configurations\u003C\u002Fstrong>\u003C\u002Fa> | \n\u003Ca href=\"#macos\" rel=\"nofollow ugc noopener\">\u003Cstrong>MacOS\u003C\u002Fstrong>\u003C\u002Fa> | \n\u003Ca href=\"#example-python-notebook\" rel=\"nofollow ugc noopener\">\u003Cstrong>Example notebooks\u003C\u002Fstrong>\u003C\u002Fa> | \n\u003Ca href=\"#faq\" rel=\"nofollow ugc noopener\">\u003Cstrong>FAQ\u003C\u002Fstrong>\u003C\u002Fa>\u003C\u002Fp>\n\u003Cp>\u003Cstrong>AirLLM\u003C\u002Fstrong> dramatically reduces inference memory usage, letting 70B large language models run on a single 4GB GPU card — without quantization, distillation, or pruning. You can even run \u003Cstrong>405B Llama 3.1\u003C\u002Fstrong> on \u003Cstrong>8GB\u003C\u002Fstrong>, and \u003Cstrong>DeepSeek-V3 (671B)\u003C\u002Fstrong> on \u003Cstrong>~12GB\u003C\u002Fstrong>.\u003C\u002Fp>\n\u003Cp>\u003Ca href=\"https:\u002F\u002Fgithub.com\u002Flyogavin\u002Fairllm\u002Fstargazers\" rel=\"nofollow ugc noopener\">\u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fgithub\u002Fstars\u002Flyogavin\u002Fairllm?style=social\" alt=\"GitHub Repo stars\" \u002F>\u003C\u002Fa>\n\u003Ca href=\"https:\u002F\u002Fpepy.tech\u002Fproject\u002Fairllm\" rel=\"nofollow ugc noopener\">\u003Cimg src=\"https:\u002F\u002Fstatic.pepy.tech\u002Fpersonalized-badge\u002Fairllm?period=total&units=international_system&left_color=grey&right_color=blue&left_text=downloads\" alt=\"Downloads\" \u002F>\u003C\u002Fa>\u003C\u002Fp>\n\u003Cp>\u003Ca href=\"https:\u002F\u002Fgithub.com\u002FLianjiaTech\u002FBELLE\u002Fblob\u002Fmain\u002FLICENSE\" rel=\"nofollow ugc noopener\">\u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FCode%20License-Apache_2.0-green.svg\" alt=\"Code License\" \u002F>\u003C\u002Fa>\n\u003Ca href=\"https:\u002F\u002Fstatic.aicompose.cn\u002Fstatic\u002Fwecom_barcode.png?t=1671918938\" rel=\"nofollow ugc noopener\">\u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002Fwechat-Anima-brightgreen?logo=wechat\" alt=\"Generic badge\" \u002F>\u003C\u002Fa>\n\u003Ca href=\"https:\u002F\u002Fdiscord.gg\u002F2xffU5sn\" rel=\"nofollow ugc noopener\">\u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fdiscord\u002F1175437549783760896?logo=discord&color=7289da\" alt=\"Discord\" \u002F>\u003C\u002Fa>\n\u003Ca href=\"https:\u002F\u002Fpypi.org\u002Fproject\u002Fairllm\u002F\" rel=\"nofollow ugc noopener\">\u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fpypi\u002Fformat\u002Fairllm?logo=pypi&color=3571a3\" alt=\"PyPI - AirLLM\" \u002F>\n\u003C\u002Fa>\n\u003Ca href=\"https:\u002F\u002Fmedium.com\u002F@lyo.gavin\" rel=\"nofollow ugc noopener\">\u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fwebsite?up_message=blog&url=https%3A%2F%2Fmedium.com%2F%40lyo.gavin&logo=medium&color=black\" alt=\"Website\" \u002F>\u003C\u002Fa>\n\u003Ca href=\"https:\u002F\u002Fgavinliblog.com\" rel=\"nofollow ugc noopener\">\u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FGavin_Li-Blog-blue\" alt=\"Website\" \u002F>\u003C\u002Fa>\n\u003Ca href=\"https:\u002F\u002Fpatreon.com\u002Fgavinli\" rel=\"nofollow ugc noopener\">\u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fendpoint.svg?url=https%3A%2F%2Fshieldsio-patreon.vercel.app%2Fapi%3Fusername%3Dgavinli%26type%3Dpatrons&style=flat\" alt=\"Support me on Patreon\" \u002F>\u003C\u002Fa>\n\u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fsponsors\u002Flyogavin\" rel=\"nofollow ugc noopener\">\u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fgithub\u002Fsponsors\u002Flyogavin?logo=GitHub&color=lightgray\" alt=\"GitHub Sponsors\" \u002F>\u003C\u002Fa>\u003C\u002Fp>\n\u003Ch2>AI Agents Recommendation:\u003C\u002Fh2>\n\u003Cul>\n\u003Cli>\u003Cp>\u003Ca href=\"https:\u002F\u002Fgodmodeai.co\" rel=\"nofollow ugc noopener\">Best AI Game Sprite Generator\u003C\u002Fa>\u003C\u002Fp>\n\u003C\u002Fli>\n\u003Cli>\u003Cp>\u003Ca href=\"https:\u002F\u002Fcrazyfaceai.com\" rel=\"nofollow ugc noopener\">Best AI Facial Expression Editor\u003C\u002Fa>\u003C\u002Fp>\n\u003C\u002Fli>\n\u003Cli>\u003Cp>\u003Ca href=\"https:\u002F\u002Fbloome.im\u002Flogin?ref=G6BYnov0\" rel=\"nofollow ugc noopener\">Bloome — build & run AI agent teams in the cloud, zero setup\u003C\u002Fa>\u003C\u002Fp>\n\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Ch2>Updates\u003C\u002Fh2>\n\u003Cp>[2026\u002F06] \u003Cstrong>v3.0\u003C\u002Fstrong>: FP8 model support + the latest models. Run \u003Cstrong>DeepSeek-V3 (671B) on ~12GB\u003C\u002Fstrong> and \u003Cstrong>Qwen3-235B on ~3GB\u003C\u002Fstrong>, plus Qwen3, Llama 3.x\u002F4, DeepSeek V2\u002FV3, Phi-4, Gemma and more — all through a single \u003Ccode>AutoModel\u003C\u002Fcode>.\u003C\u002Fp>\n\u003Cp>[2024\u002F08\u002F20] v2.11.0: Support Qwen2.5\u003C\u002Fp>\n\u003Cp>[2024\u002F08\u002F18] v2.10.1 Support CPU inference. Support non sharded models. Thanks @NavodPeiris for the great work! \u003C\u002Fp>\n\u003Cp>[2024\u002F07\u002F30] Support Llama3.1 \u003Cstrong>405B\u003C\u002Fstrong> (\u003Ca href=\"https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002Flyogavin\u002Fairllm\u002Fblob\u002Fmain\u002Fair_llm\u002Fexamples\u002Frun_llama3.1_405B.ipynb\" rel=\"nofollow ugc noopener\">example notebook\u003C\u002Fa>). Support \u003Cstrong>8bit\u002F4bit quantization\u003C\u002Fstrong>.\u003C\u002Fp>\n\u003Cp>[2024\u002F04\u002F20] AirLLM supports Llama3 natively already. Run Llama3 70B on 4GB single GPU.\u003C\u002Fp>\n\u003Cp>[2023\u002F12\u002F25] v2.8.2: Support MacOS running 70B large language models.\u003C\u002Fp>\n\u003Cp>[2023\u002F12\u002F20] v2.7: Support AirLLMMixtral. \u003C\u002Fp>\n\u003Cp>[2023\u002F12\u002F20] v2.6: Added AutoModel, automatically detect model type, no need to provide model class to initialize model.\u003C\u002Fp>\n\u003Cp>[2023\u002F12\u002F18] v2.5: added prefetching to overlap the model loading and compute. 10% speed improvement.\u003C\u002Fp>\n\u003Cp>[2023\u002F12\u002F03] added support of \u003Cstrong>ChatGLM\u003C\u002Fstrong>, \u003Cstrong>QWen\u003C\u002Fstrong>, \u003Cstrong>Baichuan\u003C\u002Fstrong>, \u003Cstrong>Mistral\u003C\u002Fstrong>, \u003Cstrong>InternLM\u003C\u002Fstrong>!\u003C\u002Fp>\n\u003Cp>[2023\u002F12\u002F02] added support for safetensors. Now support all top 10 models in open llm leaderboard.\u003C\u002Fp>\n\u003Cp>[2023\u002F12\u002F01] airllm 2.0. Support compressions: \u003Cstrong>3x run time speed up!\u003C\u002Fstrong>\u003C\u002Fp>\n\u003Cp>[2023\u002F11\u002F20] airllm Initial version!\u003C\u002Fp>\n\u003Ch2>Star History\u003C\u002Fh2>\n\u003Ca href=\"https:\u002F\u002Fstar-history.com\u002F#lyogavin\u002Fairllm&Timeline\" rel=\"nofollow ugc noopener\">\n  \u003Cpicture>\n    \u003Csource media=\"(prefers-color-scheme: dark)\" srcset=\"assets\u002Fstar-history-dark.png\">\u003C\u002Fsource>\n    \u003Cimg alt=\"Star History Chart\" src=\"https:\u002F\u002Fraw.githubusercontent.com\u002Flyogavin\u002Fairllm\u002FHEAD\u002Fassets\u002Fstar-history.png\" \u002F>\n  \u003C\u002Fpicture>\n\u003C\u002Fa>\u003Ch2>Table of Contents\u003C\u002Fh2>\n\u003Cul>\n\u003Cli>\u003Ca href=\"#quickstart\" rel=\"nofollow ugc noopener\">Quick start\u003C\u002Fa>\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"#model-compression---3x-inference-speed-up\" rel=\"nofollow ugc noopener\">Model Compression\u003C\u002Fa>\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"#configurations\" rel=\"nofollow ugc noopener\">Configurations\u003C\u002Fa>\u003C\u002Fli>\n\u003Cli>[Run on MacOS](#\u003C\u002Fli>\n\u003C\u002Ful>\n",1784240406337]