I see a lot of talk of Ollama here, which I personally don’t like because:

• The quantizations they use tend to be suboptimal

• It abstracts away llama.cpp in a way that, frankly, leaves a lot of performance and quality on the table.

• It abstracts away things that you should really know for hosting LLMs.

• I don’t like some things about the devs. I won’t rant, but I especially don’t like the hint they’re cooking up something commercial.

So, here’s a quick guide to get away from Ollama.

• First step is to pick your OS. Windows is fine, but if setting up something new, linux is best. I favor CachyOS in particular, for its great python performance. If you use Windows, be sure to enable hardware accelerated scheduling and disable shared memory.

• Ensure the latest version of CUDA (or ROCm, if using AMD) is installed. Linux is great for this, as many distros package them for you.

• Install Python 3.11.x, 3.12.x, or at least whatever your distro supports, and git. If on linux, also install your distro’s “build tools” package.

Now for actually installing the runtime. There are a great number of inference engines supporting different quantizations, forgive the Reddit link but see: https://old.reddit.com/r/LocalLLaMA/comments/1fg3jgr/a_large_table_of_inference_engines_and_supported/

As far as I am concerned, 3 matter to “home” hosters on consumer GPUs:

• Exllama (and by extension TabbyAPI), as a very fast, very memory efficient “GPU only” runtime, supports AMD via ROCM and Nvidia via CUDA: https://github.com/theroyallab/tabbyAPI

• Aphrodite Engine. While not strictly as vram efficient, its much faster with parallel API calls, reasonably efficient at very short context, and supports just about every quantization under the sun and more exotic models than exllama. AMD/Nvidia only: https://github.com/PygmalionAI/Aphrodite-engine

• This fork of kobold.cpp, which supports more fine grained kv cache quantization (we will get to that). It supports CPU offloading and I think Apple Metal: https://github.com/Nexesenex/croco.cpp

Now, there are also reasons I don’t like llama.cpp, but one of the big ones is that sometimes its model implementations have… quality degrading issues, or odd bugs. Hence I would generally recommend TabbyAPI if you have enough vram to avoid offloading to CPU, and can figure out how to set it up. So:

• Open a terminal, run git clone https://github.com/theroyallab/tabbyAPI.git

• cd tabbyAPI

• Follow this guide for setting up a python venv and installing pytorch and tabbyAPI: https://github.com/theroyallab/tabbyAPI/wiki/01.-Getting-Started#installing

This can go wrong, if anyone gets stuck I can help with that.

• Next, figure out how much VRAM you have.

• Figure out how much “context” you want, aka how much text the llm can ingest. If a models has a context length of, say, “8K” that means it can support 8K tokens as input, or less than 8K words. Not all tokenizers are the same, some like Qwen 2.5’s can fit nearly a word per token, while others are more in the ballpark of half a work per token or less.

• Keep in mind that the actual context length of many models is an outright lie, see: https://github.com/hsiehjackson/RULER

• Exllama has a feature called “kv cache quantization” that can dramatically shrink the VRAM the “context” of an LLM takes up. Unlike llama.cpp, it’s Q4 cache is basically lossless, and on a model like Command-R, an 80K+ context can take up less than 4GB! Its essential to enable Q4 or Q6 cache to squeeze in as much LLM as you can into your GPU.

• With that in mind, you can search huggingface for your desired model. Since we are using tabbyAPI, we want to search for “exl2” quantizations: https://huggingface.co/models?sort=modified&search=exl2

• There are all sorts of finetunes… and a lot of straight-up garbage. But I will post some general recommendations based on total vram:

• 4GB: A very small quantization of Qwen 2.5 7B. Or maybe Llama 3B.

• 6GB: IMO llama 3.1 8B is best here. There are many finetunes of this depending on what you want (horny chat, tool usage, math, whatever). For coding, I would recommend Qwen 7B coder instead: https://huggingface.co/models?sort=trending&search=qwen+7b+exl2

• 8GB-12GB Qwen 2.5 14B is king! Unlike it’s 7B counterpart, I find the 14B version of the model incredible for its size, and it will squeeze into this vram pool (albeit with very short context/tight quantization for the 8GB cards). I would recommend trying Arcee’s new distillation in particular: https://huggingface.co/bartowski/SuperNova-Medius-exl2

• 16GB: Mistral 22B, Mistral Coder 22B, and very tight quantizations of Qwen 2.5 34B are possible. Honorable mention goes to InternLM 2.5 20B, which is alright even at 128K context.

• 20GB-24GB: Command-R 2024 35B is excellent for “in context” work, like asking questions about long documents, continuing long stories, anything involving working “with” the text you feed to an LLM rather than pulling from it’s internal knowledge pool. It’s also quite goot at longer contexts, out to 64K-80K more-or-less, all of which fits in 24GB. Otherwise, stick to Qwen 2.5 34B, which still has a very respectable 32K native context, and a rather mediocre 64K “extended” context via YaRN: https://huggingface.co/DrNicefellow/Qwen2.5-32B-Instruct-4.25bpw-exl2

• 32GB, same as 24GB, just with a higher bpw quantization. But this is also the threshold were lower bpw quantizations of Qwen 2.5 72B (at short context) start to make sense.

• 48GB: Llama 3.1 70B (for longer context) or Qwen 2.5 72B (for 32K context or less)

Again, browse huggingface and pick an exl2 quantization that will cleanly fill your vram pool + the amount of context you want to specify in TabbyAPI. Many quantizers such as bartowski will list how much space they take up, but you can also just look at the available filesize.

• Now… you have to download the model. Bartowski has instructions here, but I prefer to use this nifty standalone tool instead: https://github.com/bodaay/HuggingFaceModelDownloader

• Put it in your TabbyAPI models folder, and follow the documentation on the wiki.

• There are a lot of options. Some to keep in mind are chunk_size (higher than 2048 will process long contexts faster but take up lots of vram, less will save a little vram), cache_mode (use Q4 for long context, Q6/Q8 for short context if you have room), max_seq_len (this is your context length), tensor_parallel (for faster inference with 2 identical GPUs), and max_batch_size (parallel processing if you have multiple user hitting the tabbyAPI server, but more vram usage)

• Now… pick your frontend. The tabbyAPI wiki has a good compliation of community projects, but Open Web UI is very popular right now: https://github.com/open-webui/open-webui I personally use exui: https://github.com/turboderp/exui

• And be careful with your sampling settings when using LLMs. Different models behave differently, but one of the most common mistakes people make is using “old” sampling parameters for new models. In general, keep temperature very low (<0.1, or even zero) and rep penalty low (1.01?) unless you need long, creative responses. If available in your UI, enable DRY sampling to tamp down repition without “dumbing down” the model with too much temperature or repitition penalty. Always use a MinP of 0.05 or higher and disable other samplers. This is especially important for Chinese models like Qwen, as MinP cuts out “wrong language” answers from the response.

• Now, once this is all setup and running, I’d recommend throttling your GPU, as it simply doesn’t need its full core speed to maximize its inference speed while generating. For my 3090, I use something like sudo nvidia-smi -pl 290, which throttles it down from 420W to 290W.

Sorry for the wall of text! I can keep going, discussing kobold.cpp/llama.cpp, Aphrodite, exotic quantization and other niches like that if anyone is interested.