---
license: apache-2.0
tags:
  - gguf
  - qwen
  - qwen3
  - qwen3-8b
  - qwen3-8b-gguf
  - llama.cpp
  - quantized
  - text-generation
  - reasoning
  - agent
  - chat
  - multilingual
base_model: Qwen/Qwen3-8B
author: geoffmunn
pipeline_tag: text-generation
language:
  - en
  - zh
  - es
  - fr
  - de
  - ru
  - ar
  - ja
  - ko
  - hi
---

# Qwen3-8B-GGUF

This is a **GGUF-quantized version** of the **[Qwen/Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B)** language model — an **8-billion-parameter** LLM from Alibaba's Qwen series, designed for **advanced reasoning, agentic behavior, and multilingual tasks**.

Converted for use with `llama.cpp` and compatible tools like OpenWebUI, LM Studio, GPT4All, and more.

> 💡 **Key Features of Qwen3-8B**:
> - 🤔 **Thinking Mode**: Use `enable_thinking=True` or `/think` for step-by-step logic, math, and code.
> - ⚡ **Non-Thinking Mode**: Use `/no_think` for fast, lightweight dialogue.
> - 🧰 **Agent Capable**: Integrates with tools via MCP, APIs, and plugins.
> - 🌍 **Multilingual Support**: Fluent in 100+ languages including Chinese, English, Spanish, Arabic, Japanese, etc.

## Available Quantizations (from f16)

These variants were built from a **f16** base model to ensure consistency across quant levels.

| Level     | Speed     | Size        | Recommendation                                                                        |
|-----------|-----------|-------------|---------------------------------------------------------------------------------------|
| Q2_K      | ⚡ Fastest | 3.28 GB     | Not recommended. Came first in the bat & ball question, no other appearances.         |
| 🥉Q3_K_S  | ⚡ Fast    | 3.77 GB     | 🥉 Came first and second in questions covering both ends of the temperature spectrum. |
| 🥇 Q3_K_M | ⚡ Fast    | 4.12 GB     | 🥇 **Best overall model.** Was a top 3 finisher for all questions except the haiku.   |
| 🥉Q4_K_S  | 🚀 Fast   | 4.8 GB      | 🥉 Came first and second in questions covering both ends of the temperature spectrum. |
| Q4_K_M    | 🚀 Fast   | 5.85 GB     | Came first and second in questions covering high temperature questions.               |
| 🥈 Q5_K_S | 🐢 Medium | 5.72 GB     | 🥈 A good second place. Good for all query types.                                     |
| Q5_K_M    | 🐢 Medium | 5.85 GB     | Not recommended, no appeareances in the top 3 for any question.                       |
| Q6_K      | 🐌 Slow   | 6.73 GB     | Showed up in a few results, but not recommended.                                      |
| Q8_0      | 🐌 Slow   | 8.71 GB     | Not recommended, Only one top 3 finish.                                               |

## Model anaysis and rankings

There are numerous good candidates - lots of different models showed up in the top 3 across all the quesionts. However, **Qwen3-8B-f16:Q5_K_M** was a finalist in all but one question so is the recommended model. **Qwen3-8B-f16:Q5_K_S** did nearly as well and is worth considering, 

The 'hello' question is the first time that all models got it exactly right. All models in the 8B range did well and it's mainly a question of what one works best on your hardware.

You can read the results here: [Qwen3-8b-analysis.md](Qwen3-8b-analysis.md)

If you find this useful, please give the project a ❤️ like.

## Usage

Load this model using:
- [OpenWebUI](https://openwebui.com) – self-hosted AI interface with RAG & tools
- [LM Studio](https://lmstudio.ai) – desktop app with GPU support and chat templates
- [GPT4All](https://gpt4all.io) – private, local AI chatbot (offline-first)
- Or directly via `llama.cpp`

Each quantized model includes its own `README.md` and shares a common `MODELFILE` for optimal configuration.

Importing directly into Ollama should work, but you might encounter this error: `Error: invalid character '<' looking for beginning of value`.
In this case try these steps:

1. `wget https://huggingface.co/geoffmunn/Qwen3-4B/resolve/main/Qwen3-4B-f16%3AQ3_K_M.gguf` (replace the quantised version with the one you want)
2. `nano Modelfile` and enter these details (again, replacing Q3_K_M with the version you want):
```text
FROM ./Qwen3-4B-f16:Q3_K_M.gguf

# Chat template using ChatML (used by Qwen)
SYSTEM You are a helpful assistant

TEMPLATE "{{ if .System }}<|im_start|>system
{{ .System }}<|im_end|>{{ end }}<|im_start|>user
{{ .Prompt }}<|im_end|>
<|im_start|>assistant
"
PARAMETER stop <|im_start|>
PARAMETER stop <|im_end|>

# Default sampling
PARAMETER temperature 0.6
PARAMETER top_p 0.95
PARAMETER top_k 20
PARAMETER min_p 0.0
PARAMETER repeat_penalty 1.1
PARAMETER num_ctx 4096
```

The `num_ctx` value has been dropped to increase speed significantly.

3. Then run this command: `ollama create Qwen3-4B-f16:Q3_K_M -f Modelfile`

You will now see "Qwen3-4B-f16:Q3_K_M" in your Ollama model list.

These import steps are also useful if you want to customise the default parameters or system prompt.

## Author

👤 Geoff Munn (@geoffmunn)  
🔗 [Hugging Face Profile](https://huggingface.co/geoffmunn)

## Disclaimer

This is a community conversion for local inference. Not affiliated with Alibaba Cloud or the Qwen team.