Update README.md
Browse filesmove-up installation instruction
README.md
CHANGED
|
@@ -16,6 +16,20 @@ base_model_relation: quantized
|
|
| 16 |
# DeepSeek-V3.1-AWQ
|
| 17 |
Base model: [DeepSeek-V3.1](https://huggingface.co/deepseek-ai/DeepSeek-V3.1)
|
| 18 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 19 |
### 【vLLM Single Node with 8 GPUs — Startup Command】
|
| 20 |
```
|
| 21 |
CONTEXT_LENGTH=32768
|
|
@@ -35,20 +49,6 @@ vllm serve \
|
|
| 35 |
--port 8000
|
| 36 |
```
|
| 37 |
|
| 38 |
-
### 【Dependencies / Installation】
|
| 39 |
-
As of **2025-08-23**, create a fresh Python environment and run:
|
| 40 |
-
|
| 41 |
-
```bash
|
| 42 |
-
# ❗there are glitches with vllm 0.10.1.1, still looking for resolutions❗
|
| 43 |
-
# ❗downgrade vllm for now ❗
|
| 44 |
-
pip install vllm==0.9.0
|
| 45 |
-
pip install transformers==4.53
|
| 46 |
-
|
| 47 |
-
# ❗patch up AWQ MoE quant config, otherwise some modules cannot be properly loaded❗
|
| 48 |
-
SITE_PACKAGES=$(pip -V | awk '{print $4}' | sed 's/\/pip$//')
|
| 49 |
-
cp awq_marlin.py "$SITE_PACKAGES/vllm/model_executor/layers/quantization/awq_marlin.py"
|
| 50 |
-
```
|
| 51 |
-
|
| 52 |
### 【Logs】
|
| 53 |
```
|
| 54 |
2025-08-23
|
|
|
|
| 16 |
# DeepSeek-V3.1-AWQ
|
| 17 |
Base model: [DeepSeek-V3.1](https://huggingface.co/deepseek-ai/DeepSeek-V3.1)
|
| 18 |
|
| 19 |
+
### 【Dependencies / Installation】
|
| 20 |
+
As of **2025-08-23**, create a fresh Python environment and run:
|
| 21 |
+
|
| 22 |
+
```bash
|
| 23 |
+
# ❗there are glitches with vllm 0.10.1.1, still looking for resolutions❗
|
| 24 |
+
# ❗downgrade vllm for now ❗
|
| 25 |
+
pip install vllm==0.9.0
|
| 26 |
+
pip install transformers==4.53
|
| 27 |
+
|
| 28 |
+
# ❗patch up AWQ MoE quant config, otherwise some modules cannot be properly loaded❗
|
| 29 |
+
SITE_PACKAGES=$(pip -V | awk '{print $4}' | sed 's/\/pip$//')
|
| 30 |
+
cp awq_marlin.py "$SITE_PACKAGES/vllm/model_executor/layers/quantization/awq_marlin.py"
|
| 31 |
+
```
|
| 32 |
+
|
| 33 |
### 【vLLM Single Node with 8 GPUs — Startup Command】
|
| 34 |
```
|
| 35 |
CONTEXT_LENGTH=32768
|
|
|
|
| 49 |
--port 8000
|
| 50 |
```
|
| 51 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 52 |
### 【Logs】
|
| 53 |
```
|
| 54 |
2025-08-23
|