Update README.md
Browse files
README.md
CHANGED
|
@@ -8,9 +8,6 @@ base_model:
|
|
| 8 |
---
|
| 9 |
# bge-base-en-v1.5-int8-ov
|
| 10 |
|
| 11 |
-
> [!WARNING]
|
| 12 |
-
> **Disclaimer**: This model is provided for evaluation purposes only. Performance, accuracy, and stability may vary. Use at your own discretion.
|
| 13 |
-
|
| 14 |
* Model creator: [BAAI](https://huggingface.co/BAAI)
|
| 15 |
* Original model: [bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5)
|
| 16 |
|
|
@@ -22,56 +19,19 @@ This is [bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) model c
|
|
| 22 |
|
| 23 |
## Quantization Parameters
|
| 24 |
|
| 25 |
-
|
| 26 |
-
|
| 27 |
-
```
|
| 28 |
-
from functools import partial
|
| 29 |
-
|
| 30 |
-
from transformers import AutoTokenizer
|
| 31 |
-
|
| 32 |
-
from optimum.intel import OVConfig, OVModelForFeatureExtraction, OVQuantizationConfig, OVQuantizer
|
| 33 |
-
|
| 34 |
-
|
| 35 |
-
MODEL_ID = "OpenVINO/bge-base-en-v1.5-fp16-ov"
|
| 36 |
-
base_model_path = "bge-base-en-v1.5"
|
| 37 |
-
int8_ptq_model_path = "bge-base-en-v1.5-int8"
|
| 38 |
-
|
| 39 |
-
model = OVModelForFeatureExtraction.from_pretrained(MODEL_ID)
|
| 40 |
-
model.save_pretrained(base_model_path)
|
| 41 |
|
| 42 |
-
|
| 43 |
-
tokenizer.save_pretrained(base_model_path)
|
| 44 |
-
|
| 45 |
-
|
| 46 |
-
quantizer = OVQuantizer.from_pretrained(model)
|
| 47 |
-
|
| 48 |
-
def preprocess_function(examples, tokenizer):
|
| 49 |
-
return tokenizer(examples["sentence"], padding="max_length", max_length=384, truncation=True)
|
| 50 |
-
|
| 51 |
-
|
| 52 |
-
calibration_dataset = quantizer.get_calibration_dataset(
|
| 53 |
-
"glue",
|
| 54 |
-
dataset_config_name="sst2",
|
| 55 |
-
preprocess_function=partial(preprocess_function, tokenizer=tokenizer),
|
| 56 |
-
num_samples=300,
|
| 57 |
-
dataset_split="train",
|
| 58 |
-
)
|
| 59 |
-
|
| 60 |
-
ov_config = OVConfig(quantization_config=OVQuantizationConfig())
|
| 61 |
-
|
| 62 |
-
quantizer.quantize(ov_config=ov_config, calibration_dataset=calibration_dataset, save_directory=int8_ptq_model_path)
|
| 63 |
-
tokenizer.save_pretrained(int8_ptq_model_path)
|
| 64 |
-
```
|
| 65 |
|
| 66 |
-
For more information on quantization, check the [OpenVINO model optimization guide](https://docs.openvino.ai/2025/openvino-workflow/model-optimization-guide/
|
| 67 |
|
| 68 |
|
| 69 |
## Compatibility
|
| 70 |
|
| 71 |
The provided OpenVINO™ IR model is compatible with:
|
| 72 |
|
| 73 |
-
* OpenVINO version 2025.
|
| 74 |
-
* Optimum Intel 1.
|
| 75 |
|
| 76 |
|
| 77 |
## Running Model Inference with [Optimum Intel](https://huggingface.co/docs/optimum/intel/index)
|
|
|
|
| 8 |
---
|
| 9 |
# bge-base-en-v1.5-int8-ov
|
| 10 |
|
|
|
|
|
|
|
|
|
|
| 11 |
* Model creator: [BAAI](https://huggingface.co/BAAI)
|
| 12 |
* Original model: [bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5)
|
| 13 |
|
|
|
|
| 19 |
|
| 20 |
## Quantization Parameters
|
| 21 |
|
| 22 |
+
Weight compression was performed using `nncf.compress_weights` with the following parameters:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 23 |
|
| 24 |
+
* mode: **INT8_ASYM**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 25 |
|
| 26 |
+
For more information on quantization, check the [OpenVINO model optimization guide](https://docs.openvino.ai/2025/openvino-workflow/model-optimization-guide/weight-compression.html).
|
| 27 |
|
| 28 |
|
| 29 |
## Compatibility
|
| 30 |
|
| 31 |
The provided OpenVINO™ IR model is compatible with:
|
| 32 |
|
| 33 |
+
* OpenVINO version 2025.3.0 and higher
|
| 34 |
+
* Optimum Intel 1.25.2 and higher
|
| 35 |
|
| 36 |
|
| 37 |
## Running Model Inference with [Optimum Intel](https://huggingface.co/docs/optimum/intel/index)
|