mistralai

/

Mistral-7B-Instruct-v0.1

like 1.81k

Follow

Mistral AI_ 12.9k

@codegood Have you found a fix? I have been struggling with the same error on Azure ML Studio. Thanks

No, not yet. Looking to get some update from the developers.

in your bnb_config remove load_in_8bit=true. or whatever bit you are trying to use.

@omidai then the model will be too big to load. That's the power of BnB.

correct, in my case, I just started with a fresh environment and installed all packages again then It was resolved. It seems there is a hidden package dependency.

@omida I've attempted this a few times using the latest packages, but it still doesn't seem to work for me. Could you please share the packages versions you're using?

cc @ybelkada

hi everyone,
To benefit from quantization load_in_8bit or load_in_4bit , please make sure to install the latest bitsandbytes package:

pip install -U bitsandbytes

I would also advise to use load_in_4bit for a more memory efficient inference. Let us know how that goes

I've attempted today again to install the latest package versions (accelerate==0.23.0 and bitsandbytes==0.41.1), but it still doesn't seem to be working, neither in 4-bit nor in 8-bit.

Hi @Ayoba
I see OK - can you share a full reproducible snippet here?

Hi @ybelkada , I'm also facing the same issue. Please help with the fix :

Steps to recreate:

!pip install -U bitsandbytes
!pip install -U git+https://github.com/huggingface/transformers.git
!pip install -U git+https://github.com/huggingface/peft.git
!pip install -U git+https://github.com/huggingface/accelerate.git
!pip install -U datasets scipy ipywidgets matplotlib

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig

base_model_id = "mistralai/Mistral-7B-v0.1"
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16
)

model = AutoModelForCausalLM.from_pretrained(base_model_id, quantization_config=bnb_config)

ERROR :
```bash

ImportError Traceback (most recent call last)
Cell In[1], line 12
4 base_model_id = "mistralai/Mistral-7B-v0.1"
5 bnb_config = BitsAndBytesConfig(
6 load_in_4bit=True,
7 bnb_4bit_use_double_quant=True,
8 bnb_4bit_quant_type="nf4",
9 bnb_4bit_compute_dtype=torch.bfloat16
10 )
---> 12 model = AutoModelForCausalLM.from_pretrained(base_model_id, quantization_config=bnb_config)

File ~/transformers/src/transformers/models/auto/auto_factory.py:565, in _BaseAutoModelClass.from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
563 elif type(config) in cls._model_mapping.keys():
564 model_class = _get_model_class(config, cls._model_mapping)
--> 565 return model_class.from_pretrained(
566 pretrained_model_name_or_path, *model_args, config=config, **hub_kwargs, **kwargs
567 )
568 raise ValueError(
569 f"Unrecognized configuration class {config.class} for this kind of AutoModel: {cls.name}.\n"
570 f"Model type should be one of {', '.join(c.name for c in cls._model_mapping.keys())}."
571 )

File ~/transformers/src/transformers/modeling_utils.py:2634, in PreTrainedModel.from_pretrained(cls, pretrained_model_name_or_path, config, cache_dir, ignore_mismatched_sizes, force_download, local_files_only, token, revision, use_safetensors, *model_args, **kwargs)
2632 if load_in_8bit or load_in_4bit:
2633 if not (is_accelerate_available() and is_bitsandbytes_available()):
-> 2634 raise ImportError(
2635 "Using load_in_8bit=True requires Accelerate: pip install accelerate and the latest version of"
2636 " bitsandbytes pip install -i https://test.pypi.org/simple/ bitsandbytes or"
2637 " pip install bitsandbytes" 2638 ) 2640 if torch_dtype is None: 2641 # We force thedtypeto be float16, this is a requirement frombitsandbytes 2642 logger.info( 2643 f"Overriding torch_dtype={torch_dtype} withtorch_dtype=torch.float16due to " 2644 "requirements ofbitsandbytes` to enable model loading in 8-bit or 4-bit. "
2645 "Pass your own torch_dtype to specify the dtype of the remaining non-linear layers or pass"
2646 " torch_dtype=torch.float16 to remove this warning."
2647 )

ImportError: Using load_in_8bit=True requires Accelerate: pip install accelerate and the latest version of bitsandbytes pip install -i https://test.pypi.org/simple/ bitsandbytes or pip install bitsandbytes`
```

Hi @sukkritsharma

I just ran :

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig

base_model_id = "mistralai/Mistral-7B-v0.1"
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16
)

model = AutoModelForCausalLM.from_pretrained(base_model_id, quantization_config=bnb_config)
tokenizer = AutoTokenizer.from_pretrained(base_model_id)

text = "Hello my name is"
input_ids = tokenizer.encode(text, return_tensors="pt").to(0)

output = model.generate(input_ids, do_sample=False, max_length=10)
print(tokenizer.decode(output[0], skip_special_tokens=True))

On a fresh new environment and it seemed to work fine on my end. Can you re-try everything on a fresh new environment?

On Windows, I had the same problem. If you try:

torch.cuda.is_available()

it would show as False because the cuda version it needs was different from the cuda version that pytorch uses.

Hence, I found this solution from here: https://github.com/TimDettmers/bitsandbytes/issues/793#issuecomment-1806979077

Instead of installing bitsandbytes from the latest pip package, install it using the following command:

python.exe -m pip install https://github.com/TimDettmers/bitsandbytes/releases/download/0.41.0/bitsandbytes-0.41.0-py3-none-any.whl

Hope this helps