yosefw/msmarco_1M_splade_index

Splade-Index

This is an index created with the splade-index library (version 0.2.0)

Installation

You can install the splade-index library with pip:

pip install "splade-index==0.2.0"

# Include extra dependencies like stemmer
pip install "splade-index[full]==0.2.0"

# For huggingface hub usage
pip install huggingface_hub

Load this Index

You can use the following code to load this SPLADE index from Hugging Face hub:

from sentence_transformers import SparseEncoder
from splade_index import SPLADE

# Download the SPLADE model that was used to create the index from the HuggingFace Hub
model_id = "naver/splade-v3-distilbert" # Enter the splade model id
model = SparseEncoder(model_id)

# Set your huggingface token if repo is private
repo_id = "yosefw/msmarco_1M_splade_index"

# Load a SPLADE index from the Hugging Face model hub
retriever = SPLADE.load_from_hub(repo_id, model=model)

Stats

This dataset was created using the following data:

Statistic	Value
Number of documents	980853
Number of tokens	188965703
Average tokens per document	192.65