Model Card for ModernBERT-base-cos

ModernBERT-base-cos is a ModernBERT-based sequence classification model specifically fine-tuned to assess the quality of summaries in a QnA context. This model is designed to evaluate how well a generated summary captures essential information needed for question-answering tasks as part of research on the "chain of summaries" approach.

Model Details

Model Description

This model evaluates the quality and completeness of summaries by providing a quality score. It helps determine whether a summary adequately captures the information needed for downstream QnA tasks, making it useful for:

• Researchers working on summarization evaluation

• QnA pipeline optimization

• Educational applications requiring assessment of student-generated summaries

• Content creation platforms where summary quality is important

• Developed by: [More Information Needed]

• Funded by [optional]: [More Information Needed]

• Shared by [optional]: [More Information Needed]

• Model type: [More Information Needed]

• Language(s) (NLP): [More Information Needed]

• License: [More Information Needed]

• Finetuned from model [optional]: [More Information Needed]

Model Sources [optional]

• Repository: [More Information Needed]
• Paper [optional]: [More Information Needed]
• Demo [optional]: [More Information Needed]

How to Get Started with the Model

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

# Load model and tokenizer
model_id = "williambrach/ModernBERT-base-cos"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id).to("cuda")

def summary_score(
    tokenizer,
    summaries: list[str],
    device: str = "cuda",
    return_tensor: bool = True,
):
    inputs = tokenizer(
        summaries, return_tensors="pt", padding=True, truncation=True
    ).to(device)
    with torch.no_grad():
        outputs = model(**inputs)
        logits = torch.sigmoid(outputs["logits"])
        if return_tensor:
            logits = logits
        else:
            logits = logits.cpu().numpy().tolist()
    return logits

# Example
texts = [
    "test",
    "Michael Jackson was a famous singer and dancer.",
    "Michael Jackson was a famous singer.",
    "Michael Jackson was a famous dancer.",
]
scores = summary_score(tokenizer, texts, return_tensor=False)
print(scores)

Hardware

[More Information Needed]

Software

[More Information Needed]

Citation [optional]

BibTeX:

[More Information Needed]