MonkeyDAnh/my-awesome-ai-detector-roberta-base-v4-human-vs-machine-finetune

my-awesome-ai-detector-roberta-base

Model Description

This model is a fine-tuned version of FacebookAI/roberta-base designed to detect whether a given text is generated by AI or written by a human. It has been sequentially fine-tuned distinct datasets to enhance its robustness and generalization capabilities across various text domains. The model aims to achieve high accuracy in distinguishing between human-produced and machine-generated content, making it a valuable tool for applications requiring AI text detection.

Intended Use: The model is intended for research and practical applications involving the detection of AI-generated text. It can be used as a component in systems for content moderation, academic integrity checks, or general content analysis.

Limitations and Bias:

• While trained on diverse datasets, the model's performance may vary on text generated by new or unseen AI models, or on highly domain-specific text not well-represented in the training data.
• Like all AI models, it may exhibit biases present in the training data. Further evaluation on diverse and representative datasets is recommended for critical applications.
• The model's predictions should be interpreted as probabilities and used as one signal among others, not as absolute truth.

Training Data

The model was sequentially fine-tuned on the following datasets:

1. artem9k/ai-text-detection-pile:

   • Subset Used: A total of 20,000 samples were utilized, specifically 10,000 rows from train-00000-of-00007-bc5952562e004d67.parquet and 10,000 rows from train-00006-of-00007-3d8a471ba0cf1c8d.parquet. This dataset contains a variety of text samples labeled as AI-generated or human-written.

2. Ateeqq/AI-and-Human-Generated-Text:

   • Subset Used: All available samples from this dataset were used. This dataset provides additional diverse text samples for robust AI detection.

3. NicolaiSivesind/human-vs-machine:

   • Subset Used: All available samples from this dataset were used, specifically from wiki-labeled.csv and research-abstracts-labeled.csv. This further enhances the model's ability to distinguish between human-produced and machine-generated content, particularly focusing on Wikipedia articles and research abstracts.

Training Procedure

Base Model

The model started with weights from FacebookAI/roberta-base.

Fine-tuning Strategy

The model was fine-tuned sequentially, meaning the output of one fine-tuning stage served as the starting point for the next.

Hyperparameters

The following hyperparameters were used during each fine-tuning phase:

• Learning Rate: 2e-5
• Per Device Train Batch Size: 8
• Per Device Eval Batch Size: 8
• Gradient Accumulation Steps: 2 (resulting in an effective batch size of 16)
• Number of Train Epochs: 3 (per fine-tuning phase)
• Weight Decay: 0.01

Hardware

All training was performed on a Google Colab GPU T4.

Training Times

• Fine-tuning on artem9k/ai-text-detection-pile: 1 hour 12 minutes 14 seconds
• Fine-tuning on Ateeqq/AI-and-Human-Generated-Text: 0 hours 30 minutes 36 seconds
• Fine-tuning on NicolaiSivesind/human-vs-machine: 1 hour 04 minutes 22 seconds

Overall Total Training Time: Approximately 2 hours 47 minutes 12 seconds across all fine-tuning stages.

Evaluation

The model's performance was evaluated on the validation splits of each respective dataset during its fine-tuning phase. The overall metrics reflect the performance of the final model after all sequential fine-tuning stages.

Metrics

• Eval Loss: 0.10
• Eval F1 Score: 0.972
• Train Loss: 0.20
• Train Samples per Second: 32
• Runtime (Evaluation): 33s

Usage

Inference

To use the model for inference, you can load it with the Hugging Face Transformers library:

from transformers import AutoModelForSequenceClassification, AutoTokenizer
import torch

# Define the model name/path
model_name = "MonkeyDAnh/my-awesome-ai-detector-roberta-base"

# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)

# Move model to GPU if available
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)

# Define label mapping (ensure this matches your model's config.id2label if available)
# Based on your ClassLabel definition (0: human-produced, 1: machine-generated)
id_to_label = model.config.id2label if hasattr(model.config, 'id2label') else {0: "human-produced", 1: "machine-generated"}

# Example text for inference
text = "This is a sample text to detect AI generation. It is written to sound very robotic and unnatural."

# Tokenize the input text
# Using max_length=512 for standard RoBERTa, adjust if your MAX_LENGTH_CHUNK was different
inputs = tokenizer(text, return_tensors="pt", truncation=True, padding=True, max_length=512).to(device)

# Perform inference
outputs = model(**inputs)

# Get predicted label ID and convert to integer
predictions_id = outputs.logits.argmax(-1).item()

# Map ID to the human-readable label
predicted_label = id_to_label[predictions_id]

print(f"Text: \"{text}\"")
print(f"Predicted Label ID: {predictions_id}")
print(f"Predicted Label: {predicted_label}")
# Expected output examples:
# Predicted Label: human-produced
# Predicted Label: machine-generated