Roadmap.md

🗺️ Tunisian NLP Leaderboard Roadmap

📌 Phase 1: Dataset Acquisition & Preparation

1. Sentiment Analysis

• Existing Dataset: TUNIZI

  • Description: A large dataset containing 100,000 Tunisian Arabizi comments annotated as positive, negative, or neutral.
  • Source: K4All Foundation

• Usage: Utilize this dataset to evaluate models' performance in sentiment classification tasks.

2. Named Entity Recognition (NER)

• Existing Dataset: ArabNER

  • Description: A comprehensive Arabic NER corpus that can be adapted for Tunisian dialects.
  • Source: ResearchGate

• Usage: Fine-tune models on this dataset to assess their ability to recognize entities in Tunisian Arabic text.

3. Corpus Coverage

• Existing Dataset: Tunisian Dialect Corpus

  • Description: A sizable collection of Tunisian dialect texts, useful for assessing vocabulary coverage.
  • Source: Hugging Face

• Usage: Evaluate models' coverage of the Tunisian dialect vocabulary using this corpus.

4. Arabizi Robustness

• Existing Dataset: TUNIZI

  • Description: Since it's in Arabizi, it can also serve to evaluate models' robustness to this writing style.
  • Source: K4All Foundation

• Usage: Assess models' robustness to Arabizi by evaluating their performance on this dataset.

5. Code-Switching

• Existing Dataset: TunSwitch

  • Description: A dataset of code-switched Tunisian Arabic speech, valuable for training and evaluating models on code-switching tasks.
  • Source: Zenodo

• Usage: Evaluate models' ability to handle code-switching between Tunisian Arabic and other languages using this dataset.

6. Typo Robustness

• Existing Dataset: TUNIZI

  • Description: Its informal nature includes typographical variations, making it suitable for evaluating models' tolerance to typos.
  • Source: K4All Foundation

• Usage: Assess models' robustness to typographical errors by evaluating their performance on this dataset.

7. Zero-Shot Transfer

• Existing Dataset: TUNIZI

  • Description: Can be used to test models' ability to generalize to tasks they weren't explicitly trained on.
  • Source: K4All Foundation

• Usage: Evaluate models' zero-shot transfer capabilities by assessing their performance on this dataset.

8. Domain Shift

• Existing Dataset: TUNIZI

  • Description: Its diverse sources provide a foundation for testing domain adaptation capabilities.
  • Source: K4All Foundation

• Usage: Assess models' ability to adapt to different domains by evaluating their performance on this dataset.

---

🧪 Phase 2: Metric Development & Evaluation Tasks

For each task, define the evaluation metric and the corresponding dataset:

Task	Metric	Dataset
Sentiment Analysis	Accuracy / F1 Score	TUNIZI
Named Entity Recognition	F1 Score	ArabNER
Corpus Coverage	Vocabulary Coverage (%)	Tunisian Dialect Corpus
Arabizi Robustness	Accuracy / F1 Score	TUNIZI
Code-Switching	Accuracy / F1 Score	TunSwitch
Typo Robustness	Accuracy / F1 Score	TUNIZI

Certainly! Here's a comprehensive roadmap to guide you through enhancing your TunisianEncoderModelsLeaderboard project, focusing on dataset acquisition, metric development, and evaluation tasks.

---

🗺️ Tunisian NLP Leaderboard Roadmap

📌 Phase 1: Dataset Acquisition & Preparation

1. Sentiment Analysis

• Existing Dataset: TUNIZI

  • Description: A large dataset containing 100,000 Tunisian Arabizi comments annotated as positive, negative, or neutral.
  • Source: K4All Foundation

• Usage: Utilize this dataset to evaluate models' performance in sentiment classification tasks.

2. Named Entity Recognition (NER)

• Existing Dataset: ArabNER

  • Description: A comprehensive Arabic NER corpus that can be adapted for Tunisian dialects.
  • Source: ResearchGate

• Usage: Fine-tune models on this dataset to assess their ability to recognize entities in Tunisian Arabic text.

3. Corpus Coverage

• Existing Dataset: Tunisian Dialect Corpus

  • Description: A sizable collection of Tunisian dialect texts, useful for assessing vocabulary coverage.
  • Source: Hugging Face

• Usage: Evaluate models' coverage of the Tunisian dialect vocabulary using this corpus.

4. Arabizi Robustness

• Existing Dataset: TUNIZI

  • Description: Since it's in Arabizi, it can also serve to evaluate models' robustness to this writing style.
  • Source: K4All Foundation

• Usage: Assess models' robustness to Arabizi by evaluating their performance on this dataset.

5. Code-Switching

• Existing Dataset: TunSwitch

  • Description: A dataset of code-switched Tunisian Arabic speech, valuable for training and evaluating models on code-switching tasks.
  • Source: Zenodo

• Usage: Evaluate models' ability to handle code-switching between Tunisian Arabic and other languages using this dataset.

6. Typo Robustness

• Existing Dataset: TUNIZI

  • Description: Its informal nature includes typographical variations, making it suitable for evaluating models' tolerance to typos.
  • Source: K4All Foundation

• Usage: Assess models' robustness to typographical errors by evaluating their performance on this dataset.

7. Zero-Shot Transfer

• Existing Dataset: TUNIZI

  • Description: Can be used to test models' ability to generalize to tasks they weren't explicitly trained on.
  • Source: K4All Foundation

• Usage: Evaluate models' zero-shot transfer capabilities by assessing their performance on this dataset.

8. Domain Shift

• Existing Dataset: TUNIZI

  • Description: Its diverse sources provide a foundation for testing domain adaptation capabilities.
  • Source: K4All Foundation

• Usage: Assess models' ability to adapt to different domains by evaluating their performance on this dataset.

---

🧪 Phase 2: Metric Development & Evaluation Tasks

For each task, define the evaluation metric and the corresponding dataset:

Task	Metric	Dataset
Sentiment Analysis	Accuracy / F1 Score	TUNIZI
Named Entity Recognition	F1 Score	ArabNER
Corpus Coverage	Vocabulary Coverage (%)	Tunisian Dialect Corpus
Arabizi Robustness	Accuracy / F1 Score	TUNIZI
Code-Switching	Accuracy / F1 Score	TunSwitch
Typo Robustness	Accuracy / F1 Score	TUNIZI
Zero-Shot Transfer	Accuracy / F1 Score	TUNIZI
Domain Shift	Accuracy / F1 Score	TUNIZI

---

🗂️ Suggested Folder Structure

To maintain organization and clarity, consider the following structure:

TunisianEncoderModelsLeaderboard/
├── datasets/
│   ├── sentiment/
│   │   └── tunizi.json
│   ├── ner/
│   │   └── arabner.json
│   ├── coverage/
│   │   └── tunisian_dialect_corpus.json
│   ├── arabizi_robustness/
│   │   └── tunizi.json
│   ├── code_switching/
│   │   └── tunswitch.json
│   ├── typo_robustness/
│   │   └── tunizi_with_typos.json
│   ├── zero_shot/
│   │   └── tunizi.json
│   └── domain_shift/
│       └── tunisian_domain_shift.json
├── scripts/
│   ├── preprocess.py
│   ├── evaluate.py
│   └── visualize.py
└── README.md

---

✅ Next Steps

1. Integrate Existing Datasets: Incorporate the datasets mentioned above into your repository, ensuring they are properly formatted and documented.

2. Develop Evaluation Scripts: Write scripts to evaluate models on each task, ensuring they are compatible with the leaderboard format.

3. Populate the Leaderboard: As models are evaluated, update the leaderboard to reflect their performance across tasks.

4. Documentation: Update the README.md file to provide clear instructions on how to use the leaderboard, contribute models, and interpret results.

---

If you need assistance with data collection, annotation guidelines, or script development, feel free to ask!