Spaces:
Runtime error
Runtime error
File size: 12,048 Bytes
bde1c71 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 |
## 🗺️ Tunisian NLP Leaderboard Roadmap
### 📌 Phase 1: Dataset Acquisition & Preparation
#### 1. **Sentiment Analysis**
* **Existing Dataset**: **TUNIZI**
* **Description**: A large dataset containing 100,000 Tunisian Arabizi comments annotated as positive, negative, or neutral.
* **Source**: [K4All Foundation](https://k4all.org/project/database-tunisian-arabizi/)
* **Usage**: Utilize this dataset to evaluate models' performance in sentiment classification tasks.
#### 2. **Named Entity Recognition (NER)**
* **Existing Dataset**: **ArabNER**
* **Description**: A comprehensive Arabic NER corpus that can be adapted for Tunisian dialects.
* **Source**: [ResearchGate](https://www.researchgate.net/publication/374279027_Named_Entity_Recognition_of_Tunisian_Arabic_Using_the_Bi-LSTM-CRF_Model)
* **Usage**: Fine-tune models on this dataset to assess their ability to recognize entities in Tunisian Arabic text.
#### 3. **Corpus Coverage**
* **Existing Dataset**: **Tunisian Dialect Corpus**
* **Description**: A sizable collection of Tunisian dialect texts, useful for assessing vocabulary coverage.
* **Source**: [Hugging Face](https://huggingface.co/collections/tunis-ai/arabic-datasets-66344cf0df31dc81eb1dcf55)
* **Usage**: Evaluate models' coverage of the Tunisian dialect vocabulary using this corpus.
#### 4. **Arabizi Robustness**
* **Existing Dataset**: **TUNIZI**
* **Description**: Since it's in Arabizi, it can also serve to evaluate models' robustness to this writing style.
* **Source**: [K4All Foundation](https://k4all.org/project/database-tunisian-arabizi/)
* **Usage**: Assess models' robustness to Arabizi by evaluating their performance on this dataset.
#### 5. **Code-Switching**
* **Existing Dataset**: **TunSwitch**
* **Description**: A dataset of code-switched Tunisian Arabic speech, valuable for training and evaluating models on code-switching tasks.
* **Source**: [Zenodo](https://zenodo.org/records/8342762)
* **Usage**: Evaluate models' ability to handle code-switching between Tunisian Arabic and other languages using this dataset.
#### 6. **Typo Robustness**
* **Existing Dataset**: **TUNIZI**
* **Description**: Its informal nature includes typographical variations, making it suitable for evaluating models' tolerance to typos.
* **Source**: [K4All Foundation](https://k4all.org/project/database-tunisian-arabizi/)
* **Usage**: Assess models' robustness to typographical errors by evaluating their performance on this dataset.
#### 7. **Zero-Shot Transfer**
* **Existing Dataset**: **TUNIZI**
* **Description**: Can be used to test models' ability to generalize to tasks they weren't explicitly trained on.
* **Source**: [K4All Foundation](https://k4all.org/project/database-tunisian-arabizi/)
* **Usage**: Evaluate models' zero-shot transfer capabilities by assessing their performance on this dataset.
#### 8. **Domain Shift**
* **Existing Dataset**: **TUNIZI**
* **Description**: Its diverse sources provide a foundation for testing domain adaptation capabilities.
* **Source**: [K4All Foundation](https://k4all.org/project/database-tunisian-arabizi/)
* **Usage**: Assess models' ability to adapt to different domains by evaluating their performance on this dataset.
---
### 🧪 Phase 2: Metric Development & Evaluation Tasks
For each task, define the evaluation metric and the corresponding dataset:
| Task | Metric | Dataset |
| ------------------------ | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
| Sentiment Analysis | Accuracy / F1 Score | [TUNIZI](https://k4all.org/project/database-tunisian-arabizi/) |
| Named Entity Recognition | F1 Score | [ArabNER](https://www.researchgate.net/publication/374279027_Named_Entity_Recognition_of_Tunisian_Arabic_Using_the_Bi-LSTM-CRF_Model) |
| Corpus Coverage | Vocabulary Coverage (%) | [Tunisian Dialect Corpus](https://huggingface.co/collections/tunis-ai/arabic-datasets-66344cf0df31dc81eb1dcf55) |
| Arabizi Robustness | Accuracy / F1 Score | [TUNIZI](https://k4all.org/project/database-tunisian-arabizi/) |
| Code-Switching | Accuracy / F1 Score | [TunSwitch](https://zenodo.org/records/8342762) |
| Typo Robustness | Accuracy / F1 Score | [TUNIZI]([https://k4all.org/project/database-tunisian](https://k4all.org/project/database-tunisian) |
Certainly! Here's a comprehensive roadmap to guide you through enhancing your **TunisianEncoderModelsLeaderboard** project, focusing on dataset acquisition, metric development, and evaluation tasks.
---
## 🗺️ Tunisian NLP Leaderboard Roadmap
### 📌 Phase 1: Dataset Acquisition & Preparation
#### 1. **Sentiment Analysis**
* **Existing Dataset**: **TUNIZI**
* **Description**: A large dataset containing 100,000 Tunisian Arabizi comments annotated as positive, negative, or neutral.
* **Source**: [K4All Foundation](https://k4all.org/project/database-tunisian-arabizi/)
* **Usage**: Utilize this dataset to evaluate models' performance in sentiment classification tasks.
#### 2. **Named Entity Recognition (NER)**
* **Existing Dataset**: **ArabNER**
* **Description**: A comprehensive Arabic NER corpus that can be adapted for Tunisian dialects.
* **Source**: [ResearchGate](https://www.researchgate.net/publication/374279027_Named_Entity_Recognition_of_Tunisian_Arabic_Using_the_Bi-LSTM-CRF_Model)
* **Usage**: Fine-tune models on this dataset to assess their ability to recognize entities in Tunisian Arabic text.
#### 3. **Corpus Coverage**
* **Existing Dataset**: **Tunisian Dialect Corpus**
* **Description**: A sizable collection of Tunisian dialect texts, useful for assessing vocabulary coverage.
* **Source**: [Hugging Face](https://huggingface.co/collections/tunis-ai/arabic-datasets-66344cf0df31dc81eb1dcf55)
* **Usage**: Evaluate models' coverage of the Tunisian dialect vocabulary using this corpus.
#### 4. **Arabizi Robustness**
* **Existing Dataset**: **TUNIZI**
* **Description**: Since it's in Arabizi, it can also serve to evaluate models' robustness to this writing style.
* **Source**: [K4All Foundation](https://k4all.org/project/database-tunisian-arabizi/)
* **Usage**: Assess models' robustness to Arabizi by evaluating their performance on this dataset.
#### 5. **Code-Switching**
* **Existing Dataset**: **TunSwitch**
* **Description**: A dataset of code-switched Tunisian Arabic speech, valuable for training and evaluating models on code-switching tasks.
* **Source**: [Zenodo](https://zenodo.org/records/8342762)
* **Usage**: Evaluate models' ability to handle code-switching between Tunisian Arabic and other languages using this dataset.
#### 6. **Typo Robustness**
* **Existing Dataset**: **TUNIZI**
* **Description**: Its informal nature includes typographical variations, making it suitable for evaluating models' tolerance to typos.
* **Source**: [K4All Foundation](https://k4all.org/project/database-tunisian-arabizi/)
* **Usage**: Assess models' robustness to typographical errors by evaluating their performance on this dataset.
#### 7. **Zero-Shot Transfer**
* **Existing Dataset**: **TUNIZI**
* **Description**: Can be used to test models' ability to generalize to tasks they weren't explicitly trained on.
* **Source**: [K4All Foundation](https://k4all.org/project/database-tunisian-arabizi/)
* **Usage**: Evaluate models' zero-shot transfer capabilities by assessing their performance on this dataset.
#### 8. **Domain Shift**
* **Existing Dataset**: **TUNIZI**
* **Description**: Its diverse sources provide a foundation for testing domain adaptation capabilities.
* **Source**: [K4All Foundation](https://k4all.org/project/database-tunisian-arabizi/)
* **Usage**: Assess models' ability to adapt to different domains by evaluating their performance on this dataset.
---
### 🧪 Phase 2: Metric Development & Evaluation Tasks
For each task, define the evaluation metric and the corresponding dataset:
| Task | Metric | Dataset |
| ------------------------ | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
| Sentiment Analysis | Accuracy / F1 Score | [TUNIZI](https://k4all.org/project/database-tunisian-arabizi/) |
| Named Entity Recognition | F1 Score | [ArabNER](https://www.researchgate.net/publication/374279027_Named_Entity_Recognition_of_Tunisian_Arabic_Using_the_Bi-LSTM-CRF_Model) |
| Corpus Coverage | Vocabulary Coverage (%) | [Tunisian Dialect Corpus](https://huggingface.co/collections/tunis-ai/arabic-datasets-66344cf0df31dc81eb1dcf55) |
| Arabizi Robustness | Accuracy / F1 Score | [TUNIZI](https://k4all.org/project/database-tunisian-arabizi/) |
| Code-Switching | Accuracy / F1 Score | [TunSwitch](https://zenodo.org/records/8342762) |
| Typo Robustness | Accuracy / F1 Score | [TUNIZI](https://k4all.org/project/database-tunisian-arabizi/) |
| Zero-Shot Transfer | Accuracy / F1 Score | [TUNIZI](https://k4all.org/project/database-tunisian-arabizi/) |
| Domain Shift | Accuracy / F1 Score | [TUNIZI](https://k4all.org/project/database-tunisian-arabizi/) |
---
### 🗂️ Suggested Folder Structure
To maintain organization and clarity, consider the following structure:
```
TunisianEncoderModelsLeaderboard/
├── datasets/
│ ├── sentiment/
│ │ └── tunizi.json
│ ├── ner/
│ │ └── arabner.json
│ ├── coverage/
│ │ └── tunisian_dialect_corpus.json
│ ├── arabizi_robustness/
│ │ └── tunizi.json
│ ├── code_switching/
│ │ └── tunswitch.json
│ ├── typo_robustness/
│ │ └── tunizi_with_typos.json
│ ├── zero_shot/
│ │ └── tunizi.json
│ └── domain_shift/
│ └── tunisian_domain_shift.json
├── scripts/
│ ├── preprocess.py
│ ├── evaluate.py
│ └── visualize.py
└── README.md
```
---
### ✅ Next Steps
1. **Integrate Existing Datasets**: Incorporate the datasets mentioned above into your repository, ensuring they are properly formatted and documented.
2. **Develop Evaluation Scripts**: Write scripts to evaluate models on each task, ensuring they are compatible with the leaderboard format.
3. **Populate the Leaderboard**: As models are evaluated, update the leaderboard to reflect their performance across tasks.
4. **Documentation**: Update the README.md file to provide clear instructions on how to use the leaderboard, contribute models, and interpret results.
---
If you need assistance with data collection, annotation guidelines, or script development, feel free to ask!
|