Create README.md

README.md

@@ -0,0 +1,310 @@
+---
+license: apache-2.0
+language: ja
+tags:
+- audio
+- automatic-speech-recognition
+- hf-asr-leaderboard
+widget:
+- example_title: CommonVoice 8.0 (Test Split)
+  src: >-
+    https://huggingface.co/datasets/japanese-asr/ja_asr.common_voice_8_0/resolve/main/sample.flac
+- example_title: JSUT Basic 5000
+  src: >-
+    https://huggingface.co/datasets/japanese-asr/ja_asr.jsut_basic5000/resolve/main/sample.flac
+- example_title: ReazonSpeech (Test Split)
+  src: >-
+    https://huggingface.co/datasets/japanese-asr/ja_asr.reazonspeech_test/resolve/main/sample.flac
+pipeline_tag: automatic-speech-recognition
+metrics:
+- wer
+model-index:
+  - name: kotoba-tech/kotoba-whisper-v1.0
+    results:
+      - task:
+          type: automatic-speech-recognition
+        dataset:
+          name: CommonVoice_8.0 (Japanese)
+          type: japanese-asr/ja_asr.common_voice_8_0
+        metrics:
+          - name: WER
+            type: WER
+            value: 59.27
+          - name: CER
+            type: CER
+            value: 9.44
+      - task:
+          type: automatic-speech-recognition
+        dataset:
+          name: ReazonSpeech (Test)
+          type: japanese-asr/ja_asr.reazonspeech_test
+        metrics:
+          - name: WER
+            type: WER
+            value: 56.62
+          - name: CER
+            type: CER
+            value: 12.60
+      - task:
+          type: automatic-speech-recognition
+        dataset:
+          name: JSUT Basic5000
+          type: japanese-asr/ja_asr.jsut_basic5000
+        metrics:
+          - name: WER
+            type: WER
+            value: 64.36
+          - name: CER
+            type: CER
+            value: 8.48
+---
+
+# Kotoba-Whisper
+_Kotoba-Whisper_ is a collection of distilled [Whisper](https://arxiv.org/abs/2212.04356) models for Japanese ASR, developed through the collaboration bewteen
+[Asahi Ushio](https://asahiushio.com) and [Kotoba Technologies](https://twitter.com/kotoba_tech).
+Following the original work of distil-whisper ([Robust Knowledge Distillation via Large-Scale Pseudo Labelling](https://arxiv.org/abs/2311.00430)), 
+we employ OpenAI's [Whisper large-v3](https://huggingface.co/openai/whisper-large-v3) as the teacher model, and the student model consists the full encoder of the 
+teacher large-v3 model and the decoder with two layers initialized from the first and last layer of the large-v3 model.
+Kotoba-Whisper is **6.3x faster than large-v3**, while retaining as low error rate as the large-v3.
+
+As the initial version, we release ***kotoba-whisper-v1.0*** trained on the `large` subset of [ReazonSpeech](https://huggingface.co/datasets/reazon-research/reazonspeech) 
+(the largest speech-transcription paired dataset in Japanese extracted from Japanese TV audio recordings), 
+which amounts 1,253 hours of audio with 16,861,235 characters of transcriptions (5 sec audio with 18 text tokens in average) after 
+those transcriptions more than 10 WER are removed (see [WER Filter](https://huggingface.co/distil-whisper/distil-large-v3#wer-filter) for detail).
+The model was trained for 8 epochs with batch size 256 with sampling rate of 16kHz, and the training and evaluation code to reproduce kotoba-whisper is available at [https://github.com/kotoba-tech/kotoba-whisper](https://github.com/kotoba-tech/kotoba-whisper).
+
+Kotoba-whisper-v1.0 achieves better CER and WER than the [openai/whisper-large-v3](https://huggingface.co/openai/whisper-large-v3) in the in-domain held-out test set
+from ReazonSpeech, and achieves competitive CER and WER on the out-of-domain test sets including [JSUT basic 5000](https://sites.google.com/site/shinnosuketakamichi/publication/jsut) and
+the Japanese subset from [CommonVoice 8.0](https://huggingface.co/datasets/common_voice) (see [Evaluation](#evaluation) for detail).
+
+- ***CER***
+
+| Model                                                                                           | CommonVoice 8.0 (Japanese) | JSUT Basic 5000 | ReazonSpeech Test |
+|:------------------------------------------------------------------------------------------------|---------------------------:|----------------:|------------------:|
+| [**kotoba-tech/kotoba-whisper-v1.0**](https://huggingface.co/kotoba-tech/kotoba-whisper-v1.0)   |                   9.44 |        8.48 |         **12.60** |
+| [openai/whisper-large-v3](https://huggingface.co/openai/whisper-large-v3)                       |                       **8.52** |            **7.18** |             15.18 |
+| [openai/whisper-medium](https://huggingface.co/openai/whisper-medium)                           |                      11.34 |            9.87 |             29.56 |
+| [openai/whisper-small](https://huggingface.co/openai/whisper-small)                             |                      15.26 |           14.22 |             34.29 |
+| [openai/whisper-tiny](https://huggingface.co/openai/whisper-tiny)                               |                      46.86 |           35.69 |             96.69 |
+
+- ***WER***
+
+| Model                                                                                           | CommonVoice 8.0 (Japanese) | JSUT Basic 5000 | ReazonSpeech Test |
+|:------------------------------------------------------------------------------------------------|---------------------------:|----------------:|------------------:|
+| [**kotoba-tech/kotoba-whisper-v1.0**](https://huggingface.co/kotoba-tech/kotoba-whisper-v1.0)   |                  59.27 |       64.36 |         **56.62** |
+| [openai/whisper-large-v3](https://huggingface.co/openai/whisper-large-v3)                       |                      **55.41** |           **59.34** |             60.23 |
+| [openai/whisper-medium](https://huggingface.co/openai/whisper-medium)                           |                      63.64 |           69.52 |             76.04 |
+| [openai/whisper-small](https://huggingface.co/openai/whisper-small)                             |                      74.21 |           82.02 |             82.99 |
+| [openai/whisper-tiny](https://huggingface.co/openai/whisper-tiny)                               |                      93.78 |           97.72 |             94.85 |
+
+- ***Latency***: As kotoba-whisper uses the same architecture as [distil-whisper/distil-large-v3](https://huggingface.co/distil-whisper/distil-large-v3),
+it inherits the benefit of the improved latency compared to [openai/whisper-large-v3](https://huggingface.co/openai/whisper-large-v3) 
+(**6.3x faster than large-v3**, see the table below taken from [distil-whisper/distil-large-v3](https://huggingface.co/distil-whisper/distil-large-v3)).
+
+| Model                                                                                        | Params / M | Rel. Latency |
+|----------------------------------------------------------------------------------------------|------------|--------------|
+| **[kotoba-tech/kotoba-whisper-v1.0](https://huggingface.co/kotoba-tech/kotoba-whisper-v1.0)**| **756**    | **6.3**      |
+| [openai/whisper-large-v3](https://huggingface.co/openai/whisper-large-v3)                    | 1550       | 1.0          |
+
+
+## Transformers Usage
+Kotoba-Whisper is supported in the Hugging Face 🤗 Transformers library from version 4.39 onwards. To run the model, first 
+install the latest version of Transformers. 
+
+```bash
+pip install --upgrade pip
+pip install --upgrade transformers accelerate
+```
+
+### Short-Form Transcription
+The model can be used with the [`pipeline`](https://huggingface.co/docs/transformers/main_classes/pipelines#transformers.AutomaticSpeechRecognitionPipeline)
+class to transcribe short-form audio files (< 30-seconds) as follows:
+
+```python
+import torch
+from transformers import pipeline
+from datasets import load_dataset
+
+# config
+model_id = "kotoba-tech/kotoba-whisper-v1.0"
+torch_dtype = torch.bfloat16 if torch.cuda.is_available() else torch.float32
+device = "cuda:0" if torch.cuda.is_available() else "cpu"
+model_kwargs = {"attn_implementation": "sdpa"} if torch.cuda.is_available() else {}
+generate_kwargs = {"language": "japanese", "task": "transcribe"}
+
+# load model
+pipe = pipeline(
+    "automatic-speech-recognition",
+    model=model_id,
+    torch_dtype=torch_dtype,
+    device=device,
+    model_kwargs=model_kwargs
+)
+
+# load sample audio
+dataset = load_dataset("japanese-asr/ja_asr.reazonspeech_test", split="test")
+sample = dataset[0]["audio"]
+
+# run inference
+result = pipe(sample, generate_kwargs=generate_kwargs)
+print(result["text"])
+```
+
+- To transcribe a local audio file, simply pass the path to your audio file when you call the pipeline (make sure the audio is sampled in 16kHz):
+```diff
+- result = pipe(sample, generate_kwargs=generate_kwargs)
++ result = pipe("audio.mp3", generate_kwargs=generate_kwargs)
+```
+
+- For segment-level timestamps, pass the argument `return_timestamps=True` and return the `"chunks"` output:
+```python
+result = pipe(sample, return_timestamps=True, generate_kwargs=generate_kwargs)
+print(result["chunks"])
+```
+
+***Sequential Long-Form:*** Kotoba-whisper is designed to be compatible with OpenAI's sequential long-form transcription algorithm. This algorithm uses a sliding window for buffered 
+inference of long audio files (> 30-seconds), and returns more accurate transcriptions compared to the [chunked long-form algorithm](#chunked-long-form).
+As default, if long audio files are passed to the model, it will transcribes with the sequential long-form transcription.
+The sequential long-form algorithm should be used in either of the following scenarios:
+
+1. Transcription accuracy is the most important factor, and latency is less of a consideration
+2. You are transcribing **batches** of long audio files, in which case the latency of sequential is comparable to chunked, while being up to 0.5% WER more accurate
+
+If you are transcribing single long audio files and latency is the most important factor, you should use the chunked algorithm
+described [below](#chunked-long-form). For a detailed explanation of the different algorithms, refer to Sections 5 of 
+the [Distil-Whisper paper](https://arxiv.org/pdf/2311.00430.pdf). The [`pipeline`](https://huggingface.co/docs/transformers/main_classes/pipelines#transformers.AutomaticSpeechRecognitionPipeline) 
+class can be used to transcribe long audio files with the sequential algorithm as follows: 
+
+
+### Chunked Long-Form
+This algorithm should be used when a single large audio file is being transcribed and the fastest possible inference is required. In such circumstances, 
+the chunked algorithm is up to 9x faster than OpenAI's sequential long-form implementation (see Table 7 of the [Distil-Whisper paper](https://arxiv.org/pdf/2311.00430.pdf)).
+To enable chunking, pass the `chunk_length_s` parameter to the `pipeline`. For distil-large-v3, a chunk length of 25-seconds
+is optimal. To activate batching over long audio files, pass the argument `batch_size`:
+
+```python
+import torch
+from transformers import pipeline
+from datasets import load_dataset
+
+# config
+model_id = "kotoba-tech/kotoba-whisper-v1.0"
+torch_dtype = torch.bfloat16 if torch.cuda.is_available() else torch.float32
+device = "cuda:0" if torch.cuda.is_available() else "cpu"
+model_kwargs = {"attn_implementation": "sdpa"} if torch.cuda.is_available() else {}
+generate_kwargs = {"language": "japanese", "task": "transcribe"}
+
+# load model
+pipe = pipeline(
+    "automatic-speech-recognition",
+    model=model_id,
+    torch_dtype=torch_dtype,
+    device=device,
+    model_kwargs=model_kwargs,
+    chunk_length_s=15,
+    batch_size=16
+)
+
+# load sample audio (concatenate instances to create a long audio)
+dataset = load_dataset("japanese-asr/ja_asr.reazonspeech_test", split="test")
+sample = {"array": np.concatenate([i["array"] for i in dataset[:20]["audio"]]), "sampling_rate": dataset[0]['audio']['sampling_rate']}
+
+# run inference
+result = pipe(sample, generate_kwargs=generate_kwargs)
+print(result["text"])
+```
+
+
+### Additional Speed & Memory Improvements
+You can apply additional speed and memory improvements to further reduce the inference speed and VRAM 
+requirements. These optimisations primarily target the attention kernel, swapping it from an eager implementation to a 
+more efficient flash attention version.
+
+#### Flash Attention 2
+
+We recommend using [Flash-Attention 2](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#flashattention-2) 
+if your GPU allows for it. To do so, you first need to install [Flash Attention](https://github.com/Dao-AILab/flash-attention):
+
+```
+pip install flash-attn --no-build-isolation
+```
+
+Then pass `attn_implementation="flash_attention_2"` to `from_pretrained`:
+
+```diff
+- model_kwargs = {"attn_implementation": "sdpa"} if torch.cuda.is_available() else {}
++ model_kwargs = {"attn_implementation": "flash_attention_2"} if torch.cuda.is_available() else {}
+```
+
+
+## Model Details
+See [https://huggingface.co/distil-whisper/distil-large-v3#model-details](https://huggingface.co/distil-whisper/distil-large-v3#model-details).
+
+
+## Evaluation
+The following code-snippets demonstrates how to evaluate the kotoba-whisper model on the Japanese subset of the CommonVoice 8.0. 
+First, we need to install the required packages, including 🤗 Datasets to load the audio data, and 🤗 Evaluate to 
+perform the WER calculation:
+
+```bash
+pip install --upgrade pip
+pip install --upgrade transformers datasets[audio] evaluate jiwer
+```
+
+Evaluation can then be run end-to-end with the following example: 
+
+```python
+import torch
+from transformers import pipeline
+from datasets import load_dataset
+from evaluate import load
+from transformers.models.whisper.english_normalizer import BasicTextNormalizer
+
+# model config
+model_id = "kotoba-tech/kotoba-whisper-v1.0"
+torch_dtype = torch.bfloat16 if torch.cuda.is_available() else torch.float32
+device = "cuda:0" if torch.cuda.is_available() else "cpu"
+model_kwargs = {"attn_implementation": "sdpa"} if torch.cuda.is_available() else {}
+generate_kwargs = {"language": "japanese", "task": "transcribe"}
+normalizer = BasicTextNormalizer()
+
+# data config
+dataset_name = "japanese-asr/ja_asr.reazonspeech_test"
+audio_column = 'audio'
+text_column = 'transcription'
+
+# load model
+pipe = pipeline(
+    "automatic-speech-recognition",
+    model=model_id,
+    torch_dtype=torch_dtype,
+    device=device,
+    model_kwargs=model_kwargs,
+    batch_size=16
+)
+
+# load the dataset and sample the audio with 16kHz
+dataset = load_dataset(dataset_name, split="test")
+transcriptions = pipe(dataset['audio'])
+transcriptions = [normalizer(i['text']).replace(" ", "") for i in transcriptions]
+references = [normalizer(i).replace(" ", "") for i in dataset['transcription']]
+
+# compute the CER metric
+cer_metric = load("cer")
+cer = 100 * cer_metric.compute(predictions=transcriptions, references=references)
+print(cer)
+```
+
+The huggingface links to the major Japanese ASR datasets for evaluation are summarized at [here](https://huggingface.co/collections/japanese-asr/japanese-asr-evaluation-dataset-66051a03d6ca494d40baaa26).
+For example, to evaluate the model on JSUT Basic5000, change the `dataset_name`:
+
+```diff
+- dataset_name = "japanese-asr/ja_asr.reazonspeech_test"
++ dataset_name = "japanese-asr/ja_asr.jsut_basic5000"
+```
+
+## Acknowledgements
+* [OpenAI](https://openai.com/) for the Whisper [model](https://huggingface.co/openai/whisper-large-v3).
+* Hugging Face 🤗 [Transformers](https://github.com/huggingface/transformers) for the model integration.
+* Hugging Face 🤗 for the [Distil-Whisper codebase](https://github.com/huggingface/distil-whisper).
+* [Reazon Human Interaction Lab](https://research.reazon.jp/) for the [ReazonSpeech dataset](https://huggingface.co/datasets/reazon-research/reazonspeech).