SindBERT, the Sailor: Charting the Seas of Turkish NLP
Paper
•
2510.21364
•
Published
•
1
AI & ML interests
Pretraining Historical Multilingual Language Models
Recent Activity
View all activity
stefan-it
authored
a
paper
about 10 hours ago
stefan-it
authored
a
paper
11 days ago
stefan-it
authored
a
paper
about 2 months ago
Post
3573
Wohoo 🥳 I have finished my 2025 GPU workstation build and I am very excited to train new awesome open source models on it.
I built my last GPU workstation 5 years ago featuring an AMD Ryzen 5900X, 64GB of G.SKILL Trident Z RGB on an ASRock X570 Taichi cooled by an Alphacool Eisbär 420. GPU was a Zotac RTX 3090 AMP Extreme. Unfortunately, I was never satisfied with the case - some Fractal Define 7, as it is definitely too small, airflow is not optimal as I had to open the front door all the time and it also arrived with a partly damaged side panel.
For my new build, I've used the following components: an outstanding new AMD Ryzen 9950X3D with 64GB of Corsair Dominator Titanium (what a name). As a huge Noctua fan - warm greetings to my Austrian neighbors - I am using the brand new Noctua NH-D15 G2 on an ASRock X870E Taichi in an amazing Lian Li LANCOOL III chassis. One joke that only NVIDIA Blackwell users will understand: you definitely need a tempered glass panel to check if your GPU cables/connectors start melting 😂 And the best is yet to come: I returned my previously bought Zotac RTX 5090 Solid to the eBay seller (because of... missing ROPs, only NVIDIA Blackwell users will again understand) and bought a Zotac 5090 AMP Extreme INFINITY (yes, the long name indicates that this is the flagship model from Zotac) from a more trustworthy source (NBB in Germany).
I am so happy to start training and fine-tuning new open source models - stay tuned!!!
I built my last GPU workstation 5 years ago featuring an AMD Ryzen 5900X, 64GB of G.SKILL Trident Z RGB on an ASRock X570 Taichi cooled by an Alphacool Eisbär 420. GPU was a Zotac RTX 3090 AMP Extreme. Unfortunately, I was never satisfied with the case - some Fractal Define 7, as it is definitely too small, airflow is not optimal as I had to open the front door all the time and it also arrived with a partly damaged side panel.
For my new build, I've used the following components: an outstanding new AMD Ryzen 9950X3D with 64GB of Corsair Dominator Titanium (what a name). As a huge Noctua fan - warm greetings to my Austrian neighbors - I am using the brand new Noctua NH-D15 G2 on an ASRock X870E Taichi in an amazing Lian Li LANCOOL III chassis. One joke that only NVIDIA Blackwell users will understand: you definitely need a tempered glass panel to check if your GPU cables/connectors start melting 😂 And the best is yet to come: I returned my previously bought Zotac RTX 5090 Solid to the eBay seller (because of... missing ROPs, only NVIDIA Blackwell users will again understand) and bought a Zotac 5090 AMP Extreme INFINITY (yes, the long name indicates that this is the flagship model from Zotac) from a more trustworthy source (NBB in Germany).
I am so happy to start training and fine-tuning new open source models - stay tuned!!!
- 2 replies
Post
1030
🇹🇷 😍 I'm very happy to finally announce my new Turkish LM called "BERT5urk":
stefan-it/bert5urk
It is a 1.42B T5-based model, trained with UL2 pretraining objective on the Turkish part of the awesome HuggingFaceFW/fineweb-2 dataset.
Feel free to check it out!
stefan-it/bert5urk
It is a 1.42B T5-based model, trained with UL2 pretraining objective on the Turkish part of the awesome HuggingFaceFW/fineweb-2 dataset.
Feel free to check it out!
-
- 1 reply
Post
3294
After running some 3DMark and FurMark benchmarks on Windows to make sure that my new 5090 is not causing melting cables [1] and some nice shots with a thermal camera (I don't think that's too much), running some fine-tuning experiments with my favorite Flair & Transformers libraries are very easy to perform.
Important steps:
Good idea is to start with a fresh Ubuntu 24.04 installation with latest CUDA 12.8 and the open NVIDIA driver - follow more advices from [2]:
I tried update from an existing Ubuntu installation with an older CUDA and driver version and it resulted in a non-startable system.
If you are using PyTorch 2.6 with built CUDA 12.6 it will result in:
But no worries! For PyTorch you need just to use a nightly 2.7 version that was built with CUDA 12.8. This can easily done via:
After that the latest Flair version can be installed and fine-tuning will work!
References:
[1]: https://www.reddit.com/r/nvidia/comments/1inpox7/rtx_50_series_12vhpwr_megathread/
[2]: https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&Distribution=Ubuntu&target_version=24.04&target_type=deb_network
Important steps:
Good idea is to start with a fresh Ubuntu 24.04 installation with latest CUDA 12.8 and the open NVIDIA driver - follow more advices from [2]:
sudo apt -y install cuda-toolkit-12-8 nvidia-openI tried update from an existing Ubuntu installation with an older CUDA and driver version and it resulted in a non-startable system.
If you are using PyTorch 2.6 with built CUDA 12.6 it will result in:
NVIDIA Graphics Device with CUDA capability sm_120 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_50 sm_60 sm_70 sm_75 sm_80 sm_86 sm_90.But no worries! For PyTorch you need just to use a nightly 2.7 version that was built with CUDA 12.8. This can easily done via:
pip install --pre torch --index-url https://download.pytorch.org/whl/nightly/cu128After that the latest Flair version can be installed and fine-tuning will work!
References:
[1]: https://www.reddit.com/r/nvidia/comments/1inpox7/rtx_50_series_12vhpwr_megathread/
[2]: https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&Distribution=Ubuntu&target_version=24.04&target_type=deb_network
- 1 reply
Post
5179
She arrived 😍
[Expect more models soon...]
[Expect more models soon...]
-
- 2 replies
Post
1586
My latest project is the outcome of the last 2+ years working with TPUs from the amazing TPU Research Cloud (TRC) program and training Encoder-only LMs with the TensorFlow Model Garden library.
👉 Link: https://github.com/stefan-it/model-garden-lms
An overview of some features:
- Cheatsheet for setting-up a TPU VM Pod (with all necessary dependencies) to pretrain LMs with TF Model Garden
- Conversion scripts that convert TF Model Garden weights to Hugging Face Transformers-compatible models
- Supported architectures include BERT, BERT with Token Dropping and TEAMS
I also released BERT-based models pretrained on the great Hugging Face FineWeb and FineWeb-Edu datasets (10BT subset). With more to come!
👉 Model Hub Link:
model-garden-lms
If you find these resources useful, please give them a like!
Made from Bavarian Oberland with ❤️ and 🥨.
👉 Link: https://github.com/stefan-it/model-garden-lms
An overview of some features:
- Cheatsheet for setting-up a TPU VM Pod (with all necessary dependencies) to pretrain LMs with TF Model Garden
- Conversion scripts that convert TF Model Garden weights to Hugging Face Transformers-compatible models
- Supported architectures include BERT, BERT with Token Dropping and TEAMS
I also released BERT-based models pretrained on the great Hugging Face FineWeb and FineWeb-Edu datasets (10BT subset). With more to come!
👉 Model Hub Link:
model-garden-lms
If you find these resources useful, please give them a like!
Made from Bavarian Oberland with ❤️ and 🥨.
stefan-it
updated
11
models
about 2 years ago
hmbert-tiny/flair-hipe-2022-topres19th-en
Token Classification
•
Updated
•
4
hmbert-tiny/flair-hipe-2022-newseye-sv
Token Classification
•
Updated
•
6
hmbert-tiny/flair-hipe-2022-newseye-fr
Token Classification
•
Updated
•
4
hmbert-tiny/flair-hipe-2022-newseye-fi
Token Classification
•
Updated
•
7
hmbert-tiny/flair-hipe-2022-newseye-de
Token Classification
•
Updated
•
6
hmbert-tiny/flair-hipe-2022-letemps-fr
Token Classification
•
Updated
•
7
hmbert-tiny/flair-icdar-nl
Token Classification
•
Updated
•
3
hmbert-tiny/flair-icdar-fr
Token Classification
•
Updated
•
5
hmbert-tiny/flair-hipe-2022-hipe2020-fr
Token Classification
•
Updated
•
4
hmbert-tiny/flair-hipe-2022-hipe2020-de
Token Classification
•
Updated
•
4
hmbert-tiny/flair-hipe-2022-ajmc-fr
Token Classification
•
Updated
•
10