ArtusDev
/

requests-exl

+---
+language: en
+tags:
+- exl2
+- exl3
+- quantization
+- requests
+- community
+---
+<style>
+  .container-dark {
+    font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", Arial, sans-serif;
+    line-height: 1.6;
+    background-color: #1e1e1e;
+    color: #d4d4d4;
+    padding: 20px;
+  }
+  .card-dark {
+    background-color: #252526;
+    border-radius: 12px;
+    padding: 24px;
+    margin-bottom: 20px;
+    box-shadow: 0 4px 12px rgba(0,0,0,0.3);
+    border: 1px solid #3c3c3c;
+  }
+  .card-dark h1 {
+    font-size: 2.2em;
+    color: #ffffff;
+    text-align: center;
+    margin-bottom: 10px;
+  }
+  .card-dark .subtitle {
+    text-align: center;
+    font-size: 1.1em;
+    color: #a0a0a0;
+  }
+  .card-dark h2 {
+    font-size: 1.5em;
+    margin-top: 0;
+    padding-bottom: 10px;
+    border-bottom: 1px solid #3c3c3c;
+    color: #c586c0;
+  }
+  .card-dark h3 {
+    font-size: 1.2em;
+    color: #d4d4d4;
+  }
+  .btn-purple {
+    display: inline-block;
+    background-color: #6A5ACD;
+    color: white !important;
+    padding: 12px 24px;
+    border-radius: 8px;
+    text-decoration: none;
+    font-weight: 600;
+    transition: background-color 0.3s ease, transform 0.2s ease;
+    text-align: center;
+  }
+  .btn-purple:hover {
+    background-color: #7B68EE;
+    transform: translateY(-2px);
+  }
+  .info-box-dark {
+    background-color: rgba(106, 90, 205, 0.1);
+    border-left: 5px solid #6A5ACD;
+    padding: 16px;
+    margin: 20px 0;
+    border-radius: 0 8px 8px 0;
+  }
+  code.inline-code-dark {
+    background-color: #3a3a3a;
+    padding: 3px 6px;
+    border-radius: 4px;
+    font-family: 'Fira Code', 'Courier New', monospace;
+    color: #4fc1ff;
+  }
+  .code-block-dark {
+    background-color: #1e1e1e;
+    color: #dcdcdc;
+    padding: 16px;
+    border-radius: 8px;
+    font-family: 'Fira Code', 'Courier New', monospace;
+    overflow-x: auto;
+    white-space: pre-wrap;
+    border: 1px solid #3c3c3c;
+  }
+  .code-block-dark .comment {
+    color: #6a9955;
+  }
+  a {
+    color: #569cd6;
+    text-decoration: none;
+  }
+  a:hover {
+    text-decoration: underline;
+  }
+  ul {
+    padding-left: 20px;
+  }
+  li {
+    margin-bottom: 8px;
+  }
+</style>
+<div class="container-dark">
+  <div class="card-dark">
+    <h1>EXL3 Quantization Requests</h1>
+    <p class="subtitle">Community hub for requesting EXL3 quants.</p>
+  </div>
+  <div class="card-dark">
+    <h2>How to Request a Quant</h2>
+    <p>To request a new model quant, please follow these steps:</p>
+    <ol>
+      <li><strong>Check Existing Quants:</strong> Before making a request, please check if an EXL3 quant already exists <a href="https://huggingface.co/models?other=exl3&sort=created" target="_blank">by exl3 tag</a> or <a href="https://huggingface.co/models?sort=created&search=exl3" target="_blank">by exl3 suffix</a>.</li>
+      <li><strong>Go to the Community Tab:</strong> Navigate to the Community Tab for this repository.</li>
+      <li><strong>Create a Model Topic:</strong> Start a new discussion with the model title. In the body, provide a direct HF link to the model you are requesting a quant for.</li>
+    </ol>
+    <div style="text-align: center; margin-top: 25px;">
+      <a href="https://huggingface.co/ArtusDev/requests-exl/discussions/new?title=[MODEL_NAME_HERE]&description=[MODEL_HF_LINK_HERE]" class="btn-purple" target="_blank">Request EXL3 Quant</a>
+    </div>
+    <div class="info-box-dark">
+      <p>Please note that not all requests can be fulfilled. The decision to quantize a model depends on available computing resources, model popularity, technical feasibility, and priority.</p>
+      <p>This is a personal, community-driven project. Your patience and understanding are appreciated ❤️.</p>
+    </div>
+  </div>
+  <div class="card-dark">
+    <h2>Can I Request EXL2 Quants?</h2>
+    <p>Being superior to EXL2 in every way (in terms of quantization quality and flexibility), EXL3 is the main target format for quantization. If you see a good reason for provisioning EXL2 quants - you can make a request with the reasoning why EXL2 should be considered for a particular model.</p>
+    <p>Keep in mind that among all quantization requests, EXL2 takes the lowest priority.</p>
+  </div>
+  <div class="card-dark">
+    <h2>About EXL3 Quantization</h2>
+    <p><strong>EXL3</strong> is a highly optimized quantization format based on QTIP designed for LLM inference on consumer GPUs. It is an evolution of the EXL2 format, offering higher quality within lower bitrates.</p>
+    <p>If you enjoy EXL quants, feel free to support <a href="https://github.com/turboderp-org/exllamav3" target="_blank"><b>EXL3 development</b></a> and a small cat working tirelessly behind it: <b>turboderp</b> (<a href="https://github.com/turboderp" target="_blank">GitHub</a>, <a href="https://ko-fi.com/turboderp" target="_blank">Ko-Fi</a>).</p>
+    <h3>Available Quantization Sizes</h3>
+    <p>To use resources optimally, quants are created in a fixed range of sizes. Custom sizes will only be considered if there is a high community demand and/or available compute.</p>
+    <ul>
+      <li><code class="inline-code-dark">2.5bpw_H6</code></li>
+      <li><code class="inline-code-dark">3.0bpw_H6</code></li>
+      <li><code class="inline-code-dark">3.5bpw_H6</code></li>
+      <li><code class="inline-code-dark">4.0bpw_H6</code></li>
+      <li><code class="inline-code-dark">4.5bpw_H6 (4.25bpw_H6 for 70b and above)</code></li>
+      <li><code class="inline-code-dark">5.0bpw_H6</code></li>
+      <li><code class="inline-code-dark">6.0bpw_H6</code></li>
+      <li><code class="inline-code-dark">8.0bpw_H8</code></li>
+    </ul>
+  </div>
+  <div class="card-dark">
+    <h2>How to Download and Use EXL Quants</h2>
+    <p>Each quantization size for a model is stored in a separate HF repository branch. You can download a specific quant size by its branch.</p>
+    <p>For example, to download the <code class="inline-code-dark">4.0bpw_H6</code> quant:</p>
+    <div class="code-block-dark">
+      <span class="comment"># Replace MODEL_NAME with the correct quant repository name</span>
+      <br>
+      git clone -b 4.0_6 https://huggingface.co/YOUR_USERNAME/ModelName
+      huggingface-cli download ArtusDev/MODEL_NAME --revision "4.0bpw_H6" --local-dir ./
+    </div>
+    <p style="margin-top: 15px;">These quants can be run with any inference client that supports the EXL3 format, such as <a href="https://github.com/theroyallab/tabbyapi" target="_blank"><b>TabbyAPI</b></a>. Please refer to <a href="https://github.com/theroyallab/tabbyAPI/wiki/01.-Getting-Started" target="_blank">documentation</a> for set up instructions.</p>
+  </div>
+  <div class="card-dark">
+    <h2>Other EXL3 Quanters</h2>
+    <p>If you don't find the model quant you're looking for, please check these other excellent community members who also provide EXL3 quants:</p>
+    <ul>
+      <li><a href="https://huggingface.co/turboderp" target="_blank"><b>@turboderp</b></a></li>
+      <li><a href="https://huggingface.co/bullerwins" target="_blank"><b>@bullerwins</b></a></li>
+      <li><a href="https://huggingface.co/MikeRoz" target="_blank"><b>@MikeRoz</b></a></li>
+    </ul>
+  </div>
+</div>