README.md

---
language: en
tags:
- exl2
- exl3
- quantization
- requests
- community
---
<style>
  .container-dark {
    font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", Arial, sans-serif;
    line-height: 1.6;
    color: #d4d4d4;
  }
  .card-dark {
    background-color: #252526;
    border-radius: 12px;
    padding: 24px;
    margin-bottom: 20px;
    box-shadow: 0 4px 12px rgba(0,0,0,0.3);
    border: 1px solid #3c3c3c;
  }
  .card-dark.card-dark-title h1 {
    font-size: 1.5em;
    color: #ffffff;
    text-align: center;
    margin-bottom: 10px;
  }
  .card-dark h1 {
    font-size: 2.2em;
    color: #ffffff;
    text-align: center;
    margin-bottom: 10px;
  }
  .card-dark .subtitle {
    text-align: center;
    font-size: 1.1em;
    color: #a0a0a0;
  }
  .card-dark h2 {
    font-size: 1.5em;
    margin-top: 0;
    padding-bottom: 10px;
    border-bottom: 1px solid #3c3c3c;
    color: #c586c0;
  }
  .card-dark h3 {
    font-size: 1.2em;
    color: #d4d4d4;
  }
  .btn-purple {
    display: inline-block;
    background-color: #6A5ACD;
    color: white !important;
    padding: 12px 24px;
    border-radius: 8px;
    text-decoration: none;
    font-weight: 600;
    transition: background-color 0.3s ease, transform 0.2s ease;
    text-align: center;
  }
  .btn-purple:hover {
    background-color: #7B68EE;
    transform: translateY(-2px);
  }
  .info-box-dark {
    background-color: rgba(106, 90, 205, 0.1);
    border-left: 5px solid #6A5ACD;
    padding: 16px;
    margin: 20px 0;
    border-radius: 0 8px 8px 0;
  }
  code.inline-code-dark {
    background-color: #3a3a3a;
    padding: 3px 6px;
    border-radius: 4px;
    font-family: 'Fira Code', 'Courier New', monospace;
    color: #c586c0;
  }
  a {
    color: #569cd6;
    text-decoration: none;
    font-weight: 600;
  }
  a:hover {
    text-decoration: underline;
  }
  ul {
    padding-left: 20px;
  }
  li {
    margin-bottom: 8px;
  }
</style>

<div class="container-dark">

  <div class="card-dark card-dark-title">
    <h1>EXL3 Quantization Requests</h1>
    <p class="subtitle">Community hub for requesting EXL3 quants.</p>
  </div>

  <div class="card-dark">
    <h2>How to Request a Quant</h2>
    <p>To request a new model quant, please follow these steps:</p>
    <ol>
      <li><strong>Check Existing Quants:</strong> Before making a request, please check if an EXL3 quant already exists <a href="https://huggingface.co/models?other=exl3&sort=created" target="_blank">by exl3 tag</a> or <a href="https://huggingface.co/models?sort=created&search=exl3" target="_blank">by exl3 suffix</a>.</li>
      <li><strong>Go to the Community Tab:</strong> Navigate to the Community Tab for this repository.</li>
      <li><strong>Create a Model Topic:</strong> Start a new discussion with the model title. In the body, provide a direct HF link to the model you are requesting a quant for.</li>
    </ol>
    <div style="text-align: center; margin-top: 25px;">
      <a href="https://huggingface.co/ArtusDev/requests-exl/discussions/new?title=[MODEL_NAME_HERE]&description=[MODEL_HF_LINK_HERE]" class="btn-purple" target="_blank">Request EXL3 Quant</a>
    </div>
    <div class="info-box-dark">
      <p>Please note that not all requests can be fulfilled. The decision to quantize a model depends on available computing resources, model popularity, technical feasibility, and priority.</p>
      <p>This is a personal, community-driven project. Your patience and understanding are appreciated ❤️.</p>
    </div>
  </div>

  <div class="card-dark">
    <h2>Can I Request EXL2 Quants?</h2>
    <p>Being superior to EXL2 in every way (in terms of quantization quality and flexibility), EXL3 is the main target format for quantization. If you see a good reason for provisioning EXL2 quants - you can make a request with the reasoning why EXL2 should be considered for a particular model.</p>
    <p>Keep in mind that among all quantization requests, EXL2 takes the lowest priority.</p>
  </div>

  <div class="card-dark">
    <h2>About EXL3 Quantization</h2>
    <p><strong>EXL3</strong> is a highly optimized quantization format based on QTIP designed for LLM inference on consumer GPUs. It is an evolution of the EXL2 format, offering higher quality within lower bitrates.</p>
    <p>If you enjoy EXL quants, feel free to support <a href="https://github.com/turboderp-org/exllamav3" target="_blank"><b>EXL3 development</b></a> and a small cat working tirelessly behind it: <b>turboderp</b> (<a href="https://github.com/turboderp" target="_blank">GitHub</a>, <a href="https://ko-fi.com/turboderp" target="_blank">Ko-Fi</a>).</p>
    <h3>Available Quantization Sizes</h3>
    <p>To use resources optimally, quants are created in a fixed range of sizes. Custom sizes will only be considered if there is a high community demand and/or available compute.</p>
    <ul>
      <li><code class="inline-code-dark"><b>2.5bpw_H6</b></code></li>
      <li><code class="inline-code-dark"><b>3.0bpw_H6</b></code></li>
      <li><code class="inline-code-dark"><b>3.5bpw_H6</b></code></li>
      <li><code class="inline-code-dark"><b>4.0bpw_H6</b></code></li>
      <li><code class="inline-code-dark"><b>4.5bpw_H6</b></code> / <code class="inline-code-dark"><b>4.25bpw_H6</b></code> (for 70b and above)</li>
      <li><code class="inline-code-dark"><b>5.0bpw_H6</b></code></li>
      <li><code class="inline-code-dark"><b>6.0bpw_H6</b></code></li>
      <li><code class="inline-code-dark"><b>8.0bpw_H8</b></code></li>
    </ul>
  </div>

  <div class="card-dark">
    <h2>How to Download and Use EXL Quants</h2>
    <p>Each quantization size for a model is stored in a separate HF repository branch. You can download a specific quant size by its branch.</p>
    <p>For example, to download the <code class="inline-code-dark">4.0bpw_H6</code> quant:</p>
    <p><b>1. Install hugginface-cli:</b></p>
    <pre><code>pip install -U "huggingface_hub[cli]"</code></pre>
    <p><b>2. Download quant by targeting the specific quant size (revision):</b></p>
    <pre><code>huggingface-cli download ArtusDev/MODEL_NAME --revision "4.0bpw_H6" --local-dir ./</code></pre>
    <p>EXL3 quants can be run with any inference client that supports the EXL3 format, such as <a href="https://github.com/theroyallab/tabbyapi" target="_blank"><b>TabbyAPI</b></a>. Please refer to <a href="https://github.com/theroyallab/tabbyAPI/wiki/01.-Getting-Started" target="_blank">documentation</a> for set up instructions.</p>
  </div>
  
  <div class="card-dark">
    <h2>Other EXL3 Quanters</h2>
    <p>If you don't find the model quant you're looking for, please check these other excellent community members who also provide EXL3 quants:</p>
    <ul>
      <li><a href="https://huggingface.co/turboderp" target="_blank"><b>@turboderp</b></a></li>
      <li><a href="https://huggingface.co/bullerwins" target="_blank"><b>@bullerwins</b></a></li>
      <li><a href="https://huggingface.co/MikeRoz" target="_blank"><b>@MikeRoz</b></a></li>
    </ul>
  </div>

</div>