Hugging Face's logo

Spaces:
ucalyptus
/
sglang-prefill-decoded-aggregation
Running

App Files Community
ucalyptus commited on
Commit
535344b
·
verified ·
1 Parent(s): ceaf146

Add 3 files

Browse files
Files changed (3) hide show
  1. README.md +7 -5
  2. index.html +312 -18
  3. prompts.txt +0 -0
README.md CHANGED
@@ -1,10 +1,12 @@
1
  ---
2
- title: Sglang Prefill Decoded Aggregation
3
- emoji: 📈
4
- colorFrom: gray
5
- colorTo: purple
6
  sdk: static
7
  pinned: false
 
 
8
  ---
9
 
10
- Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
 
1
  ---
2
+ title: sglang-prefill-decoded-aggregation
3
+ emoji: 🐳
4
+ colorFrom: blue
5
+ colorTo: gray
6
  sdk: static
7
  pinned: false
8
+ tags:
9
+ - deepsite
10
  ---
11
 
12
+ Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
index.html CHANGED
@@ -1,19 +1,313 @@
1
- <!doctype html>
2
- <html>
3
- <head>
4
- <meta charset="utf-8" />
5
- <meta name="viewport" content="width=device-width" />
6
- <title>My static Space</title>
7
- <link rel="stylesheet" href="style.css" />
8
- </head>
9
- <body>
10
- <div class="card">
11
- <h1>Welcome to your static Space!</h1>
12
- <p>You can modify this app directly by editing <i>index.html</i> in the Files and versions tab.</p>
13
- <p>
14
- Also don't forget to check the
15
- <a href="https://huggingface.co/docs/hub/spaces" target="_blank">Spaces documentation</a>.
16
- </p>
17
- </div>
18
- </body>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
19
  </html>
 
1
+ <!DOCTYPE html>
2
+ <html lang="en">
3
+ <head>
4
+ <meta charset="UTF-8">
5
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
6
+ <title>DeepSeek Deployment with SGLang: Visual Explanation</title>
7
+ <script src="https://cdn.tailwindcss.com"></script>
8
+ <link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700&display=swap" rel="stylesheet">
9
+ <style>
10
+ body {
11
+ font-family: 'Inter', sans-serif;
12
+ background-color: #f3f4f6; /* Light gray background */
13
+ }
14
+ .section-title {
15
+ font-size: 1.75rem; /* Larger section titles */
16
+ font-weight: 700;
17
+ color: #1e3a8a; /* Dark blue */
18
+ border-bottom: 2px solid #3b82f6; /* Medium blue border */
19
+ padding-bottom: 0.5rem;
20
+ margin-bottom: 1.5rem;
21
+ }
22
+ .subsection-title {
23
+ font-size: 1.25rem;
24
+ font-weight: 600;
25
+ color: #1d4ed8; /* Slightly lighter blue */
26
+ margin-top: 1rem;
27
+ margin-bottom: 0.75rem;
28
+ }
29
+ .card {
30
+ background-color: #ffffff;
31
+ border-radius: 0.75rem; /* More rounded corners */
32
+ box-shadow: 0 4px 6px -1px rgba(0, 0, 0, 0.1), 0 2px 4px -1px rgba(0, 0, 0, 0.06);
33
+ padding: 1.5rem;
34
+ margin-bottom: 1.5rem;
35
+ transition: transform 0.2s ease-in-out;
36
+ }
37
+ .card:hover {
38
+ transform: translateY(-5px);
39
+ }
40
+ .highlight {
41
+ background-color: #eff6ff; /* Light blue background for highlights */
42
+ color: #1e40af; /* Darker blue text for highlights */
43
+ padding: 0.25rem 0.75rem;
44
+ border-radius: 0.375rem;
45
+ font-weight: 600;
46
+ }
47
+ .metric {
48
+ font-size: 1.1rem;
49
+ font-weight: 700;
50
+ color: #16a34a; /* Green for positive metrics */
51
+ }
52
+ .comparison-metric {
53
+ font-size: 1rem;
54
+ font-weight: 600;
55
+ color: #52525b; /* Neutral gray for comparison details */
56
+ }
57
+ ul {
58
+ list-style-type: none; /* Remove default bullets */
59
+ padding-left: 0;
60
+ }
61
+ li {
62
+ position: relative;
63
+ padding-left: 1.75rem; /* Space for custom bullet */
64
+ margin-bottom: 0.75rem;
65
+ line-height: 1.6;
66
+ }
67
+ li::before {
68
+ content: '✓'; /* Custom checkmark bullet */
69
+ position: absolute;
70
+ left: 0;
71
+ color: #2563eb; /* Blue checkmark */
72
+ font-weight: bold;
73
+ font-size: 1.25rem;
74
+ }
75
+ .arrow {
76
+ font-size: 1.5rem;
77
+ color: #3b82f6;
78
+ margin: 0 0.5rem;
79
+ }
80
+ .gpu-icon svg {
81
+ width: 24px;
82
+ height: 24px;
83
+ fill: currentColor;
84
+ margin-right: 8px;
85
+ }
86
+ .flex-container {
87
+ display: flex;
88
+ align-items: center;
89
+ justify-content: space-around;
90
+ flex-wrap: wrap;
91
+ }
92
+ .flow-item {
93
+ text-align: center;
94
+ margin: 1rem;
95
+ padding: 1rem;
96
+ background-color: #e0e7ff;
97
+ border-radius: 0.5rem;
98
+ min-width: 150px;
99
+ }
100
+ </style>
101
+ </head>
102
+ <body class="p-4 md:p-8">
103
+ <div class="max-w-5xl mx-auto">
104
+ <header class="mb-12 text-center">
105
+ <h1 class="text-4xl font-bold text-gray-800 mb-2">Deploying DeepSeek with SGLang</h1>
106
+ <p class="text-xl text-gray-600">Achieving High Performance with PD Disaggregation & Large-scale Expert Parallelism</p>
107
+ <p class="text-sm text-gray-500 mt-1">Based on SGLang Team, May 05, 2025</p>
108
+ </header>
109
+
110
+ <section class="mb-10">
111
+ <h2 class="section-title">Key Achievements with SGLang</h2>
112
+ <div class="grid md:grid-cols-2 gap-6">
113
+ <div class="card">
114
+ <h3 class="subsection-title">🚀 Near Official Performance</h3>
115
+ <p class="text-gray-700">SGLang's implementation on 12 nodes (96 H100 GPUs) nearly matches DeepSeek's official inference throughput.</p>
116
+ <p class="mt-2">Input: <span class="metric">52.3k tokens/s per node</span></p>
117
+ <p>Output: <span class="metric">22.3k tokens/s per node</span> (for 2k token inputs)</p>
118
+ </div>
119
+ <div class="card">
120
+ <h3 class="subsection-title">💰 Cost Efficiency</h3>
121
+ <p class="text-gray-700">Translates to <span class="metric">$0.20 / 1M output tokens</span>, approximately <span class="highlight">1/5th the cost</span> of the official DeepSeek Chat API.</p>
122
+ </div>
123
+ <div class="card md:col-span-2">
124
+ <h3 class="subsection-title">⚡ Throughput Boost</h3>
125
+ <p class="text-gray-700">Optimized strategy improves output throughput by up to <span class="metric">5x</span> compared to vanilla tensor parallelism on the same resources.</p>
126
+ </div>
127
+ </div>
128
+ <div class="card mt-6">
129
+ <h3 class="subsection-title">Core SGLang Enhancements</h3>
130
+ <ul>
131
+ <li>Support for Prefill-Decode (PD) Disaggregation.</li>
132
+ <li>Large-scale Expert Parallelism (EP), including DeepEP, DeepGEMM, and EPLB.</li>
133
+ <li>Open-source implementation for community access and development.</li>
134
+ </ul>
135
+ </div>
136
+ </section>
137
+
138
+ <section class="mb-10">
139
+ <h2 class="section-title">Parallelism Design Strategies</h2>
140
+ <div class="grid md:grid-cols-2 gap-6">
141
+ <div class="card">
142
+ <h3 class="subsection-title">Attention Layers (MLA)</h3>
143
+ <p class="text-gray-700">Utilizes <span class="highlight">DP Attention</span> (Data Parallelism):</p>
144
+ <ul>
145
+ <li>Eliminates KV cache duplication across devices.</li>
146
+ <li>Significantly reduces memory overhead.</li>
147
+ <li>Supports hybrid data and tensor parallelism for flexibility.</li>
148
+ </ul>
149
+ </div>
150
+ <div class="card">
151
+ <h3 class="subsection-title">Dense FFNs</h3>
152
+ <p class="text-gray-700">Adopts <span class="highlight">Data Parallelism (DP)</span> over Tensor Parallelism (TP):</p>
153
+ <ul>
154
+ <li><span class="font-semibold">Enhanced Scalability:</span> Avoids fragmentation and ensures balanced workloads.</li>
155
+ <li><span class="font-semibold">Optimized Memory Efficiency:</span> Lower TP degree often minimizes memory, making DP favorable.</li>
156
+ <li><span class="font-semibold">Minimized Communication:</span> Reduces all-reduce operations by 50% compared to pure TP.</li>
157
+ </ul>
158
+ </div>
159
+ <div class="card">
160
+ <h3 class="subsection-title">Sparse FFNs (Mixture of Experts)</h3>
161
+ <p class="text-gray-700">Implements <span class="highlight">Expert Parallelism (EP)</span>:</p>
162
+ <ul>
163
+ <li>Distributes expert weights across multiple devices.</li>
164
+ <li>Scales memory capacity effectively.</li>
165
+ <li>Addresses challenges like irregular communication and workload imbalance using DeepEP.</li>
166
+ </ul>
167
+ </div>
168
+ <div class="card">
169
+ <h3 class="subsection-title">LM Head</h3>
170
+ <p class="text-gray-700">Employs <span class="highlight">Data Parallelism (DP)</span>:</p>
171
+ <ul>
172
+ <li>Mirrors the strategy for dense FFNs.</li>
173
+ <li>Reduces memory overhead for large vocabulary computations.</li>
174
+ <li>Simplifies communication across devices.</li>
175
+ </ul>
176
+ </div>
177
+ </div>
178
+ </section>
179
+
180
+ <section class="mb-10">
181
+ <h2 class="section-title">Prefill & Decode (PD) Disaggregation</h2>
182
+ <div class="card">
183
+ <p class="text-gray-700 mb-4">LLM inference has two phases: computation-intensive <span class="font-semibold">Prefill</span> and memory-intensive <span class="font-semibold">Decode</span>. Unified scheduling is inefficient.</p>
184
+ <h3 class="subsection-title">Problems with Unified Scheduling:</h3>
185
+ <ul>
186
+ <li>Prefill batches interrupt decode batches (delay).</li>
187
+ <li>DP Attention imbalance (increased decode latency).</li>
188
+ <li>Incompatible with DeepEP's dual dispatch modes.</li>
189
+ </ul>
190
+ <h3 class="subsection-title mt-4">SGLang's PD Disaggregation Solution:</h3>
191
+ <div class="flex-container my-4 p-4 bg-blue-50 rounded-lg">
192
+ <div class="flow-item">Input Request</div>
193
+ <div class="arrow">➔</div>
194
+ <div class="flow-item">Prefill Server<br/>(Computes KV Cache)</div>
195
+ <div class="arrow">➔</div>
196
+ <div class="flow-item">Data Transfer (RDMA)</div>
197
+ <div class="arrow">➔</div>
198
+ <div class="flow-item">Decode Server<br/>(Iterative Token Gen)</div>
199
+ </div>
200
+ <p class="text-gray-700">This separation allows tailored optimizations for each phase, maximizing GPU utilization.</p>
201
+ <h4 class="font-semibold text-gray-800 mt-3 mb-1">Key Implementation Details:</h4>
202
+ <ul>
203
+ <li><span class="highlight">Non-blocking Transfer:</span> Background data send/receive.</li>
204
+ <li><span class="highlight">RDMA-Based Transfer:</span> Efficient for non-contiguous memory.</li>
205
+ <li><span class="highlight">Flexible API Integration:</span> Supports Mooncake, NIXL.</li>
206
+ </ul>
207
+ </div>
208
+ </section>
209
+
210
+ <section class="mb-10">
211
+ <h2 class="section-title">Large-scale Expert Parallelism Optimizations</h2>
212
+ <div class="space-y-6">
213
+ <div class="card">
214
+ <h3 class="subsection-title">Expert Parallelism with DeepEP</h3>
215
+ <p class="text-gray-700">DeepEP streamlines EP by efficiently routing tokens to experts across GPUs.</p>
216
+ <p class="text-gray-700 mt-2"><span class="highlight">Normal Dispatch:</span> For prefill (long inputs, max throughput). Incompatible with CUDA Graph.</p>
217
+ <p class="text-gray-700 mt-1"><span class="highlight">Low-Latency Dispatch:</span> For decode (output tokens, min delay). Supports CUDA Graph.</p>
218
+ <p class="text-gray-700 mt-2">SGLang's <span class="font-semibold">PD Disaggregation</span> enables using both modes effectively with DP Attention.</p>
219
+ </div>
220
+
221
+ <div class="card">
222
+ <h3 class="subsection-title">DeepGEMM Integration</h3>
223
+ <p class="text-gray-700">Optimizes MoE matrix multiplications (Grouped GEMMs).</p>
224
+ <p class="text-gray-700 mt-2"><span class="highlight">Contiguous Layout Kernel:</span> For prefill (dynamic shapes). Used with DeepEP's Normal Dispatch (requires permutation).</p>
225
+ <p class="text-gray-700 mt-1"><span class="highlight">Masked Layout Kernel:</span> For decode (fixed shapes, CUDA Graph compatible). Used with DeepEP's Low-Latency Dispatch.</p>
226
+ </div>
227
+
228
+ <div class="card">
229
+ <h3 class="subsection-title">Two-batch Overlap (TBO)</h3>
230
+ <p class="text-gray-700">Splits a batch into two micro-batches to <span class="highlight">overlap computation and communication</span>.</p>
231
+ <ul>
232
+ <li>Lowers peak memory usage.</li>
233
+ <li>Addresses limited communication bandwidth in multi-node setups.</li>
234
+ <li>SGLang uses an abstraction layer (operations & yield points) for clean implementation.</li>
235
+ <li>Optimized launch order in prefill to avoid CPU-blocking by DeepEP.</li>
236
+ </ul>
237
+ </div>
238
+
239
+ <div class="card">
240
+ <h3 class="subsection-title">Expert Parallelism Load Balancer (EPLB)</h3>
241
+ <p class="text-gray-700">Addresses uneven workload distribution in MoE models.</p>
242
+ <ul>
243
+ <li>Computes optimal expert arrangement to minimize imbalance.</li>
244
+ <li>Uses redundant experts (e.g., 288 instead of 256) for flexible placement.</li>
245
+ <li>Enables diverse parallelism sizes (e.g., 12 or 72).</li>
246
+ <li>SGLang implements efficient, non-disruptive rebalancing.</li>
247
+ </ul>
248
+ <p class="mt-2 text-gray-600">Effectiveness depends on matching input distribution to serving workload (achieved via larger batches or periodic rebalancing).</p>
249
+ </div>
250
+ </div>
251
+ </section>
252
+
253
+ <section class="mb-10">
254
+ <h2 class="section-title">Evaluation Highlights</h2>
255
+ <div class="grid md:grid-cols-2 gap-6">
256
+ <div class="card">
257
+ <h3 class="subsection-title">Prefill Phase Performance</h3>
258
+ <p class="text-gray-700">On 4 nodes (32 H100s, EP32):</p>
259
+ <p>Up to <span class="metric">3.3x improvement</span> over TP16 baseline.</p>
260
+ <p>Throughput within <span class="comparison-metric">5.6% of DeepSeek's official profile</span> (assuming perfect balance).</p>
261
+ <p class="mt-1">Example: <span class="highlight">50,302 tokens/s per node</span> for 4K prompts.</p>
262
+ </div>
263
+ <div class="card">
264
+ <h3 class="subsection-title">Decode Phase Performance</h3>
265
+ <p class="text-gray-700">On 9 nodes (72 H100s, EP72):</p>
266
+ <p><span class="metric">5.2x speedup</span> over TP16 baseline.</p>
267
+ <p>With simulated MTP, throughput <span class="comparison-metric">6.6% below DeepSeek's profile</span>.</p>
268
+ <p class="mt-1">Example: <span class="highlight">22,282 tokens/s per node</span> for 2K inputs.</p>
269
+ </div>
270
+ </div>
271
+
272
+ <div class="card mt-6">
273
+ <h3 class="subsection-title">Ablation Study: Two-batch Overlap (TBO)</h3>
274
+ <p class="text-gray-700"><span class="font-semibold">Prefill:</span></p>
275
+ <ul>
276
+ <li>Supports larger batch sizes (e.g., 16k tokens/device vs 8k OOM without TBO).</li>
277
+ <li><span class="metric">27-35% throughput increase</span> by overlapping computation & communication.</li>
278
+ </ul>
279
+ <p class="text-gray-700 mt-3"><span class="font-semibold">Decode:</span></p>
280
+ <ul>
281
+ <li>Speedup contingent on batch size (e.g., <span class="metric">25.5% at 256 tokens/device</span>).</li>
282
+ <li>Most substantial speedup (<span class="metric">35%</span>) in simulated MTP with prolonged attention.</li>
283
+ </ul>
284
+ </div>
285
+
286
+ <div class="card mt-6">
287
+ <h3 class="subsection-title">Ablation Study: EPLB</h3>
288
+ <p class="text-gray-700">Delivers significant speedup by mitigating workload imbalance:</p>
289
+ <ul>
290
+ <li>Prefill: <span class="metric">1.49x speedup</span>.</li>
291
+ <li>Decode: <span class="metric">2.54x speedup</span>.</li>
292
+ </ul>
293
+ <p class="text-gray-700 mt-2">Strong correlation between <span class="highlight">workload balancedness and overall throughput</span>.</p>
294
+ <p class="text-gray-700 mt-2">Different expert distributions for prefill vs. decode support PD disaggregation for phase-specific expert placement.</p>
295
+ </div>
296
+ </section>
297
+
298
+ <section class="mb-6">
299
+ <h2 class="section-title">Conclusion</h2>
300
+ <div class="card">
301
+ <p class="text-gray-700 leading-relaxed">
302
+ SGLang, by integrating advanced techniques like Prefill-Decode Disaggregation and sophisticated Expert Parallelism strategies (DeepEP, DeepGEMM, TBO, EPLB), successfully deploys the large DeepSeek model on H100 GPUs with performance nearly matching official reports and significantly reducing costs.
303
+ The open-source nature of these components empowers the community to build upon these optimizations for efficient large-scale LLM serving.
304
+ </p>
305
+ </div>
306
+ </section>
307
+
308
+ <footer class="text-center mt-12 py-6 border-t border-gray-300">
309
+ <p class="text-gray-600">Visual summary generated based on "Deploying DeepSeek with PD Disaggregation and Large-scale Expert Parallelism on 96 H100 GPUs" by The SGLang Team.</p>
310
+ </footer>
311
+ </div>
312
+ <p style="border-radius: 8px; text-align: center; font-size: 12px; color: #fff; margin-top: 16px;position: fixed; left: 8px; bottom: 8px; z-index: 10; background: rgba(0, 0, 0, 0.8); padding: 4px 8px;">Made with <img src="https://enzostvs-deepsite.hf.space/logo.svg" alt="DeepSite Logo" style="width: 16px; height: 16px; vertical-align: middle;display:inline-block;margin-right:3px;filter:brightness(0) invert(1);"><a href="https://enzostvs-deepsite.hf.space" style="color: #fff;text-decoration: underline;" target="_blank" >DeepSite</a> - 🧬 <a href="https://enzostvs-deepsite.hf.space?remix=ucalyptus/sglang-prefill-decoded-aggregation" style="color: #fff;text-decoration: underline;" target="_blank" >Remix</a></p></body>
313
  </html>
prompts.txt ADDED
File without changes