saiga-api-cuda-internal

Runtime error

muryshev commited on Oct 1, 2023

Commit

026d071

1 Parent(s): aa9cfee

Update app.py

Files changed (1) hide show

app.py CHANGED Viewed

@@ -133,7 +133,7 @@ def generate_search_request():
         logits_all=True,
         #n_threads=12,
         verbose=True,
-        n_gpu_layers=20,
         n_gqa=8       #must be set for 70b models
     )
@@ -183,7 +183,7 @@ def generate_response():
         logits_all=True,
         #n_threads=12,
         verbose=True,
-        n_gpu_layers=20,
         n_gqa=8       #must be set for 70b models
     )

         logits_all=True,
         #n_threads=12,
         verbose=True,
+        n_gpu_layers=40,
         n_gqa=8       #must be set for 70b models
     )
         logits_all=True,
         #n_threads=12,
         verbose=True,
+        n_gpu_layers=40,
         n_gqa=8       #must be set for 70b models
     )