vllm support
do you already have a patch that would work with vllm?
Why not try the QVB8 Inference Engine?
I can running the Ripple on DGX Spark, and it works great for coding with VS Code + Cline!
https://huggingface.co/addiesam/Ripple-65B-Preview-QVB8R-1e-3
Why not try the QVB8 Inference Engine?
I can running the Ripple on DGX Spark, and it works great for coding with VS Code + Cline!https://huggingface.co/addiesam/Ripple-65B-Preview-QVB8R-1e-3
Because i am on SM_80 and not a fan of ai slop engines...
The QVB8 Inference Engine currently has an sm_70 version specifically built for running models on the V100.
Could you let me know how much GPU memory (in GB) your sm_80 machine has? I may be able to calibrate a version for it, provided the error bound isn't too loose—otherwise, it won't work properly.
The QVB8 Inference Engine supports for sm_70 and sm_80, too.
v1.0.1 is updated.