Flide is extremely fast when one input carries many questions, as the shared input is computed only once. Even with a single question, it can still be up to 3x faster, depending on the model size and input length.

I wanted to try this - essentially for my dataset I have a number of decision questions for the same input. I haven't figured out yet which local model to try first with it!

Hi HN! I'm one of the Flide team.

We built Flide from scratch to make Jev-like decision model inference up to 9.5x faster than vLLM and SGLang, while keeping exactness.

Here is the source code: https://github.com/OmniJev/Flide

Look forward to hearing your feedback and ideas!