Open-source small-parameter models used to be little more than toys, but this year is different—qwen3.5, gemma4, and lfm2.5 have all been quite impressive.

I recently deployed some small-parameter models on a Jige Wujie 14x for testing. The main use case is generating Git Commit Message, which local models can handle quite well.

I was curious about the speed and quality of different models. From what I've used so far, google/gemma-4-e4b is fairly well-balanced, performing well in both speed and quality. Due to limited time, this post only covers speed for now.

Below is the output report from the testing tool. I'll open-source the tool later (it was actually developed by the LLM grandpas too).

LM Studio Text Generation Model Benchmark Report

Test Conditions

• Start time: 2026-08-18T23:02:40+08:00
• API: http://localhost:1234 (model list source: api_v1)
• Prompt version: 2026.08.18
• Sampling: temperature=0.0, max_tokens=512, top_p=1.0, seed=42
• Rounds: 1 warm-up + 3 official runs per model × scenario
• Exclusive loading: on (only keep the current model loaded; unload immediately after testing)
• Timeout: 180s
• Output directory: C:\code\1\land-go\data\benchmarks\20260818-230240
• VRAM / RAM: not measured in this round. When LM Studio does not provide stable, readable usage data, no unreliable estimates are made.

Scenarios

• short_answer (short Q&A): one or two sentences of knowledge Q&A, mainly observing time to first token. Primarily look at ttft_ms.
• long_generation (long-text generation): request an explanatory piece of about 800 characters, mainly observing sustained generation speed. Primarily look at tokens_per_second.
• reasoning (reasoning problems): fixed-difficulty arithmetic word problems, mainly observing total time and stability. Primarily look at total_ms.
• code_generation (code generation): write a short Python function with comments, observing practical output speed. Primarily look at tokens_per_second.
• long_context (long-context Q&A): read a long passage first, then ask questions, observing time to first token and throughput under long context. Primarily look at ttft_ms.

Models Tested

• qwen3.5-4b (Q4_K_M), architecture qwen35
• lfm2.5-8b-a1b (Q4_K_M), architecture lfm2moe
• liquidai/lfm2.5-2.6b (Q5_K_M), architecture lfm2
• lmstudio-community/lfm2.5-2.6b (Q4_K_M), architecture lfm2
• google/gemma-4-e4b (Q4_K_M), architecture gemma4
• qwen/qwen3.5-2b (Q4_K_M), architecture qwen35

Skipped Models

Run Overview

• Official requests: 90, succeeded 90, failed 0
• Warm-up requests: 30

Ranked by Time to First Token (TTFT)

Lower is better. Everyday chat and input completion care more about this; don't only look at tokens/s.

Ranked by Sustained Generation Speed (tokens/s)

Higher is better. Long-form writing cares more about this. Rows where completion_tokens could not be obtained from the API will not appear here.

Ranked by Stability (Coefficient of Variation of Total Time)

Lower is better. Uses the standard deviation / mean of total time from successful official rounds; combinations with fewer than 2 samples are excluded.

Overall Recommendations

• Everyday interaction cares more about low TTFT: prefer lmstudio-community/lfm2.5-2.6b (Q4_K_M) (scenario long_context, median TTFT 2132.39 ms).
• Long-form / code cares more about sustained tokens/s: prefer lfm2.5-8b-a1b (Q4_K_M) (scenario code_generation, median 44.96 tok/s).
• Stability: qwen/qwen3.5-2b (Q4_K_M) (scenario reasoning, total time CV 0.000) has smaller fluctuation.

Detailed Summary

How to Read the Conclusions

• Don't decide "fastest" based on tokens/s alone.
• Actual model selection should also consider generation quality, VRAM usage, and failure rate, which are not in this report's speed rankings.
• After changing the prompt, bump PROMPT_VERSION and treat the new results as a new benchmark version.