LLM Benchmarks

Hardware

• CPU: Ryzen 9 5950X
• RAM: 1x32GB 2400MHz DDR4 Corsair Vengeance
• CUDA0: RTX 3080 10GB
• CUDA1: RTX A2000 6GB
• PSU: MSI Mag A850GL (850W, 80 Plus Gold)
• Motherboard: MSI Mag B550 Tomahawk WiFi Max
• Machine name: Basilisk

Methodology

1. Downloaded model weights to laptop, then transferred to Basilisk via sftp. See setup and initial tests here, or open it in a modal.
2. Installed Nvidia drivers and llama.cpp.
3. Ran each model through llama bench with various parameters using bash-fu with jsonl outputs. See full benchmark scripts below.
4. Failed runs due to OoM are being logged.
5. Will process jsonl files into one json file for this page.

Notes

• The inclusion of a model in these benchmarks does not constitute an endorsement of that model for any purpose. Some of these are good, and some of them are really dumb.
• I originally intended to benchmark more models and more parameter permutations, but it would take ~16 months to complete, so I drastically reduced the scope and optimized the scripts.
  • Replaced CSV parameters with nested bash for loops.
  • Split GPU testing into dedicated dense and MoE scripts to remove irrelevant --device CUDA0/CUDA1 and -ncmoe X params from dense.
  • Added logging of permutations and the ability to skip previously run permutations upon restart.
  • Removed the -r 30 param that would repeat each benchmark 30 times.
  • Replaced independently varying K and V cache types with linked KV cache types.
  • Removed f16 and iq4_nl from tested KV quants.
  • Removed 5120 and 10240 from tested context depths.
• This benchmark system uses random tokens; MoE models often have better performance in practice due to token correlations, causing relevant experts to remain on the GPU when running llama serve --moe-cache-mib N
• CPU-only was achieved via n_gpu_layers=0 (-ngl 0 on CLI), whereas it was left at the default =-1 for GPU
• CPU-only was achieved via device=none, whereas it was set to another value for GPU(s)
• Both ngl=0 and device=none were required for CPU-only execution
• -d sets the context depth at which the test begins
• -ncmoe configures CPU offloading for MoE and was excluded for CPU tests
• -nkvo 1 disables KV cache offloading to CPU/RAM during GPU tests
• -ctk and -ctv set the types for KV cache keys and values, respectively
• I will endeavor to expand benchmark coverage once the first batch of results are processed, whenever the right combo of convenience, ability, and motivation strikes.

Agentic Tests

The dense academic data that will fill up the table and contribute to useful graphs will take a while to generate and collate. In the meantime, I am doing some read-only agentic tasks in real harnesses and comparing inference engines, parameters, etc. (A side quest into pragmatism.)

One such agentic task benchmark compared llama.cpp with ik_llama on Flint 40B-A2.5B IQ3_XXS. Those results can be viewed here . Or view it in a modal here.

Filtering

You can filter the benchmark rows by any column. Pick a field from the dropdown and a value from the suggestions, then click Add Filter to create a removable tag. Within a single field, multiple values use OR logic (a row matches if it matches any of them); across different fields, filters use AND logic (a row must match all fields). The model field uses case-insensitive substring matching, while all other fields use case-insensitive exact matching. Clear everything with Reset Filters.