Filter by type:
Filter by tags:
qwopus3.8-27b-flash
Qwopus3.8-27B-Flash is a Qwen3.8-27B fine-tune for reasoning and agent
workloads. This Q4_K_M GGUF includes the F32 vision projector and uses
llama.cpp's embedded chat template with MTP speculative decoding.
The publisher reports a known Python code indentation issue.
Repository: localaiLicense: apache-2.0
qwopus3.8-27b-flash-q8
Qwopus3.8-27B-Flash is a Qwen3.8-27B fine-tune for reasoning and agent
workloads. This Q8_0 GGUF includes the F32 vision projector and uses
llama.cpp's embedded chat template with MTP speculative decoding.
The publisher reports a known Python code indentation issue.
Repository: localaiLicense: apache-2.0
hy4-preview
# Hy4-preview GGUF
Three GGUF builds of Hy4-Preview: https://huggingface.co/tencent/Hy4-preview
**Language / 语言:** English · 中文
**Neither file runs on stock llama.cpp.** The `hyv4` architecture is not upstream. Apply the
patches in `hy4-preview-patch/`
## English
### 1. What these are
**`Hy4-preview-Q4_K_M.gguf`** — a conventional Q4_K_M. Most tensors are Q4_K; `ffn_down_exps`
gets Q6_K on 37 layers via llama.cpp's own logic. Use this unless you are memory-constrained.
**`Hy4-preview-UD-IQ1_M.gguf`** - mixed precision with UD-IQ1_M strategy at ~2.44 bpw, roughly **half the size** for the
same model. The routed-expert `gate`/`up` projections run at 1.75 bpw (IQ1_M) and 2.0625 bpw (IQ2_XXS).
**`Hy4-preview-STQ1_0.gguf`** — mixed precision with MIX-STQ1_0 strategy at ~2.38 bpw, roughly **half the size** for the
same model. The routed-expert `gate`/`up` projections run at 1.3125 bpw (STQ1_0) on 29 layers and
2.0625 bpw (IQ2_XXS) on the other 48. See section 3.
### 2. Running them
Build a patched llama.cpp
```bash
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git checkout 0cea36222
...
Repository: localai
huihui-qwen3.8-flash-next-abliterated-q4
Huihui's abliterated Qwen3.8-Flash-Next is a vision-language
mixture-of-experts model modified to reduce refusals. This entry uses
the publisher's UD-Q4_K_XL GGUF and BF16 vision projector for text chat
and image input through llama.cpp. The default context is 32,768 tokens.
Model weights use the Qwen Community License 1.0.
Repository: localaiLicense: other
qwen3.8-flash-next-atomic-iq4
Qwen3.8 Flash Next in AtomicChat's AD-3.84bpw IQ4_XS M64 GGUF
build, with the F16 vision projector. The n-gram table occupies a
separate shard. This entry enables memory mapping and disables llama.cpp
automatic parameter fitting as required by the publisher.
Repository: localaiLicense: other
qwen3.8-flash-next-atomic-q4
Qwen3.8 Flash Next in AtomicChat's AD-4.27bpw Q4_K_M M64 GGUF
build, with the F16 vision projector. The n-gram table occupies a
separate shard. This entry enables memory mapping and disables llama.cpp
automatic parameter fitting as required by the publisher.
Repository: localaiLicense: other
huihui-qwen3.8-27b-abliterated
Huihui Qwen3.8 27B is an abliterated vision-language model published by
huihui-ai. This BF16 GGUF build includes the shared BF16 vision projector
and enables MTP speculative decoding through llama.cpp. Q4_K and Q8_0
variants are available as smaller downloads.
Repository: localaiLicense: apache-2.0
huihui-qwen3.8-27b-abliterated-q4
Huihui Qwen3.8 27B in Q4_K GGUF format, with the shared BF16 vision
projector and MTP speculative decoding through llama.cpp.
Repository: localaiLicense: apache-2.0
huihui-qwen3.8-27b-abliterated-q8
Huihui Qwen3.8 27B in Q8_0 GGUF format, with the shared BF16 vision
projector and MTP speculative decoding through llama.cpp.
Repository: localaiLicense: apache-2.0
btl-4-compact
BTL-4 Compact is Bad Theory Labs' text-only 35B mixture-of-experts model
compressed into a single 9.96 GB IQ2_XXS GGUF. Around 2.1B parameters are
active per token, and the model is tuned for agentic work, tool use, coding,
and reasoning. The compact build omits the vision tower and disables the
source model's MTP layer for compatibility with stock llama.cpp.
Repository: localaiLicense: apache-2.0
deepseek-v4-pro-0813
DeepSeek V4 Pro 0813 is DeepSeek's MIT-licensed flagship mixture-of-experts
model for agentic coding, reasoning, and long-horizon tool use. This entry
uses Unsloth's UD-Q4_K_XL GGUF build, split into 20 shards for llama.cpp.
Repository: localaiLicense: mit
pocket-35b
POCKET-35B is an Apache-2.0 Qwen3.5-family mixture-of-experts model from
FINAL-Bench/VIDRAFT, derived from Darwin-36B-Opus and packaged for stock
llama.cpp. This entry uses the quality-oriented Q4_K_M GGUF quantization.
Repository: localaiLicense: apache-2.0
pocket-35b-q3
POCKET-35B is an Apache-2.0 Qwen3.5-family mixture-of-experts model from
FINAL-Bench/VIDRAFT, derived from Darwin-36B-Opus and packaged for stock
llama.cpp. This entry uses the balanced Q3_K_M GGUF quantization.
Repository: localaiLicense: apache-2.0
pocket-35b-q2
POCKET-35B is an Apache-2.0 Qwen3.5-family mixture-of-experts model from
FINAL-Bench/VIDRAFT, derived from Darwin-36B-Opus and packaged for stock
llama.cpp. This entry uses the smaller Q2_K GGUF quantization.
Repository: localaiLicense: apache-2.0
pocket-35b-iq1
POCKET-35B is an Apache-2.0 Qwen3.5-family mixture-of-experts model from
FINAL-Bench/VIDRAFT, derived from Darwin-36B-Opus and packaged for stock
llama.cpp. This entry uses the most compact IQ1_M GGUF quantization.
Repository: localaiLicense: apache-2.0
pocket-26b
POCKET-26B is an Apache-2.0 Gemma 4 26B-A4B mixture-of-experts model from
FINAL-Bench/VIDRAFT, tuned for Korean and packaged for stock llama.cpp.
This entry uses the quality-oriented Q4_K_M GGUF quantization.
Repository: localaiLicense: apache-2.0
pocket-26b-q2
POCKET-26B is an Apache-2.0 Gemma 4 26B-A4B mixture-of-experts model from
FINAL-Bench/VIDRAFT, tuned for Korean and packaged for stock llama.cpp.
This entry uses the smaller Q2_K GGUF quantization.
Repository: localaiLicense: apache-2.0
qwen3.6-35b-a3b-uncensored-genesis-hermes-v6
Qwen3.6-35B-A3B Uncensored Genesis Hermes V6 is LuffyTheFox's multimodal,
agentic derivative of HauhauCS's uncensored Qwen3.6-35B-A3B model. It
combines Genesis tensor calibration with Hermes function-calling data while
retaining the 35B mixture-of-experts architecture, roughly 3B active
parameters per token, and the native 262K-token context window.
This entry installs the Q8_0 GGUF together with its F16 multimodal projector
for llama.cpp. The model card recommends Jinja chat templates and at least a
128K context for its thinking behavior. License: Apache-2.0.
Repository: localaiLicense: apache-2.0
qwythos-27b-v1
Qwythos-27B-v1 is an Apache-2.0 dense 27B reasoning and agentic model
derived from Qwen3.5-27B. It supports tool use, vision through the included
projector, and a one-million-token context window. This entry uses the
recommended Q4_K_M GGUF quantization; an MTP-enabled build is available as
a variant for hosts with recent llama.cpp support.
Repository: localaiLicense: apache-2.0
minicpm5-1b-claude-opus-fable5-v2-thinking
# MiniCPM5-1B-Claude-Opus-Fable5-V2-Thinking
GGUF quantizations for local deployment: **MiniCPM5-1B-Claude-Opus-Fable5-V2-Thinking-GGUF**
中文说明
**MiniCPM5-1B-Claude-Opus-Fable5-V2-Thinking** is a compact 1B **Thinking** language model built on openbmb/MiniCPM5-1B. Compared with V1, this V2 release is further fine-tuned on **Fable 5** data with a stronger focus on **tool calling / function calling**, while also improving **coding** and **instruction-following**. It keeps MiniCPM5's native Thinking chat template and XML tool-call format.
Previous version: **MiniCPM5-1B-Claude-Opus-Fable5-Thinking** (V1)
For llama.cpp / Ollama / LM Studio deployment, see the **GGUF repository**.
## Overview
## Capabilities
- **Tool calling (enhanced in V2)** — more reliable XML / function-calling style tool use on top of MiniCPM5's native format
- **Coding** — code generation, debugging, and software-engineering-style tasks
- **Instruction following** — more reliable adherence to user prompts and structured constraints
- **Thinking mode** — chain-of-thought reasoning via the MiniCPM5 chat template
- **Long context** — up to **128K tokens** (131,072 tokens per `config.json`)
...
Repository: localaiLicense: apache-2.0
bonsai-8b-1bit
Bonsai 8B (PrismML) is an end-to-end 1-bit language model built on the Qwen3-8B dense
architecture (GQA, SwiGLU, RoPE, RMSNorm, 36 layers, 65,536 context). Every weight is a
single sign bit (`-scale` / `+scale`) with one FP16 scale per group of 128 weights, for
an effective 1.125 bits/weight and a ~1.15 GB footprint (14.2x smaller than FP16) while
matching full-precision 8B instruct models at ~70.5 average across 6 benchmark
categories.
The Q1_0 quantization is only decodable by the PrismML llama.cpp fork, so this entry
runs on LocalAI's `bonsai` backend (that fork), not the stock `llama-cpp` backend.
License: Apache 2.0.
Repository: localaiLicense: apache-2.0
Page 1