Filter by type:
Filter by tags:
gemmable-4-12b-mtp
## Gemmable 4 12B
Gemmable 4 12B is a GGUF export of Gemma 4 12B fine-tuned on Fable-5 style
reasoning and assistant traces.
## Highlights
- Base model: `google/gemma-4-12B`
- Format: GGUF
- Training style: Fable-5 style reasoning and assistant traces
- Distribution: fp16 GGUF plus matching assistant GGUFs for each quant
- Intended use: local inference, coding, reasoning, and assistant workflows
## How to use
### llama.cpp
Standard load:
```bash
llama-server -m "gemmable-4-12b-fp16.gguf"
```
Speculative / draft-MTP load:
```bash
llama-server -m "gemmable-4-12b-Q4_K_M.gguf" \
--spec-draft-model "gemmable-4-12b-Q4_K_M-mtp.gguf" \
--spec-type draft-mtp \
--spec-draft-n-max 4
```
Use the matching fp16 or quantized main file with its `-mtp` companion.
### LM Studio
1. Search this repo, download target + mtp file.
2. Load target.
3. Load settings → Speculative Decoding → select mtp file file.
(Requires a llama.cpp runtime with Gemma 4 MTP support from ggml-org/llama.cpp#23398. LocalAI's pinned llama.cpp backend already carries it, so this entry runs draft-mtp out of the box.)
## GGUF / local inference notes
...
Repository: localai
gemma-4-12b-qat-hauhaucs-balanced
HauhauCS Balanced is an uncensored Gemma 4 12B model built from
quantization-aware-trained weights for chat, coding, and creative writing.
This Q4_K_M GGUF build includes the vision projector for image input.
Repository: localaiLicense: gemma
gemma-4-12b-qat-hauhaucs-balanced-mtp
HauhauCS Balanced is an uncensored Gemma 4 12B model built from
quantization-aware-trained weights for chat, coding, and creative writing.
This Q4_K_M GGUF build includes the vision projector for image input.
The MTP variant adds the publisher-provided draft head and enables
llama.cpp multi-token-prediction speculative decoding.
Repository: localaiLicense: gemma
gemma-4-12b-agentic-fable5-composer2.5-v2-3.5x-tau2
Hugging Face |
GitHub |
Launch Blog |
Documentation
License: Apache 2.0 | Authors: Google DeepMind
> [!Note]
> This model card is for the Gemma 4 12B Unified model, which is part of the Gemma 4 family of open models. Built with the same multimodal functionality as Gemma 4 E2B and E4B (text, audio, image, and video inputs), it brings native audio and vision understanding directly to local environments without the need for separate encoders. This unified approach to multimodality makes the model encoder-free, offering a deployment size that is perfect for consumer devices and streamlined local execution.
Gemma is a family of open models built by Google DeepMind. Gemma 4 models are multimodal, handling text and image input (with audio supported on E2B, E4B, and 12B) and generating text output. This release includes open-weights models in both pre-trained and instruction-tuned variants. Gemma 4 features a context window of up to 256K tokens and maintains multilingual support in over 140 languages.
...
Repository: localaiLicense: apache-2.0
gemma-4-12b-agentic-fable5-composer2.5-v2-3.5x-tau2-mtp
Gemma 4 12B Agentic v2 is an Apache-2.0 fine-tune for coding, terminal
work, multi-step tool use, and reasoning. This variant pairs the Q4_K_M
target with the upstream Q8_0 Gemma 4 MTP drafter for faster lossless
generation through llama.cpp speculative decoding.
Repository: localaiLicense: apache-2.0
gemma-4-12b-agentic-fable5-composer2.5-v2-3.5x-tau2-q8
Gemma 4 12B Agentic v2 is an Apache-2.0 fine-tune for coding, terminal
work, multi-step tool use, and reasoning. This variant uses the
near-lossless Q8_0 GGUF quantization for higher output fidelity.
Repository: localaiLicense: apache-2.0
gemma-4-12b-coder-fable5-composer2.5-v1
Hugging Face |
GitHub |
Launch Blog |
Documentation
License: Apache 2.0 | Authors: Google DeepMind
> [!Note]
> This model card is for the Gemma 4 12B Unified model, which is part of the Gemma 4 family of open models. Built with the same multimodal functionality as Gemma 4 E2B and E4B (text, audio, image, and video inputs), it brings native audio and vision understanding directly to local environments without the need for separate encoders. This unified approach to multimodality makes the model encoder-free, offering a deployment size that is perfect for consumer devices and streamlined local execution.
Gemma is a family of open models built by Google DeepMind. Gemma 4 models are multimodal, handling text and image input (with audio supported on E2B, E4B, and 12B) and generating text output. This release includes open-weights models in both pre-trained and instruction-tuned variants. Gemma 4 features a context window of up to 256K tokens and maintains multilingual support in over 140 languages.
...
Repository: localaiLicense: gemma
gemma-4-31b-scotoma-2-q4
Gemma 4 31B Scotoma 2 is a multimodal Gemma 4 31B instruction-tuned model from ReadyArt. It applies a bounded refusal edit and preference training intended to reduce repetitive prose patterns while retaining the base model's text and image capabilities. This entry uses the 18.7 GB Q4_K_M GGUF and the matching Q8_0 vision projector.
License: Apache 2.0 | Base model: Google Gemma 4 31B IT
Repository: localaiLicense: apache-2.0
gemma-4-31b-scotoma-2-q8
Gemma 4 31B Scotoma 2 is a multimodal Gemma 4 31B instruction-tuned model from ReadyArt. It applies a bounded refusal edit and preference training intended to reduce repetitive prose patterns while retaining the base model's text and image capabilities. This higher-fidelity entry uses the 32.6 GB Q8_0 GGUF and the matching Q8_0 vision projector.
License: Apache 2.0 | Base model: Google Gemma 4 31B IT
Repository: localaiLicense: apache-2.0
gemma-4-26b-a4b-it-qat
Hugging Face |
GitHub |
Launch Blog |
Documentation
License: Apache 2.0 | Authors: Google DeepMind
> [!Note]
> This model card is for the new versions of the Gemma 4 family optimized with Quantization-Aware Training (QAT), which allows preserving similar quality to bfloat16 while dramatically reducing the memory requirements to load the model.
> Four versions of the QAT checkpoints are available:
> * **Unquantized QAT checkpoints** (Q4_0): Half-precision weights extracted from the QAT pipeline, ideal for custom downstream compilation and research. Available for Gemma 4 E2B, E4B, 12B, 26B A4B, and 31B, and their drafter models.
> * **GGUF** (Q4_0): Ready-to-deploy formats for broad ecosystem compatibility. Available for Gemma 4 E2B, E4B, 12B, 26B A4B, and 31B.
> * **Mobile-optimized** (wNa8o8): A custom schema engineered explicitly for mobile hardware efficiency. It features targeted 2-bit decoding layers, optimized KV caches, and static activations to maximize VRAM savings. Available for Gemma 4 E2B and E4B.
> * **Compressed Tensors** (w4a16): QAT checkpoints serialized in the compressed-tensors format for native, optimized inference with vLLM. Available for Gemma 4 E2B, E4B, 12B
...
Repository: localaiLicense: apache-2.0
gemma-4-12b-it-q4
Gemma 4 12B IT is Google's 12B instruction-tuned model for chat,
reasoning, tool use, and image understanding. This Unsloth Q4_K_M
GGUF build includes the F16 vision projector and uses the embedded
chat template. It defaults to a 32,768-token context.
Repository: localaiLicense: apache-2.0
gemma-4-12b-it-q5
Gemma 4 12B IT is Google's 12B instruction-tuned model for chat,
reasoning, tool use, and image understanding. This Unsloth Q5_K_M
GGUF build includes the F16 vision projector and uses the embedded
chat template. It defaults to a 32,768-token context.
Repository: localaiLicense: apache-2.0
gemma-4-12b-it-q6
Gemma 4 12B IT is Google's 12B instruction-tuned model for chat,
reasoning, tool use, and image understanding. This Unsloth Q6_K
GGUF build includes the F16 vision projector and uses the embedded
chat template. It defaults to a 32,768-token context.
Repository: localaiLicense: apache-2.0
gemma-4-12b-it-q8
Gemma 4 12B IT is Google's 12B instruction-tuned model for chat,
reasoning, tool use, and image understanding. This Unsloth Q8_0
GGUF build includes the F16 vision projector and uses the embedded
chat template. It defaults to a 32,768-token context.
Repository: localaiLicense: apache-2.0
gemma-4-12b-it-qat-q4_0
Hugging Face |
GitHub |
Launch Blog |
Documentation
License: Apache 2.0 | Authors: Google DeepMind
> [!Note]
> This model card is for the new versions of the Gemma 4 family optimized with Quantization-Aware Training (QAT), which allows preserving similar quality to bfloat16 while dramatically reducing the memory requirements to load the model.
> Four versions of the QAT checkpoints are available:
> * **Unquantized QAT checkpoints** (Q4_0): Half-precision weights extracted from the QAT pipeline, ideal for custom downstream compilation and research. Available for Gemma 4 E2B, E4B, 12B, 26B A4B, and 31B, and their drafter models.
> * **GGUF** (Q4_0): Ready-to-deploy formats for broad ecosystem compatibility. Available for Gemma 4 E2B, E4B, 12B, 26B A4B, and 31B.
> * **Mobile-optimized** (wNa8o8): A custom schema engineered explicitly for mobile hardware efficiency. It features targeted 2-bit decoding layers, optimized KV caches, and static activations to maximize VRAM savings. Available for Gemma 4 E2B and E4B.
> * **Compressed Tensors** (w4a16): QAT checkpoints serialized in the compressed-tensors format for native, optimized inference with vLLM. Available for Gemma 4 E2B, E4B, 12B
...
Repository: localaiLicense: apache-2.0
security-slm-gemma-4-e2b-it-q4
Security-SLM is a compact Gemma 4 E2B fine-tune for authorized red-team,
blue-team, security operations, and AI security work. It is designed for
private and air-gapped deployments where prompts can contain sensitive
incident data, policies, or source code.
This entry uses the text-only 3.43 GB Q4_K_M GGUF release.
Repository: localaiLicense: apache-2.0
gemma-4-e2b-it-qat-q4_0
Gemma 4 E2B is a multimodal (text + image) instruction-tuned model from Google DeepMind, optimized with Quantization-Aware Training (QAT) to preserve bfloat16-level quality at a fraction of the memory. E2B is a MatFormer "effective 2B" elastic variant: it carries a larger backbone but runs at an effective 2B-parameter footprint, making it well suited to lightweight and on-device deployments. This is the official Google Q4_0 GGUF, shipped with its multimodal projector.
License: Apache 2.0 | Authors: Google DeepMind
Repository: localaiLicense: apache-2.0
gemma-4-e4b-it-qat-q4_0
Gemma 4 E4B is a multimodal (text + image) instruction-tuned model from Google DeepMind, optimized with Quantization-Aware Training (QAT) to preserve bfloat16-level quality at a fraction of the memory. E4B is a MatFormer "effective 4B" elastic variant, balancing quality and footprint for on-device and edge deployments. This is the official Google Q4_0 GGUF, shipped with its multimodal projector.
License: Apache 2.0 | Authors: Google DeepMind
Repository: localaiLicense: apache-2.0
gemma-4-e4b-hauhaucs-aggressive-q4
HauhauCS Aggressive is an uncensored derivative of Google's Gemma 4 E4B
instruction model. This Q4_K_M GGUF supports text chat and image input
through the publisher's bundled F16 multimodal projector.
Uses the embedded chat template and the publisher's sampling settings.
The default context is 8192 tokens. The publisher declares the Gemma license.
Repository: localaiLicense: gemma
gemma-4-e4b-hauhaucs-aggressive-q5
HauhauCS Aggressive is an uncensored derivative of Google's Gemma 4 E4B
instruction model. This Q5_K_M GGUF supports text chat and image input
through the publisher's bundled F16 multimodal projector.
Uses the embedded chat template and the publisher's sampling settings.
The default context is 8192 tokens. The publisher declares the Gemma license.
Repository: localaiLicense: gemma
zero-gemma4-e4b-openzero-q5-k-m
Zero Gemma4-E4B OpenZero is a Gemma 4 E4B instruction fine-tune for
local coding, research, chat, and agentic workflows. It was trained on
2,033 curated OpenZero examples and is distributed as one merged Q5_K_M
GGUF file.
License: OpenZero Community Source v1; see the model repository for terms.
Repository: localaiLicense: other
Page 1