Toolchain — safetensors, GGUF, LoRA / QLoRA (Jun 2026)

Correct, up-to-date paths for removing alignment refusal directions from model weights and deploying locally.
This is weight surgery on Hugging Face–format checkpoints — not prompt jailbreaks.

Ethics & scope: risks-and-ethics.md · Factory/agent eval gates: ../instructions/eval-driven-workflow.md

Golden rule#

Never expect to “uncensor” by quantizing or editing GGUF tensors by hand. Always ablate base safetensors, then convert.

Decision tree (2026)#

Have HF instruct safetensors?
│
├─ Want automatic + best KL/refusal trade-off?
│   └─ Heretic (pip install heretic-llm==1.4.0) → safetensors out
│
├─ Want YAML layer control / sharded 70B on one GPU?
│   └─ llm-abliteration v1.2: measure → analyze → sharded_ablate.py
│
├─ Only have GGUF + weak GPU?
│   └─ abliterate.cpp (WIP) → measurements.pt → llm-abliteration ablate → HF → GGUF
│
└─ Deploy locally?
    ├─ Single file → merge weights → convert_hf_to_gguf.py → Q4_K_M → Ollama
    ├─ Base + policy adapter → export LoRA → convert_lora_to_gguf.py → llama.cpp --lora
    └─ Download community → p-e-w/*-heretic or registry GGUF

Path 1 — Heretic → safetensors (recommended)#

Tool: p-e-w/heretic · PyPI heretic-llm==1.4.0 · Docs mintlify

What Heretic edits: mlp.down_proj, self_attn.o_proj (and MoE expert downs) via projected + norm-preserving abliteration with Optuna kernel search.

Built-in LoRA: row_normalization = "full" embeds rank-r correction tensors in the export — not the same as PEFT SGD LoRA.

2026 notes:

• 4000+ community models tagged heretic on Hugging Face
• Supports dense, many VLMs, MoE (Qwen3-MoE), Qwen3.5 hybrid, gpt-oss (PyTorch 2.6+ for MXFP4)
• Community Heretic 1.2 reports ~70% lower VRAM — pin pip install heretic-llm==1.4.0 (do not float to master)
• Pins: tools/heretic-tools-reference.md

Path 2 — llm-abliteration → safetensors (manual control)#

Tool: jim-plus/llm-abliteration · v1.2 (Jan 2026)

git clone https://github.com/jim-plus/llm-abliteration.git
pip install -r requirements.txt

# 1. Measure (4-bit OK for this step only)
python measure.py -m ./base-model -o directions.pt --quant 4bit --projected \
  --data-harmful harmful.txt --data-harmless harmless.txt

# 2. Analyze layers
python analyze.py directions.pt -c

# 3. Ablate — MUST use full-weight base (not 4-bit checkpoint)
python sharded_ablate.py gemma3-12b-it.yml --projected --normpreserve

Why sharded: loads one safetensors shard + one layer at a time — ablate 27B+ on consumer VRAM.

→ ../methods/projected-llm-abliteration.md

Path 3 — Safetensors → GGUF → Ollama#

Tools: ggml-org/llama.cpp (now Hugging Face org, Feb 2026)

git clone https://github.com/ggml-org/llama.cpp.git tools/llama.cpp
cd tools/llama.cpp

# F16 GGUF (or direct Q8_0)
python convert_hf_to_gguf.py ../../models/qwen3-abliterated \
  --outfile ../../out/qwen3-f16.gguf --outtype f16

# Quantize
./llama-quantize ../../out/qwen3-f16.gguf ../../out/qwen3-q4_k_m.gguf Q4_K_M

# Ollama
cd ../../out
ollama create qwen3-abliterated -f Modelfile

Modelfile: match chat_template from tokenizer_config.json — wrong template = gibberish after quant.

Unsloth shortcut (if you trained/repaired with Unsloth):

model.save_pretrained_gguf("out", tokenizer, quantization_method="q4_k_m")

→ ../methods/gguf-export-notes.md · ../instructions/run-locally-ollama-lmstudio.md

Path 4 — LoRA / QLoRA abliteration deploy#

Three distinct mechanisms — do not conflate:

B — Export ΔW adapter#

python scripts/export-abliteration-lora.py \
  --base ./models/Qwen3-4B-ORIGINAL \
  --abliterated ./models/Qwen3-4B-abliterated \
  --out ./adapters/abliteration-r16 \
  --rank 16

Merge for single GGUF (Ollama)#

from peft import PeftModel
merged = PeftModel.from_pretrained(base, "./adapters/abliteration-r16").merge_and_unload()
merged.save_pretrained("./merged-abliterated")
# then convert_hf_to_gguf.py

GGUF LoRA sidecar (no merge)#

# PEFT → GGUF LoRA
python convert_lora_to_gguf.py \
  --outfile abliteration-lora-f16.gguf \
  --base-model-id Qwen/Qwen3-4B-Instruct-2507 \
  --lora-path ./adapters/abliteration-r16

# Inference
./llama-cli -m base-q4_k_m.gguf --lora abliteration-lora-f16.gguf
./llama-cli -m base-q4_k_m.gguf --lora-scaled abliteration-lora-f16.gguf 0.8

Web UI: huggingface.co/spaces/ggml-org/gguf-my-lora
Examples: ggml-org GGUF LoRA adapters collection — includes abliteration LoRAs for Llama 3 / 3.1 8B.

llama-server supports multiple --lora adapters and hot reload via /lora-adapters.

→ ../techniques/lora-qlora-abliteration.md · ../methods/lora-adapter-export.md

Path 5 — GGUF-native measure (low VRAM, experimental)#

Tool: kabachuha/abliterate.cpp — WIP (~20% comply rate in author tests). Not a production agent path; prefer Heretic on safetensors. Details: ../methods/gguf-export-notes.md.

1. Replace llama.cpp tools/cvector-generator with abliterate.cpp fork
2. Run llama-cvector-generator on GGUF with harmful/harmless .txt prompts
3. convert-into-measurements.py → measurements.pt
4. Feed into llm-abliteration sharded_ablate.py on HF safetensors
5. Re-export GGUF

Use when you cannot fit HF FP16 for measure — not production-ready for agents yet.

Path 6 — QLoRA 4-bit measure (not training)#

This is not QLoRA fine-tuning — no gradient updates on refusal. For SGD QLoRA after abliteration see Jarvis pack.

Tool status matrix (Jun 2026)#

What does not remove safety#

Recommended stacks#

Related#

• tools/abliteration-tooling.md — full catalog
• research-landscape.md — papers & taxonomy
• ../methods/safetensor-abliteration-pipeline.md
• ../methods/gguf-export-notes.md