Low VRAM & low RAM abliteration

Run abliteration on 8–12 GB GPUs, 16 GB system RAM laptops, or Apple Silicon without a full FP16 model load.

First time? Complete setup-environment.md then beginner-local-model-guide.md Track A — this page adds detail.

Beginner steps (8 GB GPU, copy-paste order)#

Success at step 9: Same prompt that base refused now gets a technical answer (quality may vary).

Pick a path by hardware#

VRAM budget table (approximate)#

Add ~2–8 GB headroom for Optuna trials, KL eval, and batch autotuning unless you cap trials and enable CPU offload.

Path A — Heretic + bitsandbytes 4-bit (fastest)#

Best for one-command abliteration when VRAM is the bottleneck.

1. Config#

Use the handbook low-VRAM pin (recommended):

cp sources/heretic-tools/config.low-vram.toml config.toml

The low-VRAM profile (sources/heretic-tools/config.low-vram.toml) bakes in the projected + norm-preserving production settings (orthogonalize_direction = true + row_normalization = "full") so most 8–12 GB users get lower KL out of the box.

Why projected + norm-preserving helps on low VRAM: Raw direction removal often damages harmless capabilities (higher KL, more trials wasted, worse MMLU after). Projected keeps the refusal vector's component parallel to the harmless centroid; norm-preserving full + tiny rank-3 LoRA restores activation scales without extra memory. Result: you hit quality targets with the conservative n_trials=80 and max_batch_size=16 baked into this profile. Full math, modes table, and Heretic/llm-abliteration flags:

Projected & norm-preserving abliteration (recommended production stack + low-VRAM profile details).

Or copy upstream defaults from the repo pin and edit:

cp sources/heretic-tools/config.default.toml config.toml

quantization = "bnb_4bit"
device_map = "auto"
offload_outputs_to_cpu = true

# Tight VRAM: cap trials and batch
n_trials = 100
n_startup_trials = 30
max_batch_size = 32

# Optional: pin per-device memory (Accelerate)
# max_memory = { "0" = "7GB", "cpu" = "32GB" }

2. Run#

pip install heretic-llm==1.4.0 bitsandbytes accelerate
heretic Qwen/Qwen2.5-1.5B-Instruct
# or with config file in cwd:
heretic ./models/Qwen3-4B-Instruct-2507

3. Model picks for 8 GB VRAM#

4. Heretic LoRA inside norm-preserving export#

Heretic can embed a small LoRA correction when using full row normalization (not QLoRA training):

row_normalization = "full"
full_normalization_lora_rank = 3   # raise to 8–16 if export quality drops

This is part of projected + norm-preserving abliteration (orthogonalize the refusal direction against the harmless centroid, then restore row norms via small LoRA). The low-vram config enables these flags by default.

See:

• Projected & norm-preserving abliteration — math, modes (none/pre/full), recommended toml, when to raise full_normalization_lora_rank
• LoRA / QLoRA abliteration — adapter theory and export after norm-preserving runs
• Direct low-VRAM profile: sources/heretic-tools/config.low-vram.toml (sets orthogonalize_direction, row_normalization = "full", 4-bit + offload)

Path B — llm-abliteration 4-bit measure + sharded ablate#

Best when Heretic OOMs but you have CPU RAM for layer-wise surgery. Peak VRAM stays low because sharded_ablate.py edits one layer matrix at a time — the full model never loads into GPU memory during ablation.

Manual path: jim-plus/llm-abliteration v1.2 (projected/normpreserve). NousResearch/llm-abliteration last push ~2025-11-27 — frozen; historical YAML only, not production-pinnable.

git clone https://github.com/jim-plus/llm-abliteration.git tools/llm-abliteration
cd tools/llm-abliteration && pip install -r requirements.txt bitsandbytes

1. Measure in 4-bit#

python measure.py -m <model_path> -o directions.pt --quant 4bit \
  --data-harmful ./data/harmful.txt \
  --data-harmless ./data/harmless.txt

# Multilingual / Chinese topics — DECCP prompt sets
python measure.py -m <model_path> -o directions.pt --quant 4bit --deccp

DECCP (AUGMXNT/deccp) is a topics/dataset (Qwen2 PoC), not a peer CLI. Multilingual measure is python measure.py --deccp on jim-plus. Young’s “DECCP” GSM8K Δ -0.13 pp is that pipeline + deccp topics — still run factory JSONL gates.

2. Analyze (CPU OK)#

python analyze.py directions.pt -c

Pick middle-to-late layers in the YAML config (see repo examples).

3. Sharded ablate (low peak RAM)#

python sharded_ablate.py abliteration_config.yaml --normpreserve --projected

sharded_ablate.py processes weight shards so the full model never sits in VRAM at once. Peak demand is one layer matrix + direction vectors. Prefer this path for 20B+ checkpoints on consumer GPUs.

Full workflow: llm-abliteration-workflow.md · toolkit comparison: ../techniques/extended-abliteration-toolkit.md

Path C — Inference hooks only (zero weight edit)#

Lowest VRAM for experimentation — no permanent checkpoint.

from transformer_lens import HookedTransformer
model = HookedTransformer.from_pretrained(
    "Qwen/Qwen2.5-1.5B-Instruct",
    fold_ln=True,
    device="cuda",
)
# hook blocks.L.hook_resid_post — see methods/residual-hook-ablation.md

Use 1–3B models; store direction .pt on disk (~MB). Good for validating refusal removal before committing GPU hours to Heretic.

FailSpy/abliterator (github.com/FailSpy/abliterator) wraps the same TransformerLens pattern with activation caching, layer whitelist/blacklist, and temp vs permanent hook ablation — best for notebook prototyping; bake winning directions with Heretic or Abliterix.

→ quickstart.md · ../techniques/beyond-single-direction.md

Path D — Abliterate on cloud, infer locally quantized#

When local VRAM cannot run surgery:

1. Run Heretic on a rented 24 GB GPU (RunPod, Vast, Lambda).
2. Download safetensors output.
3. Convert + quantize locally (CPU RAM only):

git clone https://github.com/ggml-org/llama.cpp.git tools/llama.cpp
python tools/llama.cpp/convert_hf_to_gguf.py ./out/heretic-model --outfile ./out/model-f16.gguf
./tools/llama.cpp/llama-quantize ./out/model-f16.gguf ./out/model-q4_k_m.gguf Q4_K_M

4. Serve with Ollama / llama.cpp on a 8 GB RAM machine.

→ ../methods/gguf-export-notes.md

Path E — LoRA / QLoRA adapter instead of full checkpoint#

Ship a few-megabyte adapter instead of re-uploading 8B weights:

1. Full abliteration (Heretic or manual) or compute ΔW per layer.
2. Low-rank factorize ΔW → PEFT LoRA adapter.
3. Inference: base model 4-bit + adapter (Unsloth / PEFT / vLLM LoRA).

Detailed recipe: ../methods/lora-adapter-export.md · theory: ../techniques/lora-qlora-abliteration.md.

Jarvis tool-repair uses QLoRA after abliteration for command compliance — different goal, same hardware profile:

→ agentic-security-stack.md

System RAM (not VRAM) tips#

Swap thrashing: if system RAM < 32 GB and model > 8B, prefer sharded ablate or cloud GPU — do not rely on swap for measure passes.

Apple Silicon (M1/M2/M3, 16–24 GB unified)#

# Inference-only on Mac after you have a GGUF
ollama create security-agent -f Modelfile   # FROM ./model-q4_k_m.gguf

Troubleshooting#

Eval after low-VRAM run#

Same corpora as full-GPU runs — refusal removal should not depend on quant mode if KL is acceptable:

• ../data/eval/cyber-research-prompts.jsonl
• ../data/eval/hardware-factory-prompts.jsonl
• ../docs/evaluation.md

Related#