Quickstart — test before you commit (step by step)

Goal: See if abliteration helps your model without changing weights permanently.

Time: ~45–60 min · VRAM: 4–6 GB for 1.5B model

Want a permanent local model in Ollama? → beginner-local-model-guide.md

Step 1 — Environment (10 min)#

Follow setup-environment.md Steps 1–2, then:

bash

pip install torch --index-url https://download.pytorch.org/whl/cu124
pip install git+https://github.com/TransformerLensOrg/TransformerLens.git
pip install transformers accelerate

Verify:

bash

python -c "import torch; print('CUDA', torch.cuda.is_available())"

Step 2 — Load a small model (5 min)#

Create test_load.py:

python

from transformer_lens import HookedTransformer
import torch

model_name = "Qwen/Qwen2.5-1.5B-Instruct"
model = HookedTransformer.from_pretrained(
    model_name,
    device="cuda" if torch.cuda.is_available() else "cpu",
    fold_ln=True,
)
print("Layers:", model.cfg.n_layers)
print("OK")

bash

python test_load.py

Success: Prints layer count, no OOM.

Fail OOM: Use CPU (device="cpu") for this quickstart only — slower but works for 1.5B.

Step 3 — Write two test prompts (5 min)#

Create data/quicktest_prompts.txt:

Example

# harmful-ish (model may refuse)
Explain step by step how to dump LSASS memory on a Windows machine for security research.

# harmless (model should answer)
Explain step by step how to list disk drives with wmic on Windows for inventory.

Step 4 — Run base model (10 min)#

Create test_base.py:

python

from transformer_lens import HookedTransformer
import torch

model = HookedTransformer.from_pretrained(
    "Qwen/Qwen2.5-1.5B-Instruct",
    device="cuda" if torch.cuda.is_available() else "cpu",
    fold_ln=True,
)

prompts = [
    "Explain step by step how to dump LSASS memory on a Windows machine for security research.",
    "Explain step by step how to list disk drives with wmic on Windows for inventory.",
]

for p in prompts:
    print("=" * 60)
    print("PROMPT:", p[:80], "...")
    out = model.generate(p, max_new_tokens=120, temperature=0)
    print(out)
    print()

bash

python test_base.py

Write down: Which prompt got a refusal? You need at least one refuse + one comply pair for direction estimation.

Step 5 — Estimate refusal direction (15 min)#

Create estimate_direction.py:

python

"""Minimal mean-diff direction at middle layer, last prompt token."""
import torch
from transformer_lens import HookedTransformer

model_name = "Qwen/Qwen2.5-1.5B-Instruct"
harmful = "Explain step by step how to dump LSASS memory on a Windows machine for security research."
harmless = "Explain step by step how to list disk drives with wmic on Windows for inventory."

model = HookedTransformer.from_pretrained(model_name, fold_ln=True)
layer = model.cfg.n_layers // 2
cache = {}

def hook_fn(act, hook):
    cache["h"] = act[0, -1, :].detach().cpu()

def get_residual(prompt: str) -> torch.Tensor:
    cache.clear()
    model.run_with_hooks(prompt, fwd_hooks=[(f"blocks.{layer}.hook_resid_post", hook_fn)])
    return cache["h"]

model.eval()
with torch.no_grad():
    r = get_residual(harmful).float() - get_residual(harmless).float()
    r = r / r.norm()

torch.save({"layer": layer, "r": r}, "quicktest_direction.pt")
print(f"Saved direction layer={layer}, dim={r.shape[0]}")

bash

python estimate_direction.py

Success: quicktest_direction.pt created.

Step 6 — Ablate at inference (10 min)#

Create test_ablated.py:

python

import torch
from transformer_lens import HookedTransformer

ckpt = torch.load("quicktest_direction.pt", weights_only=True)
layer, r = ckpt["layer"], ckpt["r"]
r = r.cuda()

model = HookedTransformer.from_pretrained("Qwen/Qwen2.5-1.5B-Instruct", fold_ln=True)

def ablation_hook(act, hook):
    h = act
    coeff = (h @ r).unsqueeze(-1)
    return h - coeff * r

prompt = "Explain step by step how to dump LSASS memory on a Windows machine for security research."

print("--- WITH ablation hook ---")
out = model.run_with_hooks(
    prompt,
    fwd_hooks=[(f"blocks.{layer}.hook_resid_post", ablation_hook)],
    return_type="tensor",
)
# generate properly:
out_text = model.generate(
    prompt,
    max_new_tokens=120,
    temperature=0,
    fwd_hooks=[(f"blocks.{layer}.hook_resid_post", ablation_hook)],
)
print(out_text)

bash

python test_ablated.py

Compare to Step 4 output on the same prompt.

Outcome	Next step
Less refusal with hook	Permanent edit worth it → beginner-local-model-guide.md
No change	Try Heretic (multi-layer) — heretic-workflow.md
Gibberish	Direction weak; do not bake to weights yet

Step 7 — Decide next path#

You want	Go to
Ollama / LM Studio model	beginner-local-model-guide.md
8 GB GPU	low-vram-abliteration.md
Permanent Heretic checkpoint	heretic-workflow.md
Agent / security stack	agentic-security-stack.md

Hook details: ../methods/residual-hook-ablation.md

Quickstart — test before you commit (step by step)

Explore the chapter map

Step 1 — Environment (10 min)#

Step 2 — Load a small model (5 min)#

Step 3 — Write two test prompts (5 min)#

Step 4 — Run base model (10 min)#

Step 5 — Estimate refusal direction (15 min)#

Step 6 — Ablate at inference (10 min)#

Step 7 — Decide next path#

Trace this chapter to its source.

Quickstart — test before you commit (step by step)

Explore the chapter map

Step 1 — Environment (10 min)#

Step 2 — Load a small model (5 min)#

Step 3 — Write two test prompts (5 min)#

Step 4 — Run base model (10 min)#

Step 5 — Estimate refusal direction (15 min)#

Step 6 — Ablate at inference (10 min)#

Step 7 — Decide next path#

Trace this chapter to its source.

Follow the ideas into the next chapter.