MoE & hybrid architecture abliteration

Mixture-of-Experts and linear-attention hybrid models require per-expert and per-layer-type targets. Heretic handles many architectures automatically — manual pipelines must mirror its module map.

Source: Heretic directional ablation docs

Architecture → module map#

Rule: target output projections that write to the residual stream — not q_proj / gate_proj inputs.

Foundation: Projected + norm-preserving (cross-link to T03)#

All MoE / hybrid abliteration builds on the projected + norm-preserving technique (T03). See:

• projected-norm-preserving-abliteration.md — math, orthogonalize_direction, row_normalization modes, recommended production stack, low-VRAM profile.
• ../docs/advanced-techniques-catalog.md — T03 + T08 table rows (explicitly cross-linked in catalog + dedicated cross-technique guidance), config parameters, recommended MoE config snippet, and "always start MoE by reviewing T03 first" navigation note.

Why pair them?

• orthogonalize_direction = true removes only the refusal component orthogonal to the harmless centroid per expert (protects "helpful" features inside each routed expert).
• row_normalization = "full" + small LoRA rank prevents row-norm drift that would otherwise distort expert routing weights and amplify capability loss on hybrid models.
• Use the same flags for dense models; they become mandatory for stable MoE results (see catalog T03/T08 guidance and low-vram profile).

Heretic applies these per-expert automatically when the targets include experts[*].down_proj etc.

Why experts matter#

Each expert's down_proj can carry refusal writes independently. Abliterating only the shared router or one expert leaves refusal paths in inactive experts.

Heretic: iterates all experts when building the optimization graph.

Manual: loop experts in sharded_ablate.py config or custom script:

for layer in model.model.layers:
    if hasattr(layer.mlp, "experts"):
        for expert in layer.mlp.experts:
            expert.down_proj.weight.data = ablate_down_proj(
                expert.down_proj.weight.data, r_layer, alpha
            )

VRAM considerations (MoE)#

MoE abliteration on consumer GPU often requires 4-bit + cloud GPU for initial Heretic run, then GGUF for local inference.

Hybrid linear attention (Qwen3.5)#

Some layers use linear_attn instead of standard self_attn. o_proj may be absent — use linear_attn.out_proj for those layer indices only.

Verify in model:

for i, layer in enumerate(model.model.layers):
    has_sa = hasattr(layer, "self_attn") and hasattr(layer.self_attn, "o_proj")
    has_la = hasattr(layer, "linear_attn")
    print(i, "self_attn" if has_sa else "", "linear_attn" if has_la else "")

Eval notes#

• MoE models may show expert routing shift post-abliteration — monitor perplexity on domain prompts.
• KL eval per Heretic still applies; add expert-load entropy check if available.

Safetensor surgery surface#

MoE expert down_proj / w2 / output_linear tensors live in the same HF *.safetensors shards as dense models. Heretic discovers expert modules automatically; manual edits must touch every expert in each MoE layer.

→ ../methods/safetensor-abliteration-pipeline.md (tensor names, projected math, Heretic save layout)

Related#

• ../methods/moe-expert-abliteration.md
• ../methods/safetensor-abliteration-pipeline.md
• ../instructions/low-vram-abliteration.md
• projected-norm-preserving-abliteration.md (T03 foundation for per-expert projected + norm-preserve work; use with all MoE targets)
• ../docs/advanced-techniques-catalog.md (T03 + T08 explicitly cross-linked table rows + cross-technique guidance + recommended MoE config + module targets)