Submitted to arXiv on July 27, 2026 (paper 2607.24645), “Sparse Autoencoders Encode Both Concepts and Functions: The Downstream Geometry of Feature Effects” by Phu Gia Hoang, Anwoy Chatterjee, Tanmoy Chakraborty, Iryna Gurevych and Subhabrata Dutta examines a gap that has quietly limited sparse autoencoders as a practical interpretability tool. SAE features often come with clean, human-readable activation descriptions, but the causal effect of intervening on them is inconsistent: a feature with an obvious label may have a weak or unexpected effect, steering may vary across prompts or push in the opposite direction from the one intended, and picking features by activation can miss the ones that actually change the output.
Prior work mostly studied feature geometry inside the model, where the features are computed. This paper instead studies the geometry of the resulting changes in model logits. The authors introduce Feature-Effect Geometry Analysis (FEGA), an unsupervised framework that removes the same active SAE feature across many different contexts and analyzes the resulting cloud of logit changes. Across SAE variants, they find that consistent one-dimensional effects are rare - few features behave like a reusable direction you can add or subtract.
To explain the variation, the paper distinguishes value-like features, tied to static information such as factual attributes, from pointer-like features, associated with context-dependent operations. Value-like features more often show structured, low-dimensional effects, though even these typically span several directions rather than one. Pointer-like features predominantly show diffuse effects. The authors’ conclusion is precise and consequential: a feature can be genuinely interpretable and genuinely causally relevant while still providing no stable direction for steering.
For a technical business leader, this is a useful caution against the emerging pitch that interpretability features are a control surface. Reading what a model is representing and reliably steering it are different capabilities, and this work shows the second does not follow from the first. Anyone budgeting for SAE-based guardrails or behavior control should treat steering reliability as something to measure per feature, not assume.