Comparing Sparsified MLPs and Transcoders: Many Transcoder Features Are Neurons in Disguise.

arXiv:2610.24521v1 Announce Type: new Abstract: MLP neurons in transformer language models are widely understood to be polysemantic, motivating transcoders - sparse, learned approximations of MLP layers - as interpretable replacement models for mechanistic ana