Rethinking Activation Functions: Reducing Bias Can Improve Learning.

arXiv:2610.14500v1 Announce Type: cross Abstract: Many widely used activation functions, including ReLU, GELU, Swish, and Sigmoid, produce predominantly positive outputs, resulting in a nonzero post-activation mean that we term activation bias. Despite its ubi