Structural Sensitivity in Compressed Transformers: Relative Error Propagation and Layer Removal
📰 ArXiv cs.AI
arXiv:2603.20991v2 Announce Type: replace-cross Abstract: Compressing transformer weights makes large language models cheaper to deploy. But each layer's compression introduces an error. These errors accumulate as the signal passes through later layers, and how they accumulate is not well understood. We measure this directly: at each layer, we take the ratio of output to input error, calling it rho. A value below one means the layer absorbs the error; above one means it grows. Computing rho on s
Full Article
Title: Structural Sensitivity in Compressed Transformers: Relative Error Propagation and Layer Removal
Abstract:
arXiv:2603.20991v2 Announce Type: replace-cross Abstract: Compressing transformer weights makes large language models cheaper to deploy. But each layer's compression introduces an error. These errors accumulate as the signal passes through later layers, and how they accumulate is not well understood. We measure this directly: at each layer, we take the ratio of output to input error, calling it rho. A value below one means the layer absorbs the error; above one means it grows. Computing rho on s
Abstract:
arXiv:2603.20991v2 Announce Type: replace-cross Abstract: Compressing transformer weights makes large language models cheaper to deploy. But each layer's compression introduces an error. These errors accumulate as the signal passes through later layers, and how they accumulate is not well understood. We measure this directly: at each layer, we take the ratio of output to input error, calling it rho. A value below one means the layer absorbs the error; above one means it grows. Computing rho on s
DeepCamp AI