Sparse Autoencoder Decomposition of Clinical Sequence Model Representations: Feature Complexity, Task Specialisation, and Mortality Prediction
Learn how to apply sparse autoencoders to clinical sequence models for feature complexity analysis and mortality prediction, and why it matters for improving healthcare outcomes
- Train a sparse autoencoder on a clinical sequence model like FlatASCEND
- Apply the trained autoencoder to extract features from electronic health records (EHRs)
- Analyze the feature complexity and task specialization of the extracted features
- Use the extracted features to train a mortality prediction model
- Evaluate the performance of the mortality prediction model using metrics like accuracy and AUC-ROC
Data scientists and researchers in the healthcare industry can benefit from this technique to analyze and improve clinical sequence models, and healthcare professionals can use the insights to make more accurate predictions and decisions
💡 Sparse autoencoders can be used to decompose clinical sequence model representations and reveal progressive abstraction across transformer depth, leading to improved feature complexity analysis and mortality prediction
🚑 Apply sparse autoencoders to clinical sequence models to improve mortality prediction and healthcare outcomes! 📊
Key Takeaways
Learn how to apply sparse autoencoders to clinical sequence models for feature complexity analysis and mortality prediction, and why it matters for improving healthcare outcomes
Full Article
Abstract:
arXiv:2605.04072v1 Announce Type: cross Abstract: Sparse autoencoders (SAEs) have been applied to large language models and protein language models, but not systematically to electronic health record (EHR) foundation models. We train TopK SAEs on FlatASCEND, a 14.5-million-parameter autoregressive clinical sequence model, at all 10 residual stream extraction points on INSPECT (outpatient) and MIMIC-IV (ICU). SAE decomposition reveals progressive abstraction across transformer depth: layer-0 feat
DeepCamp AI