BMFM-RNA: whole-cell expression decoding improves transcriptomic foundation models

1. 1Pretrain models with whole-cell expression decoding (WCED) instead of masked language modeling (MLM)
2. 2Use a single CLS token embedding to reconstruct the entire gene vocabulary
3. 3Evaluate model performance on downstream metrics to compare WCED and MLM
4. 4Fine-tune models with WCED for specific tasks to achieve better results

ML researchers and bioinformaticians can benefit from this approach to improve the performance of their models in downstream tasks, particularly in transcriptomic analysis

💡 Whole-cell expression decoding creates a maximally informative bottleneck, leading to better cell representations and downstream task performance

🧬 WCED outperforms MLM in transcriptomic foundation models!