Efficient Fine-Tuning of Foundation Models: Generalization, Disparity, and Subgroup Adaptation

Abstract

Foundation models (FMs) perform well in vision but remain difficult to deploy when compute is limited and data are demographically imbalanced. This thesis makes two contributions. First, it studies demographic adaptation with parameter-efficient fine-tuning (PEFT) under realistic constraints where full fine-tuning is infeasible and available data are skewed. We evaluate five adaptation strategies, three pretraining paradigms, and multiple medical imaging datasets, covering 19 dataset-demographic variants, 490 configurations, and over 3,500 GPU hours. Results show that demographic adaptation can improve minority-group performance, especially with E2VPT, and clarify when PEFT offers strong efficiency--performance tradeoffs. Second, we propose Low-Rank Adaptation with Sinusoidal Projection Sampling (LoRA-SPS), a sinusoidal-projection low-rank adapter that samples from pretrained weights, decouples parameter count from embedding dimension, and improves projection rank stability while remaining mergeable for zero-overhead inference. Across vision and language tasks, LoRA-SPS matches or outperforms strong baselines using substantially fewer parameters.