Abstract
Early clinical outcome prediction is central to ICU decision sup- port, where rapidly evolving patient conditions are reflected across heterogeneous electronic health record (EHR) data, including time- series measurements and free-text clinical notes. However, existing multimodal EHR models mainly focus on temporal modeling, cross- modal fusion, or external clinical knowledge for prediction, while leaving the construction of a patient-state representation implicit. In this work, a patient-state representation denotes a learned in- termediate representation that summarizes the patient’s evolving condition (e.g., respiratory function, hemodynamic stability, or in- fection monitoring) from multimodal ICU records before outcome prediction. Because EHR datasets typically lack comprehensive annotations for predefined patient-state categories, this represen- tation must be induced from the observed modalities rather than derived from explicit patient-state labels. To address this gap, we propose MPSL, a Multimodal Patient-State Learning framework for constructing patient-state representations from multimodal ICU records for early clinical outcome prediction. Specifically, MPSL encodes time-series measurements and free-text clinical notes into multimodal token representations in a shared hidden space. It then refines multiple model-internal latent components from these to- kens, which jointly form the patient-state representation for out- come prediction. Extensive experiments demonstrate that MPSL consistently outperforms state-of-the-art baselines on MIMIC-III and MIMIC-IV for in-hospital mortality and 30-day readmission prediction, achieving AUROC gains over the best-performing base- line of 3.11/1.18 points for mortality prediction and 6.53/4.67 points for readmission prediction across the two datasets