GatorTron-S is a Megatron BERT model trained on pre-trained on synthetic clinical discharge summaries generated by SynGatorTron 5B NLG, a Megatron GPT-3 model trained on de-identified clinical free text at the University of Florida health system.