Research advances often don't translate to practical impact. In ML for small molecule drug discovery, this is at least partly because benchmark datasets don't capture important components of drug programs. We show how this can happen in ADME prediction and provide a path forward for building more realistic benchmarks from existing public data.