Developing Large-Scale Adaptive Scientific Applications with Hard to Predict Runtime Resource Requirements