A tractable physical model for the yeast polarity predicts epistasis and fitness