Doesn't this have a limit though? Say cold mastery for sorc is maxed with a infinity. Would % dmg be better?
Most builds have less -enemy resist and more +dmg, so picking which ever you have less of is often best. Usually, +% will only better if you have a lot of -resist elsewhere (CM mastery, lower resist, conviction).
Blizz sorc with max gear (doom, nw, 5/5's, 5x cold skillers, uber jewel, etc.) & infinity merc = 385 cold pen. Sundered will have -42% resist. With +30% cold dmg, an additional +15% dmg will be better than an additional -10% cold resist.
Blizz sorc with max gear (fathom, nw, 5/5's, 5x cold skillers, uber jewel, etc.)) & infinity merc = 335 cold pen. Sundered will have 12% resist. With +65% cold dmg, an additional +15% dmg will be worse than an additional -10% cold resist. However, dmg to sundered monsters is only 2.1% better than if using the +dmg%, and 8.3% worse vs non-sundered.
tldr;
With doom, the +15% dmg will be better.
With fathom, the -10% resist will be better to sundered, but worse than +15% everywhere else.