Staffmod selection is not based on available staffmods, but on the 5 staffmods in a given tier (tier is selected first).
A fail roll occurs when a "restricted" staffmod is selected.
A restricted staffmod is a staffmod not designed for this item (like Holy shield, designed for shields and not for scepters, or Poison Dagger, designed for daggers and not for wands),
or a staffmod that has already been selected previously on the same item.
When a fail roll occurs, a new roll takes place and the fail roll is ignored. 5 times.
The sixth roll applies, regarless of its result. At this step, you may have the pleasure to see a Primal helm spawning +3 Whirlwind, or a scepter Spawning +3 Holy Shield.
There is no special way to increase the chance, except increasing the number of staffmods (from 0 to 3) that has to be selected on the item.
Namely : by imbuing.
You have much more chance to see a "restricted" staffmod when 3 Staffmods are selected.
The reason is quiete simple :
assuming that, in a given tier, you have 4 regular staffmods and 1 forbidden staffmod, chance to select that forbidden staffmod is very small :
6 rolls must all select this staffmod, 1 chance out of 5 at each roll, overall chance = ( 1 / 5 ) ^ 6 ~ 0.0064 %
But after the 2 first regular staffmods have been selected, the chance for the third (and last) is much greater.
Those 2 already selected staffmods are treated as "restricted" for the last roll, giving you among 5 staffmods : 2 regular and 3 restricted.
Chance to select your desired staffmod = ( 3 / 5 ) ^ 5 x ( 1 / 5 ) ~ 1.56%
(the first 5 rolls have 3/5 chance to fail, and the last sixth roll must select your desired staffmod).
Good luck !