So I read it matters if your follower when assigned to a building is better if level 100 vs any other level. Does it matter what quality as well? Uncommon, rare, epic?

I was also wondering this.
Like for shipment output does it matter what level the follower is

Below level 100 = 50%
Level 100 = 100%

So they just have to be level 100, their rarity doesn't matter?

From what I understand, it does not.

This is correct. Also, to get the double resource yield from your work orders, they only having to be working at that place when you actually go PICK UP the work order shipment. So you can go send them on quests then when you want to pick up your supplies just put them to work in the buildings real quick and go get your double yield.

Thats a nice tipp, thanka