Can I use iron golem and take the amulet off and then replace it with another amulet and still retain the iron golem?

I mean, no Amulett no skills....so if you change it, the Golem died.

Thanks.