I was looking to buy a cheap amp but then I saw that I could also just buy a USB interface, fuck with the sound on my PC, and output to speakers or headphones?

Anyone have any experience with that latter idea? Any purchase suggestions for either item?

Budget is ~$100 I'm never going to gig and will only record as a hobby.

Focusrite 2i2 would be my choice. I'm sure you can find one for $100, or maybe a few bucks more. It's bus powered as well, so you don't need to plug it into the wall.