SLOBS has two options to encode, I am wondering if I use the x.264 option if then my gpu just handles the game?
or how does this work?

this is assuming you have a rtx card and a solid cpu setup for streaming,

or would it be better to use NVENC? wouldnt this take up the gpu resource and not use the cpu ? im kinda confused,

depends on the game i'd say.. some games are cpu heavy others are not, so u might get better performance in some games..

u can test this by streaming in x264 and check ur frame rate, then stream in nvenc and check frame rate again

nvenc has been around since 600 series i think

but here is article on turing nvenc vs x264

https://venturebeat.com/2018/10/15/nvenc-vs-x264-does-cpu-or-rtx-gpu-encoding-work-best-for-twitch/