It's better than pure melee in my opinion, but I would always choose some form of casting over not casting at all

There are pretty much two ways to go about it; either go 100% int, or get your int up to ~50 and then put the rest of the points out as if you were playing full melee.
For gear, it's best to get a combo of HM/Stats (Str/Dex) but those items tend to be pretty expensive, so I'd go for at least EE on your weapon and heal charm and then either HM or Str/Dex on your armour and main charm.
A problem with hybrid or caster samurai is the 10 base int which makes it tied for the second worst caster in the game, that doesn't mean it isn't possible though. I've seen some people do some downright amazing things with a hybrid samurai (but they had beastly gear... so it balances it out)