I have been playing Titan to level 80~ish and not experienced any real lag. Now I have a level 45 infernalist and I can see my latency jump depending on the number of summons I have. Is this pretty common?

Might be a sound issue. Search for it, you can disable some combat sounds and everything should run smoother. It's because every summon and their projectiles interacts with all the monsters and their shit so in the end it generates thousands of sounds a second which takes processing power and might cause frame drops.

https://youtu.be/tM_SoDD8szY

I think it may be server side. I have been playing for 8 hours and not any lag then all of the sudden my game plays like this.

Could be total coincidence, but since I downloaded and used the game from their standard launcher rather than steam, I have had zero latency issues.