The numbers are bit tricky, D2R has kinda unstable memory allocations, sometimes it goes straight to 6-8gb from start, sometimes it starts with 3gb and slowly goes up, I also noted that if you start the game, do nothing it will deallociate some memory after some time. Outside of reducing game files which is intrusive you cannot control it much.
What the legacy mode is doing though, it basically reduces completely your GPU core load. If you switch to modern graphics GPU load goes up, if you go into legacy GPU load (core cycles) are used much less which can reduce stuttering. So your offclients you have in legacy and main window running modern gfx.
Regarding recent performance changes, I believe that Windows 10 has a slightly better memory management and window switching fluidity for D2R compared to Win 11 and I believe also some older Nvidia drivers were stuttering bit less in the past - this is hard to measure it is more empirical feel, may vary per system.
D2:R