I think if you play in windowed mode they don't play.

Two ways. Both methods achieve it the same way by replacing video files with 0 byte files and telling the game to use these files instead of the files it usually pulls from the mpq.

Extracted Game files
1. Extract game files with cascviewer (instructions can be found elsewhere, won't repeat here.
2. Go to ~\Diablo II Resurrected\Data\hd\global\video, rename files (or delete them) and create files with the same name with 0 bytes (eg create an empty .txt file and rename to d2intro.webm)
3. Launch the game with -direct -txt parameters

Launch with a mod
1. Create a folder ~\Diablo II Resurrected\mods\DisableVideos\DisableVideos.mpq
2. Create a file called modinfo.json and copy paste the below into it

3. In the same folder, create a subfolders so you get this folder structure ~\Diablo II Resurrected\mods\DisableVideos\DisableVideos.mpq\data\hd\global\video
4. Create empty text files and rename them to the following: blizzardlogos.webm, d2intro.webm, d2x_intro.webm, logoanim.webm
5. In this folder create folders called act2, act3, act4, act5
6. In the act2 folder create a file for act02start.webm
7. In the act3 folder create a file for act03start.webm
8. In the act4 folder create a file for act04start.webm and act04end.webm
9. In the act5 folder create a file for d2x_out.webm
10. Lastly launch the game with this parameter -mod DisableVideos -txt

Congrats you just made your own mod.