Damage isn't being done at all. The portals are taking up space that the monsters need to exist in, and they can't, so they instantly die. If anything you could name it something like 'void space' damage, but someone who is more versed in the technical side of bportal can confirm. ^dubstepmammoth or ^dolphy can confirm.
Emiliaaa is right but to expand a little essentially the portals create deadspace that expands and when that intersects with a mobs position the mob is forced to poof due to a check on whether or not a monster can exist in a certain coordinate or not (the same thing that stops mobs from spawning out of bounds or inversely what allows flying mobs to cross certain kinds of terrain).
Normally this check would just stop the monster from spawning in, since it’s already there though, as a failsafe it gets the same “you do not belong here anymore” instant kill that warlock demons and other summons get when their player leaves the game. The only difference is that since it was an actual mob and not a summoned creature, it has a drop table and since it’s not out of map bounds, it still drops its items.