Martells keep their customs. Before they joined with Westeros, they were never conquered by the Targaryens. As their words say: "Unbowed, Unbent, Unbroken." Daeron I tried it, with horrible consequences. During Baelor the Blessed's reign, Dorne entered the Realm by marriage. The Dornish have their own culture and customs (women and men have equal rights in inheritance law), and the Dornish are far more sexually liberal than their neighbours. Dorne falls under the Realm and the Seven Kingdoms, but they were allowed to title themselves as Princes and Princesses instead of Lords/Ladies after the marriage pact with the Targaryens.

so according to dorne custom, if there's a prince, there must be a king
if the king died, then doran martell would be king, not a prince

No. You're misinterpreting. Dorne does not have kings/queens. Even before the Dornish joined the realm, they did not call themselves kings/queens. Only Queen Nymeria/King Mors had those titles, after that came only princes/princesses. Inheritance of the Iron Throne is done within the royal family. The Martells are a bit more pro-Targ than most (Elia was married to Rhaegar) other houses, but they don't have special inheritance rights to the Iron Throne.

They have no right to but they do.