Rugby was by FAR the toughest for me. I played varsity university rugby and by the age of 15 I was already being recruited toward the national team when it was a top 15 nation in the world, and while I only briefly considered going pro I can say it was a
daunting prospect.
In comparison @ 7th grade I was approached toward potentially playing semi-pro soccer and in hockey I found success playing junior c/junior b/ junior a summer league. I can't compare baseball because I never went far with it; likewise all-star h.s. (american) football was my highlight in that area. I was going to switch from rugby to basketball in university but I had a really bad injury that stopped me cold. I even tried my hand at bodybuilding but since I never roided ...
Based on experience competing with pros (or at least future pros) in all those sports except baseball, I think it's unfair to leave out rugby and soccer. And what I heard from Aussie rules pros is that Aussie rules demands more from your body (not sure if it computes as being harder to make professionally though).
Hockey ... thing is the talent pool wasn't as deep, at least in my time. However I thought I was pretty good until got ice time with Doug Gilmour and I gained a new perspective

Also, EVERY major pro sport demands a huge amount of experience, practice/preparation and physical effort. People crap all over baseball players but the pros I met (mostly Blue Jays when they were champs) were pretty impressive athletes.