I think the most difficult part of japanese is the grammatical structure. Knowing the word order doesn't really help since Japanese isn't ALWAYS SOV (well the verb is always at the end but since there are classifiers and grammatical markers the order of the object and subject can change). I'll admit i'm not fluent, i've only been studying for 1 1/2 years, but I still feel like i haven't learned anything really yet so I can testify that it isn't a easy language although there never really is one.
The alphabets take also some time to learn, It took me probably 1 year to be comfortable with my katakana, hiragana pace but now I think the main problem is getting into Kanji which isn't as bad as other people have said IF you practice. It just sucks that you have to learn everything in all katakana, hiragana, kanji, and romaji (to type in japanese) which does sound a little intimidating
Motivation is key, but its funny because when I first started Japanese in college my class was packed with Japanese learners now in the 4th level i'm taking the size has diminished probably by 1/2 just keep going and DON'T GIVE UP WHEN IT BECOMES TOO DIFFICULT. Japanese isn't easy or a walk in the park so have the mentality that you will be doing some work.
Good luck though

Thanks for the information.
Nice avatar btw... kind of mesmerizing.