I was majoring in SLS aka Second Language Studies in college, where we analyzed second language acquisitions, linguistics and how to teach second language to others. (I'm Japanese btw)
Learning language is all about motivation, so you need purpose that drives you to learn Japanese (such as.... if u wanna read manga/listen to j-pop in Japanese and understand 100% etc). Hence that's the main "trick".
Keep on watching Japanese shows and or listening to Japanese music,, whatever you desire. It is hard to perfect Japanese because of cultural essence that is deeply engraved in the language, so don't try to be so hard and blame yourself. Then, if you felt comfortable, start reading Japanese book (ofc first at the beginners level). Reading will greatly enhance your vocabulary.
"One day at a time". Indeed, practicing through textbook is important and helpful but in the end, it is just a "textbook", so you need somebody native Japanese that can guide you through.
The most powerful and efficient way is to live in Japan (or this can be said to learning any language and I know it is a difficult decision), immerse into Japanese society, because obviously it forces you to speak Japanese every day since Japan is monolingual country (well its kinda start changing because of the Olympics, but they are still weak in English).
Also, make Japanese friends and immerse into Japanese communities in your local area (if possible...)
頑張って下さい (がんばってください）！！:)