http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1474447/

is this really all that bad even in small amounts?

whats considered a "small amount" if so?

Possibly promote seizures in people who are susceptible to seizures. I'd say that's bad, but needs to be explored further in clinical trials. Study said any amount would increase the phenylalanine amount in the brain.

If it can be avoided, why bother having it?

from what i remember...the neurotoxicity is only serious when you lack the enzyme required to break phenylalanine into tyrosine (phenylalanine hydroxylase)

That article is from 1987 that has been disproved by a million other studies.

Aspartame is a very safe artificial sweetener, the only group who need to take precautions with aspartame are people with PKU.

https://www.facebook.com/notes/avi-bitterman/aspartame-is-safe-heres-a-wall-of-text-of-references-showing-it/697659256934115

here.