Neither is assembler is we're being technical.

If one knows assembler he could as well program in binary ... just type the program as opcodes into hex editor

if we're being technical, assembly language is in fact a language.

http://en.wikipedia.org/wiki/Assembly_language

calling binary a language is like calling the decimal numbering system a language.