Move the extra_token mechanism out of the lexer proper

Move the extra_token mechanism out of the lexer proper

Calling lex->pop_extra_token explicitly in the parser's yylex function
makes things considerably more robust.  It also allows pushing and
popping the EOF condition, necessary for backing up right at the end
of a file, like with ##{ \relative { c d e f } #} where the music
expression is checked against being a pitch, and then is backed up.