Just to clarify, the $250 NeoGeo is just the console itself with a wired arcade stick. The 35th Anniversary bundle I'm planning to get is $350 and includes Metal Slug, a wireless arcade stick, and a memory card (all in white). It is $100 more than the console itself, but with the game being $90 and the memory card being $35, it's still cheaper than buying everything separately. Personally I'm much less hesitant to spend $350 on a new-manufacture console bundled with a game and memory card than I would be to spend $400 on a used console with capacitors that have been around since I was a toddler, and have to buy a game and memory card on top of that. Not to mention the new version has HDMI out and dip switches for region selection, display modes, and even overclocking the CPU.
On a side note, it seems quite a bit of the console manufacturing costs are tied up in the arcade stick, as those are priced at $120 each.
I did see Plaion's Atari consoles on the Atari website and almost bought a 5200, but then I remembered I can't stand playing games that old. It's hard enough for me to get into NES games, so going to something even older would be a painful experience and something I'd hardly ever get any use out of.