I wouldn't want to answer this definitively without a better knowledge of which games are truly rare. I own a number of games that go for a ton of money with the perception they are rare, but who really knows?
Often times when an uncommon game is under very high demand, it makes it seem very rare. But if a lot of collectors have one, then it's not so rare. My feeling is that if a game rarely turns up, regardless of how much people want or don't want it, then it's probably rare. If it turns up often it must not be that rare, even if people ask hundreds. A lot of times high prices reflect demand and not just rarity.
So with all of that rambling, I haven't thoroughly researched which games are the most scarce that I own. I just know which ones were most expensive. Saturn Bomberman, Magic Knight Rayearth, Popful Mail, Burning Rangers, Bomberman Second Attack, and Stunt Racer 64.