I mostly buy my stuff at Book Offs or Hard Offs which are part of a huge chain of stores that sell any kind of used stuff around here, from books and games to clothes and liquor.
Apart from that, I also really like the BookMart, Takarajima, Takarasoko and A-Two chains. Since I live in Nagoya, sometimes I like to go to Meikoya, (the famous) Super Potato and Mandarake, but since they usually know exactly what they are selling, it is almost impossible to find a good deal there.
But, by far, my favorite shop is the Recycle Shop Hajimari. It is a very small, messy, dirty and smelly shop and the owner, Yagi-san, is always smoking inside. However, it is a paradise for retro gamers! I found most of my treasures (including gems like The King of Fighters 98 (CIB) for the Neo Geo CD, Star Parodia (CIB) for the PC Engine and my ultra rare Game Processor RAM Cassette for the Super Famicom) there and never paid more than the equivalent to US$5.00 in a game there.
If anyone happens to come to the Nagoya area someday, definitely check this store out!
http://sp.raqmo.com/r-hajimari/