The N64 is one of the worst post-video game crash consoles ever made. There isnt one legit RPG on the console (Dont get me started on paper mario as an RPG and tactical RPGs are as boring as the chess game they are modeled after) and its library of 296 games has about 50% of it consisting of sports and racing titles which is the gaming equivalent of landfill waste. The controller was made for a 3 armed alien and in the world of disks and games with massive space for massive content, it stuck with cartridges and limited the amount of content it could play on a single game. I will give the cartridge credit for its durability but thats as far as anything good that can be said about the N64.
That's a pretty hard knock at the racing genre. Racing games can be a lot of fun. I had a blast with San Fransisco Rush, Beetle Adventure Racing, Cruisin', and Mario Kart back in the day. Those are definitely not landfill waste. I guess if RPGs define gaming for you, then yeah, the N64 showed you no love. But, if you have a well rounded taste for other things, I don't see why you couldn't have a ton of fun with the N64. Action, Adventure, defining FPS titles, Platformers, Racing, Fighting (mostly in the form of Wrastlin') and stuff like that are covered well. No matter what label you give Paper Mario, I thought it was a great game (great series, in fact).
Does a console need to deliver on every genre before it's any good? Not really. Not if the games it does have are good, and well, superior to the competition's line up of the same games. There's some types of games I really like that are very poorly represented on the Saturn, but I still think it's a good console for what it does deliver on. If you are looking for something that has it all, PS1 was your safe bet. That doesn't mean it had the best games in every genre, so you can't really call it the best console just because it did have them. It's complicated, but the N64 was great, at least I think so.