We had a Nintendo 64 growing up, but the only game we played (and, truthfully, needed) was Smash Bros. I didn't mind the controller. I appreciated that it looked like the end of Poseidon's trident. I actually think it's better than the Game Cube controller, but that's another discussion for another time. It's not my least favorite of Nintendo's consoles, but it has issues.
-The majority of the non-sports and npn-racing games were collectathons. In fact, Quest 64 was a collectathon disguised as an RPG.
-Speaking of RPGs, there was nothing there except Ogre Battle 64. If not for Ogre Battle, the system would be an RPG flop. (I haven't played Paper Mario or Hybrid Heaven, but I never had the desire to do so.)
-Kirby 64 was basic as hell. As a Kirby fan, I was very disappointed.
-Early 3D graphics were fugly. Other consoles are guilty of this, but this was supposed to be the N64's claim to fame.