First off, there are actually two Persona 2's: Innocent Sin and Eternal Punishment. Innocent Sin was never released in the U.S. before the PSP version because of an epic fight with Hitler, among other things. So when you are considering getting the PSP game, know that you are probably going to be investing in Eternal Punishment as well. (Which at the moment is still only PS1. There's an announced version for the PSP in Japan but nothing about localizing it yet. And with the Vita's inability to transfer UMDs in the U.S., unlike Japan, who knows if they'll bother.)
Also, the Persona 4 gameplay and story is going to be much closer to what you are playing now. Persona 3 was sorta like Grand Theft Auto 3, they changed the gameplay enough that a lot of new people started there.
Not trying to skew your view of the games or try to convince you to go for one or the other, but I do know that most of my friends have loved 3 and 4 but hated 1 and 2. Actually...the only guy I know that loves them all is the guy that introduced me to the Shin Megami games...and I think he did that with Persona 3 and Digital Devil Saga for a reason...he knew I'd be hooked and go back and play them all.