Collecting physical PC games is fairly easy. While it has definitely become more popular, it's a fraction the cost of retro console collecting overall. Thrift stores still sell boxed PC games for really cheap. I definitely look all over the palce for these when i go looking for them; I've found them with board games, VHS tapes, books, and even in the electronics section. There is always Ebay as well which is going to be more expensive.
As for getting them to work, luckily many of the more popular PC titles from back in the day are available on Steam, GOG, and other online shops, and have been optimized to run on modern OS. Sometimes they take some tinkering, but overall this is the easiest method for playing classic PC games. For those games that were never released digitally, it's really a mixed bag. If it's a DOS based game, getting it to run in DOSbox is ultra easy and it works 99% of the time. Where things get tricky is with the Windows 9x OSes, and to a lessor extent Windows XP. I've got games native to all these operating systems to work in Windows 7, but doing so can often be a huge headache since most were made to run on a 32-bit OS, where most modern OS are 64-bit. Technically 32-bit should be compatible with 64, however it really boils down to how the game was programs and what it has to do to install.
Because of this, I've chosen to build a Windows 98 gaming PC using legacy hardware. It runs pretty much all mid to late 90s PC games with little fuss. I would also like to build a Windows XP gaming rig someday, but for now there are only a handful of XP games I've been unable to get running in Windows 7. With Windows 7 no longer being supported by Microsoft, it's hard to say what it will be like trying to run these games in Windows 10. I haven't really tried for the most part, but I'm sure there are guides on how to get some games running on that OS.