Popular games from the era of 'big boxes' can get expensive, but if you aren't concerned with cardboard then PC game collecting can be very cheap and easy to get into.
A few years ago I found a copy of
Microsoft Adventure for the IBM PC (arguably the first PC game ever) and decided to start visiting thrift/second-hand shops to look for old PC games: before the end of that year I had nearly 1000 titles.
My advice is to decide now what your budget is and what sorts of games you DON'T want to buy. Some people don't do magnetic floppy disks, others only do floppies and don't want optical media. Some people focus on DOS/Windows while others are all about the Commodore Amiga, Apple/Macintosh, Atari ST, Tandy 1000, MSX, etc.. Many PC game collectors are only interested in the 'big box' releases and want them to be the first version from the country of origin with all the registration cards and whatnot inside - but if you just want a functioning copy there are later editions, collection releases, foreign market versions, and so forth which can allow you to get the piece of software you want without paying top dollar.
I try not to buy Sims expansions, games designed for very young children, shareware, shovelware, mahjong/slots/go, hidden object games, MMOs, and of course anything new enough to require online registration or validation since the codes will almost certainly already be used (if the applicable server even still exists). I decided that I'll only spend up to $3.00 (USD) on a PC game that is loose or otherwise not in its box, or maybe up to $10 for a game that is complete in its original box - but then only if I recognize it as a good game. Games that look generic or downright bad have to cost less than one dollar for me to consider buying them.
Just today I made an offer to an eBay seller (who offers 'combined shipping' and knows about the USPS Media Mail rate) for nearly 200 PC games at $0.50 each and they accepted! Included in the lot are many titles I'm looking forward to such as
Ripley's Believe It Or Not: The Riddle Of Master Lu,
Jagged Alliance: Deadly Games, and
The Aftermath expansion for Red Alert.