Back a few years ago, when I was just getting into the miles and points game, I decided to overhaul my credit card collection, removing cards that were expensive and weren't great for everyday spending (like the old First USA United Airlines card) and adding new cards (like the SPG American Express).
What I didn't realize is I actually owned my old United Airlines card since the 90's, giving me 10+ years of on-time payment history. When I canceled the card, it didn't have much immediate effect on my score since it was still listed on my report. The real penalty came just about a month ago when the card's history actually fell off my credit report. Worse yet, the card fell off my report literally days before I applied to refinance my mortgage.
To give you an idea of the damage, my score on CreditKarma/TransUnion was at 772 prior to March, and it dropped to 732 a few days after the account fell off my report -- a total of 40 points. A month later, my score hasn't even started moving back up. Ouch!
Fortunately, the other credit reporting agencies didn't give me quite such a hit and my score was still sufficient to receive a good rate on my mortgage. On the flip side, I'm seriously worried about applying for additional cards in the near future.
Learn from my mistakes, and don't close your oldest account. If you want to get out of an annual fee on a card you don't use, ask your credit card company to switch you over to another no-annual-fee card under the same account. Typically, they're more than willing to do so.