The fear of making the wrong decision has skewed the way you have evaluated the options. Over time, you realize that you don't use the credit card often and realize that the annual fee is not worth paying, which leads you to regret your decision.
The intensity of the regret you feel impacts your overall experience with the bank, and you find that instead of switching plans, you end up canceling the account and moving to a different bank completely.