Not crazy at all if you objectively look at the circumstances during those 3 years. Trestman, injuries(Joe and the team), poor receivers, etc., etc. all contributed to a lack of confidence and poor play. One can only imagine how things could be if there were a few years of capable consistency at OC, receiver and TE, basic things most high performing QBs have.
i think also we were seeing the impact of all the dead money we still had tied up in players who werent on the team or were sitting on i/r - that money is now being used to pay guys like willie snead and john brown