We've actually missed the playoffs 3 out of the last 4 years.
2013: 8-8 (missed playoffs)
2014: 10-6 (made playoffs; lost in Div round)
2015: 5-11 (missed playoffs)
2016: 8-8 (missed playoffs)
Also, if you take our combined record after the SB win (including this year) then we sit at 34-36 which is the definition of mediocre. So, if we miss the playoffs again this year (very likely if we don't fix the offensive issues) then that would be missing the playoffs 4 out of the last 5 years. Most head coaches would be, at the very least, on the hot seat after that regardless of any excuses like having an injury-riddled team.
I like John Harbaugh and he's done a great job in his time here until recently, but it may be time to start thinking about going in a different direction. This is a results-oriented league and we're not getting the results that we want right now. I would say that Harbaugh is gone after next season if we fail to make the playoffs this year and next. He still has time to right the ship, but if he can't do it then it's time for a new head coach with a different philosophy.