I agree that it is a fair comparison, but I also think the division and the system are a big part of it.
In 2008, the Patriots went 11-5 with Matt Cassel at the helm. How many times has Eli Manning won 11 games or more? Only twice by my recollection. This isn't to compare Eli to Brady, but Matt fricken Cassel managed 11 wins in that division and offensive scheme.
This is my argument. I think by giving Brady so much credit you're taking away from what Belicheck does. Tom Brady is still a great player worthy of top 5 all time consideration, but having Belicheck to back him up helps tremendously. Just like Eli is boosted by having TC to back him up. Every coach brought up in the Parcells regime knows how to make do with what he has. Missing the playoffs at 11-5 is just a fluke, plain and simple. If the Dolphins weren't so good in 2008 and the Patriots had gone to the playoffs that year, Cassel could have led that team to a Super Bowl victory, just like Hostetler did for us when Simms got hurt.
If your drinking game consisted of taking a shot every time the Broncos got a 1st down, you might as well just smoke a joint.