You know, that's the direction my thoughts were first going as I wrote it, but then I realized that during both Sam and Dean's year off, the bad things let them be (except Dean there at the end, when Sam was involved). Seems unlikely that it should be so, but maybe they really could just walk away from it. It's a nice idea, in any case ....