An autopilot deferred, by itself, is no problem. In fact I'm a staunch advocate of being able to hand-fly the airplane (raw data if need be) in any configuration at any time.
But, here's that thing:
Accidents (and incidents) are usually caused by many links in the chain. These links are usually not significant risks by themselves, i.e., deferred autopilot/pack/apu, weather, long day, FNG, short runway, etc. Many of these links are inevitable, and we willingly take them on knowing they are part of the task at hand.
At some point, though, the known links eventually, add up to make an unsafe situation - especially when we consider the latent risks inherent in any flight. It is up to pilots to decide when there are already too many links in the equation and either remove some of the links, or cut the chain altogether.
Unfortunately, the type and number of links allowed is not spelled out in black and white. And, there is only ONE person on the planet who can make that call.