Stress
You won't find yourself in too many life-and-death situations, but even still, there's a lot of pressure to correctly diagnose your patients and prescribe the appropriate medication or program of treatment. If you lose a patient, that stinks. If you pass them along to the wrong specialist and they prescribe the wrong medication or treatment, that also stinks, as it's still kinda on you.
Additionally, it can just be kind of depressing to be around sick, injured, and generally miserable people all day long. You know how it is when someone won't stop complaining about something for a few minutes? Try all day, every day. Granted, most of your patients have good reason to whine, but that doesn't make it any more uplifting.