Stress
This really may depend more on your personality than on the work itself. There are plenty of professors who love the work, the limited hours of actual classroom instruction, and the additional perks and benefits so much that it overwhelms the frustration of not getting through to certain students, the lack of adequate pay, and the occasional senior prank that results in their cars being filled with Cheez Whiz.
There are others who, while they're passionate about teaching, get bogged down in all of the little things, or have a hard time self-motivating themselves to put as much time and effort into lesson-planning as they know they should. They have an absolutely critical job to perform and they know it, so the stakes are high. Boy, we're getting stressed out just thinking about it.