Stress
For those priests who got into the game earlier and didn't lock down a spouse before entering the seminary, the stress of a lonely life can be overwhelming. The more significant stress factor, though, will be dealing with those incredibly flawed, confused, rude, and sometimes unpleasant human beings at your parish.
You'll have to do that while still being calm, patient, and loving, like you promised you would be. Even if people aren't being terrible to you, you are still going to have to listen to their problems for hours on end. That's a heavy load—especially if you can't go home and complain to your spouse about it (source).