Stress
With authority comes responsibility, and with baseball that means responsibility to millions of sports fans. If you win, they'll carry you around the city hoisted on their shoulders. If you lose, they're coming after you—and not in a good way.
You may not be the one at bat or on the mound, but you'll be the first one to get the boot if the team starts to stink. Job security might make your hair fall out (if you have any left to begin with). You'll also have to deal with late nights, early mornings, long road trips, and constant decision-making.
You might want to consider doing yoga.