Stress
You may have a lot of authority at your hands, but as the human arachnid said, with great power comes a great big headache.
Regardless of the venue, chances are most people who'll be affected by your inspection don't want you there―and most of them won't even know you're coming until you get there (source). It's like a pop quiz, only failing it means they lose money, time, and maybe a lot of customers.
Many of these workers and managers are going to be friendly, if a little on edge. Some of them will be defensive and concerned, and every second you're there, they'll be wishing you away. One or two may actually be hiding something, and they won't be very helpful. Watch out for the person who smiles just a little too much.
Then there's the actual process of inspection. In your average day, you may encounter dangerous machinery, unhealthy toxic chemicals, incredibly tight spaces, mounds of dust, and nests of pests like rats or cockroaches or things right out of a horror movie. And that's just at your regular breakfast place down the street.