Odds of Hanging On
The hard part is getting in. Once you're a specialist in urology, it'll be hard to get rid of you. You'll need to work really hard, but that just comes with the job. If you play by the rules and don't do anything illegal, you'll be fine (which is usually the case). You may not have a lot of patients in a big, exciting city like New York, but it's easy to get rid of competition—just move to a different, less flashy city.
Specialists are a hot commodity, especially with the growing need for healthcare. You should have no problem holding onto your job unless, again, you're just a garbage doctor and have no idea what you're doing. But all that talk about the sheer difficulty of becoming a certified urologist pretty much negates the possibility of you being bad at your job. Rest easy.