Automated URL Discovering
Discovery of URLs in the wild tends to be done in one of three ways
- Search Engine Based. If your content gets indexed by a search engine, it's discoverable. this doesn't just include them hitting your site directly, if other sites link to the content it may be findable that way.
- "known URLS". There's a lot of URLs that are known to be things like administrative interfaces or CGIs with known issues. Tools like nikto maintain a database of them and can be used by attackers to discover systems which have that software installed.
- Brute-force. Tools like OWASP DirBuster combined with lists of common directory/file names can be used to discover content.
In terms of an untargeted attack unless you have a common vulnerable piece of software (e.g. phpmyadmin) hosted on a default URL, it's unlikely you'll have a problem as the sheer number of sites on the Internet make it unlikely that you'll face a human attacker focused on finding content on your site.
With a targeted attacker tools like nikto and dirbuster would be usuable to try and find content on your site.
If you have information that you don't want to be found I would recommend adding authentication and SSL to it to prevent unauthorised access to it.
One way would be to use a tool called ZAP, it contains a fork of DirBuster. You can use dictionary forced browsing to try common urls.
ZAP is a free tool from OWASP