Sharepoint - How to disable the new document option in a library which is supposed to be for document sets only

Edit the library web part and change Toolbar Type to 'No Toolbar'

This may not work in all cases, particularly if there are aspects of the toolbar you would like to keep.

Option 1: You could just remove creating new item permissions on that list, by breaking role inheritance for that list.

Option 2: Add event receiver on ItemAdding event and cancel item creation process by setting SPItemEventProperties.Cancel = true;. Examples are on the web.

Option 3: You can customize the Toolbars by following this link. http://blogs.msdn.com/b/syedi/archive/2008/07/19/customizing-the-list-toolbar-template-wss-3-0-moss.aspx

Link is for WSS 3.0 and MOSS, but I think you can do it for SP 2013 by replacing 12 to 15 in folder paths. I haven't tested my self