Can't set toAddresses on email when templateId defined?

Using Newly introduced method setTreatTargetObjectAsRecipient() in Winter 16, you can send email message to any email address.

    Messaging.SingleEmailMessage email = new Messaging.SingleEmailMessage();
    email.setWhatId(recordId);
    email.setTemplateId(templateId);
    email.setToAddresses(new String[] { toAddress });   

    //Set current user as target object 
    email.settargetObjectId(UserInfo.getUserId()); 
    email.setsaveAsActivity(false);

    //This method would make sure email will not be sent to user
    email.setTreatTargetObjectAsRecipient(false); 
    Messaging.sendEmail(new Messaging.SingleEmailMessage[] { email });

I am fairly certain that you have always been required to use a templateId and targetObjectId in tandem. All of my code that sends an email using a template requires a contact to be associated to the email, and I am fairly certain that this was because the system required this condition when I wrote the code a few years back. The problem is that without a targetObjectId, the template can't merge contact fields correctly, so they probably made this restriction intentionally.

You don't need to use a contact, though, as leads are also acceptable targets. If your leads are less burdened than contacts are, you might simply choose to create a new lead instead (but then you can't set a "what id", such as an opportunity). In any event, you could probably just patch your triggers to ignore any contacts created when a certain flag is true, and use a @future method to (hard) delete the contact once the email has been sent. Of course, if you have installed packages, this may not be feasible.