How should I verify a caller is from the bank or company they claim?

I would refuse to give out any personal details to anyone that called me as you can't verify who they are.
If they need to talk to you then say you can call back. You can then call through on the direct number, which if it is a large bank will be well known and on their website. You could then ask for an extension number to direct your call once you know you are through to your bank.

Should I refuse to give even partial answers to my security questions in this situation?

You should. Mutual authentication on the telephone really isn't trivial but important. One of the more common ways is always having the same person call you. A lot of banks will assign you a specific accountant who is responsible for you as a customer. Talking to him/her in person before makes it reasonably secure to talk to him on the phone by either recognizing his voice or arranging a keyword or something before.

A more simple approach: Call your bank and tell them about your concerns. Ask them for a solution; you might not be the first customer thinking about those problems.

Call the provider back on their publicly-listed phone number from another line

My concern is that I do all this without any way of checking their identity. I could be giving these details and any subsequent discussion of my account to an anonymous third party.

Both ends deciding the other is who they claim to be is not a trivial problem. As you say, caller ID can be spoofed; in any case not everyone has caller ID.

Should I refuse to give even partial answers to my security questions in this situation?

I have refused to give any details or discuss anything without first verifying the source of the call. If the information being asked for can be used to verify you, it could be used by a malicious third part to impersonate you if they obtain it.

What is a sensible alternative?

Ask for the name of the person making the call and the broad purpose, stating that you want to call them back about it.

If you have your provider's phone number, call them on that; don't call a number back that is supplied to you during the call without verifying it elsewhere using a source you trust (phone book, provider's own website) first.

I have also heard reports¹ of phishing/scam attempts which say words to the effect "This is your bank calling, please call us back on our number to discuss an important matter <click>". Ostensibly they have hung up, but the line is still active.

This is discussed further on another QA here.

If you try to dial back at that point you are still talking to the phisher. This is a problem with landlines as opposed to cell phones; an ongoing call on a cellular device will be indicated. If you were called on a land line, it is worthwhile to call back on a different line.

So, in short:

  1. Ask the person who they are or who to ask for to discuss this
  2. Thank them and end the call
  3. Look up the provider's number via a trusted source
  4. Call that number using a different line if on a landline
  5. Go through your normal authentication process

¹ For example, an article on The Register from 2009 claiming this is the case:

Scammers posing as representatives of phone service providers, such as BT, are calling up UK subscribers in an attempt to trick prospective marks into handing over credit card or bank details under threat of disconnection.

Plausibility is added to the scam by a trick designed to fool people into thinking that their line has been temporarily cut off, ostensibly under the control of the person calling them.

This happens after fraudsters are challenged to establish their identity as a representative of a telecoms carrier. In response, con men tell their prospective victims to hang up a phone and attempt to try phoning someone, claiming the line will be disconnected to prevent this. This supposedly establishes that conmen are calling on official business.

In reality, the fraudster stays on the line with the mute button on. Because the person who initiates a call is the one to terminate it, a prospective mark is left unable to make a phone call, or even obtain a dial tone.

This is based on a "BT and Ofcom warn(ing)" but I haven't found that yet.

The Money Advice Service also mentions it:

Known as the ‘no hang-up scam’, this is where the fraudster keeps the line open, spoofs a dial tone and the fraudster’s accomplice answers and impersonates whoever the victim thinks they are trying to call.