when to use nonce?
You want to use a nonce when:
- you care deeply that a transmission not be submitted twice
AND/OR
- you have reason to be afraid that re-transmission could occur by someone other than the authorized user, and that the ramifications of that will be bad
Transactions where money changes hands is a good sample use case - I care deeply that I not buy the same product twice without knowingly consenting. Also - any authorization case - both when a client transmits a credential to the server and when the server performs authentication checks to credential verification services. In either case, if a third party hears the transmission, it could be repeated at a later time in imitation of the legitimate source.
The general principle is that if I am the concerned party, I can give you a suitably unique, unpredictable piece of data, and have you transmit it back to me along with proof that you are who you say you are - the proof of your identity and this suitably unique data should be melded together in such a way that a listener couldn't replicate them without having both the unique information AND your credentials.
A web form is a great context for this, as there are plenty of good cases. But it's not the only option for the implementation of the basic concepts - for example OCSP uses a nonce in certificate lookups (totally non-web - it's all in ASN.1). I don't want to add confusion, but I want to emphasize that a nonce is a security concept, not a thing that is unique to a given technology or protocol.
Taking the questions one by one:
1.what is the "challenge" that sent to client?
Something suitably random. As usual, perfect randomness is considered impossible - but something that is unique enough that it is not repeated predictably or frequently. So as an attacker, I can't build a stockpile of useable transmissions and reuse a previously used nonce.
2.Server sends challenge to client, how? In the session? hidden input in form? url?
In a UI sense, you want to abstract it from the user, since they have no part to play in transmitting it verbatim. The way you package this is entirely related to the problem you are trying to solve. For example:
in an OSCP request/response, the nonce is a field in the ASN.1 - it has an object identifier as specified in the protocol and a value. Off the top of my head, I can't remember but it may be limited to integers.
if you are trying to attach a nonce to a webform, a hidden value in the form input parameters is a great idea. Generally sticking this in a URL is bad idea, as URLs fit into navigation and bookmarking, and this is not a repeatable piece of data - the next time I submit the form with this nonce, I should be denied, so if I have bookmarked a URL with a nonce in it, I should be very frustrated.
In XML transactions, I believe there's an option for including the nonce in the XMLDSIG schema - it's an element of its own - this is often used in SOAP web services.
3.How client responds with hashed password? is client=php?
This points back to the point that a nonce is a concept - not an implementation. Many APIs offer a nonce as a parameter to the request initiation - usually a boolean - TRUE/FALSE on providing a nonce at all.
In any case, however the client responds, there needs to be a connection between the nonce and the private credential that proves the client's identity. Hashing the nonce and the password may be an approach - but it's not an approach I've used. But then my background is largely PKI related. If you are using asymmetric keys then the typical thing would be to sign the nonce as part of the package - so the response would be a collection of information that matters for the transaction (my shopping cart order, for example), information about who I am (for example, my digital certificate or username), and my nonce. How I package this is a design detail for my application.
4.what type of security this approach provides?
Uniqueness.
It ensures that this exchange is happening "live" - that the response is not a canned response that could as easily be given by a malicious entity as by a server which pregenerated responses and it ensures I can't send the same transaction twice and have it work both times (replay attack).
There's a cost - the cost of computation and processing. In a high load situation, demanding re-generation and cryptographic binding for every transmission can get expensive. Not radically expensive, but for very high loads, even a little bit becomes a big deal. It also forces persistence on the requestor - the requestor has to make a nonce, send it, remember it, and compare it when the return response is provided. If there is no request/response nonce comparison, there is no value to providing the nonce in the first place. Which means there's a memory and computation cost there, as well. Also if you are custom building the transaction for a specialized purpose, it adds development time/cost. Obviously that's a one-time cost - but it's still work that needs doing.
A random value.
Usually as a hidden value in a form.
Normally, we don't want the protocol that @Thomas described. What we want is for you to use SSL and hash the password on the server side.
This prevents replay and blind submission (CSRF) attacks.
If you've ever bought something online and seen a page at the end that says, "Don't refresh or resubmit this page or you'll be charged twice," then you've found somebody who should have used a nonce.
By creating a (sufficiently large) random value that is attached to the form that should be submitted, you can ensure that the form is only used once. Further, CSRF is prevented because you must first visit the form page in order to generate receive a valid nonce.
An encrypted login form probably doesn't need a nonce, but a nonce is effective in hashing a password uniquely to prevent replay over an insecure channel.