HttpClient throwing "An error occurred while sending the request."
I face the exact same problem in production environment. Similar setup and about 30k http calls send from the client. Errors occur in a very infrequent way and it is very difficult to reproduce.
After reading numerous posts I think this Microsoft's bug when doing the connection pooling in HttpClient (in my solution I use httpClientFactory). You can take a look at https://github.com/dotnet/runtime/issues/26629
The approach followed to overcome this issue until Microsoft fixed the issue:
Retry policy. Polly has been used and when this exception occurs the call is retried after a couple of seconds.
Increased the request timeout to 3 minutes.
It seems to work fine for now but as I mentioned before it is hard to reproduce the error in a controlled way.
After some digging I resolved the issue. When A was sending the request to X, A was setting the Connection: keep-alive and the X was responding with Connection: Close property in header.
So after some calls A was exhaust with opened tcp connection and randomly it was throwing the error.
(Fiddler helped me to figure this out)
So all I had to do was setting ConnectionClose property of HttpClient
_client.DefaultRequestHeaders.ConnectionClose = true;