How not to lose confidence in front of supervisor?
I experienced something similar at the beginning of my PhD. It was mostly due to the fact that as I was verbalising my ideas, I'd realise their flaws, that they don't quite make sense, or that I hadn't it as figured out as I thought.
I suggest you try to verbally explain your ideas to someone else (if no one else, plushies are great listeners) before you talk to him, that'll give you extra confidence. It is important that you speak out loud, and from the beginning: don't skip right to the meat of the question, your problem may be in the fundamentals.
If this is your problem, once you get the hang of it, you'll quickly learn how to weed out some of these bad ideas, and gain confidence on your explanations.
Something I have seen my students struggle with (and probably myself too, but I don't notice it so much) is that when I ask them a question, they try to answer it as soon as possible, so they get stuck in suboptimal explanations, increase their confusion, get more stressed, and lastly blocked. He is giving you time, so take it, think carefully about what you are going to say, and explore different ways of expressing it before you start saying it. And furthermore, if you find a better way of explaining it, feel free to scratch what you just said and start all over again.
Another option is to set the the ground for a discussion by sending a long email taking your time to explain everything. And if you feel you didn't manage to get your point across, never hesitate to send it after the fact.
Lastly, for a few weeks I got the feeling that my supervisor wasn't really understanding what I am trying to do; but after I gave a 30 min presentation of my work in the department, from the bottom up, he got very excited, and now his comments are, I feel, much more spot on. The presentation included explaining my tools for people completely unfamiliar with them (to be specific, a specific technique built on deep learning for machine [but not deep] learning practitioners).
For your case, consider asking for some time, thinking on it on your own, and coming back with an explanation whenever you have it.
What you are lacking is confidence in yourself. What I see commonly is a student who has such respect (rightly so, I'll assume) for their advisor that every debate is interpreted as 'I am wrong'. This isn't correct. Academics love to debate, and fruitful discussions question every answer. So, when your advisor questions your stance, know that he or she is engaged by you, not necessarily questioning your thesis per se.
Secondly, it is important that you become confident enough to say 'I don't know'. You can't know everything, and your advisors job is to push your boundaries. Feel confident, and admit when you don't know something but retort with 'excellent point, I need to look into that'.
Those who admit a lack of knowledge often glean more respect than those who feign knowledge or become combative.
To gain a foothold, prepare well for your next meeting. Read 5 or so papers on the topic and know them well; refer to them during your discussion. Having that armory will give you more confidence than you think.
I have not tried this, but I have seen that some people are able to increase their confidence in public speaking by taking some acting classes. I don't mean you would need to make a full hobby of it, but maybe one semester would be helpful.
If you are not very comfortable talking while working out some math on a whiteboard in general, then -- it's time to form a study group! In my opinion, there are very few things in life as fun as working out math problems on a board with a friend or a small group of friends. Try out various levels: problems you could do in your sleep, problems where you're just making up nonsense as you go along, and everything in between.
Edit: Food for Thought About the Language Aspect
I had a study partner in grad school who was from China. One day he mentioned that he had to go home in order to study his Russian before we could get together. I didn't understand why he had to go home in order to do that. He explained that when he did his Russian study he would pronounce all his exercises in a very loud voice -- actually, he said that he shouted -- and he wasn't comfortable doing that in his office.
My first six months living in a foreign country I was very quiet. People who met me at that time thought I was just the quiet type. Actually, I was still getting comfortable using the language. Over time my natural personality came to the fore. It just took time. Eventually I became completely bilingual.
Some years later my family situation changed and I needed to learn some German in order to communicate with my new in-laws. But I only saw them twice a year, and without the immersion, my German never really took off. I've always been a perfectionist, but at some point I decided that German grammar was never going to make sense to me and it was more important to just crash through a sentence and get my point across as best I could. I decided that functioning in family life during visits was more important than getting it right. I discovered that you can actually train yourself to allow mistakes. I was able to come to terms with speaking pidgin German.
In grad school, in the summer, I organized free "English for Public Speaking" classes for my international fellow students and their spouses. Each student chose a poem to work on and, inspired by my friend's experience studying Russian, I asked them to recite their poems very loudly. I gave them warm-up exercises to loosen up, such as the following: "Give me a breaaak" while shaking one's head. We had a short exercise for each vowel sound, such as "How now, brown cow." Students were supposed to use a small mirror to check proper mouth shape. Even though they were already using English at a sophisticated level in their studies, I noticed certain patterns of grammar errors. We practiced those basics quite a bit, so that it would become second nature to form negations, questions, and indirect speech, with the correct auxiliary and the right word order. By making sure that these grammar basics and pronunciation basics were correct, the students' confidence really grew. (I'm not sure that they had ever had individualized feedback, when they were learning English in their home countries; and once they were in the U.S., others felt it would be impolite to correct them.) Another thing we worked on was: in preparation for the thesis defense and job talks, each student wrote a short presentation about a non-science topic, for example, one student wrote a one-page description of his home country, Singapore. When a student read his presentation to the group, the most important part was the Q & A at the end. Everyone was encouraged to ask questions. The presenter was supposed to repeat each question before responding to it.
A conversation partner might be helpful. Some universities facilitate matching people up, for example, someone from France might want to polish his English, and he would be matched up with someone who wants to improve his French.
Some towns have literacy volunteer programs, and some of these programs are set up for international students who want to improve their English.
Reading equations, and talking about math, involves some special expressions that aren't obvious from looking at what's written on the page, for example, [(x + 3)^2 -1]/x would be read "x plus 3 quantity squared minus 1 all over x." It's easier to relax when you know you're saying things in a natural way.
Questions: are you more out-going in your first language than you tend to be in English? Do you sometimes feel doubtful whether you're saying something correctly? Do you think you have a noticeable foreign accent? If so, how do you feel about that?