How to draw a filled rectangle without a border using TikZ?

The border of the rectangle is drawn because you are using the \draw command, which, by definition, draws the specified path (here the rectangle). The fact that you are adding the option fill=orange just tells TikZ that in addition it should fill the path with orange. Replacing \draw with \fill will do what you are expecting.

\documentclass{article}
\usepackage{tikz}

\begin{document}

\tikz \fill [orange] (0.1,0.1) rectangle (0.2,0.2);

\end{document}

As noted in the comments, another option is to use \draw [fill=orange,draw=none]. This amounts to telling TikZ to draw the path with an invisible color and fill it with orange, hence the result is the same. As drawing the path with an invisible color is maybe a little strange, it is perhaps more natural to use instead the general \path [fill=orange]. Note that \draw is simply a shortcut for \path [draw] and in a similar manner, \fill is a shortcut for \path [fill]. Similarly, you have the command \filldraw which is an abbreviation for \path [fill,draw]. In full generality, colors to use for either filling or drawing can be specified by replacing fill or draw by fill=color or draw=color in the options.

All these commands are documented in section 15 of the TikZ manual (and especially 15.3 and 15.4), with details and examples (accessible by the command texdoc tikz in a shell or here).

To avoid some confusion from the selected answer, another option is to use path

\path [fill=orange] (0.1,0.1) rectangle (0.2,0.2);