Shorten a line by a number of pixels

You can use similar triangles. For the main triangle, d is the hypotenuses and the extension of r is the vertical line that meets the right angle. Inside the circle you will have a smaller triangle with a hypotenuses of length r.

r/d = (x2-a0)/(x2-x1) = (y2-b0)/(y2-y1)

a0 = x2 + (x2-x1)r/d

b0 = y2 + (y2-y1)r/d

I'm not sure why you even had to introduce the circle. For a line stretching from (x2,y2) to (x1,y1), you can calculate any point on that line as:

(x2+p*(x1-x2),y2+p*(y1-y2))

where p is the percentage along the line you wish to go.

To calculate the percentage, you just need:

p = r/L

So in your case, (x3,y3) can be calculated as:

(x2+(10/L)*(x1-x2),y2+(10/L)*(y1-y2))

For example, if you have the two points (x2=1,y2=5) and (x1=-6,y1=22), they have a length of sqrt(7² + 17² or 18.38477631 and 10 divided by that is 0.543928293. Putting all those figures into the equation above:

  (x2 + (10/l)      * (x1-x2) , y2 + (10/l)      * (y1-y2))
= (1  + 0.543928293 * (-6- 1) , 5  + 0.543928293 * (22- 5))
= (1  + 0.543928293 * -7      , 5  + 0.543928293 * 17     )
= (x3=-2.807498053,y3=14.24678098)

The distance between (x3,y3) and (x1,y1) is sqrt(3.192501947² + 7.753219015²) or 8.384776311, a difference of 10 to within one part in a thousand million, and that's only because of rounding errors on my calculator.

Find the direction vector, i.e. let the position vectors be (using floats) B = (x2, y2) and A = (x1, y1), then AB = B - A. Normalize that vector by dividing by its length ( Math.Sqrt(xx + yy) ). Then multiply the direction vector AB by the original length minus the circle's radius, and add back to the lines starting position:

double dx = x2 - x1;
double dy = y2 - y1;
double length = Math.Sqrt(dx * dx + dy * dy);
if (length > 0)
{
    dx /= length;
    dy /= length;
}
dx *= length - radius;
dy *= length - radius;
int x3 = (int)(x1 + dx);
int y3 = (int)(y1 + dy);

Edit: Fixed the code, aaand fixed the initial explanation (thought you wanted the line to go out from the circle's center to its perimeter :P)