Applying non-Affine transforms to 2D polygons with textures

I think the "ugly thing" might be because texture is interpolated on triangles (demonstrated after), and a quadrangle is only divided into 2 triangles - up-left and down-right. So to solve the problem, we just need a triangulation network with much higher resolution. One way is to use ParametricPlot:

ParametricPlot[
               Evaluate[tr@{u, v}], {u, 0, 1}, {v, 0, 1},
               PlotRange -> All, Mesh -> False, Axes -> False, Frame -> False,
               PlotStyle -> {Opacity[1], Texture[ExampleData[{"Texture", "Bricks3"}]]},
               TextureCoordinateFunction -> Function[{x, y, u, v}, {u, v}]
              ] //
Show[{Graphics[{FaceForm[None], EdgeForm[Red], p1,
                EdgeForm[{Blue, AbsoluteThickness[4]}], p2}],
      # }]&

Edit:

To demonstrate that texture is interpolated on triangles, we map a regular grid:

txtr = Plot[I, {x, 0, 1},
            PlotRange -> 10 {{-1, 1}, {-1, 1}},
            GridLines -> {
                          {#, Darker[Green]} & /@ Range[-10, 10],
                          {#, GrayLevel[.8]} & /@ Range[-10, 10]
                         },
            AspectRatio -> 1, Frame -> True, FrameStyle -> Directive[Red, Thick],
            AxesStyle -> Directive[Blue, Thick],
            FrameTicks -> {
                           {Range[-10, 10], False},
                           {{#, Style[#, Black]} & /@ Range[-10, 10], False}
                          }
           ]

onto a pentagon:

Graphics[GraphicsComplex[{{0, 0}, {1, 0}, {1, 1}, {1/2, 1}, {0, 1}},
             {Texture[txtr], EdgeForm[Black],
              Polygon[Range[5],
                      VertexTextureCoordinates -> {{0, 0}, {1, 0},
                                                   {1.5, 1.3}, {.5, 1.2}, {-.7, 2}}
                     ],
              AbsolutePointSize[10], Point[Range[5]]
             }
        ]]

So we can see clearly, the pentagon is divided into 3 triangles.

You can use something designed for creating such ilusion :)

pic = Import["ExampleData/lena.tif"];
Manipulate[
 Graphics3D[{Texture@pic, Polygon[{{0, 0, 0}, {0, 1, 0}, {1, 1, 0}, {1, 0, 0}}, 
                          VertexTextureCoordinates -> {{0, 0}, {1, 0}, {1, 1}, {0, 1}}]}, 
  ViewVertical -> {0, 0, 1}, ViewVector -> {{-.2, .5, h}, {100, .5, 0}}, 
  ViewAngle -> vAngle Degree, ImageSize -> {400, 400}],
 {{vAngle, 150}, 10, 179, 1},
 {{h, .7}, 0, 2}]

This is kind of joke answer and I will probably delete it later :) But worth to show I think.