What is known about sufficient conditions for the rigidity of a convex surface?

Answer to question 1: proper compact subsets of convex surfaces are infinitesimally flexible.

In his book Pogorelov proves infinitesimal flexibility of the graph of a convex function over a strictly convex compact subset of the plane. (Basically the idea is that in order to find an infinitesimal bending of a surface with positive Gauss curvature one has to solve an elliptic PDE.) This implies infinitesimal flexibility of any proper compact subset of a strictly convex surface. Indeed, infinitesimal rigidity is projectively invariant (an amazing property given that the definition is in terms of distance). If you send to infinity a plane that cuts the surface in a neighborhood of one point, then the complement of this neighborhood becomes a graph of a convex function.

Answer to question 2: closed convex surfaces without flat pieces are infinitesimally rigid.

This is proved in the cited book of Pogorelov (Theorem 4 in Section 4.7). Clearly, any flat piece allows infinitesimal deformations: take any vector field supported inside this piece and orthogonal to its plane.

As to recent research in the field, one may consult a series of surveys by Ivanova-Karatopraklieva and Sabitov. A good textbook on the topic is the first half of volume 5 of Spivak's "Comprehensive introduction...".

One direction in which one can generalize rigidity results for convex surfaces is rigidity of hyperbolic manifolds with convex boundary. (Indeed, rigidity of a convex surface translates as rigidity of a Euclidean ball enclosed by this surface.) In this article:

Schlenker, Jean-Marc, Hyperbolic manifolds with convex boundary, Invent. Math. 163, No. 1, 109-169 (2006). ZBL1091.53019.

infinitesimal rigidity of hyperbolic manifolds with convex boundary is proved (using Pogorelov's theorem, by the way).


OP: "I'm curious about more recent research in this area." Here are two relatively recent papers. Ivan visits MO, so he may answer more definitively.

Izmestiev, Ivan. "Infinitesimal rigidity of convex surfaces through the second derivative of the Hilbert-Einstein functional II: Smooth case." arXiv:1105.5067. (2011).

"The paper is centered around a new proof of the infinitesimal rigidity of smooth closed surfaces with everywhere positive Gauss curvature."


Martinez-Maure, Yves. "Rigidity and Bellows-type Theorem for hedgehogs." (2011). Author's link.


Answer to question 1: If a convex surface is not closed, then generally it is far from rigid as it might admit infinitely many isometric deformations; however, if the surface has $\mathcal{C}^{2,1}$ regularity and positive curvature, then it becomes rigid as soon as one fixes an arbitrarily small curve segment on that surface. This has been proved very recently in a joint work with Joel Spruck, which has been accepted for publication in International Math. Research Notices (IMRN):

Rigidity of nonnegatively curved surfaces relative to a curve, arXiv:1805.02580.

This result may also be extended to nonnegatively curved surfaces under some additional conditions. The proof uses Hormander's unique continuation principle for elliptic PDEs. Our methods also yield a very short proof of Cohn-Vossen's rigidity theorem for smooth closed convex surfaces, via Hopf's maximum principle, which is included in the appendix to the paper.