Mathematical theory of aesthetics
George D Birkhoff, Aesthetic Measure, 1933
An attempt to bring the basic formal side of art within the purview of simple mathematical formula defining aesthetic measure. Contents: the basic formula; polygonal forms; ornaments and tilings; vases; diatonic chords; diatonic harmony; melody; musical quality in poetry; earlier aesthetic theories; art and aesthetics. Over 20 plates and illustrations.
A search for aesthetic* in the title at MathSciNet yields 100 hits, e.g.,
MR3751155 Lähdesmäki, Tuuli – Aesthetics of geometry and the problem of representation in monument sculpture. Aesthetics of interdisciplinarity: art and mathematics, 275–290, Birkhäuser/Springer, Cham, 2017.
MR3751140 Cohen, Mark Daniel – The geometric expansion of the aesthetic sense. Aesthetics of interdisciplinarity: art and mathematics, 29–43, Birkhäuser/Springer, Cham, 2017.
MR3644156 Pimm, David; Sinclair, Nathalie – Explaining beauty in mathematics: an aesthetic theory of mathematics [book review of MR3156013]. Math. Intelligencer 39 (2017), no. 1, 79–81.
MR3623974 Kao, Yueying; He, Ran; Huang, Kaiqi – Deep aesthetic quality assessment with semantic information. IEEE Trans. Image Process. 26 (2017), no. 3, 1482–1495.
One name and two books come to mind: Joseph Schillinger, and his two books, The Mathematical Basis of the Arts and The Schillinger System of Musical Composition.
The Mathematical Basis of the Arts is a work that aims at generalizing the concepts present in art pieces, in general, from a geometrical point of view, and how this affects the human perception mechanism. I cannot comment too much about this book because I have read it but haven't really study from it. Schillinger's works are clear, but the (natural) language he uses, and the strange mathematical context/notation make it so that some time and practice are needed to really absorb his ideas. A quick look at the table of contents should wet anyones appetite.
The Schillinger System of Musical compositon is as much a philosopical corpus about aesthetics based on geometrical/psycological/physiological arguments, as a theory of composition. It is the only work that I am aware of that has a philosophical theory of melody. That is, it throughly studies what makes a melody what it is, and not just how to write a melody. The principles that he presents and develops apply to art in general, and not only to music, however, he presents his ideas in the context of musical theory.
I should also say that his is not so much a mathematical theory of art/music/aesthetics as a general theory of aesthetics. It's just that he uses the concepts and language of mathematics to present those general concepts. In doing so, many people come to believe that he is developing a mathematical theory of art/music. He's not. As he points out, he is developing a scientific theory of how we create and perceive art, and thus, effectively, he's developing a formal theory of aesthetics, since beauty is always in the eye of the beholder.
Schillinger's work is from an epoch that believed that human intellect trumped statistical analysis. And that is how it is developed. From a rational derivation from a few basic (universal?) principles, and not from a brute-force approach to find regularities in works of art.