Detect rectangular portraits of people on images with OpenCV

For detecting rectangular portraits (headshots), I've had some success with the following methodology.

1. Rectangle Detection:
   a. Convert to grayscale
   b. Change color of image border to background color
   c. Binary Thresholding
   d. Closing Morphological Transformation
   e. Invert image if necessary
   f. Find contours
   g. Choose rectangular contours based on aspect ratio and area
2. Face Detection: Find the rectangular contours that contain headshots using Haar cascade. To be a headshot portrait, a rectangle should contain only one face with dimensions of face specified relative to size of the rectangle. For example, when using OpenCv CascadeClassifier.detectMultiScale, set minSize=(0.4 * width, 0.3 * height) for face size where height and width are the dimensions of rectangle.
3. Portrait Validation: Check for grid structure of portrait rectangles. For missing rectangles in grid, check if face exists. For existing rectangles, use grid structure to correct rectangle dimensions if necessary.

---

1. Python code for rectangle detection (It should be easy to convert to Java.)

img = cv2.imread('example.jpg')
gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)

# Remove black border by cropping
bw = 6 # border width
ht, wd = img.shape[:2] # height, width
gray = gray[bw:ht-bw, bw:wd-bw]

# HISTOGRAM -- Put histogram function here to determine the following:
bg_color = (235,235,235) # background color
thresh_value = 220

# Add back border with background color
gray = cv2.copyMakeBorder(gray, bw, bw, bw, bw, cv2.BORDER_CONSTANT, value=bg_color)

# Binary Threshold
thresh = cv2.threshold(gray, thresh_value, 255, cv2.THRESH_BINARY)[1] # orig: 235

# Closing Morphological Transformation
kernel = np.ones((5,5),np.uint8)
closing = cv2.morphologyEx(thresh, cv2.MORPH_CLOSE, kernel)

# Invert Image
closing = np.invert(closing)

# Find contours
cnts = cv2.findContours(closing, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
cnts = cnts[0] if len(cnts) == 2 else cnts[1]

# Find portraits by specifying range of sizes and aspect ratios
img_area = ht * wd
for cnt in cnts:
    x,y,w,h = cv2.boundingRect(cnt)
    if w*h < 0.005*img_area or w*h > 0.16*img_area or h/w < 0.95 or h/w > 1.55:
        continue
    cv2.rectangle(img, (x, y), (x+w, y+h), (0, 255, 0), 2)

cv2.imshow('Result', img)
cv2.waitKey(0)

---

Example 1 Result (First image is after inverting.)

Example 2 Result

---

2. Python code for face detection

def is_headshot(cnt_img):
    gray = cv2.cvtColor(cnt_img, cv2.COLOR_BGR2GRAY)
    height, width = cnt_img.shape[:2]
    min_size = int(max(0.4*width, 0.3*height))
    faces = face_cascade.detectMultiScale(gray, 
                                          scaleFactor=1.3, 
                                          minNeighbors=3, 
                                          minSize=(min_size, min_size))
    if len(faces) == 1:
        return True
    else:
        return False

face_cascade = cv2.CascadeClassifier('haarcascade_frontalface_default.xml')
x,y,w,h = cv2.boundingRect(cnt) # bounding rectangle of contour found in code above
if is_headshot(img[y:y+h, x:x+w]):
    cv2.imwrite('headshot.jpg', img[y:y+h, x:x+w])

---

3. Python code for portrait validation
The grid structure can be found using code I posted in this stackoverflow question. Loop through the results of the completed grid. Each grid element is defined by (x,y,w,h) where w and h can be the average width and height of portraits found above. Use the function box1.intersection(box2) from shapely.geometry to determine if there are missing or missized portraits. If the intersection area is small or zero, there may be a missing portrait that should then be checked with face detection. I'm open to providing more details if there is any interest.

it is not a complete answer but maybe useful.

i get the image below with the following code.

to understand the code you can refer to my old answer at http://answers.opencv.org/question/85884

if it seems promising we will try to improve it together.

#include <opencv2/highgui.hpp>
#include <opencv2/imgproc.hpp>

using namespace cv;

int main(int argc, char** argv)
{
    Mat img = imread("e:/test/twHVm.jpg");
    if (img.empty())
        return -1;

    Mat resized, gray, reduced_h, reduced_w;
    resize(img, resized, Size(), 1, 1);

    cvtColor(resized, gray, CV_BGR2GRAY);

    reduce(gray, reduced_h, 0, REDUCE_AVG);
    reduce(gray, reduced_w, 1, REDUCE_AVG);


    for (int i = 0; i < img.cols; i++)
    {
        if (reduced_h.at<uchar>(0, i) > 200) // this is experimental value
        line(resized, Point(i, 0), Point(i, img.rows), Scalar(0, 255, 0), 1);
    }

    for (int i = 0; i < img.rows; i++)
    {
        if (reduced_w.at<uchar>(i, 0) > 225) // this is experimental value
        line(resized, Point(0, i), Point(img.cols, i), Scalar(0, 255, 0), 1);
    }

    imshow("result", resized);
    waitKey(0);
    return 0;
}