Extraction of sunken portions from image in hieroglyphics

You can up-sample the input image, apply some smoothing, and find the Otsu threshold, then use this threshold to find Canny edges with different window sizes.

For the larger window (5 x 5), you get a noisy image that contains almost all the edges you need, plus noise.

For the smaller window (3 x 3), you get a less noisy image, but some of the edges are missing.

If this less noisy image is not good enough, you can try morphologically reconstructing it using the noisy image as the mask. Here, I've linked some diagonal edge segments in the noisy image using a morphological hit-miss transform and then applied the reconstruction.

Using a

Mat k = (Mat_<int>(3, 3) <<
        0, 0, 1,
        0, -1, 0,
        1, 0, 0);

kernel for linking broken edges, you get a thinner outline.

Please note that in the c++ code below, I've used a naive reconstruction.

Mat im = imread("rsSUY.png", 0);
/* up sample and smooth */
pyrUp(im, im);
GaussianBlur(im, im, Size(5, 5), 5);
/* find the Otsu threshold */
Mat bw1, bw2;
double th = threshold(im, bw1, 0, 255, THRESH_BINARY | THRESH_OTSU);
/* use the found Otsu threshold for Canny */
Canny(im, bw1, th, th/2, 5, true);  /* this result would be noisy */
Canny(im, bw2, th, th/2, 3, true);  /* this result would be less noisy */

/* link broken edges in more noisy image using hit-miss transform */
Mat k = (Mat_<int>(3, 3) <<
    0, 0, 1,
    0, -1, 0,
    0, 0, 0);
Mat hitmiss;
morphologyEx(bw1, hitmiss, MORPH_HITMISS, k);
bw1 |= hitmiss;

/* apply morphological reconstruction to less noisy image using the modified noisy image */
Mat kernel = getStructuringElement(MORPH_ELLIPSE, Size(3, 3));
double prevMu = 0;
Mat recons = bw2.clone();
for (int i = 0; i < 200; i++)
{
    dilate(recons, recons, kernel);
    recons &= bw1;
    Scalar mu = mean(recons);
    if (abs(mu.val[0] - prevMu) < 0.001)
    {
        break;
    }
    prevMu = mu.val[0];
}

imshow("less noisy", bw2);
imshow("reconstructed", recons);

waitKey();