masking image with CSS
You only need to consider one image which is the bottom part of your mask then use a simple gradient for the other part. You also don't need mask-composite. Simply adjust the size/position so that both mask don't overlap:
.img-poster {
display: block;
max-width: 100%;
-webkit-mask:
linear-gradient(#fff,#fff) top,
url(https://i.ibb.co/5WvbqgG/zmylJ.png) bottom;
-webkit-mask-size:
100% calc(100% - 30px),
auto 30px;
-webkit-mask-repeat: repeat-x;
mask:
linear-gradient(#fff,#fff) top,
url(https://i.ibb.co/5WvbqgG/zmylJ.png) bottom;
mask-size:
100% calc(100% - 30px),
auto 30px;
mask-repeat: repeat-x;
}<section class="section poster-container">
<img src="https://imagejournal.org/wp-content/uploads/bb-plugin/cache/23466317216_b99485ba14_o-panorama.jpg" alt="" class="img-poster add img-responsive">
</section>SVG + filter solution
The SVG solution for creating a serrated edge in images does not require the use of raster images of patterns with serrated edges.
Therefore, this SVG solution can be applied to any image and makes it adaptive.
Consider creating a jagged edge using the same image 2 times. Filter is applied to the lower image and because of this, it gets a little bigger. The original image is superimposed on top and therefore the teeth on the edges become visible. To leave the teeth only on the lower border, we cut off the unnecessary teeth with a mask on the lateral borders.
.container {
width:90vw;
height:90vh;
}<div class="container">
<svg version="1.1" xmlns="http://www.w3.org/2000/svg"
xmlns:xlink="http://www.w3.org/1999/xlink"
viewBox="0 0 1800 900" >
<defs>
<mask id="msk1">
<rect x="100%" y="100%" fill="white" />
<rect x="1700" y="0" width="50" height="900" fill="white" />
</mask>
<filter id="displacementFilter">
<feTurbulence type="turbulence" baseFrequency="0.01 0.01"
numOctaves="2" result="turbulence" seed="1"/>
<feDisplacementMap in2="turbulence" in="SourceGraphic"
scale="65" xChannelSelector="R" xChannelSelector="G" yChannelSelector="B"/>
</filter>
</defs>
<!-- The image is processed by a filter and therefore overall dimensions have increased due to the height of the teeth -->
<image filter="url(#displacementFilter)" xlink:href="https://i.stack.imgur.com/paWbQ.jpg.jpg" width="1700" height="850" />
<rect mask="url(#msk1)" x="1700" y="0" width="50" height="900" fill="white" />
<!-- The same original image, but without filters -->
<image x="0" xlink:href="https://i.stack.imgur.com/paWbQ.jpg.jpg" width="1700" height="850" />
</svg>
</div> Another image
.container {
width:90vw;
height:90vh;
}<div class="container">
<svg version="1.1" xmlns="http://www.w3.org/2000/svg"
xmlns:xlink="http://www.w3.org/1999/xlink"
viewBox="0 0 1800 900" >
<defs>
<mask id="msk1">
<rect x="100%" y="100%" fill="white" />
<rect x="1700" y="0" width="50" height="900" fill="white" />
</mask>
<filter id="displacementFilter">
<feTurbulence type="turbulence" baseFrequency="0.01 0.01"
numOctaves="2" result="turbulence" seed="1"/>
<feDisplacementMap in2="turbulence" in="SourceGraphic"
scale="75" xChannelSelector="R" xChannelSelector="G" yChannelSelector="B"/>
</filter>
</defs>
<!-- The image is processed by a filter and therefore overall dimensions have increased due to the height of the teeth -->
<image filter="url(#displacementFilter)" xlink:href="https://i.stack.imgur.com/kaYSe.jpg" width="1700" height="850" />
<rect mask="url(#msk1)" x="1700" y="0" width="50" height="900" fill="white" />
<!-- The same original image, but without filters -->
<image x="0" xlink:href="https://i.stack.imgur.com/kaYSe.jpg" width="1700" height="850" />
</svg>
</div>