Styling specific layer by using polygon mask in QGIS?

Not a perfect solution but you could make use of the Geometry Generator which adds a visualised line to represent the intersection. You could then set this to overlap the original line feature.

Add a new symbol layer by clicking the plus sign and select the Geometry generator as symbol layer type. Set the geoemtry type to LineString / MultiLineString and use the following expression:

intersection($geometry, geometry(get_feature( 'polygonLayer','fieldName','value'))) 

You would need to add details about your specific polygon where:

• polygonLayer is the name of your polygon layer
• fieldName is the name of the field
• value is the feature value of your specific polygon

Note that to colour the visual line, you may need to do it from the Draw effects property:

This was the result (note that the visual line did not overlap the original line completely so I modified the offset slightly):

And without the polygon:

---

---

Edit:

If you want this to be applied for each line feature intersecting a polygon feature, go to the Function Editor and use the following function (change the name of polygon example_2 to match the name of your polygon layer):

from qgis.core import *
from qgis.gui import *

@qgsfunction(args='auto', group='Custom')
def func(feature, parent):
    polygon_layer = QgsMapLayerRegistry.instance().mapLayersByName( "polygon example_2" )[0]
    feat_list = []
    geoms = QgsGeometry.fromWkt('GEOMETRYCOLLECTION()')
    for polygon_feat in polygon_layer.getFeatures():
        if feature.geometry().intersects(polygon_feat.geometry()):
            intersection = feature.geometry().intersection(polygon_feat.geometry())
            feat_list.append(intersection)
    for x in feat_list:
        geoms = geoms.combine(x)
    return geoms

Click Load then go to the Expression tab and type in func(). Hopefully the result should look like the following (using the same style properties mentioned above):

Extending on Joseph's answer I came up with this function. It accounts for different coordinate systems and I needed to lookup into two masking layers, thus it handles that too. Further more I wanted to be able to either mask the lines inside the polygons or the lines outside the polygons.

from qgis.core import *
from qgis.gui import *
from qgis.utils import iface

@qgsfunction(args='auto', group='Custom')
def mask_line_with_polygon(mask_type, line_layer_name, polygon_layer_name_1, polygon_layer_name_2, feature, parent):
    line_layer = QgsMapLayerRegistry.instance().mapLayersByName( line_layer_name )[0]

    # This is the geometry outside the polygon mask.
    outside = QgsGeometry(feature.geometry())

    polygon_layer_names = [polygon_layer_name_1, polygon_layer_name_2]
    line_feature_extent = outside.boundingBox()

    geoms = QgsGeometry.fromWkt('MultiLineString()')

    for polygon_layer_name in polygon_layer_names:
        if polygon_layer_name is None or len(polygon_layer_name) == 0:
            continue

        # If the line and the polygon layers have different projections, handle them here.
        polygon_layer = QgsMapLayerRegistry.instance().mapLayersByName(polygon_layer_name)[0]
        trs = QgsCoordinateTransform(line_layer.crs(), polygon_layer.crs())
        polygon_extent = trs.transform(line_feature_extent)
        trs = QgsCoordinateTransform(polygon_layer.crs(), line_layer.crs())

        # Go through the features in the polygon layer, but only those within the line feature bounding box.
        for feature in polygon_layer.getFeatures(QgsFeatureRequest().setFilterRect(polygon_extent)):
            polygon_geometry = QgsGeometry(feature.geometry())

            # Transform the polygon to line space.
            polygon_geometry.transform(trs)

            if outside.intersects(polygon_geometry):
                if mask_type.lower() == 'outside':
                    inside = outside.intersection(polygon_geometry)

                    if inside.isMultipart():
                        for x in inside.asMultiPolyline():
                            geoms.addPart(x)
                    else:
                        geoms.addPart(inside.asPolyline())

                outside = outside.difference(polygon_geometry)

    if mask_type.lower() == 'inside':
        if outside.isMultipart():
            for x in outside.asMultiPolyline():
                geoms.addPart(x)
        else:
            geoms.addPart(outside.asPolyline())

    return geoms

This exercise has shown me that QGIS isn't too fond of working with large data sets and this algorithm with QGIS crashing on my way too often. I suspect that the QGIS renderer doesn't like to render Geometry Generators which are time consuming.