Labeling lines that are not within polygons using field calculator

You trying to call within on a string representing a layer, not its geometry. But it won't work that way anyway. This way it would label/not label whole feature, not segments. With within it would label feature with " " in case that whole line is in polygon. In case of intersects it will label feature that somehow interact with any polygon.

Maybe there is option how to solve this with expression of Label with, but I thing this is primary for formatting label text for feature, not for label placement. I didn't figure out how to split the line geometry in this expression for the desired result.

In QGIS 3.8, there is Geometry generator in Placement tab in labels properties. With this you can generate new source geometry only for labels, in your case lines that are not intersecting polygons. Labels are than rendered only over this geometry (i.e. rendering labels over polygon i suppressed):

if(
    intersects(
        $geometry,
        aggregate('poly','collect',$geometry,intersects($geometry, geometry(@parent)))
    ),
    difference(
        $geometry,
        aggregate('poly','collect',buffer($geometry,0.1),intersects($geometry, geometry(@parent)))
    ),
    $geometry
)

Notice the buffer function, this gives space around polygon to suppress label rendering. This avoids long labels overlap polygons.

Than in Rendering tab check the Label every part of multi-part features. This keeps rendering labels on all part of a line (lines that overlap the polygon and continues).

Or in a very simple way, you can use the polygon to block the labels of the polyline.

Label polyline normally:

After using the polygon as a blocking feature, the label will be shifted automatically outside the polygon:

Your approach does not work because

• spatial joins over layers (not tests for the relations of single geometries) cannot be performed in expression dialog
• within is only true when a whole geometry falls entirely into another geometry, wich is not the fact in your case
• double quotes are for attribute names, for strings use single quotes

All in all, I doubt that this can be achieved using expressions in symbology or labelling, I recommend using the Difference tool to compute the sections of your linestring layer that lie outside the polygon (Processing Toolbox > Vector Overlay > Difference):

Labelling for the original linestring layer:

Labelling for the Difference Layer:

If you want to display your original linestring layer (without labels) and use the labels from the Difference layer I suggest setting up a "No Symbol" style for the Difference layer, label it and display both of them.