How not to switch past the edge of panes in tmux

This is a bit of a hack but might be good enough for you. From version 2.3 you can find the x and y co-ordinate of each pane's borders. For example, display -p #{pane_right} for a pane at the right-hand edge of an 80 column terminal would be 79. If you give the command to move right to the next pane, and the new pane's pane_right is, for example, 39, then you have moved left, so you will want to move back to the previous pane with select-pane -l.

You can run most tmux commands from a shell script, so create the following file mytmux in your PATH and make it executable (chmod +x mytmux):

#!/bin/bash
# https://unix.stackexchange.com/a/451473/119298
restrict(){
        case $1 in
        U)  d=-U p=pane_top    cmp=-gt ;;
        D)  d=-D p=pane_bottom cmp=-lt ;;
        L)  d=-L p=pane_left   cmp=-gt ;;
        R)  d=-R p=pane_right  cmp=-lt ;;
        *)  exit 1 ;;
        esac
        old=$(tmux display -p "#{$p}")
        tmux select-pane "$d"
        new=$(tmux display -p "#{$p}")
        [ "$new" "$cmp" "$old" ] && tmux select-pane -l
        exit 0
}
case $1 in
-restrict)shift
        restrict "${1?direction}" ;;
esac

then setup the following bindings in your ~/.tmux.conf:

bind-key -r -T prefix Up    run-shell 'mytmux -restrict U'
bind-key -r -T prefix Down  run-shell 'mytmux -restrict D'
bind-key -r -T prefix Left  run-shell 'mytmux -restrict L'
bind-key -r -T prefix Right run-shell 'mytmux -restrict R'

You will need to extend this if you want to handle multiple sessions, for example.

The PrefixArrow keys are by default bound like

bind-key -r -T prefix       Up                select-pane -U
bind-key -r -T prefix       Down              select-pane -D
bind-key -r -T prefix       Left              select-pane -L
bind-key -r -T prefix       Right             select-pane -R

The select-pane function does not have an option that tells it "don't cycle when you've reached the last pane in that direction".

Looking at the source code that is executed for select-pane, it looks like the cycling left/right and top/bottom is hard-coded in, which means that it's unlikely to be easy to stop it from behaving in this way.