Improve Tree Aspect (forest)

Here's a solution which I think meets all the stated desiderata. I assumed, from the target image, that you did not want the arc to fall short of the topmost point of the node P_2. If that is not required, you need not bother with some of the calculations involved in my code.

\documentclass[tikz, border=10pt, multi]{standalone}

We load through and calc TikZ libraries to get the arc drawn correctly.

\usetikzlibrary{through,calc}
\usepackage{forest}

Now we set up some styles. Some of these simplify your existing code. If you prefer not to use them, you can omit them.

\forestset{%

auto edge label automates the code for formatting the edge label. It creates the node, puts the contents in maths mode and decides whether to put the label to the left or right of the edge. This means that

edge label=x^2, auto edge label

will do the right thing.

  auto edge label/.style={%
    before typesetting nodes={%

Don't do anything if it's the root node.

      if level=0{}{

If the node is in the second half of it's parent's children, or if it is the middle child ...

        if={n()>(n_children("!u")/2)}{

If the node is the middle child ...

          if={n()==((n_children("!u")+1)/2)}{
            edge label/.wrap value={
              node[midway, right] {$##1$}
            },
          }{

If the node is in the second half of it's parent's children ...

            edge label/.wrap value={
              node[midway, outer sep=1.5mm, right] {$##1$}
            },
          },
        }{

If the node is in the first half of it's parent's children ...

          edge label/.wrap value={
            node[midway, outer sep=1.5mm, left] {$##1$}
          },
        }
      },
    },
  },

This is the style for nicer empty nodes. It is from page 65 of the current manual. It is part of the linguistics library. Hence, if you use that library, you can omit this definition and just apply the style out-of-the-box.

Note that I think the explanation in this part of the manual mistakenly refers to non-existent options, but I'm not certain about this.

  nice empty nodes/.style={% page 65 of the manual - this is from the linguistics library
    for tree={
      calign=fixed edge angles
    },
    delay={
      where content={}{
        shape=coordinate,
        for parent={
          for children={anchor=north}
        }
      }{}
    }
  },

Here's the style for drawing the arcs. You pass this as an option to the parent of the nodes through which the arc should be drawn.

No edges are drawn as part of the tree itself except to the second child. If there might be more than 3 children, this would need to get a bit more sophisticated. This code assumes 3 children.

Instead, the arc is drawn afterwards by calculating the appropriate points based on the position of the middle node and the default angle to the nodes when using calign=fixed edge angles. The edges to the first and third child are drawn at this time.

[This should really be a bit more sophisticated in terms of checking for various possibilities, but this should work in cases relevantly like the MWE.]

  arc below/.style={
    tikz+={%
      \clip (.center) coordinate (o) -- (!1.north) coordinate (a) |- (!2.north) coordinate (b) -| (!3.north) coordinate (c) -- cycle;
      \node [draw, circle through={(b)}] at (o) {};
      \draw [\forestoption{edge}] () -- ($(o)!1!-35:(b)$) ($(o)!1!35:(b)$) -- ();
    },
    for children={
      if n=2{}{no edge},
    }
  }
}

Simple TikZ style for convenience.

\tikzset{%
  my circle/.style={draw, circle}
}

Then we can just apply all this stuff to the tree as follows.

\begin{document}
\begin{forest}
  for tree={

From the MWE.

    l sep=4em,
    s sep=8em,

Apply two of the new styles to the whole tree.

    auto edge label,
    nice empty nodes,

Put all nodes into maths mode to save dollar signs.

    math content,
  }

Specify the arc below style for the root.

  [P_1, my circle, arc below
    [, edge label={x=0}]
    [P_2, my circle, edge label=x
      [{c-x,x}, edge label=Y]
      [{0,0}, edge label=N]
    ]
    [, edge label={x=c}]
  ]
\end{forest}
\end{document}

tree with styles

Complete code:

\documentclass[tikz, border=10pt, multi]{standalone}
\usetikzlibrary{through,calc}
\usepackage{forest}
\forestset{%
  auto edge label/.style={%
    before typesetting nodes={%
      if level=0{}{
        if={n()>(n_children("!u")/2)}{
          if={n()==((n_children("!u")+1)/2)}{
            edge label/.wrap value={
              node[midway, right] {$##1$}
            },
          }{
            edge label/.wrap value={
              node[midway, outer sep=1.5mm, right] {$##1$}
            },
          },
        }{
          edge label/.wrap value={
            node[midway, outer sep=1.5mm, left] {$##1$}
          },
        }
      },
    },
  },
  nice empty nodes/.style={% page 65 of the manual - this is from the linguistics library
    for tree={
      calign=fixed edge angles
    },
    delay={
      where content={}{
        shape=coordinate,
        for parent={
          for children={anchor=north}
        }
      }{}
    }
  },
  arc below/.style={
    tikz+={%
      \clip (.center) coordinate (o) -- (!1.north) coordinate (a) |- (!2.north) coordinate (b) -| (!3.north) coordinate (c) -- cycle;
      \node [draw, circle through={(b)}] at (o) {};
      \draw [\forestoption{edge}] () -- ($(o)!1!-35:(b)$) ($(o)!1!35:(b)$) -- ();
    },
    for children={
      if n=2{}{no edge},
    }
  }
}
\tikzset{%
  my circle/.style={draw, circle}
}
\begin{document}
\begin{forest}
  for tree={
    l sep=4em,
    s sep=8em,
    auto edge label,
    nice empty nodes,
    math content,
  }
  [P_1, my circle, arc below
    [, edge label={x=0}]
    [P_2, my circle, edge label=x
      [{c-x,x}, edge label=Y]
      [{0,0}, edge label=N]
    ]
    [, edge label={x=c}]
  ]
\end{forest}
\end{document}

EDIT

If you want to increase the separation between the first two levels of the tree further, the easiest way is to simply increase the value of l sep for the root node. Here's a deliberately exaggerated example:

\begin{forest}
  for tree={
    l sep=4em,
    s sep=8em,
    auto edge label,
    nice empty nodes,
    math content,
  }
  [P_1, my circle, arc below, l sep*=6
    [, edge label={x=0}]
    [P_2, my circle, edge label=x
      [{c-x,x}, edge label=Y]
      [{0,0}, edge label=N]
    ]
    [, edge label={x=c}]
  ]
\end{forest}

increased level separation

Here the minimum distance between the root and its children is set to be six times the usual minimum distance between levels with l sep*=6. If you prefer to add an absolute amount, you could say l sep+=<dimension>. Or if you just want to override the default l sep=<dimension> specifies the minimum distance precisely.

It is important that what l sep ensures is the minimum distance. So, if l sep is set very small for one level and a bit bigger for another, you may end up with the same distance in each case because other factors will mean that forest needs the nodes to be spaced at greater distances than either of the specified minimums.

I note that your actual target tree is not, in fact, like the one you showed in the question. In fact, the trickiest part of my code above is not required for your final tree at all.

Here's an automated version of that tree for reference. This version dispenses with the TikZ libraries since they are only needed for arc below. arc through is a new style which connects to the west and east anchors rather than passing through the north anchor. my arc determines the arc's style. This can be set within the tree using my arcs={<key list>} to determine the style. By default, it is empty and the arc is draw with the style of the current edge option. Specifying the style with my arcs allows to supplement or override the edge style. For example, the arc may be densely dashed even if the edges are solidly drawn.

\forestset{%
  arc through/.style={
    tikz+={%
      \path [\forestoption{edge}, my arc] (!1) [out=-35, in=180] to (!2.west) (!2.east) [out=0, in=-145] to (!3);
    }
  },
  my arcs/.code={%
    \tikzset{%
      my arc/.style={#1},
    }
  },
}
\tikzset{%
  my arc/.style={},
}

dashed arc through

\documentclass[tikz, border=10pt, multi]{standalone}
\usepackage{forest}
\forestset{%
  auto edge label/.style={%
    before typesetting nodes={%
      if level=0{}{
        if={n()>(n_children("!u")/2)}{
          if={n()==((n_children("!u")+1)/2)}{
            edge label/.wrap value={
              node[midway, right] {$##1$}
            },
          }{
            edge label/.wrap value={
              node[midway, outer sep=1.5mm, right] {$##1$}
            },
          },
        }{
          edge label/.wrap value={
            node[midway, outer sep=1.5mm, left] {$##1$}
          },
        }
      },
    },
  },
  nice empty nodes/.style={% page 65 of the manual - this is from the linguistics library
    for tree={
      calign=fixed edge angles
    },
    delay={
      where content={}{
        shape=coordinate,
        for parent={
          for children={anchor=north}
        }
      }{}
    }
  },
  arc through/.style={
    tikz+={%
      \path [\forestoption{edge}, my arc] (!1) [out=-35, in=180] to (!2.west) (!2.east) [out=0, in=-145] to (!3);
    }
  },
  my arcs/.code={%
    \tikzset{%
      my arc/.style={#1},
    }
  },
}
\tikzset{%
  my circle/.style={draw, circle},
  my arc/.style={},
}
\begin{document}
\begin{forest}
  for tree={
    l sep=4em,
    s sep=8em,
    auto edge label,
    nice empty nodes,
    math content,
    my arcs={densely dashed},
  }
  [P_1, my circle, arc through
    [, edge label={x=0}]
    [P_2, my circle, edge label=x
      [{c-x,x}, edge label=Y]
      [{0,0}, edge label=N]
    ]
    [, edge label={x=c}]
  ]
\end{forest}
\end{document}

EDIT ASIDE

This version is just for Clément. Although it is more than 546 characters, it is still only 644. And Kile makes the TikZ-only code 563 characters, so probably my statistics is counting characters differently.

Personally, I don't think this an advantage, but there you are.

It is not very transparent so I don't actually recommend using this.

However, the way the arc is drawn is much neater than my earlier code. What I'd probably do is base arc below on this method rather than using the through library.

The main saving in characters is achieved by eliminating automation. The edge labels are no longer placed automatically according to the position of the child relative to its siblings. So, if you add a node, you have to check if any lefts should become rights or vice-versa. Moreover, no style is used for the circles, reducing flexibility and maintainability of the code. Finally, dollar signs are used for the content of nodes rather than math content because math content, contains more characters than the number required to assign a pair of dollar signs to each node which needs them.

Ironically, the drawing of the arc actually now uses forest features more heavily and TikZ features less heavily. (y() is used with a pgfmath wrapper to get the information needed for the arc, rather than relying on the through library.)

\documentclass{standalone}
\usepackage{forest}
\usetikzlibrary{calc}
\begin{document}
\begin{forest}
  ey/.style={shape=coordinate,no edge},
  elr/.style 2 args={edge label={node[midway,outer sep=1.5mm,#1]{$#2$}}},
  el/.style={elr={left}{#1}},
  er/.style={elr={right}{#1}},
  for tree={l sep=4em,s sep=8em,calign=fixed edge angles}
  [$P_1$,draw,circle
    [,el={x=0},ey]
    [$P_2$,draw,circle,er=x,anchor=north,before drawing tree={TeX/.wrap pgfmath arg={\gdef\rs{#1}}{y("!u")-y()}},tikz={\draw(!u)--($(!u)!1!-35:(.north)$)arc(235:305:\rs pt)--(!u);}
      [${c-x,x}$,el=Y]
      [${0,0}$,er=N]]
    [,er={x=c},ey]]
\end{forest}
\end{document}

short and sour version


I know you want to stick to forest but while you wait for an answer with it, you can try to understand a poor man way of drawing your scheme. It's not so difficult:

Let's draw P1 somewhere

\node[circle,draw] (P1) {$P_1$};

Let's draw P2 at a certain known distance below P1

\node[circle, draw, on grid, below = 2cm of P1, anchor=north] (P2) {$P_2$};

with on grid and anchor=north we force a 2cm distance between P1's center and P2's north. We will need it for a perfect arc.

Now we can draw the line between P1 and P2

\draw (P1)-- node[right]{$x$} (P2);

Next, decide which angles you want for left and right children. The arm length will be 2cm starting from P1 center. At the end of right child, we draw the arc because we know initial angle, final angle and radius.

\draw (P1) -- node[right] {$x=c$} ++ (-60:2cm) arc (-60:-120:2cm);
\draw (P1) -- node[left] {$x=0$} ++ (240:2cm);

And we finish with P2 children using similar commands:

\draw (P2) -- node[right] {$N$} ++ (-60:2cm) node[below] {$0,0$};
\draw (P2) -- node[left] {$Y$} ++ (240:2cm) node[below] {$c-x,x$};

That's all. The result:

enter image description here

and complete code:

\documentclass[tikz,border=2mm]{standalone}
\usetikzlibrary{positioning}

\begin{document}
\begin{tikzpicture}
\draw (-2,-2) grid (2,2);
\node[circle,draw] (P1) {$P_1$};
\node[circle, draw, on grid, below = 2cm of P1, anchor=north] (P2) {$P_2$};
\draw (P1)-- node[right]{$x$} (P2);

\draw (P1) -- node[right] {$x=c$} ++ (-60:2cm) arc (-60:-120:2cm);
\draw (P1) -- node[left] {$x=0$} ++ (240:2cm);

\draw (P2) -- node[right] {$N$} ++ (-60:2cm) node[below] {$0,0$};
\draw (P2) -- node[left] {$Y$} ++ (240:2cm) node[below] {$c-x,x$};
\end{tikzpicture}
\end{document}

  1. The "white space" is nothing more than empty nodes. Even though you didn't add text, the nodes are created therefore in turn creating the white space. To solve this, simply add coordinate to the options for those nodes, like name=1, coordinate,...
  2. Solved by the next point.
  3. For the angles, add calign=fixed edge angles to your for tree={} options.
  4. I can't find a way to do this, not even in the manual. I think I'm missing something obvious.

In any case, here's the current result:

enter image description here