Help to reproduce a diagram in TikZ

You were loading but not using the positioning library, that is instead of right of=, say, please use right=of or right=5mm of. And one can kick out several auxiliary nodes/coordinates like (ael) and (aer), they are not at all needed.

\documentclass{standalone}
\usepackage[utf8]{inputenc}
\usepackage{tikz}
\usetikzlibrary{arrows.meta,shadows,positioning}

\tikzset{
  frame/.style={
    rectangle, draw,
    text width=6em, text centered,
    minimum height=4em,drop shadow,fill=white,
    rounded corners,
  },
  line/.style={
    draw, -{Latex},rounded corners=3mm,
  }
}

\begin{document}
\begin{tikzpicture}[font=\small\sffamily\bfseries,very thick,node distance = 4cm]
\node [frame] (agent) {Agent};
\node [frame, below=1.2cm of agent] (environment) {Environment};
\draw[line] (environment) -- ++ (3.5,0) |- (agent) 
node[right,pos=0.25,align=left] {action\\ $A_t$};
\coordinate[left=8mm of environment] (P);
\draw[thin,dashed] (P|-environment.north) -- (P|-environment.south);
\draw[line] (environment.200) -- (P |- environment.200)
node[midway,above]{$S_{i+1}$};
\draw[line,thick] (environment.160) -- (P |- environment.160)
node[midway,above]{$R_{i+1}$};
\draw[line] (P |- environment.200) -- ++ (-1.4,0) |- (agent.160)
node[left, pos=0.25, align=right] {state\\ $s_t$};
\draw[line,thick] (P |- environment.160) -- ++ (-0.8,0) |- (agent.200)
node[right,pos=0.25,align=left] {reward\\ $R_t$};
\end{tikzpicture}
\end{document}

ADDENDUM: If you want to adjust the distances of the two lines, there is no need to introduce new coordinat (However, your way to achieve this with additional coordinates is very elegant, I think. Just in the original question the coordinates (ael) and (aer) were not necessarily a good choice as they interrupted paths. In principle one could avoid introducing P in the following by switching to decorations.markings, but this might be an overkill.)

\documentclass{standalone}
\usepackage[utf8]{inputenc}
\usepackage{tikz}
\usetikzlibrary{arrows.meta,shadows,positioning}

\tikzset{
  frame/.style={
    rectangle, draw,
    text width=6em, text centered,
    minimum height=4em,drop shadow,fill=white,
    rounded corners,
  },
  line/.style={
    draw, -{Latex},rounded corners=3mm,
  }
}

\begin{document}
\begin{tikzpicture}[font=\small\sffamily\bfseries,very thick,node distance = 4cm]
\node [frame] (agent) {Agent};
\node [frame, below=1.2cm of agent] (environment) {Environment};
\draw[line] (environment) -- ++ (3.5,0) |- (agent) 
node[right,pos=0.25,align=left] {action\\ $A_t$};
\coordinate[left=12mm of environment] (P);
\draw[thin,dashed] (P|-environment.north) -- (P|-environment.south);
\pgfmathsetmacro{\Ldist}{4mm}
\draw[line] ([yshift=-\Ldist]environment.west) -- 
([yshift=-\Ldist]environment.west -| P) node[midway,above]{$S_{i+1}$};
\draw[line,thick] ([yshift=\Ldist]environment.west) -- ([yshift=\Ldist]environment.west
-|P) node[midway,above]{$R_{i+1}$};
\draw[line] ([yshift=-\Ldist]environment.west -| P) -- ++ (-12mm-\Ldist,0) |- 
([yshift=\Ldist]agent.west) node[left, pos=0.25, align=right] {state\\ $S_t$};
\draw[line,thick] ([yshift=\Ldist]environment.west -| P) -- ++ (-12mm+\Ldist,0) 
|- ([yshift=-\Ldist]agent.west) node[right,pos=0.25,align=left] {reward\\ $R_t$};
\end{tikzpicture}
\end{document}

You can remove the blank spaces in the middle by declaring (ael) and (aer) as \coordinate.

Currently Reward and State are same node in the question. Shift the State node left to get some separation between them. This can be done with positioning library:

\coordinate [left=1cm of ael] (aell) {}

Note that I have used left=1cm of ael not left of=. As @marmot mentioned, writing left of= is not using the loaded library positioning. This can reduce the number of nodes required also.

\documentclass[border=3mm]{standalone}
\usepackage[utf8]{inputenc}
\usepackage{tikz}
\usetikzlibrary{arrows,shadows,positioning}

\tikzset{
  frame/.style={
    rectangle, draw,
    text width=6em, text centered,
    minimum height=4em,drop shadow,fill=white,
    rounded corners,
  },
  line/.style={
    draw, -latex',rounded corners=3mm,
  }
}

\begin{document}
\begin{tikzpicture}[font=\small\sffamily\bfseries,very thick,node distance = 2cm]
\node [frame] (agent) {Agent};
\coordinate [below right=of agent] (aer) {};
\coordinate [below left=of agent] (ael) {};
\coordinate [left=0.9 cm of ael] (aell) {};
\node [frame, below left=of aer] (environment) {Environment};

\path [line] (agent) -| node[right, pos=1,align=left] {action\\ $A_t$} (aer) |- (environment);
\path [line] (environment.160) -| node[right,pos=1,align=left] {reward\\ $R_t$} (ael) |- (agent.200);
\path [line] (environment.200) -| node[left,pos=1,align=right]  {state\\ $S_t$} (aell) |- (agent.160);

\draw[-latex] (environment.160) -- ++(-1,0) node[above,midway]{$R_{i+1}$} coordinate(c1);
\draw[-latex] (environment.200) -- ++(-1,0) node[above,midway]{$S_{i+1}$} coordinate(c2);
\draw[dashed,shorten >=-3mm,shorten <=-3mm] (c1) -- (c2);
\end{tikzpicture}
\end{document}