What is the name of this kind of equivalence between two topological spaces?
As far as I can see there are two possibilities where we can say more:
The bijection is order preserving. As Qiaochu Yuan said in a comment, this is the same as asking that their associated locales are isomorphic. Thanks to the equivalence of sober spaces with spatial locales, this is the same as asking that their soberifications are homeomorphic. In particular, if $X$ and $Y$ are already sober (a very common condition, sober is strictly weaker than Hausdorff), this is the same as asking that $X$ and $Y$ are homeomorphic.
The bijection is order reversing. Then the lattice of opens of both spaces need to have arbitrary meets distribute over finite joins (i.e. each frame of opens must have the property that its opposite lattice is a frame too). This is a very rare situation, that happens mainly for Alexandrov spaces,although as მამუკაჯიბლაძე notes in the comments there might be other examples.
Edited after realizing that my preceding answer was false.
By elementary tools (i.e., without soberifications) it can be shown that for $T_1$-spaces this equivalence coincides with the homeomorphness.
Theorem 1. For $T_1$-spaces $X,Y$ the following conditions are equivalent:
$X$ and $Y$ are homeomorphic;
The topologies of the spaces $X,Y$ are order isomophic.
Proof. The implication $(1)\Rightarrow(2)$ is trivial.
To prove that $(2)\Rightarrow(1)$, assume that there exists an order isomorphism $i:\tau_X\to \tau_Y$ between the topologies of the spaces $X$ and $Y$.
Since $X$ is a $T_1$-space, for every $x\in X$ the open set $X\setminus\{x\}$ is a maximal element of the poset $\tau_X\setminus\{X\}$. Since $i$ is an isomorphism of the posets $\tau_X$ and $\tau_Y$, the set $i(X\setminus\{x\})$ is a maximal element of the poset $\tau_Y\setminus\{Y\}$ and hence $i(U_x)=Y\setminus \{f(x)\}$ for some point $f(x)\in Y$. The injectivity of $i$ implies that the map $f:X\to Y$ is injective. To see that $f$ is surjective, observe that for every $y\in Y$ the set $i^{-1}(Y\setminus \{y\})$ is a maximal element of the poset $\tau_X\setminus\{X\}$ and hence it coincides with the set $X\setminus\{x\}$ for some $x\in X$. It follows that $Y\setminus\{y\}=i(X\setminus\{x\})=Y\setminus \{f(x)\}$ and hence $y=f(x)$.
Therefore the map $f:X\to Y$ is bijective. To show that $f$ is continuous, it suffices to check that for every open set $U\in\tau_Y$ we get $f^{-1}[U]=i^{-1}(U)$.
To show that $f^{-1}[U]\subset i^{-1}(U)$, observe that for any $x\in f^{-1}[U]$ we get $f(x)\in U$ and hence $(Y\setminus \{f(x)\})\cup U=Y$. Taking into account that order isomorphisms of lattices preserve lattice operations, we conclude that $$(X\setminus\{x\})\cup i^{-1}(U)=i^{-1}(Y\setminus \{f(x)\})\cup i^{-1}(U)=i^{-1}(Y)=X$$ and hence $x\in i^{-1}(U)$. Therefore $f^{-1}[U]\subset i^{-1}(U)$.
To see that $i^{-1}(U)\subset f^{-1}[U]$, observe that for any $x\in i^{-1}(U)$ we get $X=(X\setminus \{x\})\cup i^{-1}(U)$ and hence $(Y\setminus\{f(x)\})\cup U=i(X\setminus\{x\})\cup U=Y$, which implies $f(x)\in U$ and hence $x\in f^{-1}[U]$.
This shows that the preimage $f^{-1}[U]=i^{-1}(U)$ of the open set $U$ is open and $f$ is continuous. By analogy we can check that the map $f^{-1}:Y\to X$ is continuous. So, $f$ is a homeomorphism.
Remark. The theorem does not generalize to $T_0$-spaces as the $T_0$-space $X=[0,1)$ with topology $\tau_X=\{[0,a):0\le a\le 1\}$ and its subspace $Y=(0,1)$ are not homeomorphic but have order isomorphic topologies.
Also there exist a $T_1$-space $X$ and a sober space $Y$ which are not homeomorphic but have order-isomorphic topologies.
Example 1s. Take any infinite set $X$ such that $X\notin X$. On the set $Y= X\cup\{X\}$ consider the topology $\tau_Y=\{\emptyset\}\cup\{Y\setminus F:F\subset X$ is finite$\}$. The space $Y$ is sober but not $T_1$ and its subspace $X$ is $T_1$ but not sober. Nonetheless the spaces $X,Y$ have order-isomorphic topologies.
Added after a comment of მამუკა ჯიბლაძე:
Theorem 1 can be generalized to $T_{Constructible}$-spaces. We recall that a set $A$ in a topological space $X$ is constructible (resp. Borel) if it belongs to the smallest algebra (resp. $\sigma$-algebra) of subsets of $X$, containing all open subsets of $X$.
A topological space $X$ is defined to be $T_{Constructible}$-space (resp. $T_{Borel}$-space) if for each $x\in X$ the singleton $\{x\}$ is a constructible (resp. Borel) subset of $X$.
By Theorem 2.1 of this paper, a topological space $X$ is $T_{Borel}$ (resp. $T_{Constructible}$) if and only if each singleton $\{x\}\subset X$ can be written as the intersection $F\cap U$ of a closed set $F\subset X$ and an (open) $G_\delta$-set $U$.
This characterization implies that the class of $T_{Constructible}$-spaces coincides with the class of $T_D$-spaces (i.e. spaces in which each singletion is an intersection of an open and a closed sets).
So, we have the implications
$$T_1\Rightarrow T_D\Leftrightarrow T_{Constructible}\Rightarrow T_{Borel}\Rightarrow T_0.$$
Observe that if the set $X$ in Example 1s is countable, then the spaces $X,Y$ from this example are $T_{Borel}$. This shows that Theorem 1 does not generalize to $T_{Borel}$-spaces.
On the other hand, it does generalize to $T_{Constructible}$-spaces (and to $T_D$-spaces as was suggested by მამუკა ჯიბლაძე).
Theorem 2. Two $T_{Constructible}$-spaces are homeomorphic if and only if their topologies are order-isomorphic.
Proof. Let $(X,\tau_X)$ be a $T_{Constructible}$-space.
A pair $(V,W)$ of open sets in $X$ is called point-determining if
$V\subset W$ and $V\ne W$;
for any open set $U\in\tau_X$ with $V\subset U\subset W$ we get $U=V$ or $U=W$;
for any open set $U\subset X$ with $V\subset U$ and $W\not\subset U$ we get $V=U$.
Using the fact that each $T_{Constructible}$-space is a $T_D$-space, one can easily check:
Lemma 1. For any point $x\in X$ the pair $(V_x,W_x)$ of the open sets $V_x=X\setminus\overline{\{x\}}$ and $W_x=V_x\cup\{x\}$ is point-determining.
Lemma 2. Each point-determining pair $(V,W)$ is equal to $(V_x,W_x)$ for a unique point $x\in X$.
Lemma 3. An open set $U\in\tau_X$ is a neighborhood of a point $x\in X$ if and only if $W_x\subset V_x\cup U$.
Now theorem 2 easily follows from Lemmas 1-3 and the observation that point-determining pairs of open sets are preserved by order-isomorphisms of topologies.
Added in an edit. The reconstruction theorems for continuous maps and homeomorphisms on sober and $T_D$-spaces can be found in this book, which also contains Example 1s (see Example 3.1 in this book).