What is meant exactly by "renormalization" in condensed matter physics, specifically in density matrix renormalization group (DMRG)?

It is instructive to come back first to NRG (Numerical Renormalization Group) proposed by Ken Wilson (Nobel prize laureate for his work on Renormalization Group in the context of critical phenomena). Consider a translation-invariant 1D quantum Hamiltonian (for example a quantum spin chain). Start with a sufficiently small system of size $\ell$ so that the Hamiltonian can be numerically diagonalized. Truncate the Hilbert space by keeping only the $M$ lowest eigenstates: ${\cal H}\longrightarrow {\cal H}_M$. Now, join two copies of this system by an interaction Hamiltonian $H_{\rm int}$ (for example an exchange coupling between nearest neighbors spins) to get a new system of size $2\ell$. The matrix associated to the full Hamiltonian $H=H_1\otimes\mathbb{I}+\mathbb{I}\otimes H_2+H_{\rm int}$ in the Hilbert space ${\cal H}_M\otimes{\cal H}_M$ can be computed numerically if $M$ is small enough. Because of the truncation, $H$ is not the true Hamiltonian of the system of size $2\ell$ but an effective one. $H$ is called the renormalized Hamiltonian because of the similarity with Wilson renormalization group where a renormalized action is obtained by integrating short-distance (i.e. high frequency) modes. In the NRG algorithm, high-energy states are not integrated out but simply thrown away. The procedure is iterated many times. For more details, I advice you to have a look to the review "The density-matrix renormalization group" (https://arxiv.org/abs/cond-mat/0409292) by U. Schollwöck.

NRG is not efficient for reasons discussed in the above-cited paper. DMRG was introduced by Steve White, former Ph.D. student of Kenneth Wilson, as an improvement of NRG. The truncated Hilbert space ${\cal H}_M$ is not constructed from the $M$ lowest eigenstates of the Hamiltonian but from the $M$ lowest eigenstates of the ground state reduced density matrix $\rho_1={\rm Tr}_2|\psi_0\rangle\langle\psi_0|$ of the full Hamiltonian. Therefore, the density matrix is said to be renormalized. With DMRG, the energy, as well as any local average, of the ground state can be estimated with a very high accuracy. Remember that a quantum phase transition occurs when the gap between the ground state and the first-excited state vanishes leading to a change of symmetry of the ground state.

To summarize, the term "Renormalization Group" is used for NRG and DMRG because an effective Hamiltonian is constructed at each step by throwing away higher-energy states that are assumed to be irrelevant. However, this renormalized Hamiltonian is used only to compute quantum averages. Critical exponents cannot be extracted from the law $J'={\cal R}(J)$ as would be done in Wilson Renormalization Group. With the DMRG algorithm, you have to run several simulations at different control parameters $\delta$ to estimate for example the magnetic critical exponent $\beta$ from the magnetization behavior close to the quantum critical point: $M\sim |\delta-\delta_c|^\beta$. You can also use Finite-Size Scaling $M\sim L^{-\beta/\nu}$ by performing several DMRG calculations for different lattice sizes.

It's basically a misnomer that has nothing to do with the "real" renormalization group. (I've heard this confirmed directly from professional practitioners.) You can come up with some vague similarities if you really stretch for it, but I don't think there's any conceptual insight to be gained from doing so.