Lags when RecyclerView scrolling

You are passing a context to adapter. First of all this could lead to memory leaks and also could be affecting your performance. Instead of passing the context into adapter, just simply get it from ViewHolder. You can always get a context reference inside RecyclerView.Adapter without a need to pass it around.

To dump the RAM after the scroll is, it shows that there are 71 memory instances originally ViewHolder.

Judging from the dump this is most probably the case.

@Override
public ViewHolder onCreateViewHolder(ViewGroup parent, int viewType) {
    Context context = parent.getContext();
    ...
}

@Override
public void onBindViewHolder(final ViewHolder holder, int position) {
    Context context = holder.itemView.getContext();
    ...
}