How to configure SFTP so it behaves like ftp chrooting user to his home directory?

vsftpd is a FTP server implementing the FTP protocol. Some extensions for encryption are available for FTP, but they are completely different from SFTP which is a subsystem of SSH.

If you want to use sftp, you need to configure a ssh server and enable the sftp subsystem (see the sshd_config man page for details). It's also possible to configure sftp with chrooted user areas.

In addition to Stephane's answer I'd like to point out that there is FTPS, too. FTPS is the classic FTP protocol over an SSL-secured connection. If you meant this you'd have to adjust your question of course, but it would be a completely different question then.

There are two variations of FTPS, one were the control channel is secured (credentials etc) and another where also the data channel is secured. However, as Stephane already pointed out, the protocols are different, including capabilities and commands.

---

Concerning your comment. You can configure in /etc/ssh/sshd_config to allow based on certain criteria only a certain directory structure. Here's an example that will confine all members of the group sftponly to the /home folder. Adjust to your needs:

Match group sftponly
    ChrootDirectory /home
    X11Forwarding no
    AllowTcpForwarding no
    ForceCommand internal-sftp
    PasswordAuthentication no

As you can see it also sets other options. Strictly speaking for the functionality you ask you'd only need this:

Match group sftponly
    ChrootDirectory /home
    ForceCommand internal-sftp

But of course these options prevent users from (ab)using other SSH facilities.

Consult man sshd_config for more details in particular on the Match directive. You can also match per-user, per-host and per (remote) address.