How to use Paramiko getfo to download file from SFTP server to memory to process it

A simple solution that still allows you to use progress callback is:

• Use BytesIO file-like object to store a downloaded file to memory;

• You have to seek file pointer back to file start after downloading it, before you start reading it.

  with io.BytesIO() as fl:
      sftp.getfo(file_name, fl, callback=printTotals)
      fl.seek(0)
      df = pd.read_csv(fl, sep=' ')
  

Though with this solution, you will end up having the file loaded to memory twice.

---

Better solution is to implement a custom file-like object. It will even allow you to download and parse the file at the same time.

class FileWithProgress:

    def __init__(self, fl):
        self.fl = fl
        self.size = fl.stat().st_size
        self.p = 0

    def read(self, blocksize):
        r = self.fl.read(blocksize)
        self.p += len(r)
        print(str(self.p) + " of " + str(self.size)) 
        return r

And use it like:

with sftp.open(file_name, "rb") as fl:
    fl.prefetch()
    df = pd.read_csv(FileWithProgress(fl), sep=' ') 

For the SFTPFile.prefetch call, refer to:
Reading file opened with Python Paramiko SFTPClient.open method is slow.

What I ended up doing was a simple version of that, unfortunately without a callback for the progress, I also needed rb for reading:

with sftp.open(file_name, 'rb') as fl:
        df = pd.read_csv(fl, sep=' ')

Anyway, Martin's answer is what I was looking for!