How can I avoid multiple nested subscriptions using RxJS operators?

You want to use pipe before subscribing. Pipe allows you to make changes to values coming down the stream before the stream emits them. Also, use mergeMap to flatten the subscribe chain. Here's an overview. This doesn't provide a full solution - not paying me enough ;) - but is sufficient to point you in the right direction:

this.prepareUpload(file).pipe(
  tap(values => console.log('hello1', values)),
  map(values => [response, filekey, data]),
  tap(values => console.log('hello2', values)),
  mergeMap(() =>
      // essential to catchError else an HTTP error response will disable this effect - if it uses HTTP - catchError essentially prevents the stream from erroring in which case it will never emit again
      this.encryptData(data, filekey).pipe(
        map(res => res), // do something with res here if you want
        catchError(() => {
          return of(null)
        })
      )
    ),
    filter(res => !!res)
    // more mergemap stuff here
  ).subscribe(values => {
    console.log(values)
  })

Hint: use the tap operator to console.log values as they are passing down the stream

PS: Syntax not checked, may be missing a comma or bracket or 2

PPS: the functions in the pipe are all RxJS operators

You can use mergeMap and filter operators from RxJs and chain your calls. You will need to create some function level variables to use during the chaining.

import { mergeMap, filter, catchError } from 'rxjs/operators`

public upload(file) {
    const gen = this.indexGenerator(); // generator function
    let range, token;
    this.prepareUpload(file)
      .pipe(
        mergeMap((values) => {
          const [response, filekey, data] = values;
          token = response.token;
          return this.encryptData(data, filekey);
        }),
        mergeMap(encryptedDataContainer => {
          const formData = this.prepareEncDataUpload(encryptedDataContainer.data, file.name)
          range = this.getRange(file.size, gen.next().value);

          return this.uploadEncryptedData(formData, token, range);
        }),
        filter(() => !!range.isFinalPart),
        mergeMap(() => {
          return this.completeUpload(encryptedDataContainer.updatedFilekey, token);
        })
        catchError((error) => {
          console.log(error);
          // handle the error accordingly.
        })
      )
      .subscribe(() => {
        console.log('success');
      });

  }