Back button animation with large title jumps

@dave's answer worked for me! Thanks! Here's the code that I used in its entirety:

FirstViewController:

override func viewWillAppear(_ animated: Bool) {
    super.viewWillAppear(animated)
    if #available(iOS 11.0, *) {
        navigationItem.largeTitleDisplayMode = .always
            navigationController?.navigationBar.prefersLargeTitles = true
        }
    }
}

SecondViewController:

override func viewWillAppear(_ animated: Bool) {
        super.viewWillAppear(animated)
        if #available(iOS 11.0, *) {
            navigationItem.largeTitleDisplayMode = .never
        }
    }
}

One should make force layout of navigation bar right after switching off large title

override func viewWillDisappear(_ animated: Bool) {
    super.viewWillDisappear(animated)
    
    navigationController?.navigationBar.prefersLargeTitles = false
    navigationController?.navigationBar.layoutIfNeeded()
}

This cancels out large navigation title immediately.

on the second view controller set the largeTitleDisplayMode to .never you won't need to set the prefersLargeTitles to false.

To clarify things here, you've to set the largeTitleDisplayMode directly for the navigationItem of the view controller, not the navigation controller!

self.navigationItem.largeTitleDisplayMode = .never // This fixes the issue
self.navigationController?.navigationItem.largeTitleDisplayMode = .never // This doesn't work / Title will stay large