How to debug Kubernetes nginx Ingress redirection from HTTP to HTTPS

In the setup you describe there are likley to be at least three processes involved in the handling of incoming http requests

internet -> AWS ELB -> Nginx Ingress Controller -> Tomcat

Any of the processes may be able to issue a 301 redirect to instruct http traffic to retry with https. A redirect in the tomcat process could either be specified by tomcat configuration or the app tomcat hosts.

I would attempt to isolate the process that is performing the redirect by experimenting to bypass the higher processes:

  1. Bypass the loadbalancer. SSH onto one of the K8s nodes and curl the ingress services' nodeport in the URL (check out Internal Endpoints on the nginx-controller service description for the right port). If you get a redirect, you know the problem is not the the loadbalancer.
  2. Bypass the ingress controllers by 'docker exec' into one of the app containers, then curl localhost:8080. If you get a redirect, you know the problem is not the the ingress controller, and must be in the tomcat config or application.

If none of this is possible due to ssh restrictions, curl availability etc... A different approach would be to turn on request logging for each of the processes involved - though this will not be appropriate for a production workload as it will lead to confidential information being output to inappropriate contexts, such as a web tier log file or cloudwatch logs.