WSGI : Truncated or oversized response headers received from daemon process

Solution 1:

The cause of the problem was numpy.

Python C extension modules, like numpy, are known to cause timeouts when used under mod_wsgi.

Source : Answer by Sean F on Timeout when reading response headers from daemon process

A similar question which I didn't find on initial search answered and explained by author of mod_wsgi says -

Some third party packages for Python which use C extension modules, and this includes scipy and numpy, will only work in the Python main interpreter and cannot be used in sub interpreters as mod_wsgi by default uses. The result can be thread deadlock, incorrect behaviour or processes crashes.

Source : Answer by Graham Dumpleton on Non-responsive apache + mod_wsgi after installing scipy

Solution

Add the below line to your httpd.conf. In my case the file was /etc/apache2/apache2.conf.

WSGIApplicationGroup %{GLOBAL}

Solution 2:

As others have mentioned, this is caused by the process crashing for a number of potential reasons. One reason being that some Python dependency isn't thread safe.

If that's the problem, one work around is to switch Apache's MPM type. The prefork type doesn't use threads, so if the problem is numpy crashing due to thread misuse, then that should fix it. The worker and event types use less memory, but also use threads, so they can encounter this error.

To determine which type you're currently using, run:

apachectl -V | grep -i mpm

If you see Server MPM: prefork then you're already using prefork, meaning the cause of the error might be something else. If it says "worker" or "event", then you can switch to prefork by running:

sudo a2dismod mpm_event
sudo a2dismod mpm_worker
sudo a2enmod mpm_prefork
sudo service apache2 restart

Note, the main downside of prefork is that, since it's not using threads, it consumes more memory.

Edit: I've since encountered this error due to other causes. Most recently, the problem is caused by a bug in Ubuntu's precompiled mod-wsgi system package as well as a bug in the Python psycopg2 package.

The solution for this was to switch from the system mod-wsgi to the Python package, as well as switch to the psycopg2-binary package:

sudo apt purge libapache2-mod-wsgi*
sudo apt install apache2-dev
pip uninstall psycopg2
pip install mod_wsgi psycopg2-binary

I also had to update my apache site configuration file by adding to the top:

LoadModule wsgi_module /usr/local/myproject/.env/lib/python2.7/site-packages/mod_wsgi/server/mod_wsgi-py27.so

And change my WSGIDaemonProcess statement to use python-home instead of python-path, to something like:

WSGIDaemonProcess myproject python-home=/usr/local/myproject/.env processes=5 threads=15 display-name=%{GROUP} user=www-data group=www-data

I've encountered this for both Python2.7 and Python3.7 and the solution is the same, but the path to the mod_wsgi.so changes.

---

Solution 3:

I have received a similar error while trying to run opencv using mod_wsgi and apache. I guess the problem was probably multiple threads with underlying C code trying to acquire GIL and failing.

I solved it by setting threads=1 and processes=32(in my case it was appropriate) in WSGIDaemonProcess directive.

PS: Late to the party but thought it could help someone.