IOError: [Errno 24] Too many open files
can't create socket
I know, it's "everything is a file" in unix so that explains "can't create socket".
I knew it was leaking file descriptors but I had to find the source. The corporate environment once again poses additional challenges. Corporate Policy prevents me from poking around on production servers.
Thankfully I have an awesome XML-RPC console built-in my application. It runs regular python code within the server process. The application and it's designers deserve far more credit than I gave them in a previous post. This feature is a god send.
>>> import os
>>> os.listdir('/proc/%s/fd' % os.getpid())
['0', '1', '2', '3', '4', '5', ... '889']Uh oh! This isn't good. What are these files?
>>> data = os.fdopen(889).read()
>>> send_mail('mycorporateemail@email.com', 'Your file', 'Empty body', attachments={'data.bin': data})
After a few misguided attempts I open this file as a with Microsoft Word. It is an attachment an agent made when submitting a new client.
I quickly need to understand how attachments are generated in our framework. It was written with layers of WSGI which is a good thing because WSGI is simple to understand.
The application uses the standard python module cgi.FieldStorage to handle form attachments. The class documentation reads:
This version opens a temporary file for reading and writing,
and immediately deletes (unlinks) it. The trick (on Unix!) is
that the file can still be used, but it can't be opened by
another process, and it will automatically be deleted when it
is closed or when the current process terminates.
Closing the "file" contained in cgi.FieldStorage, does not release the file descriptor as stated. I have not determined if this is a bug in python or my application. However, I patched my application by adding a __del__ method to our request object. The object holds a reference to the file, and my __del__ will os.close the file's fd number. This is the only time I've ever had to implement __del__. I was thankful that the request object was not involved in a circular reference leak.
