dimanche 26 juin 2016

'MySQL has gone away' with Django ORM used outside Django project

I have a Django app and a Tornado service running alongside on the same server. In the Tornado service, I'm using the Django ORM to access the MySQL database. The same database is used by the Django app.

Every page in our Django web app, when rendered on the client, establishes a persistent (WebSocket) connection to the Tornado service. That service uses the Django ORM to retrieve data and return it to the client.

The website is not heavily used, and sometimes several hours or even a day or two may pass between subsequent requests.

I'm getting the infamous '2006: MySQL has gone away' error in the Tornado service after the website has been idle for a while. I did some digging and it appears that the culprit is the connection being dropped by MySQL.

This is what baffles me, however: I'm using the same Django ORM that the Django app is using, and yet the Django app itself never fires this error. Moreover, my understanding is that the Django ORM reconnects automatically if this error occurs, which explains why I'm not seeing this error in the Django app. Why is it happening for me in Tornado, then?

As it stands now, the only way for me to bring my Tornado instance back to life is to restart the gunicorn process running it. After a restart, Tornado will work without hiccups until I leave it be for several hours.

I've read this: https://code.djangoproject.com/ticket/21597#comment:29 and some of the answers to the similar problems here on StackOverflow, but I don't think increasing timeout on MySQL solves the issue, it just makes it less likely to occur. (And anyway, my wait_timeout is set to a reasonably large value anyway - 28800. On another note, just to get it out of the way, max_allowed_packet is set to 16777216 and is unlikely to be a problem here because the call it fails on is basically just retrieving the session object from the database.)

Another solution proposed in the link above by one of the Django core developers is to explicitly close the connection: from django.db import connection; connection.close() when you know that your program is going to be idle for a long time. I don't actually know this, because I can't predict how often my clients will request pages, obviously. Closing connection after a request has been served seems like overkill to me, as well. If there's no other way, of course I'll do it, but it just seems that something's off here because it seems like Django is supposed to reconnect transparently (I actually read it somewhere but not entirely sure this is true.)

I guess my main question is, if closing connection is required, why does my Django app seem to be just fine the way it is? I'm not closing any connections in the Django app, and yet it doesn't ever raise the same error. If closing connection is not required and is done by Django automatically, why does the Django ORM used in the Tornado service throw this error then?

Just for reference, this is the code that causes this error in the Tornado app.

def get_django_session(handler):
    if not hasattr(handler, '_session'):
        engine = importlib.import_module(
                django.conf.settings.SESSION_ENGINE)
        session_key = handler.get_cookie(django.conf.settings.SESSION_COOKIE_NAME)
        handler._session = engine.SessionStore(session_key)
    return handler._session


def get_current_user(handler):
    # get_user needs a django request object, but only looks at the session
    class Dummy(object):
        pass

    django_request = Dummy()
    django_request.session = get_django_session(handler)
    user = django.contrib.auth.get_user(django_request)  # 2006: MySQL has gone away
    if user.is_authenticated():
        return user
    else:
        return None

Aucun commentaire:

Enregistrer un commentaire