I'm in New York Times

Well - my phone was published there.
You can find it by searching for Duke Theatre. A tiny typo on the ad. Someone swapped 2 digits.
Recently I'm getting more or less 3 calls a day. A surprising and quite amusing interruption while I work.
Imagine a dialogue:

Elderly gentlemen: Good evening, I'd like to make a reservation for 2 tickets for...

Me: Good evening, unfortunately you've called a wrong number. Please call...

G: But this number is in the Newspaper - answers gentlemen with a quite emphatic voice.

Me: I beg your pardon? Do you mind telling me which newspaper it was in?

G: It is in New York Times - says gentlemen with even more emphatic voice.


Moral of the story - don't trust newspapers.
Especially not - New York Times

:)

fabric low level socket error (vagrant)

Scene setup.

A freshly started vagrant maintained box. All you want is to configure vm using fabric minimising amount of vagrant magic as you already have plenty of your own deployment/configuration tools.

You build a simple fabfile.py

from fabric.api import env, local, run, task

from pprint import pprint

env.user = 'vagrant'

env.host_string = 'localhost:2222'

id_file_str = local('vagrant ssh-config | grep IdentityFile', capture=True)

env.key_filename = id_file_str.strip().split()[1]

@task()

def uname():

    pprint(env)

    run('uname -a')

And it fails when you execute fab uname.

Q: What does following exception mean when you execute it:

fabric.exceptions.NetworkError: Low level socket error connecting to host localhost on port 2222: Connection refused (tried 1 time)

A: When passing specific port on the host as a part of hostname please use ip address instead of dns name. Either of following will work. Having second as my preferred choice:

env.host_string = '127.0.0.1:2222'

env.hosts = ['127.0.0.1:2222']

I was quite amused to see why.

Fabric uses paramiko to open ssh connection. And paramiko uses python socket.getaddrinfo, socket.socket, socket.connect. Pretty normal and appropriate.

What happened in my case. localhost was decoded by getaddrinfo as: ipv6 '::1'. While '127.0.0.1' is explicitly ipv4.

I leave you to decide where is the bug.

Is it hosts file returning both ::1 and 127.0.0.1 for localhost?

Is it paramiko choosing first response from getaddrinfo and due to bad luck it is ipv6? BTW: Is it bad luck or OS decision?

Is it vagrant that should by default bind ssh redir both to ipv4 and ipv6?

Hard to say but I wouldn't mind if vagrant would just bind to both ipv4 and ipv6 :)

PS:

netstat -n -a | grep 2222

tcp4       0      0  *.2222                 *.*                    LISTEN  

Servname not supported for ai_socktype

Day as any other DevOps day.

Configuring service - this time Django Sentry - through puppet on EC2.
We wanted to have AWS mail backend for simplicity.

Pretty natural configuration. To achieve better server resources usage I've enabled gevent for sentry.

Thumbs up for fantastic tool sentry has became.

Though there was a strange glitch. When Sentry is receiving the exception for the first time - than it wants to send an email to the user.

What I was getting at this point was another internally reported exception:
From:
"sentry.plugins.sentry_mail.models._send_mail"

Exception:
"Error processing 'post_process' on 'MailProcessor': [Errno -8] Servname not supported for ai_socktype"

It was raised by:
gevent/socket.py in getaddrinfo

All seems good, no typo in config. nslookup works fine on smtp host name. Though there was one tiny glitch.
EMAIL_PORT = '587'

Apparently when port is passed as string than it's being resolved using entries /etc/services which is pretty awesome. So the quick fix was just to remove quotes:
EMAIL_PORT = 587

Idiotically simple solution for slightly strange exception and kind of cool to see socket errors on C level. Missed it for a while :)

IPVanish on OS X

VPN is one of the most useful pieces of software nowadays. Especially when you travel a lot. I.e. Paypal fraud protection is annoying when you login from Latin America.

Initially I was hosting VPN endpoint myself on few VPS servers (openvpn mostly). Not a big deal when you puppetize it, but in the end it was cheaper to pick an existing provider with maintained infrastructure that allowed me to change countries on each login. Having 30 countries to select from was way more useful in comparison to 3 that I've provisioned already for myself and had to maintain.
Imagine you can test a service from any place in the world for one set price. Pretty useful.

To my joy IPVanish decided to deliver a native OS X client. No more need to play with Tunnelblick or L2TP native configuration. Just click and run.

Well, almost...

Quick installation and PPTP & L2TP works just perfect. You select protocol, country/city/server and you are in. The only missing part was openvpn protocol. While L2TP is not bad OpenVPN is way nicer. It's transparent to all routers and works on all types of crappy Internet connections. I did wanted it to work.
But, when selecting it I was getting "Unable to connect to..." alert. 

Strange, this must be something with my configuration, and indeed it was. Logs for IPVanish are available from application menu. And I was getting:

Nov  7 10:15:45 theta.local com.ipvanish.helper.openvpn[10570] : Checking kext/openvpn permissions

Nov  7 10:15:45 theta.local com.ipvanish.helper.openvpn[10570] : Checking permissions for directory: /Applications/IPVanish.app/Contents/Resources/tun.kext

Nov  7 10:15:45 theta.local com.ipvanish.helper.openvpn[10570] : Checking permissions for directory: /Applications/IPVanish.app/Contents/Resources/tap.kext

Nov  7 10:15:45 theta.local com.ipvanish.helper.openvpn[10570] : kext/openvpn permissions checked

Nov  7 10:15:45 theta.local com.ipvanish.helper.openvpn[10570] : Retrieving configuration file

Nov  7 10:15:46 theta.local com.ipvanish.helper.openvpn[10570] : loading tun.kext

Nov  7 10:15:46 theta.local com.ipvanish.helper.openvpn[10570] : tun.kext: /Applications/IPVanish.app/Contents/Resources/tun.kext

Nov  7 10:15:48 theta.local com.ipvanish.helper.openvpn[10570] : tun.kext: /Applications/IPVanish.app/Contents/Resources/tun.kext

Nov  7 10:15:49 theta.local com.ipvanish.helper.openvpn[10570] : tun.kext: /Applications/IPVanish.app/Contents/Resources/tun.kext

Nov  7 10:15:50 theta.local com.ipvanish.helper.openvpn[10570] : Unable to load tun.kext

Nov  7 10:15:50 theta.local com.ipvanish.helper.openvpn[10570] : Unable to connect. Error: Generic server error

Now it's something more reasonable. Why would I have an issue with kext load?

Let's see if I don't have any conflicting modules:

theta:~ munhitsu$ kextstat

(...)

  124    0 0xffffff7f81dff000 0x6000     0x6000     foo.tap (1.0) <7 1="1" 4="4" 5="5">
  125    0 0xffffff7f81e05000 0x6000     0x6000     foo.tun (1.0) <7 1="1" 4="4" 5="5">
  126    3 0xffffff7f81e0b000 0x36000    0x36000    org.virtualbox.kext.VBoxDrv (4.1.23) <7 1="1" 3="3" 4="4" 5="5">
  127    0 0xffffff7f81e41000 0x8000     0x8000     org.virtualbox.kext.VBoxUSB (4.1.23) <126 1="1" 35="35" 3="3" 48="48" 4="4" 5="5" 7="7">
  128    0 0xffffff7f81e49000 0x5000     0x5000     org.virtualbox.kext.VBoxNetFlt (4.1.23) <126 1="1" 3="3" 4="4" 5="5" 7="7">
  129    0 0xffffff7f81e4e000 0x6000     0x6000     org.virtualbox.kext.VBoxNetAdp (4.1.23) <126 1="1" 4="4" 5="5">

That's it. foo.tap/tun seems strange to me. Quick search - it used to be installed by old versions of Tunnelblick. Gotch ya! Let's remove Tunnelblick and... the same.
Unfortunately removing Tunelblick bundle does not remove modules.

The proper way to do it is to:
let's disable modules:
theta:~ munhitsu$ sudo kextunload -b foo.tun
theta:~ munhitsu$ sudo kextunload -b foo.tap

let's remove them from filesystem:
theta:~ munhitsu$ sudo rm -Rf /Library/Extensions/tun.kext
theta:~ munhitsu$ sudo rm -Rf /Library/Extensions/tap.kext

And now IPVanish works perfectly with OpenVPN. Quick glimpse to the logs:
(...)
Nov  7 10:23:20 theta.local com.ipvanish.helper.openvpn[10851] : loading tun.kext

Nov  7 10:23:20 theta.local com.ipvanish.helper.openvpn[10851] : tun.kext: /Applications/IPVanish.app/Contents/Resources/tun.kext

Nov  7 10:23:20 theta.local com.ipvanish.helper.openvpn[10851] : Starting openvpn

(...)

Wed Nov  7 10:23:31 2012 Initialization Sequence Completed

Wed Nov  7 10:23:31 2012 MANAGEMENT: >STATE:1352283811,CONNECTED,SUCCESS,...

SUCCESS

pip install django-dowser

Thanks to a stimulating patch from David Joerg, from now on anybody can install django-dowser using pip.
So don't chase your memory leaks using print and scratching your head.

Instead...

1. Issue:
pip install django-dowser 

2. Add django_dowser to settings.py:
INSTALLED_APPS = ( 
#... 
 'django_dowser', 
#... 
)

3. Modify urls.py:
urlpatterns += patterns('', 
 (r'^dowser/', include('django_dowser.urls')), 
)

That's all folks
;)

Making django a better WSGI citizen

Aka Monkey patching Django with paste registry

Once upon a time there were two Django projects. A legacy one and a newly developed one. The modern one was developed independently as a separate project. They were both working on separate databases (in fact Django db backend was not used, it was just mongodb and redis backends), using different set of middlewares, different context processors… Two healthy projects.

What is also important they were both using gevent and deployed as wsgi servers.

During production deployment of the modern one we’ve decided that if we can make them both to work together using wfront, than we can save on amount of python instances being run on the same cluster (read: RAM).

In this configuration one instance of wfront wsgi app wraps couple of wsgi apps within one thread. Thread is than shared by gevent managed requests dispatch.

   +-----+    +-------------------------------------+
   |     |    |                                     |
   |     +--->| Legacy Django Project as a WSGI App |
   |     |    |                                     |
   |     |    +-------------------------------------+
   |     |
   |  W  |    +-------------------------------------+
   |     |    |                                     |
   |  F  +--->| Modern Django Project as a WSGI App |
   |     |    |                                     |
   |  R  |    +-------------------------------------+
   |     |
   |  O  |    +-------------------------------------+
   |     |    |                                     |
   |  N  +--->| Other WSGI App 1                    |
   |     |    |                                     |
   |  T  |    +-------------------------------------+
   |     |
   |     |            ...
   |     |
   |     |    +-------------------------------------+
   |     |    |                                     |
   |     +--->| Other WSGI App X                    |
   |     |    |                                     |
   +-----+    +-------------------------------------+

Idea is simple and as long as we talk about clean WSGI apps that are not cluttering globals than we are fine.

Unfortunately Django is using globals. And the first one I’ve reached was parsed settings object.
The standard way to read settings from within Django project is to use:

from django conf import settings

What it does is a creation of a lazy/proxy/cache singleton that once called loads all settings.py content to itself. Location of settings.py is taken from os.environ['DJANGO_SETTINGS_MODULE']. Thanks to it we have one namespace optimizing access for all Django apps/modules to take settings from.

What we need it to have a separate settings.py file loaded for each of our Django based wsgi apps. Certainly we have to do something about loading the settings from file name defined in os.environ['DJANGO_SETTINGS_MODULE'].

So we need to hijack following entry:

class LazySettings(LazyObject):
    def _setup(self):
        try:
            settings_module = os.environ[ENVIRONMENT_VARIABLE]
            if not settings_module: # If it's set but is an empty string.
                raise KeyError
        except KeyError:
            raise ImportError("Settings cannot be imported, because environment variable %s is undefined." % ENVIRONMENT_VARIABLE)

        self._wrapped = Settings(settings_module)

settings = LazySettings()

Than we need to make it configurable, preferably on initialization/call to Django wsgi application created by application = django.core.handlers.wsgi.WSGIHandler().

I wanted to keep surgery on minimal level so inspired by great article by Simon Willson about djng I’ve decided to use paste.registry module. It’s a nice way to have a request-specific (thread safe) objects. Greenlet level safety was already maintained by geven monkey patching.

import gevent.monkey; gevent.monkey.patch_all();

The idea behind paste.registry is to wrap existing WSGI app with registry setup middleware. Than you can use other WSGI middleware to inject some data, then another to edit it and finally wsgi app to read it. Data is passed using request object, but data access api is perfect for monkey patching of existing code.

            +------------+   +------------+   +-------------+
            |            |   |            |   |             |
            |            |   |            |   |             |
            |            |   |            |   |             |
            |            |   |            |   |             |
            |            |   |            |   |             |
            |            |   |            |   |             |
            |            |   |            |   |             |
            | WSGI       |   | WSGI       |   | WSGI        |
            |            |   |            |   |             |
            | Middleware |   | Middleware |   | Application |
            |            |   |            |   |             |
            | Registry   |   | B          |   |             |
            | Manager    |   |            |   |             |
            |            |   |            |   |             |
   Request  |            |   |            |   |             |
--------------------------------------------->|             |
            |            |   |            |   |             |
            |            |   |            |   |             |
            |            |   |            |   |             |
            |            |   |            |   |             |
            |            |   |            |   |             |
   Response |            |   |            |   |             |
<---------------------------------------------|             |
            |            |   |            |   |             |
            +------------+   +------------+   +-------------+

Let’s start from monkey patching of django.conf to go through registry defined object.

def monkey_patch_conf():
    import django.conf
    django.conf.settings = StackedObjectProxy(name="patched Settings")

Next step is to properly initialize the settings object using an extra middleware. Fortunately there is already a way to initialize Settings from specific file. Just Settings(settings_module).

class SettingsMiddleware:
    def __init__(self, app, settings_module):
        self.wrapped_app = app
        self.settings_module = settings_module

    def __call__(self, environ, start_response):
        from django.conf import Settings, settings
        real_settings = Settings(self.settings_module)

        if environ.has_key('paste.registry'):
            environ['paste.registry'].register(settings, real_settings)
        return self.wrapped_app(environ, start_response)

A final step is to initialize registry and wrap it all together.

server = WSGIServer(address, wfront.route([
                ('localhost', RegistryManager(SettingsMiddleware(application, "foo.settings")), None),
                ('*', RegistryManager(SettingsMiddleware(application, "bar.settings")), None),
             ]))

Paste RegistryManager initializes the paste registry. Than our SettingsMiddleware loads appropriate settings file and than in the end django application works on the request with having appropriate settings loaded.

            +------------+   +------------+   +-------------+
            |            |   |            |   |             |
            |            |   |            |   |             |
            |            |   |            |   |             |
            |            |   |            |   |             |
            |            |   |            |   |             |
            |            |   |            |   |             |
            |            |   |            |   |             |
            | WSGI       |   | WSGI       |   | WSGI        |
            |            |   |            |   |             |
            | Middleware |   | Middleware |   | Application |
            |            |   |            |   |             |
            | Registry   |   | Settings   |   | Django      |
            | Manager    |   |            |   |             |
            |            |   |            |   |             |
   Request  |            |   |            |   |             |
--------------------------------------------->|             |
            |            |   |            |   |             |
            |            |   |            |   |             |
            |            |   |            |   |             |
            |            |   |            |   |             |
            |            |   |            |   |             |
   Response |            |   |            |   |             |
<---------------------------------------------|             |
            |            |   |            |   |             |
            +------------+   +------------+   +-------------+

And that is al folks. At least for first global :) Next global was optimization for context_processors which patching required pretty the same steps.

Code Sample

import gevent.monkey; gevent.monkey.patch_all();

from gevent.wsgi import WSGIServer
from paste.registry import RegistryManager, StackedObjectProxy
import wfront

import logging
log = logging.getLogger(__name__)

def monkey_patch_conf():
    log.debug("monkey_patching django.conf")
    import django.conf
    django.conf.settings = StackedObjectProxy(name="patched Settings")


class SettingsMiddleware:
    """
    A middleware that loads settings from a specific file and exposes its content to whole wsgi thread, coroutine.
    Requires:
    - RegistryManager middleware
    - monkey_patch_conf
    exemplary usage: RegistryManager(SettingsMiddleware(wsgi_application, "foo.settings"))
    """
    def __init__(self, app, settings_module):
        self.wrapped_app = app
        self.settings_module = settings_module

    def __call__(self, environ, start_response):
        from django.conf import Settings, settings
        real_settings = Settings(self.settings_module)

        if environ.has_key('paste.registry'):
            environ['paste.registry'].register(settings, real_settings)
        return self.wrapped_app(environ, start_response)


def main():
    monkey_patch_conf()

    import django.core.handlers.wsgi
    application = django.core.handlers.wsgi.WSGIHandler()

    address = "0.0.0.0", 8000
    server = WSGIServer(address, wfront.route([
                ('localhost', RegistryManager(SettingsMiddleware(application, "foo.settings")), None),
                ('*', RegistryManager(SettingsMiddleware(application, "bar.settings")), None),
             ]))

    try:
        log.info( "Server running on port %s:%d. Ctrl+C to quit" % address)
        server.serve_forever()
    except KeyboardInterrupt:
        server.stop()
        log.info( "Bye bye")


if __name__ == "__main__":
    main()

Having issues with Xcode 4.1 upgrade on Lion?

So you've upgraded to Lion and you want to upgrade xcode to 4.1 as well.
Should be simple, but surprisingly while you install the upgrade of xcode you are stuck on iTunes not closing properly.
Well in fact you can close it but xcode installer seems to miss this fact.
You ask yourself what's going on?

Does it sound familiar?

The reason is simple. Even though you close iTunes (BTW: why can't xcode installer do it by itself?) there is still iTunesHelper in background blocking Xcode installer.
So all you have to do is to open Terminal and issue:
$ ps -ef | grep iTunes
  501   406    96   0 12:49AM ??         0:00.14 /Applications/iTunes.app/Contents/MacOS/iTunesHelper.app/Contents/MacOS/iTunesHelper -psn_0_180268

$ kill 406

where 406 is pid of iTunesHelper

or simpler:

$ killall  iTunesHelper

Have fun :)

PS: That's something new for Apple to have QA issues.