Matt_ptr *

I’ve always liked CherryPy. I thought its routing method was a very novel take on the application routing problem. But I never used it too much because it was too loosely coupled for my purposes at the time.

Fast forward to a few weeks ago when I hear about circuits.web, part of the circuits framework. Event driven, modeled after CherryPy, asynchronous, web-socket support, and loosely coupled — which I need this time.

I’ve spent the better part of three weeks working with circuits.web, and I can say it’s been a resounding disappointment.

First of all, documentation is non-existent. And when I say non-existent, I mean non-existent. The core circuits framework is poorly documented at best. The circuits.web component isn’t documented at all. The closest thing to documentation is a collection of asinine examples that have almost no merit for real life applications, if they even work at all.

Second, I can’t exactly tell if the project is still being worked on. When I first started, I naively looked at the dates on the recent commits to the whole circuits project. The circuits.web portion hasn’t been committed to since August. WebSockets were made a candidate recommendation in December. Nothing in circuit.web’s implementation needs to be changed? That makes me suspicious.

Lastly, the Component architecture seemed like a good idea, but it seems to be thrown together in no logical way. Case in point?

Components handle events. Controllers inherit from Components. Controllers can’t handle events. Or if they can, I couldn’t figure out a way. Who knows why — there’s no documentation.

I was trying to create an Authentication component for an admin controller. There are of course, no examples of this. The authentication examples — that is the ACL, HTTP Auth, and HTTP Auth using /etc/passwd — all use an authentication scheme that is site wide.

What I needed was a way to pick specific actions that would require authorization. I figured out a way by accident, and within 3 seconds found a way around it due to the “channel” member of the Controller.

The channel in circuits.web, I assume, specifies the routing for the dispatcher to find the object. So your Admin Controller object might have a channel of “/admin”. The problem I found, using a Component for forcing authentication, I could bypass the Auth Component by going to, “http://localhost/admin/“. Note the trailing slash.

Change the channel to “/admin/” or “admin” and get a 404 on all admin actions — but the auth component works.

And that’s where I’m at. I should have went with CherryPy on this project, but no, I guess I felt like being disappointed by Python web frameworks again.

So I’ve been toying with NLTK and generating text. I’ve written a plugin for the crappy-irc-bot project which uses bash.org as a source. I wrote something similar, which uses one of the built-in NLTK sources to generate dummy text in house. I got sick of using “Lorem ipsum…”

And yesterday, I adapted it to use some of the wild stuff that Charlie Sheen has been saying and feed it in to a Twitter account, charliesheenbot.

Right now, it’s just using a Trigram based generator, so it usually doesn’t make a lot of sense. I tried using a grammar based generator at one point, but it was even worse. Grammatically, it was valid. But as far as looking like something a human being would say, no. Turns out that this is one of the more hilarious parts of the English language. Here are two of my favorite articles on grammatically correct but meaningless sentences:

http://en.wikipedia.org/wiki/Buffalo_buffalo_Buffalo_buffalo_buffalo_buffalo_Buffalo_buffalo

http://en.wikipedia.org/wiki/James_while_John_had_had_had_had_had_had_had_had_had_had_had_a_better_effect_on_the_teacher

Seems like a good grammar (that is, a natural language grammar, for use with NLTK) would be able to generate meaningful sentences, but I have no idea how to do that with NLTK, or with anything.

Things like MegaHAL or Cleverbot, if I understand correctly, use a neural net to learn sentence structure based on user input. That’s like research project stuff. I don’t really have the drive to get something like that done. The only reason Charlie Sheenbot exists is because NLTK has all of the stuff needed for it already done.

I’m open to suggestions on improvement though. In the meantime, enjoy, I guess.

A few days ago I decided to upgrade to Python 2.7. I’m running Windows 7 64-bit — pretty sweet as far as Windows goes. ;) So, I’m thinking to myself, “I’m running a 64-bit OS, why was I running a 32-bit Python?”

While the core Python distribution is available in 64-bit, many many packages that I depend on only supply precompiled binaries for the 32-bit Python distribution. Why? I have no idea. There are two things you can do.

1. Use this site. There are a bunch of packages available with 64-bit in mind that aren’t available from the package’s maintainers. MySQL-Python, for instance.
2. Compile them yourself. The unofficial repository doesn’t have all packages on PyPI compiled for Windows. gevent is one I’ve come to depend on a lot, and it’s not available — so I had to find a way to build extensions myself. Here’s how…

Install Microsoft Visual C++ 2008

Don’t bother with MinGW. Let me say it again — DO NOT USE MINGW FOR THIS! For one, the standard mingw distro is 32-bit. I found a gcc toolchain for 64-bit Windows, but I couldn’t get it to work. The Python import lib is made for Visual Studio. There are apparently ways to convert the file to something compatible, but I spent 4-5 hours trying to get this to work to absolutely no avail. Save yourself the trouble.

Additionally, you can’t use Visual C++ 2010. Python’s distutils lib is not set up to handle it. Visual C++ express works, as long as it’s 2008.

Note that if you have Visual Studio 2008 Professional, Team Studio or whatever, you should be able to stop here. The Express editions, however, don’t have the 64-bit environment, so we need to do more stuff.

Install the Windows 7 Platform SDK

Now just called the Windows SDK. You can get it here. It’s pretty large, so be prepared. Obviously, make sure you install the 64-bit environment.

Trick distutils

distutils looks for a file called vcvarsall.bat, runs it, and gets the include and lib directories that the batch file sets up. The batch file sets up the environment based on what platform you supply to it — in this case, amd64. Unfortunately, Visual C++ Express does not have the proper files for 64-bit compilation, but you can set it up pretty easily.

vcvarsall.bat should be in a directory like: C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC

You need to create:

• The directory — C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\amd64\
• The file — C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\amd64\vcvarsamd64.bat

The Windows SDK comes with a fully working 64-bit environment, so we just need to point vcvarsamd64.bat to the new SDK — which distutils doesn’t recognize.

So in vcvarsamd64.bat put:

call "C:\Program Files\Microsoft SDKs\Windows\v7.1\Bin\SetEnv.cmd" /x64 /Release

Assuming you let the Windows SDK install in the default location.

Still not done

We have to patch distutils now. Unfortunately, the new linker doesn’t generate .manifest files by default, but distutils tries to embed a manifest file in the dll (pyd) that it just built, and *will fail* if it is unable to do so.

To fix this, add the follow line to distutils\msvc9compiler.py after line 648:

ld_args.append('/MANIFEST')

That’s it!

You should now be able build your own extensions for 64-bit Python in Windows 7! You can have PyCrypto, gevent, ZODB, and so on.

Side Note

If you’re having trouble with pip or easy_install opening up a separate console window, it’s an easy fix. It’s not necessarily a problem, but it’s annoying — the console window disappears as soon as the operation is done, whether or not it fails or completes.

The issue is that setuptools is running a 32-bit application, and Windows 7 (smartly) runs 32-bit applications in a separate process.

The fix is to uninstall setuptools and pip, and reinstall setuptools from source. Do not use ez_setup.py. I don’t know if you need to be able to build extensions before you can build setuptools, but that’s what I did. After that, you can easy_install pip, and pip will now run in a 64-bit environment too. Yay!