Matt_ptr *

I’ve always liked CherryPy. I thought its routing method was a very novel take on the application routing problem. But I never used it too much because it was too loosely coupled for my purposes at the time.

Fast forward to a few weeks ago when I hear about circuits.web, part of the circuits framework. Event driven, modeled after CherryPy, asynchronous, web-socket support, and loosely coupled — which I need this time.

I’ve spent the better part of three weeks working with circuits.web, and I can say it’s been a resounding disappointment.

First of all, documentation is non-existent. And when I say non-existent, I mean non-existent. The core circuits framework is poorly documented at best. The circuits.web component isn’t documented at all. The closest thing to documentation is a collection of asinine examples that have almost no merit for real life applications, if they even work at all.

Second, I can’t exactly tell if the project is still being worked on. When I first started, I naively looked at the dates on the recent commits to the whole circuits project. The circuits.web portion hasn’t been committed to since August. WebSockets were made a candidate recommendation in December. Nothing in circuit.web’s implementation needs to be changed? That makes me suspicious.

Lastly, the Component architecture seemed like a good idea, but it seems to be thrown together in no logical way. Case in point?

Components handle events. Controllers inherit from Components. Controllers can’t handle events. Or if they can, I couldn’t figure out a way. Who knows why — there’s no documentation.

I was trying to create an Authentication component for an admin controller. There are of course, no examples of this. The authentication examples — that is the ACL, HTTP Auth, and HTTP Auth using /etc/passwd — all use an authentication scheme that is site wide.

What I needed was a way to pick specific actions that would require authorization. I figured out a way by accident, and within 3 seconds found a way around it due to the “channel” member of the Controller.

The channel in circuits.web, I assume, specifies the routing for the dispatcher to find the object. So your Admin Controller object might have a channel of “/admin”. The problem I found, using a Component for forcing authentication, I could bypass the Auth Component by going to, “http://localhost/admin/“. Note the trailing slash.

Change the channel to “/admin/” or “admin” and get a 404 on all admin actions — but the auth component works.

And that’s where I’m at. I should have went with CherryPy on this project, but no, I guess I felt like being disappointed by Python web frameworks again.

So I’ve been toying with NLTK and generating text. I’ve written a plugin for the crappy-irc-bot project which uses bash.org as a source. I wrote something similar, which uses one of the built-in NLTK sources to generate dummy text in house. I got sick of using “Lorem ipsum…”

And yesterday, I adapted it to use some of the wild stuff that Charlie Sheen has been saying and feed it in to a Twitter account, charliesheenbot.

Right now, it’s just using a Trigram based generator, so it usually doesn’t make a lot of sense. I tried using a grammar based generator at one point, but it was even worse. Grammatically, it was valid. But as far as looking like something a human being would say, no. Turns out that this is one of the more hilarious parts of the English language. Here are two of my favorite articles on grammatically correct but meaningless sentences:

http://en.wikipedia.org/wiki/Buffalo_buffalo_Buffalo_buffalo_buffalo_buffalo_Buffalo_buffalo

http://en.wikipedia.org/wiki/James_while_John_had_had_had_had_had_had_had_had_had_had_had_a_better_effect_on_the_teacher

Seems like a good grammar (that is, a natural language grammar, for use with NLTK) would be able to generate meaningful sentences, but I have no idea how to do that with NLTK, or with anything.

Things like MegaHAL or Cleverbot, if I understand correctly, use a neural net to learn sentence structure based on user input. That’s like research project stuff. I don’t really have the drive to get something like that done. The only reason Charlie Sheenbot exists is because NLTK has all of the stuff needed for it already done.

I’m open to suggestions on improvement though. In the meantime, enjoy, I guess.

So in a rare moment of downtime today, I decided to check out the Pylons mailing list since I’m working on a *huge* project in Pylons. Somehow I missed that Pylons is merging with repoze.bfg.

What does it mean? According to Ben Bangert, it will provide, “a way to easily extend a Pylons project.” What does extending mean here? I have no idea. Apparently, in the few Pylons apps I wrote, I’ve never said, “Gee this is great, but I need Pylons to do shit that it doesn’t need to do.” I guess?

The only thing I can think of is similar to Django’s reusable apps. If I was that worried about, I’d use Django.

Anyway, what does it mean to me? Another rewrite. I started using Pylons at 0.9.4. I wrote a project for it. Then 0.9.5 came out and I rewrote it for that. Then 0.9.6 came out and I rewrote parts again.

So now I’m in the middle of one of the biggest projects I’ve ever done — and I’ve staked my reputation on Pylons — and I’m finding out that I will need to rewrite it if I plan on using the next version of Pylons.

No. I’m not doing that. The Pylons 1.0 codebase isn’t going anywhere (it seems). But it won’t grow anymore either.

I will continue to write my application for Pylons. I don’t plan on using Pyramid. I don’t care how good the architecture is. I’m not worried that it uses parts of Zope. I don’t care how simple the hello world program is. I don’t care about extensibility. I don’t care that it’s “better” than micro-frameworks because they did something “wrong” or “evil” in their design.

I don’t hate Zope, I don’t have any bias against one web framework or another, and nothing in the Defense Document applies to me. I just… don’t… care.

I’m glad the Pylons devs will get around this wall they’ve run in to. Well… until the next wall and the next rewrite, the next merge, the next whatever. I’m staying away.

I have 2 draft posts written, one of which sings the praises of SQLAlchemy, the other describes some of the pitfalls I’ve run in to with Pylons, but I can’t really think of a way to finish them. So I’ll just summarize them here:

Pylons

1. Documentation still not very good. I have 8 separate sites that I use for reference.
2. Avoid @validate. It is only useful if you don’t want custom validation error messages from FormEncode.
3. If you want orphan control in your models, know that you can define the cascade rules in your ForeignKey fields. Defining them in the call to relationship() will not handle ‘delete-orphan’ unless you use forward referencing.

SQLAlchemy

1. Setting up relationships is so simple, it’s almost criminal. Even many-to-many.
2. No really, it almost seems like witchcraft.
3. Eager loading of relationships is probably the coolest thing ever.

When I wrote raw queries in apps, I hated using joins because it blew up my result table — horizontally and vertically. When I started using ORMs I couldn’t use joins anymore, but I sometimes hated the fact that I knew a million queries were getting sent. SQLAlchemy addresses this through spells and incantations. I know some kind of post-processing is going on because not only can you load an entire hierarchy of objects, but all in one query… with joins. SQLAlchemy maintains the hierarchy and no additional queries will be run at reference time. Witches, I tell ya.

A few days ago I decided to upgrade to Python 2.7. I’m running Windows 7 64-bit — pretty sweet as far as Windows goes. ;) So, I’m thinking to myself, “I’m running a 64-bit OS, why was I running a 32-bit Python?”

While the core Python distribution is available in 64-bit, many many packages that I depend on only supply precompiled binaries for the 32-bit Python distribution. Why? I have no idea. There are two things you can do.

1. Use this site. There are a bunch of packages available with 64-bit in mind that aren’t available from the package’s maintainers. MySQL-Python, for instance.
2. Compile them yourself. The unofficial repository doesn’t have all packages on PyPI compiled for Windows. gevent is one I’ve come to depend on a lot, and it’s not available — so I had to find a way to build extensions myself. Here’s how…

Install Microsoft Visual C++ 2008

Don’t bother with MinGW. Let me say it again — DO NOT USE MINGW FOR THIS! For one, the standard mingw distro is 32-bit. I found a gcc toolchain for 64-bit Windows, but I couldn’t get it to work. The Python import lib is made for Visual Studio. There are apparently ways to convert the file to something compatible, but I spent 4-5 hours trying to get this to work to absolutely no avail. Save yourself the trouble.

Additionally, you can’t use Visual C++ 2010. Python’s distutils lib is not set up to handle it. Visual C++ express works, as long as it’s 2008.

Note that if you have Visual Studio 2008 Professional, Team Studio or whatever, you should be able to stop here. The Express editions, however, don’t have the 64-bit environment, so we need to do more stuff.

Install the Windows 7 Platform SDK

Now just called the Windows SDK. You can get it here. It’s pretty large, so be prepared. Obviously, make sure you install the 64-bit environment.

Trick distutils

distutils looks for a file called vcvarsall.bat, runs it, and gets the include and lib directories that the batch file sets up. The batch file sets up the environment based on what platform you supply to it — in this case, amd64. Unfortunately, Visual C++ Express does not have the proper files for 64-bit compilation, but you can set it up pretty easily.

vcvarsall.bat should be in a directory like: C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC

You need to create:

• The directory — C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\amd64\
• The file — C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\amd64\vcvarsamd64.bat

The Windows SDK comes with a fully working 64-bit environment, so we just need to point vcvarsamd64.bat to the new SDK — which distutils doesn’t recognize.

So in vcvarsamd64.bat put:

call "C:\Program Files\Microsoft SDKs\Windows\v7.1\Bin\SetEnv.cmd" /x64 /Release

Assuming you let the Windows SDK install in the default location.

Still not done

We have to patch distutils now. Unfortunately, the new linker doesn’t generate .manifest files by default, but distutils tries to embed a manifest file in the dll (pyd) that it just built, and *will fail* if it is unable to do so.

To fix this, add the follow line to distutils\msvc9compiler.py after line 648:

ld_args.append('/MANIFEST')

That’s it!

You should now be able build your own extensions for 64-bit Python in Windows 7! You can have PyCrypto, gevent, ZODB, and so on.

Side Note

If you’re having trouble with pip or easy_install opening up a separate console window, it’s an easy fix. It’s not necessarily a problem, but it’s annoying — the console window disappears as soon as the operation is done, whether or not it fails or completes.

The issue is that setuptools is running a 32-bit application, and Windows 7 (smartly) runs 32-bit applications in a separate process.

The fix is to uninstall setuptools and pip, and reinstall setuptools from source. Do not use ez_setup.py. I don’t know if you need to be able to build extensions before you can build setuptools, but that’s what I did. After that, you can easy_install pip, and pip will now run in a 64-bit environment too. Yay!