Matt_ptr *

Just a quick thought on SQLAlchemy today.

Everyone that reads this blog surely knows that I think it is the champion of database libraries despite the fact that time and again they have ditched API backward-compatibility between minor releases (Yes open-source world, 0.x.y is still a minor release). It infuriates me to no end, because I am usually effected and I have to nearly rewrite my application…. or blow it off and stay with an older version until the world itself stops turning.

Anyway, I’ve used SA for many types of applications, but using it in web apps always bothers me. Why? Use of the word session.

Sessions, in web applications, almost universally refer to the user’s “session” — their persistent data, stored on the webserver. Sessions, in SA-land, are roughly the equivalent of a database connection, but they’re not exactly the same. It handles every aspect of communicating with the database server, in a very smart and efficient manner, I might add.

When developing web applications using SA, it can get confusing. Pylons can initiate a project with some code to start your database Session if you choose to do so. And if you plan on using SA in your Pylons application, why wouldn’t you do this? By default, the variable for the database session is Session. Note the capital letter. session is different — that’s where your persistent user-data goes.

It’s simple enough with Pylons to end the confusion and change the variable. I like using dbs (database session). But, now the Pylons documentation and user snippets will be different from your app — which kinda adds some of that confusion back. To make things worse, Pylons’ docs usually refer to the 3rd party module docs — and in SA’s case, all of the examples in the documentation use the variable session.

Obviously this isn’t a major problem. But imagine it’s the first thing in the morning, you were up late and you haven’t gotten your caffeine fix. This could be catastrophic.

I haven’t given up on the projects that I’ve started recently. However, work has been pretty busy and looks to be getting busier. That means the jQuery Plugin Index, that I was in the midst of creating, has been stalled. I still *want* to do it. Very much so. However, I did notice that someone is working on PyPi for Google App Engine. If it gets fully implemented, I think it would be easy enough to fork that and adapt it.

In the meantime, my post on building python extensions in Windows has gotten a lot of attention. I hope it helps people out, but believe it or not, I still have trouble building certain extensions. It’s especially painful if the extension depends on a library that doesn’t have native Windows support. But hopefully, this cuts down on the need for running and maintaining a VM just so you can code fun stuff with Python.

So it looks like some bot managed to guess my FTP password and installed a malware script on to my wordpress files. This in turn caused Google to report this site as distributing malware and block it (at least in Chrome).

I’m 99% certain it was a bot, since I’ve seen the *exact* same hack done on a Drupal site that I did for work.

I’ve changed everything… hopefully this won’t happen again.

I think this is from submitting the Python Extensions post to reddit. It had a good deal of spam comments blocked.

A few days ago I decided to upgrade to Python 2.7. I’m running Windows 7 64-bit — pretty sweet as far as Windows goes. ;) So, I’m thinking to myself, “I’m running a 64-bit OS, why was I running a 32-bit Python?”

While the core Python distribution is available in 64-bit, many many packages that I depend on only supply precompiled binaries for the 32-bit Python distribution. Why? I have no idea. There are two things you can do.

1. Use this site. There are a bunch of packages available with 64-bit in mind that aren’t available from the package’s maintainers. MySQL-Python, for instance.
2. Compile them yourself. The unofficial repository doesn’t have all packages on PyPI compiled for Windows. gevent is one I’ve come to depend on a lot, and it’s not available — so I had to find a way to build extensions myself. Here’s how…

Install Microsoft Visual C++ 2008

Don’t bother with MinGW. Let me say it again — DO NOT USE MINGW FOR THIS! For one, the standard mingw distro is 32-bit. I found a gcc toolchain for 64-bit Windows, but I couldn’t get it to work. The Python import lib is made for Visual Studio. There are apparently ways to convert the file to something compatible, but I spent 4-5 hours trying to get this to work to absolutely no avail. Save yourself the trouble.

Additionally, you can’t use Visual C++ 2010. Python’s distutils lib is not set up to handle it. Visual C++ express works, as long as it’s 2008.

Note that if you have Visual Studio 2008 Professional, Team Studio or whatever, you should be able to stop here. The Express editions, however, don’t have the 64-bit environment, so we need to do more stuff.

Install the Windows 7 Platform SDK

Now just called the Windows SDK. You can get it here. It’s pretty large, so be prepared. Obviously, make sure you install the 64-bit environment.

Trick distutils

distutils looks for a file called vcvarsall.bat, runs it, and gets the include and lib directories that the batch file sets up. The batch file sets up the environment based on what platform you supply to it — in this case, amd64. Unfortunately, Visual C++ Express does not have the proper files for 64-bit compilation, but you can set it up pretty easily.

vcvarsall.bat should be in a directory like: C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC

You need to create:

• The directory — C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\amd64\
• The file — C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\amd64\vcvarsamd64.bat

The Windows SDK comes with a fully working 64-bit environment, so we just need to point vcvarsamd64.bat to the new SDK — which distutils doesn’t recognize.

So in vcvarsamd64.bat put:

call "C:\Program Files\Microsoft SDKs\Windows\v7.1\Bin\SetEnv.cmd" /x64 /Release

Assuming you let the Windows SDK install in the default location.

Still not done

We have to patch distutils now. Unfortunately, the new linker doesn’t generate .manifest files by default, but distutils tries to embed a manifest file in the dll (pyd) that it just built, and *will fail* if it is unable to do so.

To fix this, add the follow line to distutils\msvc9compiler.py after line 648:

ld_args.append('/MANIFEST')

That’s it!

You should now be able build your own extensions for 64-bit Python in Windows 7! You can have PyCrypto, gevent, ZODB, and so on.

Side Note

If you’re having trouble with pip or easy_install opening up a separate console window, it’s an easy fix. It’s not necessarily a problem, but it’s annoying — the console window disappears as soon as the operation is done, whether or not it fails or completes.

The issue is that setuptools is running a 32-bit application, and Windows 7 (smartly) runs 32-bit applications in a separate process.

The fix is to uninstall setuptools and pip, and reinstall setuptools from source. Do not use ez_setup.py. I don’t know if you need to be able to build extensions before you can build setuptools, but that’s what I did. After that, you can easy_install pip, and pip will now run in a 64-bit environment too. Yay!

I’ve touched on this a while back, but I never followed through with it. Having some down time at work, I’ve decided to jump in. I want to replace http://plugins.jquery.com.

There are numerous problems with it, and one of the reasons I got overwhelmed by this project originally is because I wanted to fix the site, rather than replace it. So now, I’ve wised up and decided to start from scratch without considering any aspect of how the site currently works.

Here are the problem areas, as I see them, in no particular order.

Browsing through plugins is ridiculously terrible

First, when you get to the page, you get a bunch of categories. Compare this with http://pypi.python.org. PyPi gives a tabular listing of 40 recently updated packages. For the Latest Releases, PJC (plugins.jquery.com) gives a full body of content for each item and goes on for a zillion pages.

Second, from the start page on PJC (with the category listing), the “Browse by Name” tab doesn’t work. The “Browse by Date” tab does work, but what date? The date the plugin was created, or the date of the last release? It turns out this is the same as the “Latest Releases” page, just the tab navigation at the top doesn’t disappear. The “All Plugins” link on the is the same as the “Browse by Name” tab and also doesn’t work.

Lastly, browsing plugins in a category gives a different layout from browsing by date. Why? It’s the same information, just sorted differently and filtered.

Searching is basically useless

Do you know why I’m surprised that people have actually used my timer plugin? Because I can’t even find it myself. Searching for “timer” yields 10 pages of results, and includes plain pages and issue tracker items.

I understand the appeal of having the bug tracker and plugin page tied together, but it’s terrible. A plugin like mine is so small that it doesn’t need a bug tracker. Not to mention that use of a bug tracker is annoying without the use of source control. The plugin author should bear the responsibility of setting up bug tracking, source control, etc. There are plenty of free sites to do that.

The search is easily bombed by adding keywords and tags (which are not moderated). So when I search for timer, the sixth result I get is for dualSlider — perfect for managing timeouts and intervals.

The Rating System

There’s no point to this. The “Top Rated” plugins all have 1-3 votes. Plugins with more votes should have more clout. But it doesn’t really matter anyway. It’s not a popularity contest.

This particular part of PJC will have no part whatsoever in my new project. If there will be any spotlighting of plugins, it will be done by moderators.

Other Data Formats

Right now, there are no RSS feeds for plugins at all. Each plugin should have its own release feed, as well as a feed for all latest releases.

Writing a plugin manager currently would involve screen scraping the existing plugin page to see if there have been any changes. Of course, you have to know the URL of the plugin because searching basically gets no where, and if by some chance you were able to search, you’d have to scrape the search page as well.

That’s why I want to have everything available as JSON. Plugin details, list of plugins by category, search results… the new site has to be highly query-able. PyPi uses XML-RPC to expose their API. JSONRPC might be an option for this, or XML-RPC, but I’ll cross that bridge when I come to it.

Categories

The Categories on PJC are terrible. Not in the way that they aren’t descriptive, but they just suck. They should be hierarchical. For example, “Widgets” and “Windows and Overlays” could fall under “User Interface.” Menus could as well.

I’m not sure how Navigation and Menus are different.

DOM should probably be a child of Utilities.

I don’t know what AJAX means for a category. If the plugin is an AJAX request helper, it should go under “Utilities” or “jQuery Extension.” If it’s something like an auto-complete widget, well it should go under Widgets.

The point is, that categories aren’t very helpful in there current state. I put my Timer plugin under jQuery Extensions, Javascript, and Utilities, leading me to believe that they could all be the same category. I don’t know why Javascript is a category actually, since jQuery encapsulates, rather than extends.

The New Site

I’ve already started. http://code.google.com/p/jqpi (the app page will be http://jquerypi.appspot.com)

Basically, I want to create PyPi for jQuery plugins. I figured using Google App Engine would be nice. Also, knowing my penchant for dragging out projects, I’m coding it for HTML5, since it will probably be widely supported by the time I’m finished.

There are a few things that I don’t know how to do with GAE though. Hierarchical categories, searching, optimization, JSONRPC or XML-RPC. I’ll figure it out eventually, but help is always appreciated. Create an issue, create a wiki page, send patches, join the project, anything. We shouldn’t have to suffer the damned plugins.jquery.com any more.

While I was working on the SiteCrawler script, I had a problem getting it to redirect output to a file. In fact it’s one of the reasons I put it on the back burner. I thought it was a Python (on Windows) problem, but it turns it’s just a Windows® Issue™:

http://support.microsoft.com/kb/321788

Even though the article is about WinXP and 2k, I thought I’d try the registry fix. Sure enough — it works! So I will probably start adding stuff to the site crawler again and may actually release it!

I’m a little up in arms about what to do with this site. I barely program any more in my free time. I want to do stuff, but I lack motivation. I have about 20 unfinished posts. By the time I get through, I don’t care to re-read or add the links I want to reference.

I’m also a little worried about WordPress. I don’t like it and I want to ditch it. However, I have a bunch of posts that get me some attention from Google. Is there a way to export my posts to static files?

On a more positive note, I got to use Django for a project at work. It’s an excellent product. Anyone developing webapps should try it, at least once. I might want to convert my site to Django CMS… dunno yet.

I realize that a lot of my posts are rants, especially when I’m dealing with PHP, and I hate sounding like a jerk/whiner/maniac…

But I just couldn’t help myself today.

Today’s gripe is about Javascript and Dates. There is a convenient Date object that handles just about everything I, as a web programmer, need when handling Dates in Javascript. However, there’s one particular thing that I think is probably the dumbest thing I’ve ever seen.

//So let's say we have to get some information about today's date.
//It's April 9th.
//I want the number of the month and the day in this format: m/d
var now = new Date();
document.write(now.getMonth() + "/" + now.getDate())

//What do I get?
//  3/9

…

There are three things wrong.

1. Getters and Setters — die, ok? Strings and Arrays have a length property, it’s not blah.getLength(). Date objects have no properties (that I’m aware of).
2. Keeping in mind that I must use a method begrudgingly, it should be getDay() instead of getDate(). To me, getDate() implies that you’re getting a full date string, or a Date Object, which you already have, which wouldn’t make sense. getDay() returns the day of the week, which should be getDayOfWeek().
3. You probably noticed that the value of getMonth() is wrong. Technically, it isn’t. getMonth() returns the zero-based month number, you know, just like how they count months in nowhere on Planet Earth! Who the hell thought up this? I know that I might want to use an array of month names because the Date Object doesn’t really provide a way to get that, but most of the time, no.

Why then, aren’t the calendar days zero based? I don’t get it.

This could be inspired from POSIX C, in which the tm structure has many of the same details — although not as getters and setters — and specifies that months are, you guessed it, numbered 0 to 11.

It also gives room for up to 2 leap seconds in the structure, which is really useful. No really, if you’re using localtime_r() for high-precision timing, or polling every second of every day to see if there might be a leap second…I’ll punch you.

Turns out urljoin() wasn’t behaving badly, I just supplied it a lousy URL. Turns out after running urlopen, the file-like object that is returned has two additional methods, one of them giving the true URL (i.e. after redirects). So far, that’s seems to have fixed the issue.

Hopefully I’ll have a release soon, but I still gotta work out some bugs.

I managed to find a solution for the problem I had yesterday, though I don’t particularly know if it’s ideal.

Originally I had thought that I would need to store the entire hierarchy of the site in a tree like structure. I figured I could just store a list of the links on a page in a dict structure and then output all of the errors when the crawl was finished. I don’t know why I was hung up on the idea that errors had to be reported as they were come across.

I was worried that memory use would be a factor, but it seems to be ok.

But there’s another issue:

    #taken from lxml.html.__init__
    def make_links_absolute(self, base, root):
        """This function exists because urljoin behaves obnoxiously.
        For example, if I'm on the page:
            http://www.example.com/some/directory/index.html, or just:

http://www.example.com/some/directory/

        And I join the relative URL: ../../abc.html
        I end up with: http://www.example.com/abc.html

        *But*
        If I'm on: http://www.example.com/some/directory  [no trailing slash]
        I end up with: http://www.example.com/../abc.html
        """

My fix for it was stripping out one “../”. Yesterday I thought that it would be a good fix. Today, I can’t figure out why I thought it would fix all cases.