Matt_ptr *

I’ve touched on this a while back, but I never followed through with it. Having some down time at work, I’ve decided to jump in. I want to replace http://plugins.jquery.com.

There are numerous problems with it, and one of the reasons I got overwhelmed by this project originally is because I wanted to fix the site, rather than replace it. So now, I’ve wised up and decided to start from scratch without considering any aspect of how the site currently works.

Here are the problem areas, as I see them, in no particular order.

Browsing through plugins is ridiculously terrible

First, when you get to the page, you get a bunch of categories. Compare this with http://pypi.python.org. PyPi gives a tabular listing of 40 recently updated packages. For the Latest Releases, PJC (plugins.jquery.com) gives a full body of content for each item and goes on for a zillion pages.

Second, from the start page on PJC (with the category listing), the “Browse by Name” tab doesn’t work. The “Browse by Date” tab does work, but what date? The date the plugin was created, or the date of the last release? It turns out this is the same as the “Latest Releases” page, just the tab navigation at the top doesn’t disappear. The “All Plugins” link on the is the same as the “Browse by Name” tab and also doesn’t work.

Lastly, browsing plugins in a category gives a different layout from browsing by date. Why? It’s the same information, just sorted differently and filtered.

Searching is basically useless

Do you know why I’m surprised that people have actually used my timer plugin? Because I can’t even find it myself. Searching for “timer” yields 10 pages of results, and includes plain pages and issue tracker items.

I understand the appeal of having the bug tracker and plugin page tied together, but it’s terrible. A plugin like mine is so small that it doesn’t need a bug tracker. Not to mention that use of a bug tracker is annoying without the use of source control. The plugin author should bear the responsibility of setting up bug tracking, source control, etc. There are plenty of free sites to do that.

The search is easily bombed by adding keywords and tags (which are not moderated). So when I search for timer, the sixth result I get is for dualSlider — perfect for managing timeouts and intervals.

The Rating System

There’s no point to this. The “Top Rated” plugins all have 1-3 votes. Plugins with more votes should have more clout. But it doesn’t really matter anyway. It’s not a popularity contest.

This particular part of PJC will have no part whatsoever in my new project. If there will be any spotlighting of plugins, it will be done by moderators.

Other Data Formats

Right now, there are no RSS feeds for plugins at all. Each plugin should have its own release feed, as well as a feed for all latest releases.

Writing a plugin manager currently would involve screen scraping the existing plugin page to see if there have been any changes. Of course, you have to know the URL of the plugin because searching basically gets no where, and if by some chance you were able to search, you’d have to scrape the search page as well.

That’s why I want to have everything available as JSON. Plugin details, list of plugins by category, search results… the new site has to be highly query-able. PyPi uses XML-RPC to expose their API. JSONRPC might be an option for this, or XML-RPC, but I’ll cross that bridge when I come to it.

Categories

The Categories on PJC are terrible. Not in the way that they aren’t descriptive, but they just suck. They should be hierarchical. For example, “Widgets” and “Windows and Overlays” could fall under “User Interface.” Menus could as well.

I’m not sure how Navigation and Menus are different.

DOM should probably be a child of Utilities.

I don’t know what AJAX means for a category. If the plugin is an AJAX request helper, it should go under “Utilities” or “jQuery Extension.” If it’s something like an auto-complete widget, well it should go under Widgets.

The point is, that categories aren’t very helpful in there current state. I put my Timer plugin under jQuery Extensions, Javascript, and Utilities, leading me to believe that they could all be the same category. I don’t know why Javascript is a category actually, since jQuery encapsulates, rather than extends.

The New Site

I’ve already started. http://code.google.com/p/jqpi (the app page will be http://jquerypi.appspot.com)

Basically, I want to create PyPi for jQuery plugins. I figured using Google App Engine would be nice. Also, knowing my penchant for dragging out projects, I’m coding it for HTML5, since it will probably be widely supported by the time I’m finished.

There are a few things that I don’t know how to do with GAE though. Hierarchical categories, searching, optimization, JSONRPC or XML-RPC. I’ll figure it out eventually, but help is always appreciated. Create an issue, create a wiki page, send patches, join the project, anything. We shouldn’t have to suffer the damned plugins.jquery.com any more.

I managed to find a solution for the problem I had yesterday, though I don’t particularly know if it’s ideal.

Originally I had thought that I would need to store the entire hierarchy of the site in a tree like structure. I figured I could just store a list of the links on a page in a dict structure and then output all of the errors when the crawl was finished. I don’t know why I was hung up on the idea that errors had to be reported as they were come across.

I was worried that memory use would be a factor, but it seems to be ok.

But there’s another issue:

    #taken from lxml.html.__init__
    def make_links_absolute(self, base, root):
        """This function exists because urljoin behaves obnoxiously.
        For example, if I'm on the page:
            http://www.example.com/some/directory/index.html, or just:

http://www.example.com/some/directory/

        And I join the relative URL: ../../abc.html
        I end up with: http://www.example.com/abc.html

        *But*
        If I'm on: http://www.example.com/some/directory  [no trailing slash]
        I end up with: http://www.example.com/../abc.html
        """

My fix for it was stripping out one “../”. Yesterday I thought that it would be a good fix. Today, I can’t figure out why I thought it would fix all cases.

So I managed to stop v4 of my web crawler from opening up a billion connections in parallel. Turns out that gevent has a Pool object and that was exactly what I needed.

Now my little script (137 lines, including a utility object and comments) will not be a sysadmin’s nightmare.

However, I now have a new problem. I described how the older versions work in my previous post, but this version is quite a bit different. Instead of using a queue or stack data structure to figure out where to go next, this version has a greenlet scrape all links from a page, filters out stuff it’s already been to, then returns the rest. The main thread then accumulates the lists when all greenlets are finished. After the accumulation — and it’s ensured that there are no duplicate links — the main thread then spawns a greenlet for each link and the main thread waits until the greenlets finish again. When there are no links returned by the greenlets, the main thread is done, and the script terminates.

The problem is, if there’s a 404 or some kind of error retrieving the page, I have no way of knowing what page that link was found on.

The only solution that I see is using a custom data structure and hope that it doesn’t kill performance.

A little while ago, I wrote a post in which I was trying to figure out a way to improve a Web Crawler script I had written — it was one of those that I never published.

Anyway, for some reason, I wrote it using Stackless Python, but I was pretty novice and didn’t make it as efficient as I could. This was version 1, and it basically went to each page, scraped all the valid links (ie, those that were on the same server and not a mailto: or something) and then went through recursively.

Version 2 was basically the same, just with cleaner code and no recursion. I decided to set up the library so I would extend the SiteCrawler class and get notified of what was going on through callbacks. While it wasn’t any faster than version 1, it did seem a bit more stable.

Version 3 I decided to change drastically and made it multithreaded. It is much, much, much faster. It works like this, there’s an input Queue, and an already checked Queue. There are 4 threads waiting for input on the input Queue, when they get it, they scrape the links, check to see if any of them are in the checked Queue, put what’s been filtered on the input Queue, and put what it just checked on the checked Queue. It seems more complicated than it is. Also the code is more complicated than it needs to be.

Anyway, version 3 works well for me when I need to test a site. It’s saved me so much time in going through and checking for broken URLs. There are a few clients with the number of pages on their site in the 300′s.

But, I recently found out about gevent, and since I have some free time at work, I wanted to play with it a little bit. If you don’t know, gevent is a package that works with the greenlet package on top of libevent. I’m always interested in concurrent programming, and new technologies involved in it. This is why I had installed Stackless at one time.

So now there’s a version 4 of the SiteCrawler script, using — you guessed it — gevent. I haven’t ironed out all of the kinks yet. I was testing a non-pooled version of the script and it basically crawled through 200 links in a matter of seconds — hopefully none of the server admins look at the logs and see 100 simultaneous connections at 3:30 PM today. I did change how the crawl is done quite a bit too. So I’m going to stop there and probably have more tomorrow or in the next few days.

Good stuff!

This is my first update in quite a while. I’m not dead. I had a few posts in draft mode, but I never finished them. The topics were:

• A web crawler script I wrote and some design issues I was having with it.
  - I actually figured out some of the problems with it and made it much more efficient. So there really wasn’t any point in finishing the post. It was more of a thinking out loud kind of deal.
• Why they should let MySQL die and why Monty Widenius should bite me.
  - This was going to be my response to Monty’s Plea to people to save MySQL from Oracle (I don’t feel like finding the link, just google it). The gist was that he probably shouldn’t have sold out to Sun, would be better concentrating on MariaDB — a supposed MySQL successor, and that generally MySQL sucks and there are better alternatives anyway. I also held him responsible for the countless headaches I endure while being forced to use MySQL, and thus decided he could bite me. :)
• How awful the jQuery plugin site is and some thoughts on how to replace it.
  - I started writing this post because I have such a hard time keeping my plugins up-to-date due to the fact that the plugin directory is terrible. I started writing the post and just went through the site listing the problems I saw. The list became so massive that it clearly was something that I couldn’t handle on my own, so the post turned in to a call to action with the suggestion of using Google App Engine to run the site. But, the number of problems is depressing and thinking about spearheading a project of such magnitude in my free time was not at all appealing.

So, the holidays came and went. The new year as well. I’ve been extraordinarily busy at work and at home, extraordinarily lazy and tired.

But I do have one small announcement. Off and on over the past year, I’ve been toying with ActionScript and Flash and Flex because a long, long time ago, I thought  I would pursue Flash as a game platform.

After some consideration, I will absolutely *NOT* be doing that.  While ActionScript, as a language, isn’t terrible, I don’t particularly like it. It’s similar to Javascript, but with more crap that you have to do. The real problem I have is with Flash as a platform. In short, it’s probably one of the worst I’ve ever seen. It’s slow, bloated, and full of known security holes. I could make a whole blog dedicated to how much I hate Flash and problems with it.

The only positive thing I can say about Flash is that it’s everywhere. Though, with all of the security problems, it may not turn out to be a good thing.

Instead — I’m trying to stay positive — I plan on focusing on some of the emerging Web App technologies as a platform. I have a feeling in 6 months, I’ll be back here griping about how ridiculous canvas, WebGl, etc. are.

I may try to write a simple game to test, or redo a certain game with flying corpses. So, until then.

I’m starting to think I’m a masochist. Examine the evidence:

• I’ve been in 2 serious relationships with crazy (and I do mean crazy) girls — both ended crappily.
• I still hand wind my guitar strings.
• I’ve worked with PHP since version 3 and I’ve had a job doing it for 5 years. I actually gave up a job doing embedded systems with C to keep doing PHP work.
• I keep thinking I want to learn Erlang.
• Lastly, and I think this is the big one, despite the fact that I’ve worked on no less than 3 failed IRC clients — I started working on an IRC bot project.

(more…)

A few months ago, I was contacted by the developer of Red Bean — a new PHP ORM library. He seems to share my dismay with the overall suckiness of every ORM library and he asked if I would give his project a looksee.

Before that, I have a few thoughts on Kohana and it’s ORM library.

(more…)

I realize that 99% of what I think would break 99% of all applications, but my dictatorship style is more of a slash and burn kind of style. Out with the old, in with the new.

• Replace the crappy dom, minidom and pulldom libs with lxml.
  XML support in python has always been kind of crummy, but lxml really changes that. It’s fast, robust, and way better than other xml libs that have been around.
• Remove Tkinter.
  It’s just taking up space.
• Remove the stuff for reading and writing audio files.
  I don’t really understand why Python and other stdlibs (I’m looking at you Windows) include stuff for outputting and/or manipulating WAV files. Yet there’s no other format support. This is something that’s best handled by a 3rd party, IMO.
• Replace telnetlib with an ssh library.
  I hope no one uses telnet anymore. I can’t remember the last time I had to telnet in to something
• Add an encryption library to the “Cryptographic Services”
  It’s just hashlib and hmac right now.
• I would also love for FTP to die, so replacing ftplib with an sftp lib might help facilitate that.

:)

So, among the Perl’s-Not-Dead crowd, there seems to be a pretty big buzz surrounding Catalyst, a web application framework that I’ve known about for a while. I first discovered it about two years ago while on one of my “I need to get out of PHP land” trips. I wasn’t very impressed with it then, so I decided to take another look today.

At first glance, their site looks nice and snazzy and web-frameworkish. Upon closer inspection, I want to rip my eyes out. The left column has a few statements that really punched my goat.

Keep It Simple, Stupid.

Thanks. You think I’m stupid. Wouldn’t, “Keep It Sublimely Simple,” have come across better?

Don’t Repeat Yourself.

I don’t know of a web framework that doesn’t say this, so no surprises. But…

There Is More Than One Way To Do It.

Am I the only one that thinks those contradict each other? If you don’t like the way something is done, then there’s another way to do it, thus introducing an element of repetition in to the equation.

*Sigh* I just don’t have the fight in me today.

Then I looked at the documentation and my head exploded. Note that the tutorial chapters are completely out of order. I’ll not bring up the fact that the CPAN color scheme is like staring at the sun for too long, so when I click on the tutorial overview, I’m greeted with a table of contents and then a *real* table of contents.

I’ve had enough. CPAN might be Perl’s strongest selling point, but to me, the I-don’t-give-a-shit-about-documentation attitude will forever keep me from using it. Not that Perl has much longer to live anyway. :P

Now, I’m going to go look at Django’s docs to be reminded of how things can be presented decently.

This has been on my mind for a long, long time. Having been spoiled with SQLAlchemy and Elixir, I am tortured with sub-par ORM libraries while at work. Granted I did get to use SA for an internal project, but the majority of the things I do involves PHP.

I’ve been stuck with it for years. It is the language of choice for people who run servers and hate web developers. So yeah, 5 years of PHP and MySQL …and using so many different libs for database abstraction has given me some room to shoot my mouth off.

Anyway, this is going to be a long post and not for the faint of heart.  I’m going to start off at the beginning…

(more…)