There are numerous problems with it, and one of the reasons I got overwhelmed by this project originally is because I wanted to fix the site, rather than replace it. So now, I’ve wised up and decided to start from scratch without considering any aspect of how the site currently works.

Here are the problem areas, as I see them, in no particular order.

Browsing through plugins is ridiculously terrible

First, when you get to the page, you get a bunch of categories. Compare this with http://pypi.python.org. PyPi gives a tabular listing of 40 recently updated packages. For the Latest Releases, PJC (plugins.jquery.com) gives a full body of content for each item and goes on for a zillion pages.

Second, from the start page on PJC (with the category listing), the “Browse by Name” tab doesn’t work. The “Browse by Date” tab does work, but what date? The date the plugin was created, or the date of the last release? It turns out this is the same as the “Latest Releases” page, just the tab navigation at the top doesn’t disappear. The “All Plugins” link on the is the same as the “Browse by Name” tab and also doesn’t work.

Lastly, browsing plugins in a category gives a different layout from browsing by date. Why? It’s the same information, just sorted differently and filtered.

Searching is basically useless

Do you know why I’m surprised that people have actually used my timer plugin? Because I can’t even find it myself. Searching for “timer” yields 10 pages of results, and includes plain pages and issue tracker items.

I understand the appeal of having the bug tracker and plugin page tied together, but it’s terrible. A plugin like mine is so small that it doesn’t need a bug tracker. Not to mention that use of a bug tracker is annoying without the use of source control. The plugin author should bear the responsibility of setting up bug tracking, source control, etc. There are plenty of free sites to do that.

The search is easily bombed by adding keywords and tags (which are not moderated). So when I search for timer, the sixth result I get is for dualSlider — perfect for managing timeouts and intervals.

The Rating System

There’s no point to this. The “Top Rated” plugins all have 1-3 votes. Plugins with more votes should have more clout. But it doesn’t really matter anyway. It’s not a popularity contest.

This particular part of PJC will have no part whatsoever in my new project. If there will be any spotlighting of plugins, it will be done by moderators.

Other Data Formats

Right now, there are no RSS feeds for plugins at all. Each plugin should have its own release feed, as well as a feed for all latest releases.

Writing a plugin manager currently would involve screen scraping the existing plugin page to see if there have been any changes. Of course, you have to know the URL of the plugin because searching basically gets no where, and if by some chance you were able to search, you’d have to scrape the search page as well.

That’s why I want to have everything available as JSON. Plugin details, list of plugins by category, search results… the new site has to be highly query-able. PyPi uses XML-RPC to expose their API. JSONRPC might be an option for this, or XML-RPC, but I’ll cross that bridge when I come to it.

Categories

The Categories on PJC are terrible. Not in the way that they aren’t descriptive, but they just suck. They should be hierarchical. For example, “Widgets” and “Windows and Overlays” could fall under “User Interface.” Menus could as well.

I’m not sure how Navigation and Menus are different.

DOM should probably be a child of Utilities.

I don’t know what AJAX means for a category. If the plugin is an AJAX request helper, it should go under “Utilities” or “jQuery Extension.” If it’s something like an auto-complete widget, well it should go under Widgets.

The point is, that categories aren’t very helpful in there current state. I put my Timer plugin under jQuery Extensions, Javascript, and Utilities, leading me to believe that they could all be the same category. I don’t know why Javascript is a category actually, since jQuery encapsulates, rather than extends.

The New Site

I’ve already started. http://code.google.com/p/jqpi (the app page will be http://jquerypi.appspot.com)

Basically, I want to create PyPi for jQuery plugins. I figured using Google App Engine would be nice. Also, knowing my penchant for dragging out projects, I’m coding it for HTML5, since it will probably be widely supported by the time I’m finished.

There are a few things that I don’t know how to do with GAE though. Hierarchical categories, searching, optimization, JSONRPC or XML-RPC. I’ll figure it out eventually, but help is always appreciated. Create an issue, create a wiki page, send patches, join the project, anything. We shouldn’t have to suffer the damned plugins.jquery.com any more.

http://support.microsoft.com/kb/321788

Even though the article is about WinXP and 2k, I thought I’d try the registry fix. Sure enough — it works! So I will probably start adding stuff to the site crawler again and may actually release it!

Originally I had thought that I would need to store the entire hierarchy of the site in a tree like structure. I figured I could just store a list of the links on a page in a dict structure and then output all of the errors when the crawl was finished. I don’t know why I was hung up on the idea that errors had to be reported as they were come across.

I was worried that memory use would be a factor, but it seems to be ok.

But there’s another issue:

    #taken from lxml.html.__init__
    def make_links_absolute(self, base, root):
        """This function exists because urljoin behaves obnoxiously.
        For example, if I'm on the page:
            http://www.example.com/some/directory/index.html, or just:

http://www.example.com/some/directory/

        And I join the relative URL: ../../abc.html
        I end up with: http://www.example.com/abc.html

        *But*
        If I'm on: http://www.example.com/some/directory  [no trailing slash]
        I end up with: http://www.example.com/../abc.html
        """

My fix for it was stripping out one “../”. Yesterday I thought that it would be a good fix. Today, I can’t figure out why I thought it would fix all cases.

Now my little script (137 lines, including a utility object and comments) will not be a sysadmin’s nightmare.

However, I now have a new problem. I described how the older versions work in my previous post, but this version is quite a bit different. Instead of using a queue or stack data structure to figure out where to go next, this version has a greenlet scrape all links from a page, filters out stuff it’s already been to, then returns the rest. The main thread then accumulates the lists when all greenlets are finished. After the accumulation — and it’s ensured that there are no duplicate links — the main thread then spawns a greenlet for each link and the main thread waits until the greenlets finish again. When there are no links returned by the greenlets, the main thread is done, and the script terminates.

The problem is, if there’s a 404 or some kind of error retrieving the page, I have no way of knowing what page that link was found on.

The only solution that I see is using a custom data structure and hope that it doesn’t kill performance.

Anyway, for some reason, I wrote it using Stackless Python, but I was pretty novice and didn’t make it as efficient as I could. This was version 1, and it basically went to each page, scraped all the valid links (ie, those that were on the same server and not a mailto: or something) and then went through recursively.

Version 2 was basically the same, just with cleaner code and no recursion. I decided to set up the library so I would extend the SiteCrawler class and get notified of what was going on through callbacks. While it wasn’t any faster than version 1, it did seem a bit more stable.

Version 3 I decided to change drastically and made it multithreaded. It is much, much, much faster. It works like this, there’s an input Queue, and an already checked Queue. There are 4 threads waiting for input on the input Queue, when they get it, they scrape the links, check to see if any of them are in the checked Queue, put what’s been filtered on the input Queue, and put what it just checked on the checked Queue. It seems more complicated than it is. Also the code is more complicated than it needs to be.

Anyway, version 3 works well for me when I need to test a site. It’s saved me so much time in going through and checking for broken URLs. There are a few clients with the number of pages on their site in the 300′s.

But, I recently found out about gevent, and since I have some free time at work, I wanted to play with it a little bit. If you don’t know, gevent is a package that works with the greenlet package on top of libevent. I’m always interested in concurrent programming, and new technologies involved in it. This is why I had installed Stackless at one time.

So now there’s a version 4 of the SiteCrawler script, using — you guessed it — gevent. I haven’t ironed out all of the kinks yet. I was testing a non-pooled version of the script and it basically crawled through 200 links in a matter of seconds — hopefully none of the server admins look at the logs and see 100 simultaneous connections at 3:30 PM today. I did change how the crawl is done quite a bit too. So I’m going to stop there and probably have more tomorrow or in the next few days.

Good stuff!

• A web crawler script I wrote and some design issues I was having with it.
  - I actually figured out some of the problems with it and made it much more efficient. So there really wasn’t any point in finishing the post. It was more of a thinking out loud kind of deal.
• Why they should let MySQL die and why Monty Widenius should bite me.
  - This was going to be my response to Monty’s Plea to people to save MySQL from Oracle (I don’t feel like finding the link, just google it). The gist was that he probably shouldn’t have sold out to Sun, would be better concentrating on MariaDB — a supposed MySQL successor, and that generally MySQL sucks and there are better alternatives anyway. I also held him responsible for the countless headaches I endure while being forced to use MySQL, and thus decided he could bite me. :)
• How awful the jQuery plugin site is and some thoughts on how to replace it.
  - I started writing this post because I have such a hard time keeping my plugins up-to-date due to the fact that the plugin directory is terrible. I started writing the post and just went through the site listing the problems I saw. The list became so massive that it clearly was something that I couldn’t handle on my own, so the post turned in to a call to action with the suggestion of using Google App Engine to run the site. But, the number of problems is depressing and thinking about spearheading a project of such magnitude in my free time was not at all appealing.

So, the holidays came and went. The new year as well. I’ve been extraordinarily busy at work and at home, extraordinarily lazy and tired.

But I do have one small announcement. Off and on over the past year, I’ve been toying with ActionScript and Flash and Flex because a long, long time ago, I thought  I would pursue Flash as a game platform.

After some consideration, I will absolutely *NOT* be doing that.  While ActionScript, as a language, isn’t terrible, I don’t particularly like it. It’s similar to Javascript, but with more crap that you have to do. The real problem I have is with Flash as a platform. In short, it’s probably one of the worst I’ve ever seen. It’s slow, bloated, and full of known security holes. I could make a whole blog dedicated to how much I hate Flash and problems with it.

The only positive thing I can say about Flash is that it’s everywhere. Though, with all of the security problems, it may not turn out to be a good thing.

Instead — I’m trying to stay positive — I plan on focusing on some of the emerging Web App technologies as a platform. I have a feeling in 6 months, I’ll be back here griping about how ridiculous canvas, WebGl, etc. are.

I may try to write a simple game to test, or redo a certain game with flying corpses. So, until then.

• I’ve been in 2 serious relationships with crazy (and I do mean crazy) girls — both ended crappily.
• I still hand wind my guitar strings.
• I’ve worked with PHP since version 3 and I’ve had a job doing it for 5 years. I actually gave up a job doing embedded systems with C to keep doing PHP work.
• I keep thinking I want to learn Erlang.
• Lastly, and I think this is the big one, despite the fact that I’ve worked on no less than 3 failed IRC clients — I started working on an IRC bot project.

If you didn’t know, IRC was invented in 1793 by the world-renouned horrible person, the Marquis De Sade. He was so horrible, in fact, that the term sadism comes from his name. The work I’m referring to is RFC1459 — arguably his most notorious and mind-boggling work. RFC1459 is part of the reason some people consider De Sade to be the devil incarnate.

RFC1459 was so bad that the IETF has refused to standardize it. An attempt was made in 2000 to update it with RFCs 2810, 2811, 2812, and 2813. After reading all of them, I can’t really tell what was updated. It’s more like someone took the original document of unholy sin and smeared it across 4 others. And even after this act of moral indecency, it’s still not standardized. It’s not helped by the fact that the “updates” weren’t even supposed to be considered a new standard and also pretty much every IRCd and client does whatever the hell they want.

Here are some things I really, really hate about IRC. I mean other than it being a complete total stab in the groin…

Ident
Does anyone actually have an actual ident daemon running? And does it actually help anything? It’s pretty much a useless addition — and why it’s mentioned in the IRC protocol (and by protocol, I mean random suggestions) I could only speculate.

Mode strings
This is probably the worst abomination of the entire spec (and by spec I mean loose collection of internet bowel movements).  So there are user modes and channels modes. A mode character can have parameters or not and a mode string can consist of any of the mode characters or a bunch of them. Because of that, you end up with this kind of crap:

MODE #liek :+om-v+b nick1 nick2 *.aol.com

Which sets nick1 as an operator, sets the channel to moderated, takes voice away from nick2, and bans aol users from the channel. Look at sections 3.1.5 and 3.2.3 in RFC2812 for some more awful ideas.  If you think parsing that is not so bad, then you should know that not every IRC-network is going to have the same modes. And to figure out what modes a channel might have…

005
RFC2812 says that numeric reply 005 is, “Sent by the server to a user to suggest an alternative server.” It’s just not true at all. In all my years of working with IRC, I’ve never seen 005 used in this capacity.

005 is used for relaying the server configuration to the client. This is a separate, 5-year old document that explains the ISUPPORT reply. Of course, it’s not official or anything. It’s not even really called ISUPPORT. The page I linked to at least tells you what IRCds are doing what. Keeping track of all that configuration info is unbelievably tedious.

NAMES
Channels on IRC are usually small, but on the off-chance that you find yourself in a warez channel, you’ll be in a sea of people. When you join in the chat, the server will send you the list of other chatters, which can take quite a while depending on the legality of the channel you’re in. The problem:

The names list can be too large to fit in to one packet, so the server will break it up. The problem is that if you’re in a big room, there’s bound to be lots of activity and the server will broadcast whatever messages it’s supposed to no matter what it’s doing. So rather than queuing a JOIN notification while it’s processing the names reply, it’ll just send the notice right in-between packets.

Yeah, me too. I also have no idea what order the names list might be in. I have no idea if it’s random, alphabetized, or predicted by Nostradamus — there’s nothing in the spec (and by spec, I mean not-spec) about it.

So how do you know what the NAMES reply is? Unlike JOIN or QUIT or KICK, when you request information from the server, chances are you’ll get a numeric reply.

Numeric Replies
By numeric, they mean a string of numbers — not actually binary numerics. There are a few things I don’t understand about numeric replies.

Why do some things have a numeric reply instead of just the text? Take NAMES for instance or WHOIS. I thought it had something to do with if the client sent the message requesting a reply. But then why is 005 considered a numeric reply? Or 001? Both are sent by the server after a connection.

Where the hell did these numbers come from? There’s some basic organization — 0xx are informative, 4xx are errors, 3xx are requested info… sometimes… 2xx are used for… uh…

Beyond that, I can’t figure out why the NAMES reply is 353 and the “End of NAMES” message is 366. MOTD Starting is 375, the actual MOTD packet is 372 and MOTD Ending is 376.

Additionally, when mentioning what reply is generated from what message, the spec (and by spec, I mean gelatinous blob of useless goo) refers to the numeric codes by a  name! Sending a WHOIS on a user generates a numeric reply of RPL_WHOISUSER — which is actually numeric 311. This makes things exceptionally hard to follow and you’re better off experimenting with viewing the raw output from the server.

Security
For a while, there seemed to be a trend in having SSL connections to servers. It makes sense for say, HTTP or FTP — *real* protocols I mean. But because SSL connections are not required  for every client, IT BASICALLY MAKES THE WHOLE THING POINTLESS!!! Yeah, now I’m yelling!

You know those popups that you get in your web browser that say, “This page contains both secure and non-secure items.” — You have the option to ignore the non-secure items. If you had that option on IRC, it would basically ignore 90% of the people on there.

Packet Length
It bothers me when connection oriented protocols (and by protocol in this case, I mean…. you get the idea…) reimplement stuff that TCP already does. Actually, I think the IRC protocol is the only one that does this.

TCP already breaks up things in to packets and lets you know when the stream terminates. Think about HTTP, you send the GET request, then stream everything down and then it terminates and you’re done. If you’re getting a 2MB html doc (god forbid), you don’t get it all at once — but no HTTP client on earth has to check for a line termination every 512 chars and then check to see if it sent the “End of HTTP Stream” message.

Unicode and I18N
The Marquis de Sade, being the awful, evil man that he was made no plans for IRC to ever be in anything but 8-bit ASCII. Which is weird because he said that hundreds of years before 8-bit ASCII encoding was invented. I have yet to find *any* information in 10 years, whether official or “official,” about how Unicode works with IRC. I’m sure non-ascii characters can be sent — but as for the server encoding, BOM, etc… it’s a mystery.

CTCP and DCC
At some point, some genius decided that instead of extending the protocol that extensions could be added by a hack in to an existing message.

There is, of course, yet another document about this “part.”

Trying to add support for these basically makes your handler for PRIVMSG 9000x bigger than it was. Also, you give yourself a massive headache and you start to think, “I must be a friggin’ masochist to be working on this.”

The biggest crap-sandwich that one must eat is that the Client-to-Client Protocol goes through the Server. I’m gonna start yelling again. How about a rule that says you can’t call it a protocol if all you do is wrap some capitalized letters with \001? I’m gonna make an HTTP variant called FATTP (Fucking Awesome Text Transfer Protocol) which takes regular HTTP verbs and wraps them with \001. Fuck you.

But DCC takes the cake. Not only is it ass-backwards and full of errors — once again, you must reimplement parts of TCP stream control in order for it to work right. DCC Send is the real culprit — and it was supposedly replaced by DCC XMIT in 1997, but good luck finding anything that uses it.

The CTCP-97 document intends to offer an official spec for formatting codes. The problem is…

mIRC
Currently the biggest piece of crap in IRC today is mIRC. For some reason, it is the standard for IRC clients. It uses a broken color formatting scheme, old DCC code, horrible scripting, a poor interface, and yet, you still have to pay for it. There were many, many times when during the development of Zephyr (the IRC client I was working on from 2002-2004), we came to a point and said, “We don’t really know what to do here… What does mIRC do?”

So, given all these reasons, why can’t I convince my friends to move to jabber or something? Don’t they know what kind of state it puts me in?

At least this time, I’m only writing a bot. And, aside from reading the IRC RFCs… *sigh* I’m enjoying it.

See? Masochism.

Before that, I have a few thoughts on Kohana and it’s ORM library.

Kohana

Kohana is a full-stack framework based on CodeIgniter, but ditches PHP4 compatibility for the sake of my sanity. Really, PHP4 style coding is the only reason I don’t use CakePHP for everything. But, I digress.

I’ll just start by saying Kohana is the best PHP framework I’ve used so far. I hit a few bumps in the road, but very few. It’s pretty quick (or Swift, as they would say) and is very straight-forward when it comes to extending the base functionality.

My biggest gripe about the framework as a whole, is that I ran in to limitations a lot. With the database library mostly, but with some of the other components as well. I’ve been finding that with most open source tools I’m using, that the next version, “will be way better.”

I had a major problem in the app I’m coding with session_start() sending out cache headers — overwriting what I was trying to send out manually. But, beyond that, it’s been pretty smooth sailing.

The ORM is actually pretty decent. I didn’t do any benchmarks, so I don’t know how efficient or deficient it is, but I do know that it does lazy loading on relations. That’s something I’ll just have to live with when it comes to PHP.

There are a few problems I have with it though.

1. There’s no way to do aggregate functions with the Database Query Builder and adding them in seems to be a low priority. (But the next version might be AWESOME!)
2. There’s no way (that I know of) to wrap a database query() result with an ORM class.
3. There are some limitations in using a one-to-many relationship with a foreign key not having part of the referenced table name in the field name. In other words, it doesn’t read from FK references, but relies on the field name — i.e., Profiles.user_id is meant to reference Users.id. But if you want to have Profiles.creator_id reference a different Users.id (say the user didn’t create their own profile), there’s no way to do that with the way model relationships are defined — again, just for one-to-many relationships.

Beyond that, it’s a pretty decent library. It’s not particularly difficult to figure out how to set up models.

Let’s steam this Bean

RedBean reminds me a bit of ADODB_Active_Record in that both are developed separately from a full-stack framework.  It’s slightly similar to Kohana’s ORM, but with a few key differences, which I’ll cover shortly. RedBean also reminds me of ADODB_Active_Record in the sense that the setup and usage are better and easier than other standalone PHP ORMs by several thousand trillion times over.

Now this is going to be unfair, because to me, the pinnacle of database abstraction and object-relation mapping is still SQLAlchemy. I doubt PHP will ever have something similar, but  I will always hope. At least until PHP dies.

Initialization

Right off, there’s something I don’t like. The RedBean library  isn’t redbean.php — it’s oodb.php. I’m not really sure why, but I could see myself being confused by this if I used RedBean for a project. What can I say? I’m simple.

I think a lot of the naming scheme is something I’d want to be different.

RedBean_Setup::kickstartDev connects to the database and creates models. This is one of the features I like about RedBean.  With ADODB_Active_Record, I have to define classes and extend, etc. RedBean creates the Models for you. Unfortunately, I can’t figure out why RedBean_Setup is it’s own class and if I had never seen documents before, I would never guess that the initialization method would be called ‘kickstartDev’. Would RedBean::connect() be too easy?

Table Creation

Another neet feature of RedBean is that it will create a database schema for any model that doesn’t already have a table. Read the docs, you’ll see it.

SQLAlchemy has something similar with Metadata().create_all(). Although with that, you create the model declaratively. RedBean will build the schema based on what fields you have set in the model. It sounds handy, but Python programmers would probably be unsettled — it’s just a different philosophy.

I didn’t have time to test this extensively, so as to what happens when you set a new field when the model is already defined, I can only speculate.

The CRUDiness

Overall, there aren’t many surprises when it comes to fetching records.

//Load a user
$user = new User($id);

//Similarly with Kohana's ORM
$user = ORM::factory('user', $id);

Kohana can also do the top syntax (i.e., new Users_Model($id)) — but the seemingly preferred way is the lower.

Inserting/Updating a row is done via Model::save(), as is the case with almost all Active Record implementations.

Fields can be accessed using properties: $object->field,  or using getters and setters (blah): $object->getField();

Deleting is somewhat inconsistent.

//Delete user
User::delete($user);

//I would think this is more readable/consistent:
$user->delete();

But it’s not a big deal.

Other Things

There are two cool functions that I think every ORM should consider — import and export. Import will set field values based on an array, and export will serialize the row to an array (note: Kohana has this too). An extension of import() is provided as importFromPost — which sure beats setting field names manually from forms.

One last thing…

RedBean has a “Locking” feature that I’ve been trying to figure out. No, I know what Locking is, but I’m just wondering why it would be necessary? A modern, transactional database is not going to have problems with multiple users. Even SQLite supports basic mutli-user functionality. Maybe someone can explain this to me?

Well, that’s about all I can think to say. There may never be a perfect ORM for PHP, but some are better than others. Based on API and not on performance, I can say in my biased, unfair, ridiculous opinion, that RedBean is better than most.

If I have a project that I might be able to use RedBean with, I’ll definitely give it a shot. So there may be a second review in the future. Who knows?

To Gabor de Mooij, RedBean’s author:
Good luck with the project — hopefully RedBean destroys Propel. :)

In fact, it’s this one: http://www.geico.com/public/scripts/jquery/jquery.timer.js

Yeah! That’s me! And all this time I thought the script was pretty useless, but I guess a certain Gecko found a good place for it.

Honestly, I may have used that script once or twice, but that’s it. I don’t use timers a lot. But, if there’s enough interest, I’m open to adding bug fixes and features and such. I’ll also accept patches, etc.

Thanks to Alex for pointing that out. Pretty neet! ;)

Well, I kinda lied. It sucks…

It works fine if you’re dealing with dates in the here and now, but say you’re accepting user input for a date-of-birth field. strtotime() returns a signed integer, and if you look carefully, you’ll see that strtotime() will fail on early PHP5 revisions on Windows for dates before 1/1/1970.

So that’s problem 1. Problem 2 is going past 1/19/2038.  You would think that it wouldn’t be a problem on 64-bit systems, but I have no idea if that’s the case or not — I can’t seem to find any documentation on 64-bit versions of PHP using 64-bit timestamps.

Luckily, there’s a new DateTime class for PHP 5.2 and up. The constructor of the DateTime class accepts anything that strtotime() can, and DateTime::format() accepts a date() style format. So that’s easy enough. Hopefully, you deduce by my lack of griping, that Pre-epoch and post-2038 dates will work. I didn’t test to see how far back or forward, it can go. Far enough for my uses.

Why the new class isn’t listed on the date() and strtotime() pages — I have no idea. Hopefully now I won’t have another Epoch Fail.