Site Crawler Chronicles, Part 4: I might be dumb
2010-03-23 3:08 pm ∴ Uncategorized ∴ by matt -

Turns out urljoin() wasn’t behaving badly, I just supplied it a lousy URL. Turns out after running urlopen, the file-like object that is returned has two additional methods, one of them giving the true URL (i.e. after redirects). So far, that’s seems to have fixed the issue.

Hopefully I’ll have a release soon, but I still gotta work out some bugs.