Trapped at Berkeley

I just upgraded one of my home computers to Microsoft Windows Vista Business and discovered that my Canon PowerShot G2 camera was unsupported. I hadn’t bothered to check whether my camera was supported; it seems most digital camera drivers are just glorified USB Mass Storage wrappers and support for those is generally a given.

Unfortunately, Canon’s drivers are a bit more complex, and the lack of Windows Vista compatibility was confirmed on Canon’s technical support page. Their recommended solution was to use a Compact Flash reader to access the memory card directly.

Unmoved and lacking a CF reader, I tried changing various settings and messing with the latest version of Canon’s WIA driver, but I couldn’t get it to work. On a hunch, though, I downloaded Canon’s TWAIN driver (Canon provides the WIA driver for Windows 2000 and XP systems, and the TWAIN driver for older systems) and gave that a go. Lo and behold, I was able to access and download the photos on my camera, albeit through a outdated GUI interface.

So, what’s the lesson to be learned? Don’t trust hardware manufacturers when they make a blanket statement that hardware isn’t ‘compatible’. You’ll often find that there are work-arounds for getting your hardware to work. Of course, also don’t expect them to come to your rescue when something goes wrong.

(10:38:13) Me: i have to learn harvard asm
(10:38:16) Me: *sigh*
(10:38:35) Friend: lol
(10:38:48) Friend: they gonna have like ivy league jokes in the asm
(10:38:58) Friend: woudl you like some tea and crumpets sir
(10:38:59) Friend: har har
(10:39:02) Friend: indeed i do
(10:39:11) Friend: have you seen the latest football game with yale
(10:39:22) Friend: i say… we did very well in it
(10:39:26) Friend: yes… very well indeed

I’ve written a Perl program to spider Facebook. I was looking for a way to quickly generate statistics about the University of California, Berkeley student population, and I figured that since almost everybody had a Facebook account, I could dump all of Facebook’s information into a database and generate reports from that information. Since this program has proven useful, I’ve decided to release it to the general public.

How It Works

If you’re unfamiliar with the term spider, I recommend that you read the Wikipedia page on web spiders for a thorough discussion of how a spider works. In a nutshell, my program goes to a Facebook user’s profile, scans their friends list for other profiles, visits each of their profiles, scanning their friends list, and so on. Along the way, my program also scans a user’s profile for information, parses it, and inserts it into a SQL database.

Features

I’m only aware of one other Facebook spider: a Perl script written by Michael Kelly. However, Michael’s script only collects information about user’s friends. My script captures all the information available in a user’s profile (except for the ‘About Me’ field). Furthermore, my script provides the following enhancements:

• Multi-threaded support. Each user’s profile is processed in its own thread. The total number of threads can be set using a command-line parameter, and the program uses semaphores to enforce the maximum number of threads.
• SQL database storage. My script stores user information in a SQL database ordered by Facebook UID. I’ve used relatively simple queries throughout the script, so any SQL database should be supported (i.e., MySQL and PostgreSQL should work). However, I’ve chosen SQLite3 as the default database. If you wish to use another database type, install the appropriate DBD driver and modify the database handle line to use that driver.
• Easy data processing. Since all data is stored in a SQL database, it should be relatively easy to write programs that query the database for information.
• Sleep between threads. It’s possible to provide a value, in seconds, that my script should wait before spawning a new thread. This should prevent the script from overloading the Facebook servers.

Quick Start

Assuming you have all the necessary Perl modules and sqlite3 installed:

1. Create a SQLite3 database:
   $ sqlite3 database.db ‘CREATE TABLE userdata ( uid integer, name, friends, school, status, sex, concentration, residence, hometown, highschool, screenname, mobile, website, lookingfor, interestedin, relationshipstatus, politicalviews, interests, clubsjobs, favoritemusic, favoritemovies, favoritebooks );’
2. Create a facebook.conf:
   $ cp facebook.conf.sample facebook.conf
   $ vim facebook.conf
3. Start the script:
   $ ./facebook.pl -t 2 -s 10 -f database.db [SOME FACEBOOK UID]

I Want It!

The script has been removed at Facebook’s request.

Notes

I haven’t tested the script lately, but it should still work. If it doesn’t, post a comment, and I’ll release an update.

Since my script parses the HTML returned from Facebook, if Facebook makes any changes to their profile layouts, I’ll have to make major modifications to the code.

Future

I’m in the process of designing an interface to Facebook that resembles Google Maps. Users will be able to interactively visualize their friend network, and clicking a user’s “node” should bring up their Facebook profile in a new window. More details will be forthcoming.

I’ve uploaded pictures taken by CalLUG (UC Berkeley GNU/Linux Users Group) staff during the Richard M. Stallman and Chris DiBona speeches. Please click on either of the following thumbnails to browse the complete album.

Richard M. Stallman Speech

Chris DiBona Speech

Now that I’ve survived my first wave of midterms, I plan on devoting more time to my extracurricular activities.

I’ve noticed that my daily del.icio.us script for Wordpress is broken, so I’ll be updating it to work again, and I’ll be finally addressing the nagging issue of time sync in that script. I also just finished setting up a Xen server for the System Administration for the Web class, so I’ll be sharing my experiences building a Debian Xen server via a comprehensive Xen 2.0.7 HOWTO — one cannot begin to describe the lack of documentation for Xen. Lastly, I’ll be updating some projects that are currently off-limits to the public. I’ll be releasing more information about them when they reach a usable state.

The UC Berkeley Linux Users Group will be hosting a speech by Chris DiBona, Open Source Program Manager for Google and a former editor of Slashdot, on October 17 from 6-8 PM at 306 Soda. As with all CalLUG events, the speech is open to the public and there will be no cost to attend.

If you’re interested in Linux and learning about the latest Google technology, be sure to come!

If you’re a regular reader of this blog, you’ve probably noticed that this site has been experiencing a lot of downtime in the past few days. My server hosting provider has recently installed a new firewall, which has been randomly blocking incoming Internet access. While it appears that they have finally been able to take control of the situation, I am greatly disappointed by their slow response time and their lack of communication with their customers.

I’m currently searching for a better VPS hosting provider. Any suggestions?

T-Mobile offers unlimited Internet access for compatible cell phones through their $5.99 a month T-MobileWeb package. The service allows you to check your email, chat on popular instant messaging networks, and download games and applications.

The email access is very useful. When I’m on campus without my laptop, I use my cell phone to check my personal and academic inboxes for new messages, and I can respond to important messages.

On the other hand, the ability to download games and applications is very disappointing. So far, every decent game I’ve found requires an additional fee of at least $5. You’d think that paying $5.99 a month would entitle you to download a game or two for free. Furthermore, there are no applications currently available to download.

I don’t think I’ll be paying for T-MobileWeb anymore.

UPDATE: I should note that the aforementioned games and applications were found by browsing T-MobileWeb’s main menu. You may use T-MobileWeb to open regular URLs and download J2ME applications, depending upon your phone. A nice selection of J2ME midlets can be found here.

Microsoft’s Xbox 360 will also display games in wide-screen, high-definition format. It will be powered by three 3.2-gigabyte processors, compared with a single 733-megahertz processor in the current Xbox.
Eric A. Taub, The New York Times

Three 3.2-gigabyte processors? The new Xbox must incorporate some new computer methodology.

Why can’t a great newspaper like the New York Times hire people who actually understand technology to report about it?

I’ve released a new version of my Daily del.icio.us Links Script for Wordpress. This new version introduces a code cleanup and the ability to show the tags associated with a link as well as links to your del.icio.us page of those tags. Please refer to the latest “Links” post (see below) to see an example of the new version’s output.

I’m aware of the timing issue associated with the script — del.icio.us uses Zulu time, which makes it hard to determine which links should fall under which day. Once I’m done with finals, I’ll hopefully have time to release a new version with a fix for this bug.

UPDATE: I’ve moved hosting of my Daily del.icio.us Links Script to labs.evilcoder.com. Find the script here.