Most of the programming I do is packet replication and automation. My workflow is to first analyze the handshakes and protocols in FireFox with the Tamper Data add-on. I use WireShark to nail down the interaction details. Finally, I recreate everything using the Twisted library in Python.

Here’s a boilerplate to quickly recreate packets using Twisted. It requests and saves a copy of Google.com. Simply replace with your favorite URL, or un-comment the two lines and comment the other getPage to send a POST request.

from twisted.web import client
from twisted.internet import reactor
 
agent = "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.12) Gecko/2009070611 Firefox/3.0.12 (.NET CLR 3.5.30729)"
headers = {'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'Accept-Language':'en-us,en;q=0.5', 'Accept-Charset':'ISO-8859-1,utf-8;q=0.7'}
headers_post = headers.copy()
headers_post['Content-Type'] = 'application/x-www-form-urlencoded'
 
def success(data):
    print "Success!"
    f = open('save.html','w')
    f.write(data)
    f.close()
    reactor.stop()
 
def failure(fail):
    print fail.getErrorMessage()
    reactor.stop()
 
def request():
    url = "http://google.com/"
    d = client.getPage(url, headers=headers, agent=agent)
    #post_data = ""
    #d = client.getPage(url, postdata=post_data, method="POST", headers=headers_post, agent=agent)
    d.addCallback(success).addErrback(failure)
 
request()
reactor.run()

To make it continually request the same page, change the success method to this:

def success(data):
    print "Success!"
    f = open('save.html','w')
    f.write(data)
    f.close()
    reactor.callLater(0.5, request)

I got an email today:

Hello,

Unfortunately but your data is damaged un-fixable

Thank you,
www.000webhost.com

Data damaged un-fixable! But I’m smart; I make backups. I found the file in my backup folder, extracted the SQL, then opened it to verify everything was fine. I only make backups every few months, but I had made one this month on the July 7th. When I do backups I backup all my websites and carefully label them and file them away on one of my hard drives, then I burn a copy of my entire backup folder to a DVD, just in case my backup hard drive dies. Sadly, this past month when I was making backups, I mislabeled one. Wouldn’t you know it had to be the one with data damaged un-fixable. The SQL file belonged to a different site, and my blog’s data really is gone.

Google cache has some of the old posts and I’ll work on moving the ones that matter which are really only two posts. This backup is from November 2008. Not much lost since I haven’t been active lately, but data loss is never fun.

When downloading a file from Perl, there’s many ways to go about it. You can use LWP, libcurl, wget, curl the program, etc. My shell hoster doesn’t include many perl bindings so I can’t use LWP or libcurl. I use `curl -s $url` to download urls. The only problem with using any of the above mentioned methods is that they all block Irssi until the page is downloaded. That means the client freezes and doesn’t respond to pings during this time. To remedy this, fork can be used. But fork is rather screwy in Irssi; if you use it wrong, it will create 2 instances of Irssi.

The fix? Use someone else’s code that works. Here’s mine, released under the GNU GPL. It can easily be adapted, and should be, to use LWP or libcurl if your client supports those.

use strict;
use Irssi;
use POSIX;
 
sub async_curl ($$$) {
        my ($geturl, $callback, $argref) = @_;
        my ($reader, $writer);
        pipe($reader, $writer);
        my $pid = fork();
        if ($pid > 0) {
                close($writer);
                Irssi::pidwait_add($pid);
                my $pipetag;
                my @pargs = ($reader, \$pipetag, $callback, $argref);
                $pipetag = Irssi::input_add(fileno($reader), INPUT_READ, \&pipe_input, \@pargs);
        } else {
          my $content;
          eval {
                my $data = `curl -s "$geturl"`;
                print($writer $data);
                close($writer);
          };
          POSIX::_exit(1);
        }
}
 
sub pipe_input ($$$$) {
        my ($reader, $pipetag, $callback, $argref) = @{$_[0]};
        my @lines = <$reader>;
        close($reader);
        Irssi::input_remove($$pipetag);
        my $text = join("", @lines);
        $callback->($text, $argref);
}

Using this code is very simple. Here’s a quick example:

sub callback_method {
        my ($content, $argref) = @_;
        my ($arg1, $arg2) = @{$argref};
        # Here we have the $content from the url
        # and the two args we wanted back
}
 
# Gather up any arguments we want to piggy-back onto the callback
my @args = ('test1', 'test2');
# Call async_curl with a reference to our callback method
# and a reference to our piggy-back args
async_curl("http://www.google.com/", \&callback_method, \@args);

Here’s a snippet for uploading a file via FTP using the Twisted Python library available from TwistedMatrix.

from twisted.protocols.basic import FileSender
from twisted.protocols.ftp import FTPClient
from twisted.internet.protocol import ClientCreator
from twisted.internet import reactor
 
def fileTransferFail(failure):
    failure.printTraceback()
    reactor.stop()
 
def cbStore(consumer, filename):
    fs = FileSender()
    d = fs.beginFileTransfer(open(filename, 'r'), consumer)
    d.addCallback(lambda _: consumer.finish()).addErrback(fileTransferFail)
    return d
 
def connectionMade(ftpClient, filename, uploadpath = None):
    if uploadpath is None:
        uploadpath = filename
    d1, d2 = ftpClient.storeFile(uploadpath)
    d1.addCallback(cbStore, filename).addErrback(fileTransferFail)
    d2.addCallback(lambda _: reactor.stop())
    return d2
 
def sendFile(host, port, username, password, filename, uploadpath):
    creator = ClientCreator(reactor, FTPClient, username, password)
    d = creator.connectTCP(host, port)
    d.addCallback(connectionMade, filename, uploadpath).addErrback(fileTransferFail)
    return d
 
if __name__ == '__main__':
    username = 'GoogleUser'
    password = 'GoogleSuperSecretPassword'
    host = 'google.com'
    port = 21
    filename = 'index.html'
    uploadto = '/public_html/index.html'
    sendFile(host, port, username, password, filename, uploadto)
    reactor.run()

I get so many hits to my site every day from people looking for it, so here it is.

HaloHBF v0.1

I use Google constantly. Not just Google Web Search, but Google Calendar, Google Reader, Google Mail, Google Analytics, Google Docs, Google Scholar, Goog411, Google Maps, YouTube, and I’m even writing this post in Google Chrome. I have my browser set to open three tabs on startup: Gmail, Calendar, and Reader. I check for new emails, update my calendar, and then read the news, all without leaving Google. It scares me how much of my internet life relies on Google. Yet it leaves all competition in the dust, so what choice do I have?

Google in Google Chrome

I spent an hour or so today reading the latest issue of Asimov’s Science Fiction at the library. This publication is filled with short little gems and I highly recommend reading an issue at your library or *gasp* purchasing a subscription. The stories I’ve read are not what I’d consider revolutionary for the genre, but it makes for nice light reading in the evening. I find it hard to finish a full length novel as I am constantly switching tasks as I commute back and forth from college and I just never seem to find time to sit down with a book and read. Reading a short 5-30 page story in Asimov’s Science Fiction lets me get through a whole plot in one sitting instead of my sparse readings of novels that drag on for weeks or months.

Asimovs Science Fiction

I’ve been using the latest trunk builds of Chromium for about a week now. One feature I’ve noticed is the Bookmark Manager, which alone should be enough reason to try out the latest snapshot.

Chromium Bookmark Manager

So what else has changed? Take a look at the issues list and see for yourself. There’s also the code review site where you can take a look at some of the actual chunks of code that changed along with a description of what was changed. And finally, there’s the ViewVC site where you can see the svn check ins and their descriptions.

So what are you waiting for? Dump your Google Chrome beta, grab the latest trunk, and post your thoughts on it below!

Is Google Chrome Dev Channel not bleeding edge enough for you either? Time to start using Chromium snapshots. These builds are released multiple times daily and contain the latest bug fixes and patches the Chromium team has been working on. While technically they can introduce new and worse bugs, using these can at least assuage the old ones.

To get started, head to http://build.chromium.org/buildbot/snapshots/chromium-rel-xp/ and grab the latest build at the bottom. Download and install the mini_installer.exe. Now close all your copies of Chrome/Chromium.
Copy
C:\Documents and Settings\YOUR USERNAME\Local Settings\Application Data\Google\Chrome\User Data
over into
C:\Documents and Settings\YOUR USERNAME\Local Settings\Application Data\Chromium\User Data

Now goto Start > Programs > Chromium and run Chromium. Click the little wrench in the corner and goto Options and ‘Make chromium your default browser’.

Enjoy your bleeding edge copy of Chromium and be sure to report any bugs to the Google Code Chromium project.

Chromium

I have an mp3 player. No video, no wma, no ra, no ogg, just mp3. So I have to convert all my media to mp3 to work on my player. Most media I use can simply be listened to and the video, while nice to have, is not necessary. I have written a batch file that will convert any media supported by ffmpeg to mp3.

This script requires FFMPEG. Go here and download FFmpeg-svn-XXXXXX.7z. In it should be ffmpeg.exe. Make sure it is in the same folder as the batch file you create in the following steps.

Save the following as a batch file, such as ConvertToMp3.bat. Drag and drop media files onto it to convert to mp3.

%~d1
cd %~dp1
ffmpeg -i "%~f1" -vn -acodec libmp3lame -ab 96000 "%~dpn1.mp3"
pause