Odie5533 » Python

Twisted Python RDT/RTSP Library and Stream Downloader

odie5533 — Tue, 01 Jun 2010 03:09:51 +0000

Twisted is a great framework for Python, but they don’t support RDT or RTSP protocol used for streaming Real video. The RDT/RTSP handler in MPlayer is a bit buggy and suffers from disconnect problems when receiving RTSP packets after switching to raw mode for RDT (though perhaps such occurrences are out of specification, they do still occur). Thus, I set off to create my own.

The library includes support for RTSP, RDT, SDPP, ASMRP, and RMFF, and it is all in pure Python and Pythonically coded where possible (see sdpp.py). It requires Python 2.6 and the Twisted Python library. The library has only one known bug, and I can’t for the life of me find it. Perhaps another set of eyes will effect a solution.

To use the library to simply download an RTSP stream: python rdt.py -u -f

If you like this post, please comment. Commenting is fast, anonymous, and greatly appreciated. Download the Twisted Python RDT-RTSP Library here

Automatically Download Songs from Grooveshark

odie5533 — Mon, 31 May 2010 01:11:42 +0000

Because manually downloading songs from Grooveshark is too tedious, here’s a nice process to automatically save all the songs you listen to on Grooveshark. Unlike other methods, this does not use any extra bandwidth so you don’t need to listen to the song and download it, once is enough.

Update After doing some searching, I’ve found a FireFox plugin caled Video DownloadHelper which performs much the same function as the program I’ve written, but is more stable and integrated directly into FireFox. Its only downside is that it doesn’t automatically download the songs as you listen, you need to start downloading the ones you want. But it is more stable than my program and much more user friendly. Be sure to use Mp3tag (listed below) to quickly rename all the files you’ve downloaded.

Download and install the following pre-requisites:

Python 2.6.5 – http://www.python.org/ftp/python/2.6.5/python-2.6.5.msi
dpkt 1.7 – http://dpkt.googlecode.com/files/dpkt-1.7.win32.exe
pypcap 1.1 – pcap-1.1.win32-py2.6.exe

Save this Python program into a folder of its own: groovegrab.py

Run the program using python groovegrab.py

As you listen to songs on Grooveshark, the folder will populate with mp3 files. Each are named Track_X.mp3. But there’s a really neat, easy way to rename them. Download Mp3tag and in the program go to File > Change Directory and choose the directory where the mp3 files are. Select all the mp3 files in the main pane and go to Convert > Tag to Filename.

Enjoy your perfectly named mp3 files from a Grooveshark listening session.

Thanks to dugsong for dpkt and pypcap, and thanks to Andrew Brampton for his tutorial on dpkt/pypcap!

If you liked this post, please comment. Commeting is fast, anonymous, and greatly appreciated.

Wiki Parser v0.1 – Read Wikipedia Offline

odie5533 — Thu, 15 Apr 2010 04:13:00 +0000

Wiki Parser is a Python program which takes a Wikipedia dump and hosts it locally, offline. To save disk space, Wiki Parser keeps the dump compressed and serves articles to a web browser without decompressing the entire file. I originally created it to serve up Wikipedia articles from a compressed version, but seek times on the compressed file were too great and the program really only works on smaller Wikipedias such as the Simple English Wikipedia.

Download the latest dump of Simple English Wikipedia here Download Wiki Parser v0.1 here

Using Twisted Python library to replicate packets

odie5533 — Sat, 01 Aug 2009 13:43:54 +0000

Most of the programming I do is packet replication and automation. My workflow is to first analyze the handshakes and protocols in FireFox with the Tamper Data add-on. I use WireShark to nail down the interaction details. Finally, I recreate everything using the Twisted library in Python.

Here’s a boilerplate to quickly recreate packets using Twisted. It requests and saves a copy of Google.com. Simply replace with your favorite URL, or un-comment the two lines and comment the other getPage to send a POST request.

from twisted.web import client
from twisted.internet import reactor
 
agent = "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.12) Gecko/2009070611 Firefox/3.0.12 (.NET CLR 3.5.30729)"
headers = {'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'Accept-Language':'en-us,en;q=0.5', 'Accept-Charset':'ISO-8859-1,utf-8;q=0.7'}
headers_post = headers.copy()
headers_post['Content-Type'] = 'application/x-www-form-urlencoded'
 
def success(data):
    print "Success!"
    f = open('save.html','w')
    f.write(data)
    f.close()
    reactor.stop()
 
def failure(fail):
    print fail.getErrorMessage()
    reactor.stop()
 
def request():
    url = "http://google.com/"
    d = client.getPage(url, headers=headers, agent=agent)
    #post_data = ""
    #d = client.getPage(url, postdata=post_data, method="POST", headers=headers_post, agent=agent)
    d.addCallback(success).addErrback(failure)
 
request()
reactor.run()

To make it continually request the same page, change the success method to this:

def success(data):
    print "Success!"
    f = open('save.html','w')
    f.write(data)
    f.close()
    reactor.callLater(0.5, request)

Python FTP upload using Twisted

odie5533 — Mon, 22 Dec 2008 03:23:01 +0000

Here’s a snippet for uploading a file via FTP using the Twisted Python library available from TwistedMatrix.

from twisted.protocols.basic import FileSender
from twisted.protocols.ftp import FTPClient
from twisted.internet.protocol import ClientCreator
from twisted.internet import reactor
 
def fileTransferFail(failure):
    failure.printTraceback()
    reactor.stop()
 
def cbStore(consumer, filename):
    fs = FileSender()
    d = fs.beginFileTransfer(open(filename, 'r'), consumer)
    d.addCallback(lambda _: consumer.finish()).addErrback(fileTransferFail)
    return d
 
def connectionMade(ftpClient, filename, uploadpath = None):
    if uploadpath is None:
        uploadpath = filename
    d1, d2 = ftpClient.storeFile(uploadpath)
    d1.addCallback(cbStore, filename).addErrback(fileTransferFail)
    d2.addCallback(lambda _: reactor.stop())
    return d2
 
def sendFile(host, port, username, password, filename, uploadpath):
    creator = ClientCreator(reactor, FTPClient, username, password)
    d = creator.connectTCP(host, port)
    d.addCallback(connectionMade, filename, uploadpath).addErrback(fileTransferFail)
    return d
 
if __name__ == '__main__':
    username = 'GoogleUser'
    password = 'GoogleSuperSecretPassword'
    host = 'google.com'
    port = 21
    filename = 'index.html'
    uploadto = '/public_html/index.html'
    sendFile(host, port, username, password, filename, uploadto)
    reactor.run()