I’ve written about web crawling with Perl, but there are some severe limitations to that. The biggest being that it doesn’t (easily) support AJAX. Enter the WatiN web application testing framework for .Net. Best of all, it’s a *free* framework and supports multiple browsers -currently IE, Firefox. I’ve read rumor somewhere that support for Chrome is coming, but I can neither confirm nor deny this.
Getting started with WatiN is amazingly simple.
//launch a new IE browser
using (IE browser = new IE('http://www.example.com')) {
//now we have access to the browser object
//filling a textbox and clicking a button is as easy as
browser.TextField(Find.ByName("some_textbox")).TypeText('foobar');
browser.Button(Find.ByName("submit_button")).Click();
//we can also access the full html of the page to perform regex matches, scrapes, etc...
string fullPageSource = browser.Html;
}
One issues that I ran into was getting a ThreadStateException when the browser was launched. This is quickly remedied by adding the [STAThread] decorator to your programs Main() entry point. More detail is available here.
Hopefully this helps someone out!
Well, as much as I like messing around in Linux, it’s just that – messing around. From a software development standpoint, Windows still rules. Visual Studio just can’t be beat. But that is a post for another day (soon). For now, I want to talk about how I used C# and VLC to feed my currently playing music to Twitter.
I like VLC – it plays pretty much everything and does some amazing things. It’s much more powerful than it appears to be on the surface. And I’ve posted in the past about grabbing the current track information from VLC using Perl, but honestly, since it’s a simple XML file, nearly any language can do that. Today, that language is C#.
Up until now, if song that I like comes on one of the stations that I stream, I’ve been grabbing a pen and piece of paper to jot it down for later purchase. Bleck! This is time consuming and distracting. I know that I can capture the information that I want from VLC using code, but how to easily flag songs that I like? Well, while hanging out on Twitter, it hit me – Twitter can be used to do just that!
Here’s the run down – Since I don’t want all of these songs to appear in my regular stream, I created a new Twitter account called gkurts_music. When a song comes on that I like and my application has sent the details to Twitter, I can simply hop over to that page (really, I just keep a tab open to it), and mark it as a favorite. Now I have an easy to view list of my favorite tracks. Once I purchase them, I can simply un-favorite that tweet. Simple!
Now the fun part – getting the track info to Twitter. For this, I used the Twitterizer .Net library. It’s robust and comes with some great examples.
First, we visit dev.twitter.com (as my normal user) and added a new application to my account. This is easy and is pretty self explanatory, so I won’t go into further detail. Now on to the application.
First, we need to Authorize our application to post to our new accounts stream. This is very easily done and the following code is taken almost line-for-line from the Twitterizer example. Why re-invent the wheel?
public partial class frmAuthenticate : Form
{
private string requestToken;
public frmAuthenticate()
{
InitializeComponent();
}
/// <summary>
/// Launch the form and set things up.
/// </summary>
/// <param name="sender"></param>
/// <param name="e"></param>
private void frmAuthenticate_Load(object sender, EventArgs e)
{
OAuthTokenResponse request = OAuthUtility.GetRequestToken(
ConfigurationManager.AppSettings[Constants.AuthConsumerKeyName],
ConfigurationManager.AppSettings[Constants.AuthConsumerSecretName]);
requestToken = request.Token;
//set up our message and link to the Twitter oAuth page.
llFetchAuthCode.Text = Constants.MsgFirstUseMessage;
llFetchAuthCode.LinkArea = new LinkArea(llFetchAuthCode.Text.IndexOf(Constants.MsgFirstUseLinkText), Constants.MsgFirstUseLinkText.Length);
llFetchAuthCode.Tag = OAuthUtility.BuildAuthorizationUri(request.Token).AbsoluteUri;
}
/// <summary>
/// launch the browser to the Twitter oAuth page.
/// </summary>
/// <param name="sender"></param>
/// <param name="e"></param>
private void linkLabel1_LinkClicked(object sender, LinkLabelLinkClickedEventArgs e)
{
Process.Start((string)llFetchAuthCode.Tag);
}
/// <summary>
/// Validate our authentication code and set the appropriate properties, if valid.
/// </summary>
/// <param name="sender"></param>
/// <param name="e"></param>
private void btnDoAuth_Click(object sender, EventArgs e)
{
try
{
//get our access tokens.
OAuthTokenResponse accessTokens = OAuthUtility.GetAccessToken(
ConfigurationManager.AppSettings[Constants.AuthConsumerKeyName],
ConfigurationManager.AppSettings[Constants.AuthConsumerSecretName],
requestToken,
txtAuthCode.Text);
//write our settings to our application exe.config file.
Configuration appConfig = ConfigurationManager.OpenExeConfiguration(ConfigurationUserLevel.None);
appConfig.AppSettings.Settings.Add(Constants.AuthAccessTokenName, accessTokens.Token);
appConfig.AppSettings.Settings.Add(Constants.AuthAccessTokenSecret, accessTokens.TokenSecret);
appConfig.AppSettings.Settings.Add(Constants.AuthUserId, accessTokens.UserId.ToString(CultureInfo.CurrentCulture));
appConfig.AppSettings.Settings.Add(Constants.AuthUserScreenName, accessTokens.ScreenName);
appConfig.Save();
ConfigurationManager.RefreshSection("appSettings"); //MAGIC
MessageBox.Show(string.Format(Constants.MsgThanksAuthentication, accessTokens.ScreenName));
Close();
}
catch (TwitterizerException ex)
{
MessageBox.Show(
string.Format(Constants.MsgFailedAuthentication, ex.ErrorDetails.ErrorMessage));
}
}
private void txtAuthCode_TextChanged(object sender, EventArgs e)
{
btnDoAuth.Enabled = true;
}
}
Now, all this does is launches a browser to Twitter asking permission for the application to post to our stream. Once the user enters the code, the account is verified and credentials are pulled in from Twitter.
Next, in the main form of the application, we build our OAuthTokens object and set its properties accordingly:
private void AuthorizeUser()
{
// try it three times.
for (int i = 0; i < 3; i++)
{
if (IsUserAuthenticated())
{
break;
}
new frmAuthenticate().ShowDialog(this); //show the form
}
//if we still aren't authenticated, drop out.
if (!IsUserAuthenticated())
{
MessageBox.Show(this, Constants.MsgFailedLogin, Constants.MsgMessageBoxTitle, MessageBoxButtons.OK);
Application.Exit();
}
//get our auth details from the app.config file and create our oAuthTokens object.
_oAuthTokens = new OAuthTokens
{
AccessToken = ConfigurationManager.AppSettings[Constants.AuthAccessTokenName],
AccessTokenSecret = ConfigurationManager.AppSettings[Constants.AuthAccessTokenSecret],
ConsumerKey = ConfigurationManager.AppSettings[Constants.AuthConsumerKeyName],
ConsumerSecret = ConfigurationManager.AppSettings[Constants.AuthConsumerSecretName]
};
decimal userId = decimal.Parse(ConfigurationManager.AppSettings[Constants.AuthUserId]);
try
{
//update our user object. Why did they name the method 'Show'? This is misleading.
_user = TwitterUser.Show(_oAuthTokens, userId);
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
}
Just call this method in the form_load event. Simple.
Now that the user is authenticated and authorized, we need to get the track info:
private string[] GetCurrentTrack()
{
string artist = "";
string title = "";
string nowPlaying = "";
var returnValue = new string[2];
// make a webrequest to the status.xml service for vlc
var request = (HttpWebRequest)WebRequest.Create(txtServiceUrl.Text);
var response = (HttpWebResponse)request.GetResponse();
Stream responseStream = response.GetResponseStream();
var reader = new XmlTextReader(responseStream);
while (reader.Read())
{
switch (reader.NodeType)
{
// We read each element and see if the name matches what we are looking for.
// If it matches, then we advance the reader so we can get the value of the element.
case XmlNodeType.Element:
if (reader.Name == "title")
{
reader.Read();
title = reader.Value;
}
if (reader.Name == "artist")
{
reader.Read();
artist = reader.Value;
}
if (reader.Name == "now_playing")
{
reader.Read();
nowPlaying = reader.Value;
}
break;
}
}
if (string.IsNullOrEmpty(nowPlaying) && string.IsNullOrEmpty(artist) && string.IsNullOrEmpty(title))
{
//something didn't work... let's just bail out instead of doing something nice and graceful.
return returnValue;
}
// if the now_playing field is populated, we are most likely listening to a stream which populates
// the now_playing field with artist and title info instead of the proper track info fields. Why?
if (!(string.IsNullOrEmpty(nowPlaying)))
{
string[] currentTrack = Regex.Split(nowPlaying, " - ");
returnValue[0] = currentTrack[0];
returnValue[1] = currentTrack[1];
}
else
{
returnValue[0] = artist;
returnValue[1] = title;
}
return returnValue;
}
Once we have the track info, we can send it to our Twitter stream. I decided to do this inside a Timer objects tick event. This way, I can have it poll VLC every 10 seconds (or however often I want to check it).
private void TimerTick(object sender, EventArgs e)
{
_counter--;
UpdateCountdown(_counter);
if (_counter == 0)
{
//get the currently playing track
string[] current = GetCurrentTrack();
string currentArtist = current[0];
string currentTitle = current[1];
//see what the last track played was
string[] last = GetLastPlayedTrackForStartup();
string lastArtist = last[0];
string lastTitle = last[1];
//if the two tracks differ, then get do an update.
if (currentArtist != lastArtist && currentTitle != lastTitle)
{
//except if the track matches an entry in our ignore list - for ads.
if (IsTrackInIgnoreList(current))
{
AppendToUpdateList("Ignoring Ad " + currentArtist + " - " + currentTitle);
SaveLastPlayedTrackToTextFile(currentArtist, currentTitle);
}
else
{
AppendToUpdateList(string.Format("Sending {0} by {1}...", currentTitle, currentArtist));
string status = string.Format("Now listenting to \"{0}\" by \"{1}\"", currentTitle,
currentArtist);
TwitterStatus.Update(_oAuthTokens, status);
SaveLastPlayedTrackToTextFile(currentArtist, currentTitle);
}
}
//reset the counter and update our API calls
_counter = Constants.MagicCountdownTime;
UpdateTwitterAPIText(GetApiInfo());
}
}
This probably looks pretty convoluted, but once you step through it, it really is pretty simple. All in all, I have 3 forms (I created one for setting up and ignore list for ads), a Utils class, a Constants class, and a few other misc. things scattered about.
You can grab the entire project on its page here. It still needs some finishing but I’m working on that and will be updating it regularly. Once I get some of the final bugs worked out, I hope to add support for some other media players in the future. Your comments and suggestions are always welcome!
GM-Notify integrates very well into the new Notification Applet and uses notify-osd for nice messages..
http://bleedingpaper.com/gm-notify/
Apr 10
20
Love the new button layout in Ubuntu 10.04? Yeah, me neither. Thankfully, it’s absurdly easy to fix it. You can run this from a terminal window or from the Run dialog (Alt+F2).
gconftool-2 --type string --set /apps/metacity/general/button_layout "menu:minimize,maximize,close"
Bada-bing!
My bank, while I’m very pleased with them overall, has a not-so-hot online banking system. My biggest (and simplest) gripe is the inability to get a daily email with the balance of my checking account. I’ve tried multiple ways with limited success, until I did it in Perl.
First, I outlined the steps required to log in, then used that as comments in my script. Next, I used WWW::Mechanize to process the pages and simple regexes to grab the data that I’m looking for.
Here is my Perl Module (Banking.pm) to do the entire process:
package Banking;
use Moose;
has 'Url' => (is=>'rw', isa=>'Str', required=>1);
has 'UserLogin' => (is=>'rw', isa=>'Str', required=>1);
has 'Password' => (is=>'rw', isa=>'Str', required=>1);
sub GetChecking {
my $self = shift;
use WWW::Mechanize;
# set some variables
my $url = $self->Url;
my $user = $self->UserLogin;
my $pass = $self->Password;
# security question/answer can be one of three things
# These should really be moved to a property...
# Use a regex below to grab the part of the string to see which question that we are being prompted with.
# These are not the real q/a's that I have, by the way.
my $challengeFound = "Please answer the challenge question to continue logging in";
my %challenge;
$challenge{'color'} = "Answer to question one";
$challenge{'vehicle'} = "Answer to question two";
$challenge{'pet'} = "Answer to question three";
# time to do the work...
print "Initializing Mechanize...\n";
my $mech = WWW::Mechanize->new();
$mech->get($url);
print "Got $url\n";
$mech->form_number(1);
$mech->field("j_username", $user);
$mech->click();
my $response = $mech->content();
print "Clicked \"login\"\n";
## We are being challenged for our security answer.
## Use a simple regex to see what question they are asking
## and send the appropriate response.
if ($response =~ m/$challengeFound/) {
while (my ($question, $answer) = each %challenge) {
if ($response =~ m/$question/) {
$mech->form_number(1);
$mech->field("answer", $answer);
$mech->click();
print "Answered challenge question: $question\n";
last;
}
}
}
$mech->form_number(1);
$mech->field("password", $pass);
$mech->click();
print "Entered password\n";
##this is supposed to remove all of the html tags, but it has
##a tendency to act up and do stupid stuff.
$response = $mech->content(format=>'text');
##strip any remaining html
$response =~ s/<[^>]+>//g;
##replace multiple spaces with one
$response =~ s/\s+/ /g;
##remove leading whitespace
$response =~ s/^ //;
##remove trailing whitespace
$response =~ s/ $//;
##remove html entity quotes
$response =~ s/&amp;amp;amp;#34;/"/g;;
## grab our checking account line
my $checkString = "";
if ($response =~ m/Protection(.*?)Checking/) {
$checkString = $1;
print "Got \$checkString $1\n";
}
my @checkBal = split(' ', $checkString);
## log off
$mech->follow_link(text => 'Exit');
#return balance, available, and ytd-interest.
return ($checkBal[1], $checkBal[2], $checkBal[0]);
}
1;
Now all I have to do is call it from a script (daily.pm) and throw that into my crontab to fire daily at, say 7:00AM.
#!/usr/bin/perl
use lib '/home/gkurts/scripts';
use Mail::Sendmail;
use Banking;
my $fortune = `fortune`;
my $msg = "";
## get our checking balance from the online banking website
my $banking = new Banking(
Url => "https://www.mybankingurl.com/login",
UserLogin => "mylogin",
Password => "mypassword",
);
my ($balance, $available, $ytdinterest) = $banking->GetChecking();
$msg .= "Our Bank Balance:\n";
$msg .= "\tBalance: $balance\n";
$msg .= "\tAvailable: $available\n";
$msg .= "\tYTD Interest: $ytdinterest\n";
## add a fortune for cuteness.
$msg .= "\n\n$fortune";
## send the email using Mail::Sendmail
$mail{body} = $msg;
$mail{subject} = "Daily Summary";
$mail{from} = "my\@email.com";
$mail{to} = "my\@email.com";
sendmail(%mail);
So I’ve already done a post on getting track information from Rythmbox, but VLC felt cold and left out, so here it is. The process is pretty much the same, just tweaked a bit for VLC.
First, go into VLC, Tools -> Preferences, click the “All” radio button (near the bottom), Interface, click “Main interfaces” and check “HTTP…”. If you’re feeling adventurous, choose “HTTP” from the list and you can specify port (default is 8080), address to listen on, etc.. I’m not doing all of that.
Now, restart VLC and crank up a stream or MP3. You should be able to navigate to http://localhost:8080/requests/status.xml and see an XML file representing the current track information.
If you are listening to an MP3, the “Artist” and “Title” fields should be populated. If you are streaming music, the artist and title will appear in the now_playing field as “Artist – Title”. I have no clue why this is. It just is.
Simple enough.. Here is my Perl script to grab the info, thrash it about, and do what we want with it. Simple stick it in cron to run every minute (or whatever) – */1 * * * * /home/myuser/scripts/tuneUpdater.pl
#!/usr/bin/perl
use XML::Simple;
use LWP::Simple;
use DBI;
my $date = `date +"%D %r"`;
chomp $date;
## URL to vlc status.xml file
my $url = "http://localhost:8080/requests/status.xml";
my $data = get($url);
## exit if VLC isn't running
if ($data eq "") {
exit(0);
}
## get the xml and read in the appropriate values
my $xs = XML::Simple->new(SuppressEmpty => 1);
my $root = $xs->XMLin($data);
my $information = $root->{'information'};
my $meta = $information->{'meta-information'};
## this is what we are after
my $title = $meta{'title'};
my $artist = $meta{'artist'};
## if the title or artist is blank, we must be streaming, so grab the
## now_playing field.
if (($title eq "") || ($artist eq "")) {
my $nowplaying = $meta->{'now_playing'};
($artist, $title) = split(/\ -\ /, $nowplaying);
}
## strip out Amped FM "Enhanced MP3" stuff.
$title =~ s/\ Enhanced\ MP3//;
## if we got this far safely, connect to the database.
$dbh = DBI->connect('DBI:mysql:mydb;host=somewebhost', 'luser', 'notreally')
|| die "Could not connect to database: $DBI::errstr";
my ($last_artist, $last_title) = &getLastSong();
## update the db if the artist or title differ from the last one updated.
if (($last_artist ne $artist) || ($last_title ne $title)) {
&updateLastSong($artist, $title);
}
$dbh->disconnect();
## update the database to add the current artist and title
sub updateLastSong {
my ($artist, $title) = @_;
$sql_update = "INSERT INTO music_history (id, title, artiist, played) VALUES (null, '$title', '$artist', '$date')";
$out_update = $dbh->prepare($sql_update);
$out_update->execute;
$out_update->finish;
}
## get the last artist and title played.
sub getLastSong {
$sql_last = "SELECT artiist, title FROM music_history order by id desc limit 1";
$out_last = $dbh->prepare($sql_last);
$out_last->execute;
my ($last_artist, $last_title) = $out_last->fetchrow_array;
$out_last->finish;
return ($last_artist, $last_title);
}
Feel free to contact me with any questions or comments!
For no real reason other than saying that I did it, I wanted to have the last 10 songs that I’ve played on my PC listed in the sidebar of my blog. Simple enough..
Here’s what I have: I’m a fan of Last.FM, plus I have about 25Gigs of mp3s that I like to randomize. I also like simplicity, so I use RhythmBox on my Ubuntu desktop. It has a built-in interface to Last.FM (among others) and handles my mp3 library with ease. It also is easily extensible.
At first, I was going to write my own plugin to get the song info, then I found the NowPlaying XML plugin. Perfect! Simply unzip it to your ~/.gnome2/rhythmbox/plugins directory and you’re set (you may have to create the directory first, as did I). Fire up RhythmBox, click “Edit”, “Plugins” and make sure that your NowPlaying XML plugin is checked. Play a song and it should create a file at /tmp/nowplaying.xml.
The /tmp/nowplaying.xml file looks something like this:
<?xml version="1.0" ?><nowplaying><song><album>Saturate</album><track-number>0</track-number><title>Polyamorous</title><artist>Breaking Benjamin</artist><genre></genre><duration>177</duration><bitrate>0</bitrate></song></nowplaying>
Yep, all one line, but that’s okay. It has the data that we want…
Now, to get it into WordPress. First, I created a table in my blog database:
create table music_history (id int primary key not null auto_increment, title nvarchar(30), artiist nvarchar(30), played nvarchar(30));
Yes, I mis-spelled artist, and yes, I probably should have used a datetime data type for played, but work with me here…
Next, I created a Perl script on my workstation called updatePlayHistory.pl that looks something like this:
#!/usr/bin/perl
use Date::Format;
use XML::Simple;
$now = time2str("%D %r", time);
$file = '/tmp/nowplaying.xml';
## make sure that the file exists, if not just exit.
unless (-e $file) { exit(0); }
$xml = new XML::Simple;
## read the xml file in
$data = $xml->XMLin("/tmp/nowplaying.xml");
## get our song object
$song = $data->{song};
## get the song details
$title = $song->{title};
$artist = $song->{artist};
## set up our connection
use DBI;
$dbh = DBI->connect('DBI:mysql:blogdatabase;host=myhost', 'dbuser', 'dbpass')
|| die "Could not connect to the database: $DBI::errstr";
## get the last song inserted and see if it is the same as the current one
$sql_last = "SELECT id, artiist, title FROM music_history order by id desc limit 1";
$out_last = $dbh->prepare($sql_last);
$out_last->execute;
($id, $lastArtist, $lastTitle) = $out_last->fetchrow_array;
$out_last->finish;
## if its the same one, exit out
if (($lastArtist eq $artist) && ($lastTitle eq $title)) {
$dbh->disconnect();
exit(0);
}
## insert the song
$sql_insert = "INSERT INTO music_history (title, artiist, played) VALUES ('$title', '$artist', '$now')";
$out_insert = $dbh->prepare($sql_insert);
$out_insert->execute;
$out_insert->finish;
$dbh->disconnect();
The comments should be pretty self explanatory. Tweak as needed. Now, ‘chmod +x’ your script edit your crontab (‘crontab -e’) to run it every minute:
* * * * * * /home/gkurts/scripts/updatePlayHistory.pl
Last, we need to tweak WordPress to display the last 10 played songs. For this, I just copied some of the existing code for one of the widgets and modified it to select the last 10 played and loop through them..
<?php
$songs = $wpdb->get_results("SELECT id, title, artiist, played from music_history order by id desc limit 10");
?>
<div class="widget">
<h3>Song History <span style="font-size: 10pt">(Last 10 played)</span></h3>
<?php if($songs) : foreach($songs as $song) : ?>
<ul>
<li><b><?php echo $song->title; ?></b> by <b><?php echo $song->artiist; ?></b><p style="font-size: 7pt; margin-top: 1px; padding-left: 5px;"><?php echo $song->played; ?></p></li>
</ul>
<div class="fixed"></div>
<?php endforeach; endif; ?>
</div>
That should be it. It’s really pretty simple. If you use this or have any questions, let me know!
In order to get Adobe AIR running on a 64-bit version of Ubuntu (may work with other distros – YMMV. I’ve tested this on 9.10 and 10.04), perform the following steps:
Jan 10
1
Why can’t things just be simple? Really, it’s only a few steps… Surely the folks at VMWare could fix this. Ugh!
When cloning a Linux virtual machine (Ubuntu Server, in my case – YMMV based on distro), the clone’s networking fails to start up. Doing an ifconfig only shows the loopback adapter and no amount of rebooting, adding / removing the adapter in the VM’s settings, or cussing will make it work.
Thanks to this post I found by Jamis Buck (thanks!), it’s easily fixed (okay, his post was for VMWare Fusion, but it works equally well in VMWare Workstation):
Again, why can’t they fix this?!?! Oh well, enough ranting for the day.