tail -f findings.out

I find, not too infrequently, that I need to perform operations across several directories, or across several servers, at once. To best facilitate this, I usually open 3-5 terminals, and then move them around so that I can see them all at once. Why not automate this, I thought. Instead of hard coding sizes and positions for the terminals and assigning the sequence to a keypress, I thought I would do something more useful and write a Python script to find out this information based on the current display, and launch the terminals appropriately.

[EDIT, 2009-10-07] I recommend skipping this sort of approach and instead use a terminal emulator like Terminator that lets you open as many terminals as you want within one window at the same time. It’s a lot cleaner.

This has been completed, but the script still has a lot to be desired. You still have to set as a variable whether dual monitors are being used (could not find a way to determine this programatically yet), and it only works with single monitor or TwinView in Linux. You can also set what sort of terminal you want to use (gnome-terminal is default), what side you want them to open on for dual monitor setups (right is default), and what directory they should be in (user’s home dir is default). It gets your screen’s dimensions, calculates where the terminals should be and how they should be separated, and launches them.

I then added this to my .fluxbox/keys file:

So now I just press <Alt> + q, and four gnome-terminals launch perfectly spaced on my right monitor!

The code is quite ugly right now, but as it stands:

Master Ian (whom I would link to, if his blog were not continually DOWN), showed me a very simple yet useful command:

.

You give it the path to a file, it gives you the filename and extension. If you also give it the extension, it just gives you the filename. Thus:

sam@ZenSam:~$ basename unique-ip-list.csvunique-ip-list.csvsam@ZenSam:~$ basename unique-ip-list.csv .csvunique-ip-list

This could be quite useful in shell scripting, to grab a list of desired for various purposes. One implementation I thought of right off: If you wanted to throw an error message for the usage of a command in a script, instead of having it display “See /usr/bin/zip for details” or something similar, you could run it through basename, and get a better looking name.

I have spent some time thinking about what to do with my Domainspotter program. I believe that it can be further developed into something quite interesting, and somewhat useful. As a result, I decided to create a site to house its documentation, feature roadmap, source code, and other related items. This can now be viewed at domainspotter.org.

There is not a whole lot there right now, but I hope to keep adding to it. This will minimally include the feature map as I have it planned now, and access to the SVN code base. I am still working on the content for the site, but it will mainly link to my roadmap in Trac for the project, as well as the code browser. It will be the home for all things related to Domainspotter.

In the longer term, I will also create a frontend to view results from domainspotter, updated weekly. In addition, I will also be adding various statistics and graphing displays to show the results in different ways.

So check there soon, hopefully things of interest will be found!

In an effort to learn Python better, I will be writing a number of relatively small scripts to perform set tasks. I find that giving myself some goal and having to code towards it is more effective in learning syntax than going through coding exercises in a tutorial.

For the first such project, I wanted to determine how many words in a current English dictionary are not taken as domain names. This could have become a rather complicated exercise, but I limited it to the following:

• single English words, using the AGID-4 dictionary
• .com and .net TLDs only

There are many domains made with acronyms, combinations of names, made up words, and many other forms. But I was curious just how many of the more common terms in the language were still available.

The code as it stands is at the end of this post.

Before you look at how I did it, however (which is far from optimal anyway), I would encourage you to try out the exercise on your own, given the basic requirements stated above. I found that this program was perfect for learning more Python, since it required learning how to perform a number of disparate activities, such as reading and parsing files, using external modules, performing web lookups, and displaying desired data. These are, of course, some of the most general tasks most programs of use must perform.

A few notes. The whois lookups was the only section of the program that I assumed would require something outside of core Python. I came across rwhois.py, a module for performing whois lookups. I still have the code to implement this listed in the program as it stands for illustration, but found it did not meet my needs. It could not locate many domains I confirmed were taken using the simple command

. So, I decided to fall back on something I knew: the command

! This seemed to work fine for my needs.

Additionally, while I know the code could stand for optimization all over the place, I was curious how long the web lookups would take. To do this, I modified the lookup function to the following:

import timeself.totalchecked = 0self.availdomains = []times = []for potdomain in self.potdomains:start = time.time()runthis = "whois %s" % potdomaincheck = commands.getoutput(runthis).split('\n')if check[8] == 'No match for "%s".' % potdomain.upper():self.availdomains.append(potdomain)else:passself.totalchecked+=1end = time.time()timeforquery = end - starttimes.append(timeforquery)import pdbpdb.set_trace()

I ran this against a much shortened dictionary file (the top 10 lines of the real one). Once the debugger launched, I did the following:

(Pdb) print sum(times) / len(times)0.884103870392

This gave me the average time per web query. Now, note that I found this out after I started the final run for the first time… And since the actual dictionary file has 112,505 entries, I estimated the entire series should be done in around 27.6 hours. So I started my final run at around 7:32pm Saturday evening, and expected it to run until around 11pm Sunday evening. But instead I got an error during the night:

  File "domainspotter.py", line 125, in lookupif check[8] == 'No match for "%s".' % potdomain.upper():IndexError: list index out of range

This, I am pretty certain, was due to my internet connection dropping for a moment. This not being an unlikely event, I should add in handling for such situations, and perhaps write to the results output in chunks, instead of all at once! So the complete list will have to wait. But, running against the beginning of the list showed there are some exciting domains still out there:

Domains that seem to be available:

AMusD.netAbelmoschus.netAberdonian.netAbkhas.comAbkhas.netAbkhasian.comAbkhasian.netAbkhazian.netAbnaki.netAbramis.net

One last note you might already be asking yourself about: The AGID-4 dictionary contains proper and common nouns. I left the proper ones capitalized. It could be argued I should

them first, but I don’t think one is able to buy a domain name with merely differing capitalization.

[EDIT] The code base of domainspotter has changed around a lot since I posted this. Instead of having a big block of code here that is static, check out the latest form on my Trac site.

While I won’t go into all the details and best practices of Python debugging here (because I do not know them), Chris Blunck shared a very easy and useful method with me that can be used in many situations. First, the code:

import pdbpdb.set_trace()

That’s it. All you have to do is put those two lines just before a line where, for example, an error is being thrown, or where you are not sure what the state of things is. pdb is the “Python Debugger”, appropriately enough. There are several ways to invoke and use this module, other than what I listed above. The format I gave is most useful for setting a breakpoint. When your code runs and reaches where you have placed those two lines, you enter the debugging command line. You can check variable values as they exist at the breakpoint, execute code, whatever you want to try to understand the error you are examining.

An example:

>>> list = [1,2,3,4,5,6]>>> import pdb>>> pdb.set_trace()--Return--> (1)()->None(Pdb) print list[2]3

As can be easily seen in this very simple case, once the breakpoint is reached and one enters the debugging command line, it is trivially easy to, for example, verify variable values as compared to what you were expecting them to be. If desired, you can also continue executing the code after the breakpoint by pressing “c” on the pdb command line.

For details on other ways to use pdb, see the official doc. This separate guide has very useful information for Python debugging, specifically with pdb, for beginners.