tail -f findings.out

When a server is getting low on disk space, you need to find out what is taking up that space, and fast. I previously would run this command in /:

This would show all the directories in / and their size. Then I would run it on the largest directory found by that command, and so on until I knew what was up. This is slow. And there is a better way! I created a script that looks for directories in your current directory (or in a directory you pass it). If there are directories in the directory, it looks to see how large they are. It then does the same process on the largest directory found, continuing until there are no more child directories. It also accepts an option to prompt you at each step whether or not you want to continue. This makes it much faster to zero in on the culprit.

Example output:

With prompting:

Latest code can be found in my Trac. [EDIT, 2009-02-18: Added rounding of sizes and proper error checking in dir recursion based on suggestion by Dorantor. Thanks!] Current code:

Continue Reading “Tracking down used disk space: Follow largest directories”

On a few occasions, I have needed to debug python code in some production application that I couldn’t easy get installed on my local box, either because it was old, or had a very custom setup, or I just didn’t have time and the problem needed to be fixed. I would copy over the python file I needed to mess with into my home folder, but it would almost definitely need to import modules that wouldn’t be found when I ran the script, as they too were in the original location. It wasn’t always clear exactly where, however.

I could run a recursive find command for the module(s) in a directory above where I grabbed the troubled file, but if there is a really large tree of directories in the application, that might take a while. Here’s an alternative. Open a python prompt in the directory where the problematic file was originally, try to import what it needs to import, and then run this for each imported module:

An example:

Now you know exactly what you need to copy (note: grab the .py, not the .pyc, of course). Or if you don’t want to copy all the files, you can just add this to the file you are editing for debugging:

And then the originals can be imported as desired. Except for really large projects, one or two appended paths will probably contain all you need.

My last post was on how I used Samba to share files from my central media server to other machines on my local network. Seemed great at first, but having used it for a little while, I was rather disappointed. I had Rhythmbox looking at the mounted drive as my library. It started scanning the files once the directory was mounted. I could play them, but after a few minutes, Rhythmbox would freeze scanning the files. In addition, I could not even list the directory that contained the mounted directory. Something was getting frozen. This kept happening more and more, and was quite frustrating.

I then tried of SSHFS for the same purpose, and it has been working much better. If you can SSH to the machine with the files you want to share, then you don’t have to make any changes to that computer. Just a few steps on the machine that will be accessing the remote store, and you are ready to go. From the top:

• Create the dir which to which the remote dir will be mounted (e.g. /home/myuser/Music)
• Make sure you have ssh access on remote machine from the local machine
• On the local machine:

  • 1	sudo apt-get install sshfs

  • 1	sudo modprobe fuse

  • 1	sudo usermod -G fuse -a youruser

  • 1	sudo chgrp fuse /dev/fuse

  • Logout of myuser on the local machine, then back in

  • 1	sshfs 192.168.1.6:/dir/on/remote/server LOCALDIR

will make sure you have that kernel module loaded. You have to be in the fuse group to perform the mount, sudo or not. I am not exactly sure why you have to change the group of the fuse device, but you definitely have to to get it to work.

At this point, you should be able to view all the files in /dir/on/remote/server from LOCALDIR. Performance wise, I had great success. Pointing Rhythmbox to the dir I mounted, it scanned all the files in a timely manner, and I was able to play them as if they were on the local machine without issue.

If you wish to unmount the share, run

. One problem that has been suggested for which I do not have a fix yet is how to handle if the remote machine has a problem, reboots, or the network connection is otherwise dropped. I am not sure how this would be handled, since the mount would no longer work. fuse being a kernel module, this may cause freezing. If I find this addressed somewhere, I will post it.

Additional information and links:

• General HowTo
• Good reference
• Post on changing /dev/fuse owner

Some people have difficulty renaming lots of files at once on the command line. I used to experience this issue a lot, especially since “mv” was the command recommended. While I can appreciate why mv makes sense for renaming, it still was not intuitive for me. The original file is, in a sense, gone. Renaming a file can be seen as making a new file.

However, there is another command, nearly as available, which can rename things quite easily. You will never guess its name. Something simple, and descriptive of the action of renaming files…: rename! It syntax is easy:

rename 'PERL REGEX' FILELIST

The FILELIST can use regex as well. So if I wanted to renamed all of my Mahler “Symphony No. 1″ files (they all started with “_- Mahler”), I would simply run this in the folder the files were in:

rename 's/^\_\-\ Mahler/Mahler/' *.mp3

Voilà! Now all my file names have been fixed in one fell swoop. An explanation of regex is not something I want or can do right now, but internet tutorials and articles on it are legion.