tail -f findings.out » CLI

Bag of useful Bash aliases and functions

Samuel Huckins — Sun, 15 May 2011 01:10:23 +0000

Here are a number of assorted aliases and functions for Bash that I’ve made in the last few months and found to be useful. Enjoy!

Sysadminish

These two print the current value of the swappiness parameter and tail recent syslog messages:

1 2	alias swappy="sudo /sbin/sysctl vm.swappiness" alias sysl="sudo tail /var/log/syslog"

This converts unix timestamps (e.g. 1305401097) that you might find in logfiles to a more easily readable date format:

1
2
3

function fromunixtime {
perl -e "require 'ctime.pl'; print &ctime($1);"
}

The first is simply a quick way to check internet connectivity. Next are some super short ping aliases to avoid Ctrl+c all the time when you just need to verify that something responds. Pass them a URL to get one or three ping attempts:

1
2
3

alias pingoo="ping -c 3 www.google.com"
alias p1="ping -c 1 "
alias p3="ping -c 3 "

Pass a webpage URL and it will download the source to a file named after the domain (if you’re passing a root page, e.g. http://www.cnn.com/) or the page name (e.g. contact.html):

1
2
3

function getsrc {
wget $1 -O `echo "$1" | sed 's/\/$//' | awk -F/ '{print $NF }'`
}

Sometimes you just need a big file. It doesn’t have to be interesting or contain any cosmological insights, but it does need to be a certain size. Pass that size in megabytes and this will give you just that, via a file full of zeroes:

1
2
3

function fakefile {
perl -e "print '0' x 1024 x 1024 x $1" > $1-MB-fake-file.txt
}

This handy one extracts all email addresses in a file and gives you a unique sorted list:

1
2
3
4

alias findallemails="perl -wne'while(/[\w\.]+@[\w\.]+\w+/g){print \"$&\n\"}' "
function findemails {
findallemails $1 | sort | uniq
}

Listing and searching

When you invoke ack-grep it’s case-sensitive by default. It also ignores certain files and directories, based on their nature in some cases and in all cases where it doesn’t know what the filetype is. This incantation removes those limitations and is shorter to boot:

1	alias ack="ack-grep -i -u"

This series allows you to list only directories or only files. You can pass a directory to list the contents or it will use the current directory as default:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

# Feeds file and directory filtered listers:
function lister {
ls -l `if [ "$1" == '' ]; then echo '.'; else echo "$1"; fi`
}
# List only directories:
function lsd {
lister $1 | egrep '^d'
}
# More memorable version:
alias lsdirs=lsd
# List only files:
function lsf {
lister $1 | egrep -v '^d'
}
# More memorable version:
alias lsfiles=lsf

Media and Fun

These two require identify be installed, part of ImageMagick. Install the imagemagick package for your system first. Pass the filename of an image and the first will provide a range of information while the second will just provide the resolution:

1 2	alias imginfo="identify -format '-- %f -- \nType: %m\nSize: %b bytes\nResolution: %wpx x %hpx\nColors: %k'" alias imgres="identify -format '%f: %wpx x %hpx\n'"

Examples:

Sometimes you just need to jack in. Or at least fill your screen with impressive looking output:

1
2

alias rmatrix='echo -ne "\e[31m" ; while true ; do echo -ne "\e[$(($RANDOM % 2 + 1))m" ; tr -c "[:print:]" " " < /dev/urandom | dd count=1 bs=50 2> /dev/null ; done'
alias gmatrix='echo -ne "\e[32m" ; while true ; do echo -ne "\e[$(($RANDOM % 2 + 1))m" ; tr -c "[:print:]" " " < /dev/urandom | dd count=1 bs=50 2> /dev/null ; done'

Counting columns in a CSV from the CLI on Linux

Samuel Huckins — Sun, 09 Jan 2011 15:49:38 +0000

In wrangling some DB exports recently I needed the number of columns in multiple CSVs. It’s easy enough to just open them in OpenOffice and check the final column number, but I wanted a CLI utility. I even entertained some notions of wc having a magical option to provide this, but no luck there.

So here’s a simple function I added to my .bashrc to do the trick:

1
2
3
4
5
6
7
8
9

# Displays number of columns (intended for CSV-type files),
# pass in separator then filename:
function colcnt {
DELIM=$1;
FILE=$2;
echo "# of lines | Columns in $FILE"
echo "(>1 row means there are other separators in the file):"
awk -F"$DELIM" '{print NF}' "${FILE}" | sort | uniq -c
}

It’s in a function because Bash aliases don’t accept parameters. It uses the variable NF from awk which is the number of fields in the current record. The number of fields is sorted then unique entries are counted, providing a way to detect rows with different field separators or rows with missing columns.

Say you have a simple csv file called foo.csv:

1
2
3
4
5
6
7
8

"i am","some cols","yay"
"i am","some cols","yay"
"i am","some cols","yay"
"i am","some cols","yay"
"i am"|"messssssed"|"up"
"i am","some cols","yay"
"i am","some cols","yay"
"i am"|"messssssed"|"up"

After you add the function above to your .bashrc open a new shell (I found that sourcing .bashrc did not always update the function definition) and run this:

1
2
3
4
5

> colcnt , foo.csv
# of lines | Columns in foo.csv
(>1 row means there are other separators in the file):
2 1
6 3

This tells us that six lines had three columns and two lines had one column (as defined by the separator we passed in). If the CSV was consistently formatted (without the pipe-delimited line) you would only see one result row, being the total number of lines followed by the number of columns each line has (8 3).

If there’s a better/built-in way to do this please share! So far it’s been easy to use and helpful.

Handy tips for bash scripting: screen reminder and logging

Samuel Huckins — Sun, 19 Dec 2010 20:37:55 +0000

Often I find myself writing simple wrapper scripts for long-running tasks, like migrating a database or performing some repair or optimization process. I mainly want to ensure proper logging is kept in order to determine how long each step takes and so I have a record (aside from avoiding executing each step manually more than once).

Screen reminder

Since these operations are also usually important I want to make sure I run them in screen so they don’t screech to a halt if my connection happens to timeout or otherwise disconnect. The problem is that I create these scripts pretty organically, primarily aggregating commands I run step by step on test systems, adding some helpful output, etc. During the creation process I am jumping between different terminal views so running in screen isn’t on my mind. To save myself some heartburn from the potential of starting the real thing without being in screen I’ve taken to adding this to the beginning of such scripts:

[EDIT, 2010-12-19] Thanks to visitor Dominic I know a better way to implement this. I advise setting a variable indicating whether you are testing or not (this can be useful for other switches in your scripts as well). If not in testing, then check whether the user is running in screen or not. This allows us to be reminded only when we need it, which I much prefer! Here’s a version of what I just described:

1
2
3
4
5
6
7
8
9
10
11

#!/bin/bash
# Set to 1 if you are ready to run in production
PRODUCTION=1

if [[ $PRODUCTION ]]; then
if [[ `echo $STY` == "" ]]; then
echo "WARNING: You are not running in \
screen. Please start screen and re-run."
exit
fi
fi

If you aren’t in screen then $STY contains nothing. Just what we needed! What follows is the first version, which I want to preserve as an example of the handy select function. [END EDIT]

1
2
3
4
5
6
7
8

#!/bin/bash
echo "Are you running in screen?"
select yn in "Yes" "No"; do
case $yn in
Yes ) break;;
No ) exit;;
esac
done

This will provide a reminder to select no, enter screen, and run the script again:

By the way, in case you haven’t been using select, I recommend it. It provides a numerical selection instead of users entering words or letters. This gets rid of the need for you to parse input, exit based on invalid entries, etc.

Dating your logs and watching them too

In a lot of the same situations as the ones described above, I’ll have some basic log requirements I want to cover and usually want to determine the total time for a script to run. I also want to print out the logging entries at the same time as recording them to a file. All easily accomplished. Here’s a quick example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

#!/bin/bash
LOG="tee -a scriptX.log"
START=`date +'%s'`
function DATE {
date +'%Y-%m-%d';
}
function DT {
date +'%Y-%m-%d %H:%M:%S';
}

echo -e "-- `DATE`: Beginning operations!" | $LOG
sleep 1
echo -e "-- `DT`: Detailed operation log 1" | $LOG
sleep 1
echo -e "-- `DT`: Detailed operation log 2" | $LOG
sleep 1
DURATION=$[ `date +'%s'`-$START ]
echo -e "-- `DATE`: Operations complete!" | $LOG
echo -e "-- Operations took $DURATION seconds." | $LOG

Resulting in:

[EDIT, 2011-05-15] Thanks to reading gullevek I realized I had this all wrong. You have to wrap the date invocations in functions so that when you call `DT` and `DATE` they are run each time. [END EDIT] This approach is useful for variables that will change through your script’s execution, like timestamps, without having to rewrite the same potentially complicated invocation each time.

START and DURATION, on the other hand, can be defined as plain variables and then simply called as $START and $DURATION since we only want them to be evaluated once (when bash evaluates the definition line). The %s format argument passed to date results in an output of seconds from the Unix epoch. This is done simply because it makes it easy to calculate duration by subtracting two of these. On that note, observe the format for arithmetic operations used to define DURATION. Bash is pretty picky about this way to do mathematical operations, so watch what you are doing.

I use the facilities described above pretty frequently. If you do much Bash scripting, you should consider adding the ones that look useful as snippets in your editor of choice.

Improved random password generator script

Samuel Huckins — Sun, 07 Nov 2010 15:33:01 +0000

[EDIT, 2010-11-10:] After being kindly corrected by Chris Jones, I recommend instead using a readily available utility that is more feature rich to boot: pwgen.[/EDIT]

I have been using this script I wrote about a few years back to generate random passwords. It works, but it’s limited. I’ve written a new one that doesn’t have requirements other than Python 2.5 or higher, is more flexible and a bit easier to follow: randpass.py.

This script is very easy to use. Download it somewhere you keep handy scripts and make an alias like this:

1	alias randpass="~/code/code_homerepo/sysadmin/misc/randpass.py"

Source your .bashrc file (. ~/.bashrc) and try it out to quickly generate a password:

Dealing with “could not create shared memory segment” from postgres on Ubuntu

Samuel Huckins — Mon, 22 Mar 2010 14:44:49 +0000

After installing postgresql-8.4 from the Ubuntu repos on an Ubuntu 9.10 machine, I received the following when the daemon tried to start up:

1
2
3
4
5
6
7

* The PostgreSQL server failed to start. Please check the log output:
FATAL: could not create shared memory segment: Invalid argument
DETAIL: Failed system call was shmget(key=5433001, size=35233792, 03600).
HINT: This error usually means that PostgreSQL's request for a shared memory segment exceeded your kernel's SHMMAX parameter. You can either reduce the request size or reconfigure the kernel with larger SHMMAX. To reduce the request size (currently 35233792 bytes), reduce PostgreSQL's shared_buffers parameter (currently 4096) and/or its max_connections parameter (currently 13).
If the request size is already small, it's possible that it is less than your kernel's SHMMIN parameter, in which case raising the request size or reconfiguring SHMMIN is called for.
The PostgreSQL documentation contains more information about shared memory configuration.
[fail]

Fail indeed. Here’s the fix:

1
2
3
4

sudo cp /etc/sysctl.conf /etc/sysctl.conf-orig &&
sudo sh -c 'echo "#\n# For postgres\nkernel.shmmax = 104857600" >> /etc/sysctl.conf' &&
sudo sysctl -p &&
sudo /etc/init.d/postgresql-8.4 start

This increases a particular kernel parameter to allow postgres enough memory to start. Yay for poor defaults…

Converting multiple images into one PDF on Linux

Samuel Huckins — Sun, 21 Feb 2010 18:12:26 +0000

There are a number of ways you can go about converting image files into PDFs, most simply by opening a given image and printing it. You can select “Print to file” instead of a printer device, then select PDF format and give it a name. But what to do if you have a folder full of images and want to make one big PDF of them? Here’s how you can do that:

1 2	cd FOLDER_WITH_IMAGES convert *.jpg images.pdf

This creates a single PDF, images.pdf, containing all JPG files in the directory. It uses the very handy convert utility. Thanks to reader andreas for pointing out this much simpler approach!

[Edit, 2011-08-18] Thanks to reader A. Syukri for pointing out that the above approach can crash when there are too many input files. Here’s the longer approach that should be more stable:

1
2
3
4
5
6
7
8
9
10

cd FOLDER_WITH_IMAGES
FILES=$( find . -type f -name "*jpg" | cut -d/ -f 2)
mkdir temp && cd temp
for file in $FILES; do
BASE=$(echo $file | sed 's/.jpg//g');
convert ../$BASE.jpg $BASE.pdf;
done &&
pdftk *pdf cat output ../FINAL_NAME.pdf &&
cd ..
rm -rf temp

This loops through all the .jpg images in the directory and converts them to a PDF file of the same name. Once that is done they are all combined into FINAL_NAME.pdf by pdftk, a handy PDF utility. The temp dir business is there to make the temp PDF file removal easier.

If you need to do this frequently you can create a bash function and pass in the folder containing the JPGs. You might want to also consider passing in the extension(s) desired if that varies for you.

Fun with ASCII art

Samuel Huckins — Mon, 14 Dec 2009 06:06:02 +0000

No, in fact ASCII art isn’t dead, it’s all the rage. Now that that’s established, check out these handy programs for generating your own ASCII art messages:

1	figlet -f smslant Boxes rules. \| boxes -d parchment -a hcvc

Figure 1. boxes provides boundless entertainment.

1	figlet -f standard Also figlet. \| boxes -d mouse -a hcvc

Figure 2. figlet and its inherent awesomeness.

Spice up those bland Bash scripts with entertaining comment formats!

But seriously, these utilities can make for more interesting banners, alert messages, and so on:

boxes: Creates ASCII art boxes around entered text
figlet: Creates large ASCII art versions of entered characters

Just because it’s the command line doesn’t mean it has to be boring!

Viewing all users on a Linux system

Samuel Huckins — Sun, 04 Oct 2009 20:35:59 +0000

There are a number of widely-used and stable utilities on Linux systems that allow you to view information related to users. You can see who’s logged in with who, get info on a particular user with finger, see who you are with whoami and see who logged in last with last and lastlog. And there are commands to create, remove, and change users and their groups.

But what if you just want to see all the users defined on the entire system? Looking in /home won’t show you users that don’t have a home dir or one not located there. All users are indeed listed in /etc/passwd, but having to look through this file every time you just want a list of users is tedious. I haven’t found a utility that performs this role.

This was somewhat surprising, as it’s not an uncommon need. Perhaps I’ve missed some essential program along my Linux journey. If so, feel free to enlighten me in a comment In the meantime, let’s fill this gap:

1
2
3
4
5
6
7
8
9
10
11
12

# Prints all users, divided by login ability and homedir:
function userinfo {
echo "----- Users that can login -----"
awk -F":" '!/bin\/false/ { print "username: " $1 ", uid: " $3 ", homedir: " $6 }' /etc/passwd

echo -e "\n----- And have /home dir -----"
awk -F":" '!/bin\/false/ && /\/home/ { print "username: " $1 ", uid: " $3 ", homedir: " $6 }' /etc/passwd

echo -e "\n----- Users that can't login -----"
awk -F":" '/\/bin\/false/ { print "username: " $1 ", uid: " $3 ", homedir: " $6 }' /etc/passwd
echo ""
}

This function should work on any *nix system. Place it inside your ~/.bashrc, reload that (. ~/.bashrc), and try out “userinfo”:

Three sections are displayed showing usernames, user IDs, and homedirs for users in /etc/passwd who:

Don’t have /bin/false for a shell (i.e. users that can login)
Also have a homedir under /home (i.e. the users that aren’t for system processes)
Do have /bin/false for a shell (i.e. users that can’t login)

Note that the users that can login category only shows users to which one could switch with “sudo su – USERNAME” from a shell. This doesn’t mean anything about whether they can login via SSH or other access methods.

Get useful image information on the command line

Samuel Huckins — Sun, 27 Sep 2009 03:20:57 +0000

ImageMagick is an incredible set of programs that allow you to get all sorts of information on image files from the command line, as well as batch edit and generally alter images of many varieties. You can read all about these facilities on ImageMagick’s well-made documentation site.

What I often want, however, is simple: a command to get commonly desired information from any image file. Alias to the rescue!

1	alias imginfo="identify -format '-- %f -- \nType: %m\nSize: %b bytes\nResolution: %wpx x %hpx\nColors: %k'"

This produces a succinctly helpful output:

Often this is all I need for basic tasks. And if more is needed there’s always the full ImageMagick suite.

You can alter the format to whatever you prefer, using the wide variety of variables.

Useful Bash functions to determine OS and more

Samuel Huckins — Mon, 07 Sep 2009 19:07:09 +0000

In a number of my Bash aliases I need to check two constants: what sort of OS I’m on and whether it’s a production box or not. I use the former for aliases that allow me to install and search for packages across distributions, as described here. The latter I use to determine whether I am allowed to use aliases for turning off the machine and the like. The code I used to determine these before was repetitive and hard to work with. So I created functions to determine the values, allowing all the aliases to reference a single source.

The first function determines the OS I am on, in the general categories of “debian” and “redhat”:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

function cur_os {
# Attempts to check whether the local box is running Debian/Ubuntu
# (considered together as "debian") or Red Hat/CentOS (considered
# together as "redhat".
# Make sure we have egrep
EGREP_VER=`egrep --version | head -n 1`
if [[ "${EGREP_VER:0:8}" != "GNU grep" ]] ; then
echo "egrep isn't installed, sorry."
exit 1
fi
# Make sure we have /etc/issue
if [[ ! -r '/etc/issue' ]] ; then
echo "/etc/issue isn't readable."
exit 1
fi
# Run checks
DEB_OS=`egrep -i 'Ubuntu|Debian' /etc/issue`
RH_OS=`egrep -i 'CentOS|Red Hat' /etc/issue`
if [[ ${#DEB_OS} -gt 0 ]] ; then
CUR_OS="debian"
elif [[ ${#RH_OS} -gt 0 ]] ; then
CUR_OS="redhat"
else
CUR_OS="unknown"
fi
echo $CUR_OS
}

I chose these two categories because the package handler is the same within each. This could of course be expanded to more systems, these just happen to be the flavors I am always on. Just after I define this function I also include:

1	cur_os > /dev/null

This way the variable CUR_OS is populated each time Bash loads but the output doesn’t dirty up the prompt. Having this variable allows the definition of fairly elegant functions such as:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

function inst () {
# Installs passed packages.
case "$CUR_OS" in
'debian')
sudo apt-get install $@
;;
'redhat')
sudo yum install $@
;;
*)
echo "You are not on a supported OS."
;;
esac
}

This next function determines whether the current machine is one I consider non-production:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

function local_box {
# Checks if the current machine is one of my own machines (and thus
# less risky to run various commands on).
LOCAL_BOXES=( 'ZenSam' 'B74kb0x' 'MediaServer' 'shuckins-alienware' 'laptop-shuckins' )
CURRENTBOX=`hostname`
CUR_BOX_LOCAL=0
for i in ${LOCAL_BOXES[@]}; do
if [[ $i == ${CURRENTBOX} ]] ; then
CUR_BOX_LOCAL=1
fi;
done
echo $CUR_BOX_LOCAL
}
# So that CUR_BOX_LOCAL is set initially:
local_box > /dev/null

The idea is that LOCAL_BOXES holds hostnames of machines that aren’t as sensitive to things like shutting down. Then when you create aliases for such potentially destructive actions you define a safe form for the potentially sensitive boxes:

1
2
3
4
5
6
7

if [[ $CUR_BOX_LOCAL == 1 ]] ; then
alias off="sudo shutdown -h now"
alias restart="sudo shutdown -r now"
else
alias off="echo 'You are on a remote box!'"
alias restart="echo 'You are on a remote box!'"
fi