Reinier Post's commandline scripts

This collection of commandline tools for Unix variants, developed for my own use, was started in the 1990s and is still being used, maintained, and extended.

All of these scripts have been used on Linux; many on FreeBSD, Cygwin and Solaris; most take the -h argument as a help option.

• 2mp3 - convert a couple of formats, such as .wav, .mp4 and .mid, to MP3
• add - replace numbers on lines with cumulative totals, per column (can be a useful postprocessing step)
• aggr - aggregate numbers on the input lines - a generalization of add
• allcharstwice - print only the lines in which every character occurs exactly twice - see also isograms
• append - append a file to another
• apt-rdepends-ultimates - lists the ultimate dependencies in apt-rdepends results
• a42a5 - print-on-2-pages PostScript filter
• asuser - a simple shorthand for sudo
• bsn2name - convert BSNs to HBC names
• bodyof - strip headers and quotes from e-mail/Usenet messages
• collatz - print positive numbers with the lengths of their hailstone sequences
• collatz2dot - generate GraphViz input based on hailstone sequencess
• copy-skype-shots - copy images from one directory to the other, renaming them and keeping one copy for each
• copy-camera-media - copy images from certain cameras or phones plugged in with USB on Linux
• count - count numbers of identical lines in input
• cpto - cp to the first argument (so as to allow use with xargs)
• csv2csv - convert CSV files between table and matrix form
• csv2sqlite - import certain CSV files into a SQLite database
• csv2dot - convert comma separated values to dot format for visualization
• csvcombine - combine several comma separated value files into one
• csvproj - join and project CSV files relationally
• csvsql - execute a SQL statement on CSV tables
• sqlexec2csv - execute SQL statements in the given files
• csvtrans - perform various transformations on directed graphs represented as comma separated value files
• dbinstkeys - list key constraints satisfied by a database instance (work in progress)
• diffall - diff among a whole set of files
• diffcluster - which files in a set are equal
• diffingly - run a command that takes a file and diff the result against the file
• diffsorted - diff after sorting
• diffafter - diff after filtering, e.g. diffafter 'rev|sort' f1 f2 | rev
• diffzips - diff the contents of two ZIP files
• deword-html - somewhat clean up Microsoft Word HTML (proof of concept)
• edit-distances - list the string distances between all pairs of input lines
• textdist - list the string distances between all pairs of input lines
• envperl - change header (shebang line) of Perl scripts so they work in a Unix/Linux environment
• rightperl - change header (shebang line) of Perl scripts to point to the presently selected version of Perl
• right - a generalization (right perl is rightperl)
• explorer-here - launch a Windows Explorer in the given or current directory (useful on Cygwin)
• except - print lines in first file that do not occur in second (even when the files are not sorted)
• only - print lines in first file that also occur in second (even when the files are not sorted)
• factoids - generate "factoids" an infobot can use, in a way similar to how an infobot does it - useful with the xchat log processing scripts
• flip - solve the 24 game (which was played with "flippos" ("pogs")) when this script was written)
• fizzbuzz - Fizz buzz (in Perl)
• ff - find files (a trivial wrapper)
• fmb - format mail body, a fmt variant that handles quoted material
• get-swf - download the Flash application embedded in an HTML file (if any) (I wrote this to collect Flash games on a USB stick)
• getmailaddr - get email addresses from text file(s) (quick & dirty)
• getmboxaddr - get email addresses from Unix mailbox files (with a real mailbox parser)
• geturls - get URLs from text file(s) (quick & dirty)
• gulp - like cat, output the input, but after swallowing it whole
• grep-mbox - select messages from a Unix mailbox file
• selectmail - the same, but with a real mailbox parser behind it
• google - Google search a term and return the URLs of the results
• hdr2col - "flatten" headers in output that uses them, such as ls -R
• hist - tcsh history with human-readable timestamps
• html4images1 - generate basic HTML showing all images in a directory tree
• html4images2 - generate fancy HTML showing all images in a directory tree
• img2bb - generate bounding boxes for use with images for LaTeX
• img2eps - convert images for use with LaTeX
• ipnum2name - convert IP addresses to DNS names in input (convenient for webserver logs etc.)
• impatiently - impatiently execute a command line (aborts when output does not appear soon enough)
• innosetupfiles - generate an Inno Setup listing of files
• isblack - whether each image supplied as argument is entirely black
• isograms - print the words in the input in which all characters occur equally often - see also allcharstwice
• length - prepend each line with the number of bytes on it (minus line end characters)
• lf - output the name of the last modified file (i.e. non-directory)
• lm - output the name and/or modification time of the last modified file or directory
• ln-r - recursively replace files in one directory tree with links to the corresponding files in another, when identical
• ln-r - recursively replace files in one directory tree with links to the corresponding files in another, when identical
• loo - count lines and characters, like wc, but ignoring whitespace
• mailmanage - reconfigure Mailman mailing list settings, output subscribers and messages
• maillog2csv - join up pairs of entries for the same message (in /var/log/maillog)
• make2csv - output GNU make -prR output in CSV format processable by csvtrans and/or csv2dot
• mimetypes - sort and print MIME type mappings collected from various sources
• 2mp3 - convert audio/video files (in various formats) to MP3
• mysql-rename-tables - systematically rename tables in a MySQL database dump (not foolproof!)
• mysql-unfold-inserts - insert newlines into INSERT statements in a MySQL database dump (not foolproof!)
• mvto - mv to the first argument (so as to allow use with xargs) (I didn't know of GNU mv's -t option)
• lastlonger - last on all available entries
• lastaggr - summarize last -wxaiF output
• previsprefix - similar, but more specialized
• paths2tree - print a (sorted) list of paths as a tree
• perltighten - quick and dirty empty line remover for Perl scripts
• pdfcat - concatenate PDF files (using pdfinfo and pdflatex)
• phprefs2dot - generate a(n approximate) call graph for PHP applications (quick & dirty)
• picasaweb - download Picasaweb images in various resolutions
• printable - leave only printable characters - useful in combo with xchatlog2body
• pingable - leave only pingable hosts
• plot-matrix - plot 3-column CSV files (x, y, value)
• plot-metrics - plot CSV files wih metrics values
• procmailsum - summarize a procmail logfile
• psgrep - list and optionally kill processes by name, like pgrep/pkill
• pschanges - list processes gone or new since the previous run
• rawlinsonize - munge text in a way that leaves it surprisingly readable, as first observed by Graham Rawlinson
• renamep - rename files (similar to rename
• rev - rev, in case it is missing
• realdirs - a variant on realpath
• rdesktop - a disk-sharing wrapper around /usr/bin/rdesktop
• rdesktop-maximized - wrapper around rdesktop that automatically picks a display and fills it, using xdisplaygeometries
• repair-with - apply a text filter as an in-place edit on a list of files (generalizes tidy-us)
• rm0 - rm unless nonempty
• shortest - omit input lines if a prefix of them also occurs as an input line
• skype-w-video - wrapper around Skype to enable video on my Linux systems
• sort-by-mtime - sort filenames given on input by last modification time
• splitx - split input into files by regular expressions (only because I did not know about csplit)
• squeeze - squeeze an image using seam carving with ImageMagick resize
• sshall - execute a command with ssh on multiple hosts
• substrings - which lines in the input are substrings of which other lines
• svn-win - command-line interface to the svn.win.tue.nl admin UI
• svn-win-add-user - command-line interface to the svn.win.tue.nl admin UI
• tex-clean - remove files generated by the LaTeX build process (originally by Tom Verhoeff)
• tidy-us - apply tidy to a list of HTML files
• timestampingly - precede output with timestamps (useful for logging)
• today - print the current day
• tosecond - print the current second
• touchpad - en/disable the Dell Precision touchpad
• uncomment - strip comments from a script (quick & dirty, supports various languages)
• undup-mbox - select and de-duplicate messages from a Unix mailbox file; selectmail can also do this
• uniqer - remove duplicate lines from text files (like uniq) even on unsorted input
• unixifn - change filenames to be easier for Unix tools
• urlheadinfo - given a list of URLs, get HEAD info on each of them (mainly to see if they are alive)
• urlheads4feed2csv - given an output of urlheadinfo -j, output only the URLs for RSS/Atom feeds (based on analyzing their page contents)
• vncdisplay - return the display number for a VNC server, starting one if none was running
• vpn - connect to vpn3.tue.nl or another openconnect-compatible VPN server; does not yet support vpn-student.tue.nl
• websearch - command line Web search from my Lynx; no longer used or maintained
• whereare - like where, but supporting wildcards in its arguments
• whohas - whois stripping host and/or subdomain
• words - print the normalized words in the input
• wordgen - primitive word generator (varies on words supplied on input)
• xchatlogsplit - split a set of xchat logfiles
• xchatlog2body - leave only the body text in xchat logfiles
• xdisplaygeometries - use xrandr to list the current $DISPLAY's actual display geometries
• xfirstfreedisplay - determine the first $DISPLAY on which no X server is running
• xfreerdp-on-extra-monitors - start /usr/bin/xfreerdp on all monitors except the first
• xonjointdisplay - determine the X display geometry of the joint screens
• xonlargestdisplay - determine the X display geometry of the highest-resolution screen
• xpath-rp - evaluate XPath queries on XML documents
• xpaste2window - types pasted text into the currently focused X window
• xrandr-mode-from-cvt - add or remove an X mode line using xrandr and cvt
• xref - list cross-references of terms (i.e. file names or derivations of them) in collections of files
• yes - yes, in case it is missing
• zip-r - archive one or more files/directories into a ZIP file with the expected name
• 7z-r - archive one or more files/directories into a 7z file with the expected name

Some utilities related to email:

• maillog2csv - pair up the lines in an MTA log (e.g. /var/log/mail.log) and output as CSV
• spamexperts-log-rename - rename an exported SpamExperts logfile (CSV Export OutgoingLogSearch.csv) based on its timestamps
• nondelivered-possibly-nonspam - select possibly legitimate undelivered messages in an exported SpamExperts logfile; requires csvsql

Some utilities related to Ubuntu package management:

• apt-mtime-for - show the modification time of a package on the repository server
• apt-ppa-inconsistencies - list packages that should probably be reinstalled from their PPA
• dpkgdiff - diff a file against the version provided by an Ubuntu package
• dpkg-unused-kernels - list installed kernels, except the current one

Some utilities for Apache web server usage analysis:

• accesses-per-minute - count accesses per minute
• accesslog-aggregate - aggregate logs in various ways
• apache-virtual-sites - list virtual sites
• apache-virtual-sites-l-execute - postprocess apache-virtual-sites -l output
• file-age-alert - output warnings for overaged files (useful in cron jobs)
• log-nr-of-httpd-processes - log the number of httpd processes
• nr-h - rewrite numbers in the input to use suffixes K, M, G, T, P
• nr-of-httpd-processes - print the number of httpd processes
• selenium-linkchecker - check website links with Selenium
• with-rotated - list a logfile with/or its rotated predecessors
• zcatorcat - cat uncompressed (except when stdin)

See also:

• some utilities for webhosting
.

Author: Reinier Post, TU Eindhoven. Use and redistribution permitted, but entirely at your own responsibility. Publication/distribution of modified versions permitted provided that any such modifications are communicated to me first.