View Source of articles/perl-cli.php
In the interest of open source, and in the interest of keeping Mr. Webmaster honest, here is the PHP source of articles/perl-cli.php. (This encourages him to write clean, easy-to-understand code...well, relatively clean and easy-to-understand, that is.) If you become interested in PHP, you can read more about it at PHP: Hypertext Preprocessor. You will also see plenty of HTML code, which is what makes the World Wide Web go 'round. You can read more about HTML (and CSS, another technology used on this site) here, at the official page of the very official World Wide Web Consortium (W3C). Anyway, enough rambling, here's the code!
<?php include("../header1.php"); ?>
<?php $THIS_FILE = "articles/perl-cli.php"; ?>
<title>Jeff T. - Using Perl on the Command Line</title>
<style>
div.code {
font-family: monospace;
font-size: small;
margin-left: 50px;
margin-right: 10px;
border: 1px solid gray;
}
</style>
<?php include("../header2.php"); ?>
<?php include("localheader.php"); ?>
<h1>Using Perl on the Command Line</h1>
<h3>Abstract</h3>
<p>Perl has become a very popular scripting and text-processing language, yet relatively few programmers and command-line geeks are aware of the usefulness of Perl from within an interactive, command-line interface. In this tutorial, I introduce several powerful command-line switches, including -e, -n, -p, -l, -a, and -F. I also demonstrate <code>BEGIN{}</code> and <code>END{}</code> blocks. I illustrate all new concepts with a variety of examples.</p>
<h3>Audience</h3>
<p>You are expected to be familiar with Perl. I will explain some of the more esoteric language features used below, but you should already be pretty savvy with the Perl language. If you want to learn Perl, I recommend O'Reilly's excellent books, <i><a href="http://www.oreilly.com/catalog/learnperl4/">Learning Perl</a></i> and <i><a href="http://www.oreilly.com/catalog/pperl3/">Programming Perl</a></i>. Also, most of the information in these books is available from the perl manpages--and, furthermore, it's almost as digestible as the books are. Start with <a href="http://www.its.unimelb.edu.au/manuals/perl5/">perl(1)</a> and go from there.</p>
<h3>The -e switch: inline scripting</h3>
<p>Perl's -e switch enables <i>inline</i> or <i>literal</i> scripting. With -e, the script which perl executes is actually on the command-line itself, rather than being contained in a file. So we can say things like this:</p>
<div class="code"><b>$</b> perl -e 'print 1+1 . "\n"'</div>
<p>(<b>Note:</b> the initial <b>$</b> character signifies the shell prompt, and is not part of the command. This is true for all the examples in this tutorial.)</p>
<p>Of course, if your perl script is that simple, you can just use bash operators (assuming you're using bash--and if you're not, why not?):</p>
<div class="code"><b>$</b> echo $((1+1))</div>
<p>Then again, perl is a little more useful for simple arithmetic operations, because bash doesn't support floating-point numbers:
<div class="code">
<b>$</b> echo $((22/7))<br/>
3<br/>
<b>$</b> perl -e 'print 22/7 . "\n"'<br/>
3.14285714285714</div>
<p>Nevertheless, -e by itself is not very useful. It is, however, an essential part of more advanced command-line tricks.</p>
<h3>The -n switch: implicit loops</h3>
<p>Perl's -n switch implicitly wraps the literal script (i.e. the script specified on the command-line itself) in a <code>while (<>) { }</code> loop. In other words, -n tells perl: "do the following for every line of input."</p>
<p>For example, I have a few shell scripts for a project I worked on recently. Some of these shell scripts call perl:</p>
<div class="code">
<b>$</b> grep perl *.sh<br/>
arrange.conns.sh: perl -pe 's/ [<>?] / /;s/ t \d+ \d+ -?\d+//' |<br/>
arrange.conns.sh: perl -ne '<br/>
arrange.conns.sh: perl -e 'while (<>) {<br/>
arrange.conns.sh: perl -lne '<br/>
arrange.conns.sh: perl -e '<br/>
cvec.norm.sh:perl -pe 's/(^DIR1|^DIR2|^CONC|^SEQ)/\n$1/' |<br/>
cvec.norm.sh: perl -ne 'BEGIN{$/="\n\n"} print if /^SEQ|^CONC/' |<br/>
cvec.norm.sh: perl -lne 'BEGIN{$/="\n\n"} @a=(split(/\n/,$_)); print if scalar(@a) > 1'</div>
<p>Let's say we wanted to count the number of characters on each of these lines. First, we get rid of the file name. We could do this with <code>`cut -d: -f2-`</code>, but we'll simply use GNU grep's -h option instead. Note how -ne scripts often make heavy use of the $_ variable, which contains each line of input.</p>
<div class="code">
<b>$</b> grep -h perl *.sh | perl -ne 'print length() . "\n"'<br/>
51<br/>
15<br/>
26<br/>
14<br/>
14<br/>
46<br/>
54<br/>
76</div>
<h3>The -l switch: automatic newline handling</h3>
<p>As it turns out, there's a slightly simpler way to do the above:</p>
<div class="code">
<b>$</b> grep -h perl * | perl -lne 'print length'<br/>
50<br/>
14<br/>
25<br/>
13<br/>
13<br/>
45<br/>
53<br/>
75</div>
<p>The -l switch silently removes the newline character (or whatever $/ is set to, as we will see below) from the end of each line of input, and silently adds a newline character to the end of each line of output. Note that the answers are all reduced by one because each line is shorter by one character: the newline at the end.</p>
<h3>The -p switch: implicit looping and printing</h3>
<p>Perl's -p switch is the same thing as the -n switch with an additional shortcut: a 'print' statement is implied to print the $_ variable. I find this particularly useful for simply applying a Perl regex style substitution to each line. For example, say I have <a href="tcpdump.txt">this (sanitized) output</a> from <a href="http://www.tcpdump.org/">tcpdump</a>:</p>
<div class="code">
<b>$</b> cat tcpdump.txt<br/>
1177350516.293578 64.233.100.100.4663 > 152.2.100.100.25: .<br/>
1177350516.293598 152.23.100.100.3398 > 64.81.100.100.80: .<br/>
1177350516.293590 208.111.100.100.80 > 152.23.100.100.2706: .</div>
<p>And say I want to modify the local IP address (the one starting with 152.2 or 152.23) so that only the /24 subnet is printed. (This might seem contrived, but I have actually done things like this in my research.)</p>
<div class="code">
<b>$</b> perl -pe 's/ (152\.23?\.\d+)\.\d+\.\d+\b/ $1/' tcpdump.txt<br/>
1177350516.293578 64.233.100.100.4663 > 152.2.100: .<br/>
1177350516.293598 152.23.100 > 64.81.100.100.80: .<br/>
1177350516.293590 208.111.100.100.80 > 152.23.100: .</div>
<p>Or, say I want to separate the port number (the 5th dot-separated field) from the IP address:</p>
<div class="code">
<b>$</b> perl -pe 's/ (\d+\.\d+\.\d+\.\d+)\.(\d+)\b/ $1 $2/g' tcpdump.txt<br/>
1177350516.293578 64.233.100.100 4663 > 152.2.100.100 25: .<br/>
1177350516.293598 152.23.100.100 3398 > 64.81.100.100 80: .<br/>
1177350516.293590 208.111.100.100 80 > 152.23.100.100 2706: .</div>
<h3>BEGIN/END Blocks</h3>
<p>What if we have some code that we want to execute after (or before) the implicit while loop in a -ne or -pe script? I bet you're not surprised that Perl gives us a way to do exactly this: <code>BEGIN{}</code> and <code>END{}</code> blocks.</p>
<p>The most common use for this feature (at least that I've found) is summing a bunch of numbers in a file. First, let's create such a file:</p>
<div class="code">
<b>$</b> perl -e 'print "$_\n" for (1..1000)' > nums.txt</div>
<p>Now let's sum them up. If you're as clever as <a href="http://www.maztravel.com/maz/explain/counting.html">Carl Friedrich Gauss was in first grade</a>, you'll know what the answer should be. But for the rest of us, we'll accept some help from Perl:</p>
<div class="code">
<b>$</b> perl -lne '$c += $_; END{ print $c; }' nums.txt<br/>
500500</div>
<p>Let's have some more fun. Let's first generate a file with 1000 pseudorandom numbers from 0 to 100:</p>
<div class="code">
<b>$</b> perl -e 'print(rand(100) . "\n") for (1..1000)' > rand.txt</div>
<p>Now let's average all of these numbers. If the pseudorandom number generator does a good job of picking numbers from a uniform distribution, we would expect the average to be near 50, right? (This is statistics stuff; don't worry if you don't understand it.) Well, let's see how well it does:</p>
<div class="code">
<b>$</b> perl -lne '$c += $_; END{ print $c/$.; }' rand.txt<br/>
50.1172772862758</div>
<p>Not bad. If you're feeling ambitious, try doing the same thing with more and more samples and see what happens.</p>
<h3>The -a switch: automatic field splitting</h3>
<p>Perl's -a switch stands for "autosplit". When -a is specified, Perl implicitly splits the input by whitespace, and stores the results in @F, as if you had typed <code>@F=(split);</code> immediately after getting input. This behavior is sort of like the standard <a href="http://www.gnu.org/software/coreutils/manual/html_node/cut-invocation.html#cut-invocation"><code>cut</code></a> utility (part of the GNU <a href="http://www.gnu.org/software/coreutils/">coreutils</a>), with two exceptions. First, <code>cut</code> can only split on single characters, whereas Perl's autosplit switch can split on any regular expression. Second, <code>cut</code> merely prints the columns, whereas Perl can do more complex processing.</p>
<p>For example, let's count the total number of bytes used in a directory:</p>
<div class="code">
<b>$</b> ll<br/>
total 80<br/>
-rw-r--r-- 1 jsterrel jsterrel 2948 Oct 31 13:37 j1<br/>
-rw-r--r-- 1 jsterrel jsterrel 2948 Oct 31 13:35 lorem.txt<br/>
-rw-r--r-- 1 jsterrel jsterrel 3893 Oct 31 15:53 nums.txt<br/>
-rw-r--r-- 1 jsterrel jsterrel 2180 Oct 31 16:02 perl-cli.txt<br/>
-rw-r--r-- 1 jsterrel jsterrel 16909 Oct 31 15:59 rand.txt<br/>
-rw-r--r-- 1 jsterrel jsterrel 434 Oct 31 15:41 tcpdump.txt<br/>
<b>$</b> ll | perl -lane '$c += $F[4]; END{ print $c; }'<br/>
29312</div>
<p>You know how the `ps auxwww` command often prints a lot of columns you don't care about? Don't you hate that? Yeah, me too. Let's cut it down to the essentials.</p>
<div class="code">
<b>$</b> ps auxwww | head -n3<br/>
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND<br/>
jsterrel 428 4.3 1.4 381096 14288 ?? S 10Oct07 13:10.01 /Applications/Utilities/Terminal.app/Contents/MacOS/Terminal -psn_0_3145729<br/>
jsterrel 14973 2.5 27.1 2327152 284428 ?? S 20Oct07 227:02.88 /Applications/Safari.app/Contents/MacOS/Safari -psn_0_16646145<br/>
<b>$</b> ps auxwww | head -n3 | perl -lape '$_ = join(" ", @F[ 1,10..$#F ])'<br/>
PID COMMAND<br/>
14973 /Applications/Safari.app/Contents/MacOS/Safari -psn_0_16646145<br/>
4009 /Applications/Adium.app/Contents/MacOS/Adium -psn_0_11927553</div>
<p>The <code>$#F</code> in the above command is a Perl shortcut meaning "the index of the last element of the <code>@F</code> array". The full expression inside the brackets of <code>@F[ ... ]</code> specifies an <i>array slice</i>: multiple elements of the array, forming an array in themselves. Thus, the larger <code>@F</code> expression means "the array consisting of the second element of <code>@F</code> as well as all fields from the 11th to the last".</p>
<p>Lastly, note that you don't have to split just on whitespace. You can specify any regular expression to the -a option by the -F option. See the <a href="http://www.its.unimelb.edu.au/manuals/perl5/perlrun.html">perlrun(1)</a> for more details.</p>
<h3>"Chunk" Processing: changing the input record separator</h3>
<p>Often, in a Unix-style environment, you're dealing with data in which the records are separated by newline characters. In other words, one line is one record. Sometimes, however, it is more convenient to deal with records stored in more general <i>chunks</i> of data. For example, I often deal with chunks of lines, where chunks are separated by a blank line (or "\n\n"). Thankfully, Perl can operate on such data with relatively minor tweaks. Specifically, we must change the $/ variable, or the <i>input record separator</i>.</p>
<p>Let's go through an extended example with a file of junk "Lorem Ipsum" text, generated from <a href="http://www.lipsum.com/">lipsum.com</a>. The text is arranged in paragraphs. There are 5 paragraphs, each of which is on a single line, and paragraphs are separated by a blank line. You can view the file <a href="lorem.txt">here</a>. First things first, let's split the paragraphs by sentence:</p>
<div class="code">
<b>$</b> perl -pe 's/\. /.\n/g' lorem.txt > j1</div>
<p>Now, let's count the number of sentences per paragraph:</p>
<div class="code">
<b>$</b> perl -lpe 'BEGIN{ $/ = "\n\n"; } @c = (split /\n/); $_ = @c' j1<br/>
11<br/>
16<br/>
11<br/>
15<br/>
16</div>
<p>Note that there is an implicit <code>scalar()</code> around the last <code>@c</code>. (We actually could have done it just as easily without splitting the paragraphs into sentences. I'll leave that as an exercise to the reader.) Now let's calculate the average sentence length, in characters, over the entire text:</p>
<div class="code">
<b>$</b> grep -v '^$' j1 | perl -lne '$c += length; END{ print $c/$.; }'<br/>41.6666666666667</div>
<p>Nice. One more thing: let's calculate the average sentence length (in characters) per paragraph:</p>
<div class="code">
<b>$</b> perl -lpe 'BEGIN{$/ = "\n\n"} $c = 0; @s = (split /\n/); $c += length foreach @s; $_ = $c/@s' j1<br/>
30.0909090909091<br/>
52.125<br/>
33.6363636363636<br/>
46.6666666666667<br/>
40</div>
<p>Very cool. If you're feeling ambitious, calculate the average number of words per sentence, per paragraph.</p>
<h3>Conclusion</h3>
<p>Perl can be a very valuable multi-purpose tool to add to your command-line toolbox. In this tutorial, we covered the essential -e switch, the implicit line loopers -n and -p, the auto-field-splitting -a switch, <code>BEGIN{}</code> and <code>END{}</code> blocks, and chunk-style processing by changing the input-record-separator variable.</p>
<?php include("../footer.php"); ?>
viewsource.php: Last Modified: 08/13/07@20:34:26 | Size: 2780 bytes | View Source