[<--] [Cover] [Table of Contents] [Concept Index] [Program Index] [-->]

Web dsl.org

Analyzing Text

There are many ways to use command-line tools to analyze text in various ways, such as counting the number of words in a text, creating a concordance, and comparing texts to see if (and where) they differ. There are also other tricks you can do with text that count as analysis, such as finding anagrams and palindromes, or cutting up text to generate unexpected combinations of words. This chapter covers all these topics.

Counting Text: Counting words, lines and characters.
Text Concordance: Making a text concordance.
Relevant Text: Finding similar or relevant text.
Anagrams: Finding anagrams.
Palindromes: Finding palindromes.
Text Cutups: Cutting up text.

Counting Text

Use the "word count" tool, wc, to count characters, words, and lines in text.

Give the name of a file as an argument; if none is given, wc works on standard input. By default, wc outputs three columns, displaying the counts for lines, words, and characters in the text.

To output the number of lines, words, and characters in file `outline', type:
```
$ wc outline [RET]
```

The following subsections describe how to specify just one kind of count with wc, and how to count text in Emacs.

NOTE: You can get a count of how many different words are in a text, too -- see Making a Concordance of a Text. To count the average length of words, sentences, and paragraphs, use style (see Checking Text for Readability).

Counting Characters: Counting characters.
Counting Words: Counting words.
Counting Lines: Counting lines.
Counting Number: Counting the number of some text string.
Counting Emacs: Counting pages in Emacs.

Counting the Characters in a Text

Use wc with the `-c' option to specify that just the number of characters be counted and output.

To output the number of characters in file `classified.ad', type:
```
$ wc -c classified.ad [RET]
```

Counting the Words in a Text

Use wc with the `-w' option to specify that just the number of words be counted and output.

To output the number of words in the file `story', type:
```
$ wc -w story [RET]
```

To output counts for several files, first concatenate the files with cat, and then pipe the output to wc.

To output the combined number of words for all the files with a `.txt' file name extension in the current directory, type:
```
$ cat *.txt | wc -w [RET]
```

NOTE: To read more about concatenation with cat, see Concatenating Text.

Counting the Lines in a Text

Use wc with the `-l' option to specify that just the number of lines be counted and output.

To output the number of lines in the file `outline', type:
```
$ wc -l outline [RET]
```

Counting the Occurrences of Something

To find the number of occurrences of some text string or pattern in a file or files, use grep to search the file(s) for the text string, and pipe the output to wc with the `-l' option.

To find the number of lines in the file `outline' that contain the string `chapter', type:
```
$ grep chapter outline | wc -l [RET]
```

NOTE: For more recipes for searching text, and more about grep, see Searching Text.

Counting Lines per Page in Emacs

The count-lines-page function in Emacs outputs in the minibuffer the number of lines on the current page (as delimited by pagebreak characters, if any---see Paginating Text), followed by the number of lines in the buffer before the line that point is on, and the number of lines in the buffer after point.

To count the number of lines per page in the current buffer in Emacs, type:
```
C-x l
```

Emacs outputs the number of lines per page of the current buffer in the echo area.

For example, if the output in the minibuffer is

Page has 351 lines (69 + 283)

this means that the current page contains 351 lines, and point is on line number 70 -- there are 69 lines before this line, and 283 lines after this line.

Making a Concordance of a Text

A concordance is an index of all the words in a text, along with their contexts. A concordance-like functionality -- an alphabetical listing of all words in a text and their frequency -- can be made fairly easily with some basic shell tools: tr, sort, and uniq.

To output a word-frequency list of the text file `naked_lunch', type:
```
$ tr ' ' ' [RET]
> ' < naked_lunch | sort | uniq -c [RET]
```

These commands translate all space characters to newline characters, outputting the text with each word on its own line; this is then sorted alphabetically, and that output is passed to uniq, which outputs only the unique lines -- that is, all non-duplicate lines -- while the `-c' option precedes each line with its count (the number of times it occurs in the text).

To get a word frequency count -- that is, the total number of different words in a text -- just pipe the output of the frequency list to wc with the `-l' option. This counts all the lines of its input, which in this case will be the list of unique words, one per line.

To output a count of the number of unique words in the text file `naked_lunch', type:
```
$ tr ' ' ' [RET]
> ' < naked_lunch | sort | uniq -c | wc -l [RET]
```

Text Relevance

The following recipes show how to analyze a given text for its relevancy to other text, either to keywords or to whole files of text.

You can also use the diff family of tools to analyze differences in text; those tools are especially good for comparing different revisions of the same file (see Comparing Files).

Relevance Sorting: Sorting text by relevance.
Remembrance Agent: An Emacs tool for finding relevant text.

Sorting Text in Order of Relevance

Debian: `rel'
WWW: http://www.johncon.com/

Use rel to analyze text files for relevance to a given set of keywords. It outputs the names of those files that are relevant to the given keywords, ranked in order of relevance; if a file does not meet the criteria, it is not output in the relevance listing.

Moreover, and are far more intensive the pancreaticdiabetes but more of the pancreatic diabetes.Article: https://www.gulickhhc.com/drugs/erectile-dysfunction/tadapox.htm

rel takes as an option the keyword to search for in quotes; you can build a boolean expression by grouping multiple keywords in parentheses and using any of the following operators between them:

OPERATOR DESCRIPTION

| Logical "or."
& Logical "and."

! Logical "not."
Give as arguments the names of the files to rank.

To rank the files `report.a', `report.b', and `report.c' in order of relevance to the keywords `saving' and `profit', type:
```
$ rel "(saving & profit)" report.a report.b report.c [RET]
```

Give the name of a directory tree to analyze all files in the directory tree.

To output a list of any files containing either `invitation' or `request' in the `~/mail' directory, ranked in order of relevancy, type:
```
$ rel "(invitation | request)" ~/mail [RET]
```
To output a list of any files containing `invitation' and not `wedding' in the `~/mail' directory, ranked in order of relevancy, type:
```
$ rel "(invitation ! wedding)" ~/mail [RET]
```
To output a list of any files containing `invitation' and `party' in the `~/mail' directory, ranked in order of relevancy, type:
```
$ rel "(invitation & party)" ~/mail [RET]
```

Listing Relevant Files in Emacs

Debian: `remembrance-agent'
WWW: http://www.media.mit.edu/~rhodes/RA/

The purpose of the Remembrance Agent is to analyze the text you type in an Emacs session and, in the background, find similar or relevant passages of text within your other files. It then outputs in a smaller window a list of suggestions -- those files that it has found -- which you can open in a new buffer.

When installing the Remembrance Agent, you create three databases of the files to use when making relevance suggestions; when remembrance-agent is running, it searches these three databases in parallel, looking for relevant text. You could create, for example, one database of saved email, one of your own writings, and one of saved documents.

To toggle the Remembrance Agent in the current buffer, type:
```
C-c r t
```

When remembrance-agent is running, suggested buffers will be displayed in the small `*Remembrance*' buffer at the bottom of the screen. To open a suggestion in a new buffer, type C-c r number, where number is the number of the suggestion.

To open the second suggested file in a new buffer, type:
```
C-c r 2
```

Finding Anagrams in Text

Debian: `an'

An anagram is a word or phrase whose characters consist entirely of all the characters of a given word or phrase -- for example, `stop' and `tops' are both anagrams of `pots'.

Use an to find and output anagrams. Give as an argument the word or quoted phrase to use; an writes its results to the standard output.

To output all anagrams of the word `lake', type:
```
$ an lake [RET]
```
To output all anagrams of the phrase `lakes and oceans', type:
```
$ an 'lakes and oceans' [RET]
```

To limit the anagrams output to those containing a given string, specify that string with the `-c' option.

To output only anagrams of the phrase `lakes and oceans' which contain the string `seas', type:
```
$ an -c seas 'lakes and oceans' [RET]
```

To print all of the words that some or all letters in a given word or phrase can make, use the `-w' option. This outputs words that are not anagrams, since anagrams must contain all of the letters of the other word or phrase.

To output all of the words that can be made from the letters of the word `seas', type:
```
$ an -w seas [RET]
```

This command outputs all of the words that can be formed from all or some of the characters in `seas', including `see' and `as'.

Finding Palindromes in Text

A palindrome is a word that reads the same both forwards and backwards; for example, "Mom," "madam," and "nun" are all palindromes.

To find palindromes in a file, use this simple Perl "one-liner," and substitute file for the name of the file to check:

perl -lne 'print if $_ eq reverse' file

To check for palindromes in the standard input, specify `-' as the file name to check. This is useful for putting at the end of a pipeline.

To output all of the palindromes in the system dictionary, type:
```
$ perl -lne 'print if $_ eq reverse' /usr/dict/words [RET]
```

Text Cut-Ups

A cut-up is a random rearrangement of a physical layout of text, made with the intention of finding unique or interesting phrases in the rearrangement. Software for rearranging text in random ways has existed since the earliest text-processing tools; the popularity of these tools will never die.

The cut-up technique in literature was discovered by painter Brion Gysin and American writer William S. Burroughs in 1959; they believed it brought the montage technique of painting to the written word.

"All writing is in fact cut-ups," Burroughs wrote.(21) "A collage of words read heard overheard ... [u]se of scissors renders the process explicit and subject to extension and variation."

These recipes describe a few of the common ways to make text cut-ups; more free software tools for making cut-ups are listed at http://dsl.org/comp/cutups.shtml.

Simple Cutups: Simple text cutups.
Advanced Cutups: Advanced text cutups.
Emacs Cutups: Cutups in Emacs.

Making Simple Text Cut-Ups

WWW: http://dsl.org/comp/tinyutils/

To perform a simple cut-up of a text, use cutup. It takes the name of a file as input and cuts it both horizontally and vertically along the middle, rearranges the four sections to their diagonally opposite corners, and then writes that cut-up to the standard output. The original file is not modified.

To make a cut-up from a file called `nova', type:
```
$ cutup nova [RET]
```

Making Random Word Cut-Ups

Debian: `dadadodo'
WWW: http://www.jwz.org/dadadodo/

No simple cut-up filter, Jamie Zawinski's dadadodo uses the computer to go one step beyond -- it generates passages of random text whose structure and characters are similar to the text input you give it. The program works better on larger texts, where more subtleties can be analyzed and hence more realistic-looking text is output.

Give as an argument the name of the text file to be used; by default, dadadodo outputs text to standard output until you interrupt it by typing C-c.

To output random text based on the text in the file `nova', type:
```
$ dadadodo nova [RET]
```

This command will output passages of random text based on the text in the file `nova' until it is interrupted by the user.

You can analyze a text and save the analysis to a file of compiled data; this analysis can then be used to generate random text when the original input text is not present. The following table describes this and other dadadodo options.

OPTION DESCRIPTION

-c integer Generate integer sentences (default is 0, meaning "generate an infinite amount until interrupted").
-l file Load compiled data in file and use it to generate text.

-o file Output compiled data to file file for later use.
-p integer Pause for integer seconds between paragraphs.

Making Cut-Ups in Emacs

The dissociated-press function in Emacs makes random cut-ups of the current buffer in a new buffer called `*Dissociation*'; the original buffer is not modified. The text in the new buffer is generated by combining random portions of the buffer by overlapping characters or words, thus (usually) creating plausible-sounding sentences. It pauses occasionally and asks whether or not you want to continue the dissociation.

To generate a Dissociated Press cut-up from the current buffer, type:
```
M-x dissociated-press [RET]
```

Give a positive argument to the dissociated-press function to specify the number of characters to use for overlap; give a negative argument to specify the number of words for overlap.

To generate a Dissociated Press cut-up from the current buffer, always overlapping by three characters, type:
```
C-u 3 M-x dissociated-press [RET]
```
To generate a Dissociated Press cut-up from the current buffer, always overlapping by one word, type:
```
C-u -1 M-x dissociated-press [RET]
```

[<--] [Cover] [Table of Contents] [Concept Index] [Program Index] [-->]

OPERATOR	DESCRIPTION
`\|`	Logical "or."
`&`	Logical "and."
`!`	Logical "not."

OPTION	DESCRIPTION
`-c integer`	Generate `integer` sentences (default is 0, meaning "generate an infinite amount until interrupted").
`-l file`	Load compiled data in `file` and use it to generate text.
`-o file`	Output compiled data to file `file` for later use.
`-p integer`	Pause for `integer` seconds between paragraphs.