[svn-upgrade] Tagging scowl (7.1)

[deb_pkgs/scowl.git] / 7.1 / r / alt12dicts / README-orig

Introduction

12dicts is a collection of English word lists. It differs in several
important ways from most of the other free word lists you can download.

  * The 12dicts lists are oriented towards common words. If you're
    looking for myriads of archaic, scientific or computer jargon words,
    you should look elsewhere.
  * The 12dicts lists have been rigorously checked for errors. (This is
    not to say that they are error-free, merely that enough care has
    been taken that errors are rather infrequent.)
  * 12dicts contains a variety of lists, of different sizes and
    characteristics. One size does not fit all. Because each list has
    different characteristics, I do not recommend combining them, except
    as noted below.

Originally, 12dicts was composed of lists derived from a specific set of
12 source dictionaries. In addition to these "classic" lists, 12dicts
now includes lists derived from other sources. It would perhaps be
appropriate to rename 12dicts to something more generic, such as BAWL
(Beale's Assorted Word Lists), but I have not done so in order to
preserve continuity.

A quick summary of the 12dicts lists and their characteristics is as
follows:

+---------------------------------------------------------+
|               |3esl |6of12|2of12|2of4brif|5desk|2of12inf|
|---------------+-----+-----+-----+--------+-----+--------|
|Size           |21877|32153|41236|60387   |61406|81520   |
|---------------+-----+-----+-----+--------+-----+--------|
|Abbreviations  |Y    |Y    |N    |N       |N    |N       |
|---------------+-----+-----+-----+--------+-----+--------|
|Acronyms       |Y    |Y    |N    |N       |Y    |N       |
|---------------+-----+-----+-----+--------+-----+--------|
|British English|N    |N    |N    |Y       |N    |N       |
|---------------+-----+-----+-----+--------+-----+--------|
|Hyphenations   |Y    |Y    |Y    |N       |N    |N       |
|---------------+-----+-----+-----+--------+-----+--------|
|Inflections    |N    |N    |N    |Y       |N    |Y       |
|---------------+-----+-----+-----+--------+-----+--------|
|Names          |Y    |Y    |N    |N       |Y    |N       |
|---------------+-----+-----+-----+--------+-----+--------|
|Phrases        |Y    |Y    |N    |N       |N    |N       |
+---------------------------------------------------------+

The remainder of this document is organized as follows:

  * This release
  * The classic 12dicts lists
      + The 6of12 and 2of12 lists
      + The 2of12inf list
  * The 3esl list
  * The 2of4brif list
  * The 5desk list
  * How 12dicts came to be
  * Conclusions

This release

This is release 4.0 of 12dicts, released Jan. 18, 2003. It differs from
previous versions by containing three additional lists which are not
derived from the "classic" 12dicts sources. Changes to the classic lists
are limited to error corrections.

The classic 12dicts lists

The 12dicts project began as the n-dicts projects, n being a variable
whose value finally stabilized as 12. The purpose of the project was to
create a list of words approximating the common core of the vocabulary
of American English.

The methodology of the project was to record and correlate the words
listed in a number of small dictionaries. The number of dictionaries so
recorded is now 12, comprising 8 ESL (English as a Second Language)
dictionaries and 4 "desk dictionaries". The dictionaries chosen vary
widely by publisher, by style, by completeness and by depth. In this
version of 12dicts, all of them are dictionaries of American English
(three from British publishers). The smallest of them contains about
20,000 entries, and the largest 46,000. (All totaled, there are about
75,000 entries, many of which appear in only a single dictionary.) All
but two of them were published in the last seven years.

The 6of12 and 2of12 lists

I initially tried two different ways of winnowing the 12dicts data to
produce lists of common words. Both produced interesting results. One
list, the 6of12 list, contains all words and phrases listed in 6 of the
12 dictionaries. One way of describing this list is that it contains
those words and phrases which a (seeming) majority of lexicographers
believe are relevant to people learning English, and/or to everyday
usage. This list contains about 32,000 words and phrases. The other
list, the 2of12 list, is more inclusive in that it includes words listed
in as few as two of the source dictionaries, but less inclusive in that
it excludes items of various sorts, including multiword phrases, proper
names and abbreviations. This list contains about 41,000 words. It is
perhaps more suitable for use in areas like spell checking or word games
than the 6of12 list. (Honesty compels me to admit that neither of these
lists is, by itself, a good choice for spell checking, due to the
absence of inflections, proper names, Roman numerals, etc.)

A third list, 2of12inf.txt, developed later, is of a rather different
character, and is discussed in a later section.

A more precise description of the criteria by which the above lists were
composed is as follows:

6of12 list word selection

  * The 6of12 list contains all non-excluded words and phrases which
    appear in 6 or more of the source dictionaries.
  * Prefixes and suffixes are excluded. Abbreviations are included;
    however, if they are entirely lower-case and alphabetic, they are
    terminated with a colon (":") so they can be easily distinguished
    from regular words.
  * Inflections of included words are not themselves included unless
    they are separately defined or irregular.
  * It sometimes occurs that a word is listed in several forms (e.g.,
    with and without hyphenation) in 6 or more dictionaries, even though
    no single form is so listed. In this case, if one spelling is
    clearly more accepted, this spelling and this spelling only is
    listed. If all spellings seem equally accepted, one spelling has
    been selected arbitrarily for inclusion.
  * The 6of12 list contains a significant number of words which do not
    meet either criterion 1 or 4 above. These words, sometimes called
    "signature words", are discussed below. All of these words are
    listed in at least one of the source dictionaries.
  * In addition to the ":" suffix discussed above, other special suffix
    characters are used to mark words with certain characteristics, as
    discussed below.

2of12 list word selection

  * The 2of12 list contains all non-excluded words which appear in at
    least 2 of the source dictionaries.
  * This list excludes capitalized words, multiword phrases, and
    abbreviations, as well as prefixes and suffixes. It does not exclude
    hyphenated words or contractions. If a word occurs in both a
    hyphenated and an unhyphenated form, the unhyphenated form is
    listed, even if the hyphenated form is generally preferred.
  * The list excludes spellings which are considered (by a majority of
    the dictionaries listing it) to be non-American usage. It also
    excludes secondary spellings which are mentioned by fewer than four
    of the source dictionaries.
  * Inflections of included words are not themselves included unless
    they are separately defined, or irregular.
  * Several of the source dictionaries include listings for obscure
    currencies, such as ringgit, khoum and ngwee. I was unable to regard
    such words as part of the English "core vocabulary", and so I
    required citation in over a third of the dictionaries for inclusion
    of monetary units. A side-effect was the elimination of the word 
    lepton, which, in addition to its use in particle physics, is also
    .01 Greek drachmas.
  * This list also includes a small number of signature words, as
    discussed below.

Signature words

As indicated, both lists have been augmented with words (and, in the
case of the 6of12 list, phrases) which fail to meet the formal
requirements for inclusion. In the case of the 6of12 list, 1024 words
were added (about 3 % of the total). These are all words which, in the
judgment of the compiler, are as familiar as many of the words which met
the criteria for inclusion. Examples of some of the sorts of words which
were added are:

  * Words of the same category as other included words. An example is
    the astrological sign Cancer, which alone of all the astrological
    signs fails to appear in 6 or more of the dictionaries. Similarly
    added were the omitted holidays Thanksgiving and Christmas Eve.
  * Vulgarities, sexual terms and insults. Some such words were already
    included, but most of the source dictionaries were quite squeamish
    about them. These words are very widely known indeed; I hold that
    any list of "common" words which does not include the infamous
    f-word is simply discredited thereby. Some may feel that it would
    have been better to leave some or all of these terms unmentioned.
    Nevertheless, the expression of blasphemy, unwarranted contempt and
    perverse lust, whether in words or in deeds, is a very human trait.
    Suppressing the evidence of these aspects of the human condition in
    our language makes no more sense than excluding leprosy, gangrene
    and dementia, no matter how unpleasant they may be to contemplate.
  * Conventional conversational phrases so common as to be practically
    invisible to native speakers. Examples are thank you, good night,
    uh-huh, of course and gesundheit.
  * Sports terminology, especially for football and baseball. (If I, who
    am practically sports-blind, noticed this deficiency, it must be of
    major proportions indeed.)

Note that the signature words in the 6of12 list can be identified via
the suffix character "+", and eliminated if desired.

A much smaller set of words (49) was added to the 2of12 list. These were
of two sorts:

  * Signature words from the 6of12 list which were not already present
    in the 2of12 list, and which are not excluded due to being
    abbreviations, phrases, etc.
  * Inflections of irregular verbs not explicitly mentioned in 2 source
    dictionaries, such as outfought and reheard.

Annotations

Some of the 6of12 list entries are annotated with a suffix character,
giving additional information about the associated word. The annotations
can be easily removed with an editor or script if they are unwanted.

These annotations are:

: The word is an otherwise unmarked abbreviation. This suffix may appear
  in combination with another suffix.                                   
& The word is primarily a non-American usage.                           
# The word is generally held to be a variant or less preferred form of  
  another word.                                                         
< This form of a word is held to be the primary form by fewer           
  dictionaries than some other form of the word.                        
^ This form of the word was selected arbitrarily from a set of variants,
  none of which was clearly preferred.                                  
= Roughly, this indicates a "second class" word, as described below.    
+ The word is a signature word.                                         

The reasons a word might be marked with the = annotation are:

  * The word is an inflection which was defined in the same entry as the
    base word.
  * The word is a derived word (-ly, -ness or -er/or) which was not
    defined in a separate entry.
  * The word appeared in a list of undefined words with a common prefix,
    such as un- or re-.

The words in the 2of12 list are not annotated.

The 2of12inf list

The 2of12inf list is of a rather different character from the two
original "classic" lists. Conceptually, it is simple. It consists of all
the words in the 2of12 list, plus their inflections, amounting to about
81,000 words. This list may be more useful than the other lists for
applications like word games. It was created to help Kevin Atkinson in
his Aspell and SCOWL projects (for which, follow this link). Unlike the
6of12 and 2of12 lists, this list is not based exclusively on the
contents of my 12 source dictionaries, and for this reason it has, I
feel, less authority than the other classic 12dicts lists. It also
probably has a significantly higher error rate than the other lists, for
reasons explained below.

The criteria defining the 2of12inf list are as follows:

  * The 2of12inf list contains all non-excluded words which appear in at
    least 2 of the source dictionaries.
  * This list excludes capitalized words, multiword phrases,
    abbreviations, contractions, hyphenated words and single-letter
    words, as well as prefixes and suffixes.
  * The list does not exclude secondary spellings, non-American usages
    or monetary units.
  * The list includes inflections of all included words. Any inflection
    mentioned or clearly implied by any of the source dictionaries is
    included (i.e., two citations are not required). Additionally, some
    inflections have been added from other sources.
  * Plurals of "uncountable" nouns were included, annotated with the "%"
    suffix character. See below for an extended discussion of the
    inclusion of these words.
  * Signature words from the other lists, plus their inflections, were
    added. No other signature words were added.

Though the 2of12inf list still consists mostly of very common words,
criteria 3 through 5 above cause the 2of12inf list to contain a greater
proportion of unfamiliar and unusual words than the other classic
12dicts lists.

The 2of12inf list was not derived directly from the 12 source
dictionaries. The starting point was a subset of Kevin Atkinson's AGID
list, a list of words, parts of speech and inflections derived from
public-domain sources, notably Moby Words and WordNet. (See the file
agid.txt in the 12dicts archive, which is a copy of the AGID "readme",
for more information on the antecedents of AGID.) 2of12inf was created
by a process of editing the AGID subset to remove spurious entries and
those which reflected a more esoteric English vocabulary than the other
12dicts lists, and to add inflections which AGID failed to identify.
This process required significantly less effort than would have been
needed to derive the list directly from the source dictionaries.
Unfortunately, a side effect of the process is that the result is likely
to be somewhat less reliable than the other 12dicts lists. In
particular, Moby Words is notoriously unreliable, and I find it unlikely
that I have successfully identified all the spurious inflections its use
has introduced. It is my hope in the future to release another edition
of 2of12inf which is not derived from AGID, and therefore not "infected"
by Moby Words.

After the first version of the 2of12inf list was released, I replaced
one of the source dictionaries, officially an international dictionary
but in actuality rather British in its orientation, with a more American
dictionary by the same publisher. It was not practical (nor necessarily
desirable) for me to go through the list removing inflections endorsed
only by the superseded dictionary. For this reason, the 2of12inf list
has a slightly more international character than the other 12dicts
lists. It is not altogether clear that this is a bad thing.

Selection of inflections

Ideally, the 2of12inf list would contain only inflections listed in one
of the 12dicts source dictionaries. This proved not to be practical. The
reason for this has to do with the nature of these sources, which are
mostly ESL dictionaries. An ESL dictionary might well list the word 
esophagus, but, because an English learner is unlikely to need to talk
about this organ in the plural, it will probably not bother to list the
plural form esophagi. For words of this sort, I therefore needed to
obtain their inflections from other sources. Obviously, the decisions on
when to include additional inflections were judgment calls, as were the
choices of which inflections to add.

Adjectival inflections (comparatives and superlatives) proved to be an
especially annoying problem. Only 2 of my 12 source dictionaries
provided remotely reliable information of this sort. In fact, such
information is sparse and inconsistent in most dictionaries of any size.
I relied on a small set of additional dictionaries for this information,
which was mostly disjoint from the sources for plurals and verb forms.
Several of these sources were Scrabble(r)-related, and therefore
inclined to include forms of little plausibility such as iller/illest or
fertiler/fertilest. Accordingly, I ended up rejecting some of the
documented inflections on grounds of implausibility. I have no doubt
that, in the process, I made a number of errors of both inclusion and
exclusion and, in any case, many of the forms listed have no connection
with any of the 12dicts source dictionaries.

One additional problem in the creation of the 2of12inf list was that of
"uncountable" nouns and their plurals. Some English dictionaries,
especially ESL dictionaries, as well as other linguistic sources attest
to the existence of nouns which cannot be counted, or used in the
plural. Examples of such nouns include mud, rayon, oregano, chess,
fairness, wisdom, aluminum, training, materialism and chickenpox. This
is an entirely commonsense notion, but a difficulty is the fact that the
boundary between the countable and the uncountable is extremely vague
and ill-defined. For example, the word coffee is ordinarily uncountable,
but not when ordering in a restaurant, as is the word symmetry, except
in physics or math. In general, it is possible to contrive a context
where use of the plural of any noun whatsoever is reasonable.

An alternate position, therefore, is that in fact no nouns are
uncountable, and that any noun which is not already plural possesses a
plural. This position is especially useful in the context of word games,
where words such as zeals and anthraxes may produce large scores. For
this reason, the official Scrabble dictionaries list words such as 
thens, onces and mankinds, which most people find rather implausible.
The fact that the 2of12inf list might well be useful in gaming contexts,
together with the fact that the boundary between countable and
uncountable nouns is so ill-defined, served as a powerful argument for
inclusion of all plural forms, whether commonly used or not, while its
derivation from ESL sources argued for including only the plurals of
countable nouns, however distinguished.

In the end, I was unable to resolve this dilemma, and adopted a
compromise. The 2of12inf list includes all plurals, but with the plurals
of uncountable nouns marked, making it easy to remove them if they are
not wanted. That left the issue of how to establish countability. Six of
my source dictionaries included information on countability, which was
adequate to decide the status of most of the included nouns. As for the
rest, as usual, I used my best judgment. I will confess to occasionally
overriding the source dictionaries when I believed they were clearly
incorrect. (For instance, I chose not to mark the word hatreds as an
uncountable plural, in defiance of the opinion of all my sources, on the
grounds that it has been used in too many news stories from Bosnia to be
considered unusual.) It is interesting to note that most of the plurals
I added from auxiliary sources were of words considered uncountable.

The difficulties listed above, and the fact that I was forced to
exercise personal judgment frequently in creating it, emphasizes a
fundamental difference between this list and the other classic 12dicts
lists. I have tried to make the 6of12 and 2of12 lists reflect only the
source dictionaries, and to keep my own judgments and opinions out of
the picture (except for my addition of signature words). This has proved
impossible to achieve for the 2of12inf list, which accordingly
represents a less authoritative and more arbitrary collection.
Additionally, the 2of12inf list has undergone less proofreading and
validation than the other lists, and I suspect the error rate is
considerably higher than the idealistic goal of 0.02 % I advocate
elsewhere in this document. Nevertheless, I hope it may prove to be of
some use and interest.

I wish to offer my special thanks to Kevin Atkinson, for supplying me
with the AGID list, and for encouraging me to add the inflections. Of
course, any errors that remain in the 2of12inf list are my own
responsibility, and should not be blamed on Kevin, AGID, or even on
Moby.

The 3esl list

The 3esl list represents another attempt to produce an English "core
vocabulary" list. It is about 2/3 of the size of the 6of12 list, which
it resembles in terms of the sorts of words included.

The 3esl list is a far more subjective list than any of the classic
12dicts lists. It was compiled from 3 small ESL dictionaries, using the
same criteria for eligibility as the 6of12 list. I started with a list
composed of all words from the smallest of the 3 sources, plus all words
contained in both of the others. This list was then edited in the
following ways:

 1. I removed alternate spellings for included words, such as grey and 
    off-stage. I also removed very similar synonyms for the same
    concept, for instance, removing cable television as a duplicate of 
    cable TV.
 2. I added one form of each word which would have been included if the
    sources had agreed on spelling, such as shortchange and back seat.
 3. I removed some words which were present in the smallest of the
    sources but seemed too esoteric, such as the symbols of chemical
    elements. I did this only for words which were not present in the
    other sources.
 4. I added some words which were present in only one of the two larger
    sources, but which seemed appropriate to add. These words were
    frequently of the sort added to the 6of12 list as signature words,
    as well as some inflections that often function as words with
    meanings of their own, such as comforting and notes.

All of these changes were quite subjective in nature, and quite
numerous. Probably more than 10 % of the candidate words were added or
removed in this way. For this reason, it is pointless to speak of
signature words for this list; the composition of the list is too
arbitrary for the term to make any sense. (I will note that the list is
still not entirely arbitrary, as I added only words found in some form
in one of the sources, and removed no words present in two of the
sources other than duplicates. Thus, words like front page were not
added, no matter how familiar, and words such as lugubrious were not
removed, despite clearly not being part of any "core vocabulary".)

Like the 6of12 list, the 3esl list marks lower-case abbreviations with a
":" suffix, to prevent them from being mistaken for regular English
words.

One final note on this list. The 3esl list contains about 1500 words not
present in the 6of12 list. Because these two lists have the same rules
for the kinds of words included, one could easily combine the two to
produce a slightly larger list including a number of words whose
omission from 6of12 is rather surprising. Be warned that in a few cases,
the spelling chosen for words with multiple spellings is different in
the two lists, and I would recommend that the duplicates be removed.
(I'll be happy to provide a list of the duplicates if anyone wants one.)

The 2of4brif list

All of the classic 12dicts lists are unabashedly oriented towards
American English. I've received a few expressions of interest in a
British English list. The result is the 2of4brif list. This list was
compiled from 4 large "international" ESL dictionaries, published by
British publishers. To this American, they are more British than they
are international; quite possibly, they seem more American than
international to British readers. It is interesting to note that,
although there were only a third as many sources for this list as for
the 12dicts lists, these dictionaries resembled each other far more
closely than their American counterparts, which could mean that the
2of4brif list is as good an approximation of a "core" British English
vocabulary as the 6of12 list is for American English. (Or, alternately,
it may simply mean that my choice of sources was too narrow.)

This criteria for inclusion in this list were basically those of the
2of12inf list. In particular, inflections are included for all words,
but hyphenated words, contractions, phrases, proper names and
abbreviations are all excluded. One important difference between the two
is the way in which inflections were determined for inclusion. The
2of12inf list includes some inflections found in one (or even none) of
its sources. Further, as discussed in detail above, it includes plurals
for words which are not normally considered to have plurals. The
2of4brif list differs in both of these regards. It includes only
inflections endorsed by two or more of the sources, specifically
excluding any plural forms for nouns listed as uncountable.

The 2of4brif list includes no signature words as such. I made a small
number of adjustments for consistency, such as making sure that -ise and
-ize spellings were equally represented, and adding plurals for ordinal
numbers. (Why fourteenth would be defined as a fraction, but not 
seventeenth, I must simply regard as a mystery.) These edits were so
few, and so clearly harmless, that I have not marked them.

Prospective users of the 2of4brif list should realize that it was
compiled by an American. If my sources contained any glaring errors (and
most dictionaries have a few), I might well not have noticed, and
perpetuated them in the list. The fact that two citations were required
is some protection against such an event, but no guarantee.

As the 2of4brif list is very similar in makeup to the 2of12inf list, a
user who wants a larger, more international list than either could
reasonably merge the two. If you do this, you should remove the unusual
plurals (marked with a "%") from the 2of12inf list in the process, for
consistency.

The 5desk list

I created the 5desk list in an attempt to do a better /usr/dict/words
(about which I offer many harsh criticisms elsewhere in this document).
The sorts of words admitted are the same sorts that /usr/dict/words
contains. Though somewhat larger in size than most versions of /usr/dict
/words, this is still a short word list, striving for inclusion of words
one is likely to encounter rather than the complete jargon of every
possible scientific, artistic or occult endeavor.

5desk was assembled primarily from five "desk dictionaries". It was
augmented by words from five minor sources, including a "vocabulary
builder" and a collection of proper names. The list excludes prefixes,
suffixes, phrases, hyphenated words, contractions and most abbreviations
and acronyms. There was no requirement for multiple listings; all
qualifying words from each of the sources were included. Inflections of
included words were not included themselves except when irregular, or
separately defined. Variant and non-American spellings were not
excluded, and no signature words were added.

Words commonly considered to be abbreviations/acronyms were included if
they contained at least one upper case character, and were defined with
an explicit part of speech. This excluded items like Mr and Feb, which
are abbreviations in the classic sense, but allowed words like DNA and 
ATM, which are used far more frequently than that which they abbreviate.
While there is a trend in modern dictionaries to list such words as
nouns (or occasionally verbs, adverbs, etc.), it is a trend in progress,
and rather inconsistently applied. For this reason, the set of such
words in the 5desk list is somewhat incoherent, including SPCA but not 
PETA, AIDS but not SIDS, KGB but not CIA, and PDQ but not ASAP.

One class of commonly-used words is regrettably absent from the 5desk
list, because I was unable to find a satisfactory source for them. This
is the class of commercial names such as Exxon, Tylenol, Pepsi and Chevy
. This is probably forgivable, as this class of names is as ephemeral
and transitory as teenage slang. The one-time household words Kool,
Ovaltine, Philco and Ipana serve now only as answers to trivia
questions, with modern wonders like Starbucks, Google, Ritalin and TiVo
taking their place on the tongues of the trendy.

The 5desk list has clearly moved beyond any "core vocabulary" concept.
It includes quite esoteric words (ogee, pleonastic), very uncommon
spellings (thiamine, yuppy), and obscure geographical and historical
names (Paricutin, Nevelson). Like /usr/dict/words, it is frequently
inconsistent and arbitrary, but I hope at the least I have avoided
including spelling errors, and overlooking the stuff of everyday
conversation. Perhaps it will be useful as a compromise between basic
lists such as 3esl, and truly massive lists like Mendel Cooper's ENABLE.

How 12dicts came to be

It may have occurred to some to wonder about how something like the
n-dicts project came to be (though I assume that anyone who bothers to
download this archive must already have some idea that such a project
could be of interest).

Some years ago, there was a post to the sci.crypt Usenet newsgroup, on
the subject of creating PGP passphrases using randomly selected entries
from a supplied list of very short words. (If this sounds interesting,
follow this link for an expanded version of the post.) The word list,
which was extracted from /usr/dict/words on some UNIX system, seemed to
me ill-suited to its intended purpose. It included arcane acronyms (
bstj, fmc), misspellings (diety, ouvre) and words of amazing obscurity (
bhoy, kombu). I decided I could do better (and eventually did). This
caused me to start downloading English word lists, of which there are
many, from the Internet. I was not impressed by the overall quality of
these lists, and the few which were high-quality were all-inclusive,
burying the everyday words under a mountain of archaisms and esoterica.
The flaws of the vast majority of these lists are worth recounting:

  * Failure to proofread. Many of these lists are littered with
    misspellings and typos, sometimes approaching gibberish. (I presume,
    for instance, that the bizarre string nondploe, which was found in a
    purported Scrabble word list, is a typo for something more or less
    legitimate, but I have no idea what.) Working on my own lists has
    helped me understand that 100 % accuracy is a very demanding goal,
    seldom actually achieved, but I still feel it reasonable to expect
    no more than 1 or 2 errors per 10,000 words.
  * Acceptance of completely undocumented lazy spellings, such as 
    bullseye and courtmartial.
  * Failure to respect capitalization.
  * Failure to distinguish abbreviations from other entries.
  * Treating esoteric computer jargon, and especially UNIX jargon, as
    everyday English. (Beware any list which includes bitblt, emacs,
    inode or lvalue.)
  * Apparently random word selection. For instance, the most common
    version of /usr/dicts/words contains a large set of apparently
    randomly chosen personal names (uncapitalized, of course, and
    missing wanda, marge, polly and sid).
  * Inconsistent inflection. Some lists include all inflections of their
    vocabulary, while others include only singulars and infinitives.
    Either policy is fine, and has its advantages. I am personally very
    annoyed when inflected forms appear at random. I find this generally
    happens when a compiler merges several lists with different
    characteristics, with no attempt to reconcile their divergent
    styles.
  * Omission of everyday words. I've seen a purported general-purpose
    list that includes bremsstrahlung, yet omits log and beer. Or that
    includes saxophone but not sax, and rhinoceros but not rhino. Of
    course, due to my original purpose in seeking out common short
    words, I found this especially annoying.

One result of my frustration with this situation was my working with
Mendel Cooper on ENABLE (for further information, check out this link),
which was close to unique in having an active caretaker, one clearly
concerned with quality, and in being oriented towards American rather
than British English. But ENABLE is an all-encompassing list and, even
if it had been complete at the time I started my search for a list of
common words, it would not have been what I wanted for that reason.

I finally decided that only starting from scratch with a systematic
approach was likely to get me what I was looking for, and that
dictionaries intended for non-native speakers of English were the best
possible source for words that are in some cases so familiar that we
never think of them. This has led to the 12dicts lists, which I hope
have managed to avoid the flaws recited above.

(I should acknowledge one form of inconsistency exhibited by the 12dicts
lists, which is that sometimes related words are spelled inconsistently.
For instance, the 2of12 list contains both broadminded and 
broad-mindedness. This generally occurs as a result of the methodology
used to build the lists. In the case of broadminded, only one source
dictionary listed broadmindedness, which was therefore excluded. I felt
unequal to trying to correct these inconsistencies, some of which are
real and not mere artifacts of 12dicts, such as the contrast between 
self-conscious and unselfconscious.)

Conclusions

When I released the first version of 12dicts in 1999, I assumed I was
done with it. It hasn't worked out that way. Before I declare it
finished for a second time, there are a few more things I'd like to
accomplish.

  * As mentioned above, I would like to rework the 2of12inf list to
    remove the dependency on the Moby lists.
  * As may be seen by inspecting the table of file characteristics, the
    12dicts files now form a spectrum of word lists, with contents
    ranging from the extremely common to the mildly esoteric. I would
    like to extend the spectrum further by applying the 12dicts
    methodology to dictionaries of larger size. Whether I will ever get
    the time for a project this large remains to be seen. If it ever
    comes to pass, it will probably be released separately from 12dicts
    itself, as anything larger than the 5desk list will be too large to
    even pretend to represent a "core English" vocabulary. (Even the
    5desk list itself is too large for that purpose.)
  * It is possible that in the future the "n" of n-dicts will increase
    again, but, in fact, consideration of an additional dictionary now
    generally ends with the discovery that its vocabulary matches
    12dicts pretty closely. At the very least, this phenomenon gives me
    hope that the 12dicts lists have now fulfilled their basic purpose.

The 12dicts lists were compiled by Alan Beale. I explicitly release them
to the public domain, but request acknowledgment of their use.
(Actually, the dependency of the 2of12inf list on AGID prevents its
release into the public domain. However, I do not impose any additional
requirements on its use beyond those imposed by AGID and its sources, as
described in agid.txt.) Feel free to send comments, suggestions,
inquiries and/or large sums of money to me at biljir@pobox.com. If you
find 12dicts useful, I'd love to hear about it.