--- /dev/null
+Introduction
+
+12dicts is a collection of English word lists. It differs in several
+important ways from most of the other free word lists you can download.
+
+ * The 12dicts lists are oriented towards common words. If you're
+ looking for myriads of archaic, scientific or computer jargon words,
+ you should look elsewhere.
+ * The 12dicts lists have been rigorously checked for errors. (This is
+ not to say that they are error-free, merely that enough care has
+ been taken that errors are rather infrequent.)
+ * 12dicts contains a variety of lists, of different sizes and
+ characteristics. One size does not fit all. Because each list has
+ different characteristics, I do not recommend combining them, except
+ as noted below.
+
+Originally, 12dicts was composed of lists derived from a specific set of
+12 source dictionaries. In addition to these "classic" lists, 12dicts
+now includes lists derived from other sources. It would perhaps be
+appropriate to rename 12dicts to something more generic, such as BAWL
+(Beale's Assorted Word Lists), but I have not done so in order to
+preserve continuity.
+
+A quick summary of the 12dicts lists and their characteristics is as
+follows:
+
++---------------------------------------------------------+
+| |3esl |6of12|2of12|2of4brif|5desk|2of12inf|
+|---------------+-----+-----+-----+--------+-----+--------|
+|Size |21877|32153|41236|60387 |61406|81520 |
+|---------------+-----+-----+-----+--------+-----+--------|
+|Abbreviations |Y |Y |N |N |N |N |
+|---------------+-----+-----+-----+--------+-----+--------|
+|Acronyms |Y |Y |N |N |Y |N |
+|---------------+-----+-----+-----+--------+-----+--------|
+|British English|N |N |N |Y |N |N |
+|---------------+-----+-----+-----+--------+-----+--------|
+|Hyphenations |Y |Y |Y |N |N |N |
+|---------------+-----+-----+-----+--------+-----+--------|
+|Inflections |N |N |N |Y |N |Y |
+|---------------+-----+-----+-----+--------+-----+--------|
+|Names |Y |Y |N |N |Y |N |
+|---------------+-----+-----+-----+--------+-----+--------|
+|Phrases |Y |Y |N |N |N |N |
++---------------------------------------------------------+
+
+The remainder of this document is organized as follows:
+
+ * This release
+ * The classic 12dicts lists
+ + The 6of12 and 2of12 lists
+ + The 2of12inf list
+ * The 3esl list
+ * The 2of4brif list
+ * The 5desk list
+ * How 12dicts came to be
+ * Conclusions
+
+This release
+
+This is release 4.0 of 12dicts, released Jan. 18, 2003. It differs from
+previous versions by containing three additional lists which are not
+derived from the "classic" 12dicts sources. Changes to the classic lists
+are limited to error corrections.
+
+The classic 12dicts lists
+
+The 12dicts project began as the n-dicts projects, n being a variable
+whose value finally stabilized as 12. The purpose of the project was to
+create a list of words approximating the common core of the vocabulary
+of American English.
+
+The methodology of the project was to record and correlate the words
+listed in a number of small dictionaries. The number of dictionaries so
+recorded is now 12, comprising 8 ESL (English as a Second Language)
+dictionaries and 4 "desk dictionaries". The dictionaries chosen vary
+widely by publisher, by style, by completeness and by depth. In this
+version of 12dicts, all of them are dictionaries of American English
+(three from British publishers). The smallest of them contains about
+20,000 entries, and the largest 46,000. (All totaled, there are about
+75,000 entries, many of which appear in only a single dictionary.) All
+but two of them were published in the last seven years.
+
+The 6of12 and 2of12 lists
+
+I initially tried two different ways of winnowing the 12dicts data to
+produce lists of common words. Both produced interesting results. One
+list, the 6of12 list, contains all words and phrases listed in 6 of the
+12 dictionaries. One way of describing this list is that it contains
+those words and phrases which a (seeming) majority of lexicographers
+believe are relevant to people learning English, and/or to everyday
+usage. This list contains about 32,000 words and phrases. The other
+list, the 2of12 list, is more inclusive in that it includes words listed
+in as few as two of the source dictionaries, but less inclusive in that
+it excludes items of various sorts, including multiword phrases, proper
+names and abbreviations. This list contains about 41,000 words. It is
+perhaps more suitable for use in areas like spell checking or word games
+than the 6of12 list. (Honesty compels me to admit that neither of these
+lists is, by itself, a good choice for spell checking, due to the
+absence of inflections, proper names, Roman numerals, etc.)
+
+A third list, 2of12inf.txt, developed later, is of a rather different
+character, and is discussed in a later section.
+
+A more precise description of the criteria by which the above lists were
+composed is as follows:
+
+6of12 list word selection
+
+ * The 6of12 list contains all non-excluded words and phrases which
+ appear in 6 or more of the source dictionaries.
+ * Prefixes and suffixes are excluded. Abbreviations are included;
+ however, if they are entirely lower-case and alphabetic, they are
+ terminated with a colon (":") so they can be easily distinguished
+ from regular words.
+ * Inflections of included words are not themselves included unless
+ they are separately defined or irregular.
+ * It sometimes occurs that a word is listed in several forms (e.g.,
+ with and without hyphenation) in 6 or more dictionaries, even though
+ no single form is so listed. In this case, if one spelling is
+ clearly more accepted, this spelling and this spelling only is
+ listed. If all spellings seem equally accepted, one spelling has
+ been selected arbitrarily for inclusion.
+ * The 6of12 list contains a significant number of words which do not
+ meet either criterion 1 or 4 above. These words, sometimes called
+ "signature words", are discussed below. All of these words are
+ listed in at least one of the source dictionaries.
+ * In addition to the ":" suffix discussed above, other special suffix
+ characters are used to mark words with certain characteristics, as
+ discussed below.
+
+2of12 list word selection
+
+ * The 2of12 list contains all non-excluded words which appear in at
+ least 2 of the source dictionaries.
+ * This list excludes capitalized words, multiword phrases, and
+ abbreviations, as well as prefixes and suffixes. It does not exclude
+ hyphenated words or contractions. If a word occurs in both a
+ hyphenated and an unhyphenated form, the unhyphenated form is
+ listed, even if the hyphenated form is generally preferred.
+ * The list excludes spellings which are considered (by a majority of
+ the dictionaries listing it) to be non-American usage. It also
+ excludes secondary spellings which are mentioned by fewer than four
+ of the source dictionaries.
+ * Inflections of included words are not themselves included unless
+ they are separately defined, or irregular.
+ * Several of the source dictionaries include listings for obscure
+ currencies, such as ringgit, khoum and ngwee. I was unable to regard
+ such words as part of the English "core vocabulary", and so I
+ required citation in over a third of the dictionaries for inclusion
+ of monetary units. A side-effect was the elimination of the word
+ lepton, which, in addition to its use in particle physics, is also
+ .01 Greek drachmas.
+ * This list also includes a small number of signature words, as
+ discussed below.
+
+Signature words
+
+As indicated, both lists have been augmented with words (and, in the
+case of the 6of12 list, phrases) which fail to meet the formal
+requirements for inclusion. In the case of the 6of12 list, 1024 words
+were added (about 3 % of the total). These are all words which, in the
+judgment of the compiler, are as familiar as many of the words which met
+the criteria for inclusion. Examples of some of the sorts of words which
+were added are:
+
+ * Words of the same category as other included words. An example is
+ the astrological sign Cancer, which alone of all the astrological
+ signs fails to appear in 6 or more of the dictionaries. Similarly
+ added were the omitted holidays Thanksgiving and Christmas Eve.
+ * Vulgarities, sexual terms and insults. Some such words were already
+ included, but most of the source dictionaries were quite squeamish
+ about them. These words are very widely known indeed; I hold that
+ any list of "common" words which does not include the infamous
+ f-word is simply discredited thereby. Some may feel that it would
+ have been better to leave some or all of these terms unmentioned.
+ Nevertheless, the expression of blasphemy, unwarranted contempt and
+ perverse lust, whether in words or in deeds, is a very human trait.
+ Suppressing the evidence of these aspects of the human condition in
+ our language makes no more sense than excluding leprosy, gangrene
+ and dementia, no matter how unpleasant they may be to contemplate.
+ * Conventional conversational phrases so common as to be practically
+ invisible to native speakers. Examples are thank you, good night,
+ uh-huh, of course and gesundheit.
+ * Sports terminology, especially for football and baseball. (If I, who
+ am practically sports-blind, noticed this deficiency, it must be of
+ major proportions indeed.)
+
+Note that the signature words in the 6of12 list can be identified via
+the suffix character "+", and eliminated if desired.
+
+A much smaller set of words (49) was added to the 2of12 list. These were
+of two sorts:
+
+ * Signature words from the 6of12 list which were not already present
+ in the 2of12 list, and which are not excluded due to being
+ abbreviations, phrases, etc.
+ * Inflections of irregular verbs not explicitly mentioned in 2 source
+ dictionaries, such as outfought and reheard.
+
+Annotations
+
+Some of the 6of12 list entries are annotated with a suffix character,
+giving additional information about the associated word. The annotations
+can be easily removed with an editor or script if they are unwanted.
+
+These annotations are:
+
+: The word is an otherwise unmarked abbreviation. This suffix may appear
+ in combination with another suffix.
+& The word is primarily a non-American usage.
+# The word is generally held to be a variant or less preferred form of
+ another word.
+< This form of a word is held to be the primary form by fewer
+ dictionaries than some other form of the word.
+^ This form of the word was selected arbitrarily from a set of variants,
+ none of which was clearly preferred.
+= Roughly, this indicates a "second class" word, as described below.
++ The word is a signature word.
+
+The reasons a word might be marked with the = annotation are:
+
+ * The word is an inflection which was defined in the same entry as the
+ base word.
+ * The word is a derived word (-ly, -ness or -er/or) which was not
+ defined in a separate entry.
+ * The word appeared in a list of undefined words with a common prefix,
+ such as un- or re-.
+
+The words in the 2of12 list are not annotated.
+
+The 2of12inf list
+
+The 2of12inf list is of a rather different character from the two
+original "classic" lists. Conceptually, it is simple. It consists of all
+the words in the 2of12 list, plus their inflections, amounting to about
+81,000 words. This list may be more useful than the other lists for
+applications like word games. It was created to help Kevin Atkinson in
+his Aspell and SCOWL projects (for which, follow this link). Unlike the
+6of12 and 2of12 lists, this list is not based exclusively on the
+contents of my 12 source dictionaries, and for this reason it has, I
+feel, less authority than the other classic 12dicts lists. It also
+probably has a significantly higher error rate than the other lists, for
+reasons explained below.
+
+The criteria defining the 2of12inf list are as follows:
+
+ * The 2of12inf list contains all non-excluded words which appear in at
+ least 2 of the source dictionaries.
+ * This list excludes capitalized words, multiword phrases,
+ abbreviations, contractions, hyphenated words and single-letter
+ words, as well as prefixes and suffixes.
+ * The list does not exclude secondary spellings, non-American usages
+ or monetary units.
+ * The list includes inflections of all included words. Any inflection
+ mentioned or clearly implied by any of the source dictionaries is
+ included (i.e., two citations are not required). Additionally, some
+ inflections have been added from other sources.
+ * Plurals of "uncountable" nouns were included, annotated with the "%"
+ suffix character. See below for an extended discussion of the
+ inclusion of these words.
+ * Signature words from the other lists, plus their inflections, were
+ added. No other signature words were added.
+
+Though the 2of12inf list still consists mostly of very common words,
+criteria 3 through 5 above cause the 2of12inf list to contain a greater
+proportion of unfamiliar and unusual words than the other classic
+12dicts lists.
+
+The 2of12inf list was not derived directly from the 12 source
+dictionaries. The starting point was a subset of Kevin Atkinson's AGID
+list, a list of words, parts of speech and inflections derived from
+public-domain sources, notably Moby Words and WordNet. (See the file
+agid.txt in the 12dicts archive, which is a copy of the AGID "readme",
+for more information on the antecedents of AGID.) 2of12inf was created
+by a process of editing the AGID subset to remove spurious entries and
+those which reflected a more esoteric English vocabulary than the other
+12dicts lists, and to add inflections which AGID failed to identify.
+This process required significantly less effort than would have been
+needed to derive the list directly from the source dictionaries.
+Unfortunately, a side effect of the process is that the result is likely
+to be somewhat less reliable than the other 12dicts lists. In
+particular, Moby Words is notoriously unreliable, and I find it unlikely
+that I have successfully identified all the spurious inflections its use
+has introduced. It is my hope in the future to release another edition
+of 2of12inf which is not derived from AGID, and therefore not "infected"
+by Moby Words.
+
+After the first version of the 2of12inf list was released, I replaced
+one of the source dictionaries, officially an international dictionary
+but in actuality rather British in its orientation, with a more American
+dictionary by the same publisher. It was not practical (nor necessarily
+desirable) for me to go through the list removing inflections endorsed
+only by the superseded dictionary. For this reason, the 2of12inf list
+has a slightly more international character than the other 12dicts
+lists. It is not altogether clear that this is a bad thing.
+
+Selection of inflections
+
+Ideally, the 2of12inf list would contain only inflections listed in one
+of the 12dicts source dictionaries. This proved not to be practical. The
+reason for this has to do with the nature of these sources, which are
+mostly ESL dictionaries. An ESL dictionary might well list the word
+esophagus, but, because an English learner is unlikely to need to talk
+about this organ in the plural, it will probably not bother to list the
+plural form esophagi. For words of this sort, I therefore needed to
+obtain their inflections from other sources. Obviously, the decisions on
+when to include additional inflections were judgment calls, as were the
+choices of which inflections to add.
+
+Adjectival inflections (comparatives and superlatives) proved to be an
+especially annoying problem. Only 2 of my 12 source dictionaries
+provided remotely reliable information of this sort. In fact, such
+information is sparse and inconsistent in most dictionaries of any size.
+I relied on a small set of additional dictionaries for this information,
+which was mostly disjoint from the sources for plurals and verb forms.
+Several of these sources were Scrabble(r)-related, and therefore
+inclined to include forms of little plausibility such as iller/illest or
+fertiler/fertilest. Accordingly, I ended up rejecting some of the
+documented inflections on grounds of implausibility. I have no doubt
+that, in the process, I made a number of errors of both inclusion and
+exclusion and, in any case, many of the forms listed have no connection
+with any of the 12dicts source dictionaries.
+
+One additional problem in the creation of the 2of12inf list was that of
+"uncountable" nouns and their plurals. Some English dictionaries,
+especially ESL dictionaries, as well as other linguistic sources attest
+to the existence of nouns which cannot be counted, or used in the
+plural. Examples of such nouns include mud, rayon, oregano, chess,
+fairness, wisdom, aluminum, training, materialism and chickenpox. This
+is an entirely commonsense notion, but a difficulty is the fact that the
+boundary between the countable and the uncountable is extremely vague
+and ill-defined. For example, the word coffee is ordinarily uncountable,
+but not when ordering in a restaurant, as is the word symmetry, except
+in physics or math. In general, it is possible to contrive a context
+where use of the plural of any noun whatsoever is reasonable.
+
+An alternate position, therefore, is that in fact no nouns are
+uncountable, and that any noun which is not already plural possesses a
+plural. This position is especially useful in the context of word games,
+where words such as zeals and anthraxes may produce large scores. For
+this reason, the official Scrabble dictionaries list words such as
+thens, onces and mankinds, which most people find rather implausible.
+The fact that the 2of12inf list might well be useful in gaming contexts,
+together with the fact that the boundary between countable and
+uncountable nouns is so ill-defined, served as a powerful argument for
+inclusion of all plural forms, whether commonly used or not, while its
+derivation from ESL sources argued for including only the plurals of
+countable nouns, however distinguished.
+
+In the end, I was unable to resolve this dilemma, and adopted a
+compromise. The 2of12inf list includes all plurals, but with the plurals
+of uncountable nouns marked, making it easy to remove them if they are
+not wanted. That left the issue of how to establish countability. Six of
+my source dictionaries included information on countability, which was
+adequate to decide the status of most of the included nouns. As for the
+rest, as usual, I used my best judgment. I will confess to occasionally
+overriding the source dictionaries when I believed they were clearly
+incorrect. (For instance, I chose not to mark the word hatreds as an
+uncountable plural, in defiance of the opinion of all my sources, on the
+grounds that it has been used in too many news stories from Bosnia to be
+considered unusual.) It is interesting to note that most of the plurals
+I added from auxiliary sources were of words considered uncountable.
+
+The difficulties listed above, and the fact that I was forced to
+exercise personal judgment frequently in creating it, emphasizes a
+fundamental difference between this list and the other classic 12dicts
+lists. I have tried to make the 6of12 and 2of12 lists reflect only the
+source dictionaries, and to keep my own judgments and opinions out of
+the picture (except for my addition of signature words). This has proved
+impossible to achieve for the 2of12inf list, which accordingly
+represents a less authoritative and more arbitrary collection.
+Additionally, the 2of12inf list has undergone less proofreading and
+validation than the other lists, and I suspect the error rate is
+considerably higher than the idealistic goal of 0.02 % I advocate
+elsewhere in this document. Nevertheless, I hope it may prove to be of
+some use and interest.
+
+I wish to offer my special thanks to Kevin Atkinson, for supplying me
+with the AGID list, and for encouraging me to add the inflections. Of
+course, any errors that remain in the 2of12inf list are my own
+responsibility, and should not be blamed on Kevin, AGID, or even on
+Moby.
+
+The 3esl list
+
+The 3esl list represents another attempt to produce an English "core
+vocabulary" list. It is about 2/3 of the size of the 6of12 list, which
+it resembles in terms of the sorts of words included.
+
+The 3esl list is a far more subjective list than any of the classic
+12dicts lists. It was compiled from 3 small ESL dictionaries, using the
+same criteria for eligibility as the 6of12 list. I started with a list
+composed of all words from the smallest of the 3 sources, plus all words
+contained in both of the others. This list was then edited in the
+following ways:
+
+ 1. I removed alternate spellings for included words, such as grey and
+ off-stage. I also removed very similar synonyms for the same
+ concept, for instance, removing cable television as a duplicate of
+ cable TV.
+ 2. I added one form of each word which would have been included if the
+ sources had agreed on spelling, such as shortchange and back seat.
+ 3. I removed some words which were present in the smallest of the
+ sources but seemed too esoteric, such as the symbols of chemical
+ elements. I did this only for words which were not present in the
+ other sources.
+ 4. I added some words which were present in only one of the two larger
+ sources, but which seemed appropriate to add. These words were
+ frequently of the sort added to the 6of12 list as signature words,
+ as well as some inflections that often function as words with
+ meanings of their own, such as comforting and notes.
+
+All of these changes were quite subjective in nature, and quite
+numerous. Probably more than 10 % of the candidate words were added or
+removed in this way. For this reason, it is pointless to speak of
+signature words for this list; the composition of the list is too
+arbitrary for the term to make any sense. (I will note that the list is
+still not entirely arbitrary, as I added only words found in some form
+in one of the sources, and removed no words present in two of the
+sources other than duplicates. Thus, words like front page were not
+added, no matter how familiar, and words such as lugubrious were not
+removed, despite clearly not being part of any "core vocabulary".)
+
+Like the 6of12 list, the 3esl list marks lower-case abbreviations with a
+":" suffix, to prevent them from being mistaken for regular English
+words.
+
+One final note on this list. The 3esl list contains about 1500 words not
+present in the 6of12 list. Because these two lists have the same rules
+for the kinds of words included, one could easily combine the two to
+produce a slightly larger list including a number of words whose
+omission from 6of12 is rather surprising. Be warned that in a few cases,
+the spelling chosen for words with multiple spellings is different in
+the two lists, and I would recommend that the duplicates be removed.
+(I'll be happy to provide a list of the duplicates if anyone wants one.)
+
+The 2of4brif list
+
+All of the classic 12dicts lists are unabashedly oriented towards
+American English. I've received a few expressions of interest in a
+British English list. The result is the 2of4brif list. This list was
+compiled from 4 large "international" ESL dictionaries, published by
+British publishers. To this American, they are more British than they
+are international; quite possibly, they seem more American than
+international to British readers. It is interesting to note that,
+although there were only a third as many sources for this list as for
+the 12dicts lists, these dictionaries resembled each other far more
+closely than their American counterparts, which could mean that the
+2of4brif list is as good an approximation of a "core" British English
+vocabulary as the 6of12 list is for American English. (Or, alternately,
+it may simply mean that my choice of sources was too narrow.)
+
+This criteria for inclusion in this list were basically those of the
+2of12inf list. In particular, inflections are included for all words,
+but hyphenated words, contractions, phrases, proper names and
+abbreviations are all excluded. One important difference between the two
+is the way in which inflections were determined for inclusion. The
+2of12inf list includes some inflections found in one (or even none) of
+its sources. Further, as discussed in detail above, it includes plurals
+for words which are not normally considered to have plurals. The
+2of4brif list differs in both of these regards. It includes only
+inflections endorsed by two or more of the sources, specifically
+excluding any plural forms for nouns listed as uncountable.
+
+The 2of4brif list includes no signature words as such. I made a small
+number of adjustments for consistency, such as making sure that -ise and
+-ize spellings were equally represented, and adding plurals for ordinal
+numbers. (Why fourteenth would be defined as a fraction, but not
+seventeenth, I must simply regard as a mystery.) These edits were so
+few, and so clearly harmless, that I have not marked them.
+
+Prospective users of the 2of4brif list should realize that it was
+compiled by an American. If my sources contained any glaring errors (and
+most dictionaries have a few), I might well not have noticed, and
+perpetuated them in the list. The fact that two citations were required
+is some protection against such an event, but no guarantee.
+
+As the 2of4brif list is very similar in makeup to the 2of12inf list, a
+user who wants a larger, more international list than either could
+reasonably merge the two. If you do this, you should remove the unusual
+plurals (marked with a "%") from the 2of12inf list in the process, for
+consistency.
+
+The 5desk list
+
+I created the 5desk list in an attempt to do a better /usr/dict/words
+(about which I offer many harsh criticisms elsewhere in this document).
+The sorts of words admitted are the same sorts that /usr/dict/words
+contains. Though somewhat larger in size than most versions of /usr/dict
+/words, this is still a short word list, striving for inclusion of words
+one is likely to encounter rather than the complete jargon of every
+possible scientific, artistic or occult endeavor.
+
+5desk was assembled primarily from five "desk dictionaries". It was
+augmented by words from five minor sources, including a "vocabulary
+builder" and a collection of proper names. The list excludes prefixes,
+suffixes, phrases, hyphenated words, contractions and most abbreviations
+and acronyms. There was no requirement for multiple listings; all
+qualifying words from each of the sources were included. Inflections of
+included words were not included themselves except when irregular, or
+separately defined. Variant and non-American spellings were not
+excluded, and no signature words were added.
+
+Words commonly considered to be abbreviations/acronyms were included if
+they contained at least one upper case character, and were defined with
+an explicit part of speech. This excluded items like Mr and Feb, which
+are abbreviations in the classic sense, but allowed words like DNA and
+ATM, which are used far more frequently than that which they abbreviate.
+While there is a trend in modern dictionaries to list such words as
+nouns (or occasionally verbs, adverbs, etc.), it is a trend in progress,
+and rather inconsistently applied. For this reason, the set of such
+words in the 5desk list is somewhat incoherent, including SPCA but not
+PETA, AIDS but not SIDS, KGB but not CIA, and PDQ but not ASAP.
+
+One class of commonly-used words is regrettably absent from the 5desk
+list, because I was unable to find a satisfactory source for them. This
+is the class of commercial names such as Exxon, Tylenol, Pepsi and Chevy
+. This is probably forgivable, as this class of names is as ephemeral
+and transitory as teenage slang. The one-time household words Kool,
+Ovaltine, Philco and Ipana serve now only as answers to trivia
+questions, with modern wonders like Starbucks, Google, Ritalin and TiVo
+taking their place on the tongues of the trendy.
+
+The 5desk list has clearly moved beyond any "core vocabulary" concept.
+It includes quite esoteric words (ogee, pleonastic), very uncommon
+spellings (thiamine, yuppy), and obscure geographical and historical
+names (Paricutin, Nevelson). Like /usr/dict/words, it is frequently
+inconsistent and arbitrary, but I hope at the least I have avoided
+including spelling errors, and overlooking the stuff of everyday
+conversation. Perhaps it will be useful as a compromise between basic
+lists such as 3esl, and truly massive lists like Mendel Cooper's ENABLE.
+
+How 12dicts came to be
+
+It may have occurred to some to wonder about how something like the
+n-dicts project came to be (though I assume that anyone who bothers to
+download this archive must already have some idea that such a project
+could be of interest).
+
+Some years ago, there was a post to the sci.crypt Usenet newsgroup, on
+the subject of creating PGP passphrases using randomly selected entries
+from a supplied list of very short words. (If this sounds interesting,
+follow this link for an expanded version of the post.) The word list,
+which was extracted from /usr/dict/words on some UNIX system, seemed to
+me ill-suited to its intended purpose. It included arcane acronyms (
+bstj, fmc), misspellings (diety, ouvre) and words of amazing obscurity (
+bhoy, kombu). I decided I could do better (and eventually did). This
+caused me to start downloading English word lists, of which there are
+many, from the Internet. I was not impressed by the overall quality of
+these lists, and the few which were high-quality were all-inclusive,
+burying the everyday words under a mountain of archaisms and esoterica.
+The flaws of the vast majority of these lists are worth recounting:
+
+ * Failure to proofread. Many of these lists are littered with
+ misspellings and typos, sometimes approaching gibberish. (I presume,
+ for instance, that the bizarre string nondploe, which was found in a
+ purported Scrabble word list, is a typo for something more or less
+ legitimate, but I have no idea what.) Working on my own lists has
+ helped me understand that 100 % accuracy is a very demanding goal,
+ seldom actually achieved, but I still feel it reasonable to expect
+ no more than 1 or 2 errors per 10,000 words.
+ * Acceptance of completely undocumented lazy spellings, such as
+ bullseye and courtmartial.
+ * Failure to respect capitalization.
+ * Failure to distinguish abbreviations from other entries.
+ * Treating esoteric computer jargon, and especially UNIX jargon, as
+ everyday English. (Beware any list which includes bitblt, emacs,
+ inode or lvalue.)
+ * Apparently random word selection. For instance, the most common
+ version of /usr/dicts/words contains a large set of apparently
+ randomly chosen personal names (uncapitalized, of course, and
+ missing wanda, marge, polly and sid).
+ * Inconsistent inflection. Some lists include all inflections of their
+ vocabulary, while others include only singulars and infinitives.
+ Either policy is fine, and has its advantages. I am personally very
+ annoyed when inflected forms appear at random. I find this generally
+ happens when a compiler merges several lists with different
+ characteristics, with no attempt to reconcile their divergent
+ styles.
+ * Omission of everyday words. I've seen a purported general-purpose
+ list that includes bremsstrahlung, yet omits log and beer. Or that
+ includes saxophone but not sax, and rhinoceros but not rhino. Of
+ course, due to my original purpose in seeking out common short
+ words, I found this especially annoying.
+
+One result of my frustration with this situation was my working with
+Mendel Cooper on ENABLE (for further information, check out this link),
+which was close to unique in having an active caretaker, one clearly
+concerned with quality, and in being oriented towards American rather
+than British English. But ENABLE is an all-encompassing list and, even
+if it had been complete at the time I started my search for a list of
+common words, it would not have been what I wanted for that reason.
+
+I finally decided that only starting from scratch with a systematic
+approach was likely to get me what I was looking for, and that
+dictionaries intended for non-native speakers of English were the best
+possible source for words that are in some cases so familiar that we
+never think of them. This has led to the 12dicts lists, which I hope
+have managed to avoid the flaws recited above.
+
+(I should acknowledge one form of inconsistency exhibited by the 12dicts
+lists, which is that sometimes related words are spelled inconsistently.
+For instance, the 2of12 list contains both broadminded and
+broad-mindedness. This generally occurs as a result of the methodology
+used to build the lists. In the case of broadminded, only one source
+dictionary listed broadmindedness, which was therefore excluded. I felt
+unequal to trying to correct these inconsistencies, some of which are
+real and not mere artifacts of 12dicts, such as the contrast between
+self-conscious and unselfconscious.)
+
+Conclusions
+
+When I released the first version of 12dicts in 1999, I assumed I was
+done with it. It hasn't worked out that way. Before I declare it
+finished for a second time, there are a few more things I'd like to
+accomplish.
+
+ * As mentioned above, I would like to rework the 2of12inf list to
+ remove the dependency on the Moby lists.
+ * As may be seen by inspecting the table of file characteristics, the
+ 12dicts files now form a spectrum of word lists, with contents
+ ranging from the extremely common to the mildly esoteric. I would
+ like to extend the spectrum further by applying the 12dicts
+ methodology to dictionaries of larger size. Whether I will ever get
+ the time for a project this large remains to be seen. If it ever
+ comes to pass, it will probably be released separately from 12dicts
+ itself, as anything larger than the 5desk list will be too large to
+ even pretend to represent a "core English" vocabulary. (Even the
+ 5desk list itself is too large for that purpose.)
+ * It is possible that in the future the "n" of n-dicts will increase
+ again, but, in fact, consideration of an additional dictionary now
+ generally ends with the discovery that its vocabulary matches
+ 12dicts pretty closely. At the very least, this phenomenon gives me
+ hope that the 12dicts lists have now fulfilled their basic purpose.
+
+The 12dicts lists were compiled by Alan Beale. I explicitly release them
+to the public domain, but request acknowledgment of their use.
+(Actually, the dependency of the 2of12inf list on AGID prevents its
+release into the public domain. However, I do not impose any additional
+requirements on its use beyond those imposed by AGID and its sources, as
+described in agid.txt.) Feel free to send comments, suggestions,
+inquiries and/or large sums of money to me at biljir@pobox.com. If you
+find 12dicts useful, I'd love to hear about it.
projects / deb_pkgs / scowl.git / blobdiff