Spell Checking Oriented Word Lists (SCOWL)
-Revision 6
-August 10, 2004
-by Kevin Atkinson
+Revision 7.1 (SVN Revision 161)
+January 6, 2011
+by Kevin Atkinson (kevina@gnu.org)
The SCOWL is a collection of word lists split up in various sizes, and
other categories, intended to be suitable for use in spell checkers.
However, I am sure it will have numerous other uses as well.
-The latest version can be found at http://wordlist.sourceforge.net/
+The latest version can be found at http://wordlist.sourceforge.net/.
The directory final/ contains the actual word lists broken up into
various sizes and categories. The r/ directory contains Readmes from
the various sources used to create this package.
+The misc/ contains a small list of taboo words, see the README file
+for more info. The speller/ directory contains scripts for creating
+spelling dictionaries for Aspell and Hunspell.
+
The other directories contain the necessary information to recreate the
word lists from the raw data. Unless you are interested in improving the
words lists you should not need to worry about what's here. See the
Except for the special word lists the files follow the following
naming convention:
- <spelling category>-<classification>.<size>
+ <spelling category>-<sub-category>.<size>
Where the spelling category is one of
english, american, british, british_z, canadian,
- variant_0, varaint_1, variant_2
-Classification is one of
+ variant_0, varaint_1, variant_2,
+ british_variant_0, british_variant_1,
+ canadian_variant_0, canadian_variant_1,
+Sub-category is one of
abbreviations, contractions, proper-names, upper, words
And size is one of
10, 20, 35 (small), 40, 50 (medium), 55, 60, 70 (large),
Where description is one of:
roman-numerals, hacker
-When combining the words lists the "english" spelling category should
-be used as well as one of "american", "british", "british_z" (british
-with ize spelling), or "canadian". Great care has been taken so that
-that only one spelling for any particular word is included in the main
-list. When two variants were considered equal I randomly picked one
-for inclusion in the main word list. Unfortunately this means that my
-choice in how to spell a word may not match your choice. If this is
-the case you can try including the "variant_0" spelling category which
+The perl script "mk-list" can be used to create a word list of the
+desired size, it usage is:
+ ./mk-list [-f] [-v#] <spelling categories> <size>
+where <spelling categories> is one of the above spelling categories
+(the english and special categories are automatically included as well
+as all sub-categories) and <size> is the desired desired size. The
+"-v" option can be used to used to also include the appropriate
+variants file up to level '#'. The normal output will be a sorted
+word list. If you rather see what files will be included, use the
+"-f" option.
+
+When manually combining the words lists the "english" spelling
+category should be used as well as one of "american", "british",
+"british_z" (british with ize spelling), or "canadian". Great care
+has been taken so that that only one spelling for any particular word
+is included in the main list (with some minor exceptions). When two
+variants were considered equal I randomly picked one for inclusion in
+the main word list. Unfortunately this means that my choice in how to
+spell a word may not match your choice. If this is the case you can
+try including one of the "variant_0" spelling categories which
includes most variants which are considered almost equal. The
-"variant_1" spelling category include variants which are also
-generally considered acceptable, and "variant_2" contains variants
-which are seldom used.
+"variant_0" spelling category corresponds mostly to American variants,
+while the "british_variant_0" and "canadian_variant_0" are for British
+and Canadian variants, respectively. The "variant_1" spelling
+categories include variants which are also generally considered
+acceptable, and "variant_2" contains variants which are seldom used
+and may now even be considered correct. There is no
+"british_variant_2" or "canadian_variant_2" spelling category since
+the distinction would be almost meaningless.
The "abbreviation" category includes abbreviations and acronyms which
are not also normal words. The "contractions" category should be self
To give you an idea of what the words in the various sizes look like
here is a sample of 25 random words found only in that size:
-10: began both buffer cause collection content documenting easiest
- equally examines expecting first firstly hence inclining
- irrelevant justified little logs necessarily ought sadly six
- thing visible
-
-20: chunks commodity contempt contexts cruelty crush dictatorship
- disgusted dose elementary evolved frog god hordes notion overdraft
- overlong overlook phoning poster recordings sand skull substituted
- throughput
-
-35: aliasing blackouts blowout bluntness corroborated derrick
- dredging elopements entrancing excising fellowship flagpole
- germination glimpse gondola guidebook madams minimalism minnows
- partisans petitions shelling swarmed throng welding
-
-40: altercation blender castigation chump coffeehouse determiners
- doggoning exhibitor finders flophouse gazebo lumbering masochism
- mopeds poetically pubic refinance reggae scragglier softhearted
- stubbornness teargassed township underclassman whoosh
-
-50: accumulative adulterant allegorically amorousness astrophysics
- camphor coif dickey elusiveness enviousness fakers fetishistic
- flippantly headsets liefs midyears myna pacification persiflage
- phosphoric pinhole sappy seres unrealistically unworldly
-
-55: becquerel brickie centralist cine conveyancing courgette
- disarmingly garçon gobstopper infilling insipidity
- internationalist kabuki lyrebirds obscurantism rejigged
- revisionist satsuma slapper sozzled sublieutenants teletext vino
- wellness wracking
-
-60: absorber acceptableness adventurousness antifascists arrhythmia
- audiology cartage cruses fontanel forelimbs granter hairlike
- installers jugglery lappets libbers mandrels micrometeorite
- mineshaft reconsecrates saccharides smellable spavined sud timbrel
-
-70: atomisms benedict carven coxa cyanite detraining diazonium
- dogberry dogmatics entresol fatherlessnesses firestone imprecator
- laterality legitimisms maxwell microfloppies nonteaching pelerine
- pentane pestiferousness piscator profascist tusche twirp
-
-80: cotransfers embrangled forkednesses giftwrapped globosity hatpegs
- hepsters hermitess interspecific inurbanities lamiae
- literaehumaniores literatures masulas misbegun plook prerupt
- quaalude rosanilin sabbatism scowder subreptive thumbstalls
- understrata yakows
-
-95: anatropal anientise bakshi brouzes corsie daimiote dhaw dislikened
- ectoretina fortuitisms guardeen hyperlithuria nonanachronistic
- overacceleration pamphletic parma phytolith starvedly
- trophoplasmic ulorrhagia undared undertide unplunderously
- unworkmanly vasoepididymostomy
-
-And here is a rough count on the number of words in the "english"
-spelling category for each size:
-
- Size Words Proper Names Running Total
-
- 10 5,000 5,000
- 20 8,700 14,000
- 35 34,500 200 48,000
- 40 6,000 500 55,000
- 50 23,200 17,200 95,000
- 55 7,500 103,000
- 60 16,000 12,800 132,000
- 70 45,100 34,300 211,000
- 80 137,000 30,400 379,000
- 95 198,000 51,800 628,000
-
-(The "Words" column does not include the proper name count.)
+10: advertised agreeing artificial bucket changes closest currently finding
+ implications learning liable obvious partial peace planet preparing
+ produced regulations shortly tries under unnecessary vacations vast wind
+
+20: accomplishes addict baffles blink chapel corrections depresses dripping
+ erased infant interfere launch nicking novels paranoid passport pursued
+ recruitment rectifying relaxed sixteen sundry tab undergone withdraws
+
+35: adores affixes brisks caking conciliates decimates discretionary
+ dispatches forensics glorify gridiron healed hurling kelp massacring
+ necks pits placarding pyramids ratting recreates renovated sandals shirks
+ subtract
+
+40: demoed dichotomy dilapidation disheveled ebullience estimable finagling
+ hemorrhoid lazily medalists mintiest motherboards ostracism pornographers
+ predilections remarries southbound steamrolled sympathizers tads tampons
+ tattletale upchucked vainly viscous
+
+50: bootless brawler bulkhead canoeist declassifying farthings hake hectors
+ helpmate hermitage humanoid kitsch mercerize pawnshops pleasingly
+ retrorockets scurrilously solemnizes superficiality symbiosis tangelo
+ timetabling unenviable unmoral unreconstructed
+
+55: beachfront bicarbonate caff campanologists execrably fab fightback
+ firebricks insipidity laboriousness megawatts mirthlessly misnames
+ nymphos photocell potholed psychoactive psychoanalytically schoolmarmish
+ simulacra subeditors supremo sweated turbocharges yogic
+
+60: assayer banteringly besmeared brazer chromatin cremes deciliters
+ doubtfulness enshrinement ephemerally fibular globalist gypper
+ legitimatized mensch mopers oversea pantyliner paratyphoid redivide
+ rehabilitative salesladies sensualists superposition univalves
+
+70: adactylous anticapitalist bezant bister boraginaceous civically cossacks
+ cousinly curricle dekaliter grippingly grugrus gurging hermaphroditism
+ levanted magnetizer nonapplicable panegyrists parametrize radomes
+ refilter ruinations teths truistic uts
+
+80: bodikin buhrs covetiveness diarch disaccharidases drumbeater empusas
+ flyings hyperexcitability hyperpolarizations janizaries overwash
+ physiocrats postform postsecondary preambulate puzzlehead remixer
+ snoutier tetrathlons toothdrawing triff unaffectionate wearish yawy
+
+95: actinophone aerobious anadenia biochemics chromatopathia ciclatouns
+ gaspiest guapinol hagigah interdorsal melanotekite minnicking
+ nonretrenchment overloftily oystriges peltandra retromaxillary
+ subterraqueous transphysically unconfidential unvalidating upspew
+ verminlike vetiveria yerth
+
+And here is a count on the number of in each spelling category
+(american + english spelling category):
+
+ Size Words Names Running Total %
+ 10 4,427 15 4,442 0.7
+ 20 8,122 0 12,564 1.9
+ 35 37,251 224 50,039 7.7
+ 40 6,802 503 57,344 8.8
+ 50 24,505 15,455 97,304 14.9
+ 55 6,555 0 103,859 15.9
+ 60 13,633 775 118,267 18.1
+ 70 35,507 7,747 161,521 24.8
+ 80 143,791 33,293 338,605 51.9
+ 95 227,056 86,814 652,475 100.0
+
+(The "Words" column does not include the name count.)
Size 35 is the recommended small size, 50 the medium and 70 the large.
-Sizes 70 and below contain words found in most dictionaries while the
-80 size contains all the strange and unusual words people like to use
-in word games such as Scrabble (TM). While a lot of the the words in
-the 80 size are not used very often, they are all generally considered
-valid words in the English language. The 95 contains just about every
-English word in existence and then some. Many of the words at the 95
-level will probally not be considered valid english words by most
-people. I don't recommend anyone use levels above 70 for spell
-checking as they contain rarely used words which can hide misspellings
-of similar more commonly used words. For example the word "ort" can
-hide a common typo of "or". No one should need to use a size larger
-than 80, the 95 size is labeled insane for a reason.
+For spell checking I recommend using 60. Sizes 70 and below contain
+words found in most dictionaries while the 80 size contains all the
+strange and unusual words people like to use in word games such as
+Scrabble (TM). While a lot of the the words in the 80 size are not
+used very often, they are all generally considered valid words in the
+English language. The 95 contains just about every English word in
+existence and then some. Many of the words at the 95 level will
+probably not be considered valid English words by most people. I use
+the 60 size for the English dictionary for Aspell, and I don't
+recommend anyone use levels above 70 for spell checking. Levels above
+70 contain rarely used words which can hide misspellings of similar
+more commonly used words. For example the word "ort" can hide a
+common typo of "or". No one should need to use a size larger than 80,
+the 95 size is labeled insane for a reason.
Accents are present on certain words such as café in iso8859-1 format.
CHANGES:
+From Revision 7 to 7.1 (January 6, 2011)
+
+ Updated to revision 5.1 of Varcon which corrected several errors.
+
+ Fixed various problems with the variant processing which corrected a
+ few more errors.
+
+ Added several now common proper names and some other words now
+ in common use.
+
+ Include misc/ and speller/ directory which where in SVN but left
+ out of the release tarball.
+
+ Other minor fixes, including some fixes to the taboo word lists.
+
+From Revision 6 to 7 (December 27, 2010)
+
+ Updated to revision 5.0 of Varcon which corrected many errors,
+ especially in the British and Canadian spelling categories. Also
+ added new spelling categories for the British and Canadian spelling
+ variants and separated them out from the main variant_* categories.
+
+ Moved Moby names lists (3897male.nam 4946fema.len 21986na.mes) to 95
+ level since they contain too many errors and rare names.
+
+ Moved frequently class 0 from Brian Kelk's Wordlist from
+ level 60 to 70, and also filter it with level 80 due to, too many
+ misspellings.
+
+ Many other minor fixes.
+
From Revision 5 to 6 (August 10, 2004)
Updated to version 4.0 of the 12dicts package.
Fixed a bug which caused variants of words to incorrectly appear in
the non-variant lists.
- Moved rarly used inflections of a word into higher number lists.
+ Moved rarely used inflections of a word into higher number lists.
Added other inflections of a words based on the following criteria
If the word is in the base form: only include that word.
Updated to the latest version of many of the source dictionaries.
Removed the DEC Word List due to the questionable licence and
- because removing it will not seriously decrese the quality of SCOWL
+ because removing it will not seriously decrease the quality of SCOWL
(there are a few less proper names).
From Revision 4 to 4a (April 4, 2001)
Added words in the Ispell word list at the 65 level.
Other changes due to using more recent versions of various sources
- included a more accurete version of AGID thanks to the word of
+ included a more accurate version of AGID thanks to the word of
Alan Beale
From Revision 2 to 3 (August 18, 2000)
Renamed special-unix-terms to special-hacker and added a large
- number of communly used words within the hacker (not cracker)
+ number of commonly used words within the hacker (not cracker)
community.
Added a couple more signature words including "newbie".
COPYRIGHT, SOURCES, and CREDITS:
-The collective work is Copyright 2000-2004 by Kevin Atkinson as well
+The collective work is Copyright 2000-2011 by Kevin Atkinson as well
as any of the copyrights mentioned below:
- Copyright 2000-2004 by Kevin Atkinson
+ Copyright 2000-2011 by Kevin Atkinson
Permission to use, copy, modify, distribute and sell these word
lists, the associated scripts, the output created from the scripts,
4.0 of his 12dicts package. Like his other stuff the 2of4brif is also
in the public domain.
-The 60 level includes Brian's frequency class 0 and all words
-appearing in at least 2 of the 12 dictionaries as indicated by the
-12Dicts package. A large number of names are also included: The 4,946
-female names and the 3,897 male names from the MWords package.
+The 60 level includes all words appearing in at least 2 of the 12
+dictionaries as indicated by the 12Dicts package.
-The 70 level includes the 74,550 common dictionary words and the
-21,986 names list from the MWords package The common dictionary words,
+The 70 level includes Brian's frequency class 0 and the 74,550 common
+dictionary words from the MWords package. The common dictionary words,
like those from the 12Dicts package, have had all likely inflections
added. The 70 level also included the 5desk list from version 4.0 of
-the 12Dics package which is the public domain
+the 12Dics package which is the public domain.
The 80 level includes the ENABLE word list, all the lists in the
ENABLE supplement package (except for ABLE), the "UK Advanced Cryptics
There are no other restrictions: I would like to see the list
distributed as widely as possible.
-The 95 level includes the 354,984 single words and 256,772 compound
-words from the MWords package, ABLE.LST from the ENABLE Supplement,
-and some additional words found in my part-of-speech database that
-were not found anywhere else.
+The 95 level includes the 354,984 single words, 256,772 compound
+words, 4,946 female names and the 3,897 male names, and 21,986 names
+from the MWords package, ABLE.LST from the ENABLE Supplement, and some
+additional words found in my part-of-speech database that were not
+found anywhere else.
Accent information was taken from UKACD.
FUTURE PLANS:
-There is a very nice frequency analyse of the BNC corpus done by
-Adam Kilgarriff. Unlike Brain's word lists the BNC lists include part
-of speech information. I plan on somehow using these lists as Adam
-Kilgarriff has given me the OK to use it in SCOWL. These lists will
-greatly reduce the problem of inflected forms of a word appearing at
-different levels due to the part-of-speech information.
+The process of "sort"s, "comm"s, and Perl scripts to combine the many
+word lists and separate out the variant information is inexact and
+error prone. The whole things needs to be rewritten to deal with
+words in terms of lemmas. When the exact lemma is not known a best
+guess should be made. I'm not sure what form this should be in. I
+originally thought this should be some sort of database, but maybe I
+should just slurp all that data into memory and process it in one
+giant perl script. With the amount of memory available these days (at
+least 2 GB, often 4 GB or more) this should not really be a problem.
+
+In addition, there is a very nice frequency analyze of the BNC corpus
+done by Adam Kilgarriff. Unlike Brain's word lists the BNC lists
+include part of speech information. I plan on somehow using these
+lists as Adam Kilgarriff has given me the OK to use it in SCOWL.
+These lists will greatly reduce the problem of inflected forms of a
+word appearing at different levels due to the part-of-speech
+information.
-I also plan on perhaps putting the data in a database and use SQL
-queries to create the wordlists instead of tons of "sort"s, "comm"s,
-and Perl scripts.
+There is frequency information for some other corpus such as COCA
+(Corpus of Contemporary American English) and ANS (American National
+Corpus) which I might also be able to use. The formal will require
+permission, and the latter is of questionable quality.
RECREATING THE WORD LISTS:
In order to recreate the word lists you need a modern version of Perl,
bash, the traditional set of shell utilities, a system that supports
-symbolic links, and quite possibly GNU Make. Once you have downloaded
-all the necessary raw data in the r/ directory you should be able to
-type "rm final/* && make all" and the word lists in the final/
-directory should be recreated. If you have any problems fell free to
-contact me; however, unless you are interested in improving the
-scripts used, I will likely ignore you as there should be little need
-for anyone not interested in improving the word list to do so.
+symbolic links, and quite possibly GNU Make. The easiest way to
+recreate the word lists is to checkout SVN revision 161 (or tag
+scowl-7.1) and simply type "make" (see http://wordlist.sourceforge.net).
+You can try to download all the pieces manually, but you may not get
+the same result since the latest version of some parts used to create
+SCOWL may not have been released yet.
The src/ directory contains the numerous scripts used in the creation
of the final product.
-The r/ directory contains the raw data used to
-create the final product. In order for the scripts to work various
-word lists and databases need to be created and put into this
-directory. See the README file in the r/ directory for more
-information.
+The r/ directory contains the raw data used to create the final
+product. If you checkout from SVN this directory should be populated
+automatically for you. If you insist on doing it the hard way see the
+README file in the r/ directory for more information.
The l/ directory contains symbolic links used by the actual scripts.
Finally, the working/ directory is where all the intermittent files go
that are not specific to one source.
-
projects / deb_pkgs / scowl.git / blobdiff