1 Spell Checking Oriented Word Lists (SCOWL)
2 @`if [ "$SCOWL_VERSION" ]; then echo -n "Version $SCOWL_VERSION"; fi`
3 @`git log --pretty=format:'%cd [%h]' -n 1 --`
4 by Kevin Atkinson (kevina@gnu.org)
6 The SCOWL is a collection of word lists split up in various sizes, and
7 other categories, intended to be suitable for use in spell checkers.
8 However, I am sure it will have numerous other uses as well.
10 The latest version can be found at http://wordlist.aspell.net/.
12 The directory final/ contains the actual word lists broken up into
13 various sizes and categories. The r/ directory contains Readmes from
14 the various sources used to create this package.
16 The misc/ contains a small list of taboo words, see the README file
17 for more info. The speller/ directory contains scripts for creating
18 spelling dictionaries for Aspell and Hunspell.
20 The other directories contain the necessary information to recreate the
21 word lists from the raw data. Unless you are interested in improving the
22 words lists you should not need to worry about what's here. See the
23 section on recreating the words lists for more information on what's
26 Except for the special word lists the files follow the following
28 <spelling category>-<sub-category>.<size>
29 Where the spelling category is one of
30 english, american, british, british_z, canadian, australian
31 variant_1, variant_2, variant_3,
32 british_variant_1, british_variant_2,
33 canadian_variant_1, canadian_variant_2,
34 australian_variant_1, australian_variant_2
35 Sub-category is one of
36 abbreviations, contractions, proper-names, upper, words
38 10, 20, 35 (small), 40, 50 (medium), 55, 60, 70 (large),
39 80 (huge), 95 (insane)
40 The special word lists follow are in the following format:
41 special-<description>.<size>
42 Where description is one of:
43 roman-numerals, hacker
45 The perl script "mk-list" can be used to create a word list of the
46 desired size, its usage is:
47 ./mk-list [-f] [-v#] <spelling categories> <size>
48 where <spelling categories> is one of the above spelling categories
49 (the english and special categories are automatically included as well
50 as all sub-categories) and <size> is the desired size. The
51 "-v" option can be used to also include the appropriate
52 variants file up to level '#'. The normal output will be a sorted
53 word list. If you rather see what files will be included, use the
56 When manually combining the words lists the "english" spelling
57 category should be used as well as one of "american", "british",
58 "british_z" (british with ize spelling), "canadian" or "australian".
59 Great care has been taken so that only one spelling for any particular
60 word is included in the main list (with some minor exceptions). When
61 two variants were considered equal I randomly picked one for inclusion
62 in the main word list. Unfortunately this means that my choice in how
63 to spell a word may not match your choice. If this is the case you
64 can try including one of the "variant_1" spelling categories which
65 includes most variants which are considered almost equal. The
66 "variant_1" spelling category corresponds mostly to American variants,
67 while the "british_variant_1", "canadian_variant_1" and
68 "australian_variant_1" are for British, Canadian and Australian
69 variants, respectively. The "variant_2" spelling categories include
70 variants which are also generally considered acceptable, and
71 "variant_3" contains variants which are seldom used and may not even
72 be considered correct. There is no "british_variant_3",
73 "canadian_variant_3" or "australian_variant_3" spelling category since
74 the distinction would be almost meaningless.
76 The "abbreviation" category includes abbreviations and acronyms which
77 are not also normal words. The "contractions" category should be self
78 explanatory. The "upper" category includes upper case words and proper
79 names which are common enough to appear in a typical dictionary. The
80 "proper-names" category includes all the additional uppercase words.
81 Finally the "words" category contains all the normal English words.
83 To give you an idea of what the words in the various sizes look like
84 here is a sample of 25 random words found only in that size:
86 @`src/rand-samples | iconv -f iso-8859-1 -t utf-8`
88 And here is a count on the number of words in each spelling category
89 (american + english spelling category):
93 (The "Words" column does not include the name count.)
95 Size 35 is the recommended small size, 50 the medium and 70 the large.
96 Sizes 70 and below contain words found in most dictionaries while the
97 80 size contains all the strange and unusual words people like to use
98 in word games such as Scrabble (TM). While a lot of the words in the
99 80 size are not used very often, they are all generally considered
100 valid words in the English language. The 95 contains just about every
101 English word in existence and then some. Many of the words at the 95
102 level will probably not be considered valid English words by most
105 For spell checking I recommend using size 60. This size is the
106 largest size that I am fairly confident does not contain any
107 misspellings or invalid words. In addition an effort is made to
108 exclude valid yet problematic words (such as "calender") from the 60
109 size that are likely to be a misspelling of a more common word. The
110 70 size is reasonable for those wanting a larger list and don't mind a
111 few errors. The 80 or larger sizes are not reasonable for spell
114 Accents are present on certain words such as café in iso8859-1 format.
118 From Version 2019.10.06 to 2020.12.07
124 Bump irregardless, froward (+ derivatives) and perpend to level 70.
126 From Version 2018.04.16 to 2019.10.06
130 Remove compare's and fail's.
132 From Version 2017.08.24 to 2018.04.16
136 Fix build problems on macOS.
138 From Version 2017.01.22 to 2017.08.24
142 From Version 2016.11.20 to 2017.01.22
146 From Version 2016.06.26 to 2016.11.20
148 New Australian spelling category thanks to the work of Benjamin
149 Titze (btitze@protonmail.ch)
153 From Version 2016.01.19 to 2016.06.26
157 Updated to Version 6.0.2 of 12dicts
161 From Version 2015.08.24 to 2016.01.19
165 Clarified README to indicate why the 60 size is the preferred size
168 Remove some very uncommon possessive forms.
170 Change "SET UTF8" to "SET UTF-8" in hunspell affix file.
172 From Version 2015.05.18 to 2015.08.24 (Aug 24, 2015)
176 From Version 2015.04.24 to 2015.05.18 (May 18, 2015)
178 Added some new words found to have a high frequency in the COCA
179 corpus. (http://corpus.byu.edu/coca/).
181 Fix en spelling suggestions for 'alot' and 'exersize' in hunspell
182 dictionary (upstreamed from the changes made in Firefox).
184 From Version 2015.02.15 to 2015.04.24 (April 24, 2015)
186 Added some new words.
188 Convert hunspell dictionary to UTF-8 in order to handle smart
191 From Version 2015.01.28 to 2015.02.15 (February 15, 2015)
193 Added a large number of neologisms (newly invented words)
194 such as "selfie" and "smartwatch" thanks to Alan Beale.
196 Various other new words.
198 Clean up the special-hacker category by removing some words that
199 didn't exist in the Google Book's Corpus (1980 - 2008) and
200 originated from the "Unofficial Jargon File Word Lists".
202 From Version 2014.11.17 to 2015.01.28 (January 28, 2015)
204 Various new words, many from analyzing the Google Book's Corpus
205 (1980 - 2008). See http://app.aspell.net/lookup-freq.
207 Moved some uncommon words that can easily hide a misspelling of a
208 more common word to level 70. (calender, adrenalin and Joesph)
210 Removed several -er and -est forms from adjectives that were so
211 uncommon that they were not found anywhere is the Google Book's
212 Corpus (1980 - 2008).
214 From Version 2014.08.11.1 to 2014.11.17 (November 17, 2014)
218 Fix typo in Hunspell readme.
220 From Version 2014.08.11 to 2014.08.11.1 (August 13, 2014)
222 Forgot to mention this important change from 7.1 to 2014.08.11:
224 Shifted the variant levels up by one: variant_0 is now variant_1,
225 variant_1 is now variant_2, and variant_2 is now variant_3.
227 Other minor fixes in this README.
229 No changes to the contents of the lists.
231 From Revision 7.1 to Version 2014.08.11 (August 11, 2014)
233 Added some missing possessive forms.
235 Added some new words and proper names.
237 Clean up the categories (words, upper, proper-names etc) so that they
240 Convert documentation to UTF-8. For now, the wordlist are still in
241 ISO-8859-1 to prevent compatibility problems.
243 Add schema and scripts for creating a SQLite database from SCOWL.
244 Add some utility and library functions using them. This database is
245 used by the new web app's (http://app.aspell.net/lookup & create).
247 Enhance speller/make-hunspell-dict. The biggest improvement is that
248 it that it now generates several more dictionaries in addition to
249 the official ones. These additional dictionaries are ones for
250 British English and larger dictionaries that include up to SCOWL
253 From Revision 7 to 7.1 (January 6, 2011)
255 Updated to revision 5.1 of Varcon which corrected several errors.
257 Fixed various problems with the variant processing which corrected a
260 Added several now common proper names and some other words now
263 Include misc/ and speller/ directory which were in SVN but left
264 out of the release tarball.
266 Other minor fixes, including some fixes to the taboo word lists.
268 From Revision 6 to 7 (December 27, 2010)
270 Updated to revision 5.0 of Varcon which corrected many errors,
271 especially in the British and Canadian spelling categories. Also
272 added new spelling categories for the British and Canadian spelling
273 variants and separated them out from the main variant_* categories.
275 Moved Moby names lists (3897male.nam 4946fema.len 21986na.mes) to 95
276 level since they contain too many errors and rare names.
278 Moved frequently class 0 from Brian Kelk's Wordlist from
279 level 60 to 70, and also filter it with level 80 due to, too many
282 Many other minor fixes.
284 From Revision 5 to 6 (August 10, 2004)
286 Updated to version 4.0 of the 12dicts package.
288 Included the 3esl, 2of4brif, and 5desk list from the new 12dicts
289 package. The 3esl was included in the 40 size, the 2of4brif in the
290 55 size and the 5desk in the 70 size.
292 Removed the Ispell word list as it was a source of too many errors.
293 This eliminated the 65 size.
295 Removed clause 4 from the Ispell copyright with permission of Geoff
298 Updated to version 4.1 of VarCon.
300 Added the "british_z" spelling category which is British using the
303 From Revision 4a to 5 (January 3, 2002)
305 Added variants that were not really spelling variants (such as
306 forwards) back into the main list.
308 Fixed a bug which caused variants of words to incorrectly appear in
309 the non-variant lists.
311 Moved rarely used inflections of a word into higher number lists.
313 Added other inflections of a words based on the following criteria
314 If the word is in the base form: only include that word.
315 If the word is in a plural form: include the base word and the plural
316 If the word is a verb form (other than plural): include all verb forms
317 If the word is an ad* form: include all ad* forms
318 If the word is in a possessive form: also include the non-possessive
320 Updated to the latest version of many of the source dictionaries.
322 Removed the DEC Word List due to the questionable licence and
323 because removing it will not seriously decrease the quality of SCOWL
324 (there are a few less proper names).
326 From Revision 4 to 4a (April 4, 2001)
328 Reran the scripts on a never version of AGID (3a) which fixes a bug
329 which caused some common words to be improperly marked as variants.
331 From Revision 3 to 4 (January 28, 2001)
333 Split the variant "spelling category" up into 3 different levels.
335 Added words in the Ispell word list at the 65 level.
337 Other changes due to using more recent versions of various sources
338 included a more accurate version of AGID thanks to the work of
341 From Revision 2 to 3 (August 18, 2000)
343 Renamed special-unix-terms to special-hacker and added a large
344 number of commonly used words within the hacker (not cracker)
347 Added a couple more signature words including "newbie".
349 Minor changes due to changes in the inflection database.
351 From Revision 1 to 2 (August 5, 2000)
353 Moved the male and female name lists from the mwords package and the
354 DEC name lists form the 50 level to the 60 level and moved Alan's
355 name list from the 60 level to the 50 level. Also added the top
356 1000 male, female, and last names from the 1990 Census report to the
357 50 level. This reduced the number of names in the 50 level from
360 Added a large number of Uppercase words to the 50 level.
362 Properly accented the possessive form of some words.
364 Minor other changes due to changes in my raw data files which have
365 not been released yet. Email if you are interested in these files.
367 COPYRIGHT, SOURCES, and CREDITS:
371 The variant word lists were created from a list of variants found in
372 the 12dicts supplement package as well as a list of variants I created
375 The Readmes for the various packages used can be found in the
376 appropriate directory under the r/ directory.
380 The process of "sort"s, "comm"s, and Perl scripts to combine the many
381 word lists and separate out the variant information is inexact and
382 error prone. The whole things needs to be rewritten to deal with
383 words in terms of lemmas. When the exact lemma is not known a best
384 guess should be made. I'm not sure what form this should be in. I
385 originally thought this should be some sort of database, but maybe I
386 should just slurp all that data into memory and process it in one
387 giant perl script. With the amount of memory available these days (at
388 least 2 GB, often 4 GB or more) this should not really be a problem.
390 In addition, there is a very nice frequency analyze of the BNC corpus
391 done by Adam Kilgarriff. Unlike Brian's word lists the BNC lists
392 include part of speech information. I plan on somehow using these
393 lists as Adam Kilgarriff has given me the OK to use it in SCOWL.
394 These lists will greatly reduce the problem of inflected forms of a
395 word appearing at different levels due to the part-of-speech
398 There is frequency information for some other corpus such as COCA
399 (Corpus of Contemporary American English) and ANS (American National
400 Corpus) which I might also be able to use. The former will require
401 permission, and the latter is of questionable quality.
403 RECREATING THE WORD LISTS:
405 In order to recreate the word lists you need a modern version of Perl,
406 bash, the traditional set of shell utilities, a system that supports
407 symbolic links, and quite possibly GNU Make. The easiest way to
408 recreate the word lists is to checkout the corresponding Git version
409 (see the version string at the start of the file) and simply type
410 "make" (see http://wordlist.aspell.net). You can try to download all
411 the pieces manually, but this method is not no longer tested nor
414 The src/ directory contains the numerous scripts used in the creation
415 of the final product.
417 The r/ directory contains the raw data used to create the final
418 product. If you checkout from Git this directory should be populated
419 automatically for you. If you insist on doing it the hard way see the
420 README file in the r/ directory for more information.
422 The l/ directory contains symbolic links used by the actual scripts.
424 Finally, the working/ directory is where all the intermittent files go
425 that are not specific to one source.