1 Spell Checking Oriented Word Lists (SCOWL)
3 Thu Aug 24 14:36:19 2017 -0400 [2614b88]
4 by Kevin Atkinson (kevina@gnu.org)
6 The SCOWL is a collection of word lists split up in various sizes, and
7 other categories, intended to be suitable for use in spell checkers.
8 However, I am sure it will have numerous other uses as well.
10 The latest version can be found at http://wordlist.aspell.net/.
12 The directory final/ contains the actual word lists broken up into
13 various sizes and categories. The r/ directory contains Readmes from
14 the various sources used to create this package.
16 The misc/ contains a small list of taboo words, see the README file
17 for more info. The speller/ directory contains scripts for creating
18 spelling dictionaries for Aspell and Hunspell.
20 The other directories contain the necessary information to recreate the
21 word lists from the raw data. Unless you are interested in improving the
22 words lists you should not need to worry about what's here. See the
23 section on recreating the words lists for more information on what's
26 Except for the special word lists the files follow the following
28 <spelling category>-<sub-category>.<size>
29 Where the spelling category is one of
30 english, american, british, british_z, canadian, australian
31 variant_1, variant_2, variant_3,
32 british_variant_1, british_variant_2,
33 canadian_variant_1, canadian_variant_2,
34 australian_variant_1, australian_variant_2
35 Sub-category is one of
36 abbreviations, contractions, proper-names, upper, words
38 10, 20, 35 (small), 40, 50 (medium), 55, 60, 70 (large),
39 80 (huge), 95 (insane)
40 The special word lists follow are in the following format:
41 special-<description>.<size>
42 Where description is one of:
43 roman-numerals, hacker
45 The perl script "mk-list" can be used to create a word list of the
46 desired size, its usage is:
47 ./mk-list [-f] [-v#] <spelling categories> <size>
48 where <spelling categories> is one of the above spelling categories
49 (the english and special categories are automatically included as well
50 as all sub-categories) and <size> is the desired size. The
51 "-v" option can be used to also include the appropriate
52 variants file up to level '#'. The normal output will be a sorted
53 word list. If you rather see what files will be included, use the
56 When manually combining the words lists the "english" spelling
57 category should be used as well as one of "american", "british",
58 "british_z" (british with ize spelling), "canadian" or "australian".
59 Great care has been taken so that only one spelling for any particular
60 word is included in the main list (with some minor exceptions). When
61 two variants were considered equal I randomly picked one for inclusion
62 in the main word list. Unfortunately this means that my choice in how
63 to spell a word may not match your choice. If this is the case you
64 can try including one of the "variant_1" spelling categories which
65 includes most variants which are considered almost equal. The
66 "variant_1" spelling category corresponds mostly to American variants,
67 while the "british_variant_1", "canadian_variant_1" and
68 "australian_variant_1" are for British, Canadian and Australian
69 variants, respectively. The "variant_2" spelling categories include
70 variants which are also generally considered acceptable, and
71 "variant_3" contains variants which are seldom used and may not even
72 be considered correct. There is no "british_variant_3",
73 "canadian_variant_3" or "australian_variant_3" spelling category since
74 the distinction would be almost meaningless.
76 The "abbreviation" category includes abbreviations and acronyms which
77 are not also normal words. The "contractions" category should be self
78 explanatory. The "upper" category includes upper case words and proper
79 names which are common enough to appear in a typical dictionary. The
80 "proper-names" category includes all the additional uppercase words.
81 Finally the "words" category contains all the normal English words.
83 To give you an idea of what the words in the various sizes look like
84 here is a sample of 25 random words found only in that size:
86 10: allow apology borrow commented commenting confirming device field film
87 forgotten happen industrial insist kept log present processing register
88 representative seems string style suspected tie tying
90 20: accuse advisable browsed dialings emphasizing farmers fatuous fighting
91 graduates honesty intentions judged mainframes mechanisms mirrors
92 newspapers partition phoned poison prevalent settings smart spokesman
95 35: accrued bankrupts beetling broadsides colonel compactest dissolved
96 drunkenly encumbrances engagement erects halved imprisonment miscellany
97 neurons neutralized nobly perking postured raisin ripples sprinklings
98 streaking verdicts wedged
100 40: blabbermouths boardinghouses boardrooms bopping checklist
101 compartmentalized condos famished formidably interconnecting litigates
102 mecca nudists polygraphing pristine puppeteers quarterfinal rethinks
103 silencers soreness stepchildren toeholds trundle typecasts upstaging
105 50: attributive barbacoa bondsman caduceus colluded consanguinity enfeeble
106 faker firetraps jocundly kestrel macerating piggishness quadruplicates
107 ragweed reassign rebelliously refract skywriting splodge storied turbot
108 umbels unlearning wanderlust
110 55: androgyny basques bendiest busking constitutionalism conventioneer coshes
111 creditworthiness determinedly fuckheads garçon goalless groundwater
112 innocuousness millionairess preservationists professionalizes raunchily
113 shebeens shitheads stripy stroppiest superciliously tinplate wholefoods
115 60: anapestic anodizes asphodel bargainers berkelium broncs carotene daemonic
116 defilers enshrinement instate mintage nonacceptance ovular quarterstaff
117 recharter reconsigning recontaminating reexaminations supersaturation
118 tantalization unharness unharnessed widener wonkiest
120 70: benignantly berretta bittiness blacklegged breadsticks cinquefoils
121 crossruffs epifocal extralegally frequentative gadgeteers isomorph
122 labeler oppositional phanerogams pleochroism pourable preestablish resiny
123 salpinxes singularizes sprit summarizations technic tracheid
125 80: amoristic anglify antiterrorists boileries deboshing evidents exenterates
126 formulizations graphitizing hamshackle infallibilist mohican multeities
127 mvule nabobesses panini pheochromocytomata quadruplexed rampick resculpt
128 schmooz scotias staws sterlingness valiantness
130 95: additionist anchoretism antiepithelial appropre casewood colocephalous
131 flusteration haldu hishes interpervasiveness knutty nonencroachment
132 occluse orthoformic pentateuch plumbership preconflict saumont
133 slopselling tangleproof transbaikal unaccordance uniphase unmixableness
137 And here is a count on the number of words in each spelling category
138 (american + english spelling category):
140 Size Words Names Running Total %
141 10 4,426 13 4,439 0.7
142 20 8,125 0 12,564 1.9
143 35 37,259 222 50,045 7.6
144 40 6,853 491 57,389 8.7
145 50 25,225 18,222 100,836 15.3
146 55 6,489 0 107,325 16.3
147 60 14,268 779 122,372 18.6
148 70 35,418 7,911 165,701 25.2
149 80 144,259 33,372 343,332 52.2
150 95 227,669 86,649 657,650 100.0
153 (The "Words" column does not include the name count.)
155 Size 35 is the recommended small size, 50 the medium and 70 the large.
156 Sizes 70 and below contain words found in most dictionaries while the
157 80 size contains all the strange and unusual words people like to use
158 in word games such as Scrabble (TM). While a lot of the words in the
159 80 size are not used very often, they are all generally considered
160 valid words in the English language. The 95 contains just about every
161 English word in existence and then some. Many of the words at the 95
162 level will probably not be considered valid English words by most
165 For spell checking I recommend using size 60. This size is the
166 largest size that I am fairly confident does not contain any
167 misspellings or invalid words. In addition an effort is made to
168 exclude valid yet problematic words (such as "calender") from the 60
169 size that are likely to be a misspelling of a more common word. The
170 70 size is reasonable for those wanting a larger list and don't mind a
171 few errors. The 80 or larger sizes are not reasonable for spell
174 Accents are present on certain words such as café in iso8859-1 format.
178 From Version 2017.01.22 to 2017.08.24
182 From Version 2016.11.20 to 2017.01.22
186 From Version 2016.06.26 to 2016.11.20
188 New Australian spelling category thanks to the work of Benjamin
189 Titze (btitze@protonmail.ch)
193 From Version 2016.01.19 to 2016.06.26
197 Updated to Version 6.0.2 of 12dicts
201 From Version 2015.08.24 to 2016.01.19
205 Clarified README to indicate why the 60 size is the preferred size
208 Remove some very uncommon possessive forms.
210 Change "SET UTF8" to "SET UTF-8" in hunspell affix file.
212 From Version 2015.05.18 to 2015.08.24 (Aug 24, 2015)
216 From Version 2015.04.24 to 2015.05.18 (May 18, 2015)
218 Added some new words found to have a high frequency in the COCA
219 corpus. (http://corpus.byu.edu/coca/).
221 Fix en spelling suggestions for 'alot' and 'exersize' in hunspell
222 dictionary (upstreamed from the changes made in Firefox).
224 From Version 2015.02.15 to 2015.04.24 (April 24, 2015)
226 Added some new words.
228 Convert hunspell dictionary to UTF-8 in order to handle smart
231 From Version 2015.01.28 to 2015.02.15 (February 15, 2015)
233 Added a large number of neologisms (newly invented words)
234 such as "selfie" and "smartwatch" thanks to Alan Beale.
236 Various other new words.
238 Clean up the special-hacker category by removing some words that
239 didn't exist in the Google Book's Corpus (1980 - 2008) and
240 originated from the "Unofficial Jargon File Word Lists".
242 From Version 2014.11.17 to 2015.01.28 (January 28, 2015)
244 Various new words, many from analyzing the Google Book's Corpus
245 (1980 - 2008). See http://app.aspell.net/lookup-freq.
247 Moved some uncommon words that can easily hide a misspelling of a
248 more common word to level 70. (calender, adrenalin and Joesph)
250 Removed several -er and -est forms from adjectives that were so
251 uncommon that they were not found anywhere is the Google Book's
252 Corpus (1980 - 2008).
254 From Version 2014.08.11.1 to 2014.11.17 (November 17, 2014)
258 Fix typo in Hunspell readme.
260 From Version 2014.08.11 to 2014.08.11.1 (August 13, 2014)
262 Forgot to mention this important change from 7.1 to 2014.08.11:
264 Shifted the variant levels up by one: variant_0 is now variant_1,
265 variant_1 is now variant_2, and variant_2 is now variant_3.
267 Other minor fixes in this README.
269 No changes to the contents of the lists.
271 From Revision 7.1 to Version 2014.08.11 (August 11, 2014)
273 Added some missing possessive forms.
275 Added some new words and proper names.
277 Clean up the categories (words, upper, proper-names etc) so that they
280 Convert documentation to UTF-8. For now, the wordlist are still in
281 ISO-8859-1 to prevent compatibility problems.
283 Add schema and scripts for creating a SQLite database from SCOWL.
284 Add some utility and library functions using them. This database is
285 used by the new web app's (http://app.aspell.net/lookup & create).
287 Enhance speller/make-hunspell-dict. The biggest improvement is that
288 it that it now generates several more dictionaries in addition to
289 the official ones. These additional dictionaries are ones for
290 British English and larger dictionaries that include up to SCOWL
293 From Revision 7 to 7.1 (January 6, 2011)
295 Updated to revision 5.1 of Varcon which corrected several errors.
297 Fixed various problems with the variant processing which corrected a
300 Added several now common proper names and some other words now
303 Include misc/ and speller/ directory which were in SVN but left
304 out of the release tarball.
306 Other minor fixes, including some fixes to the taboo word lists.
308 From Revision 6 to 7 (December 27, 2010)
310 Updated to revision 5.0 of Varcon which corrected many errors,
311 especially in the British and Canadian spelling categories. Also
312 added new spelling categories for the British and Canadian spelling
313 variants and separated them out from the main variant_* categories.
315 Moved Moby names lists (3897male.nam 4946fema.len 21986na.mes) to 95
316 level since they contain too many errors and rare names.
318 Moved frequently class 0 from Brian Kelk's Wordlist from
319 level 60 to 70, and also filter it with level 80 due to, too many
322 Many other minor fixes.
324 From Revision 5 to 6 (August 10, 2004)
326 Updated to version 4.0 of the 12dicts package.
328 Included the 3esl, 2of4brif, and 5desk list from the new 12dicts
329 package. The 3esl was included in the 40 size, the 2of4brif in the
330 55 size and the 5desk in the 70 size.
332 Removed the Ispell word list as it was a source of too many errors.
333 This eliminated the 65 size.
335 Removed clause 4 from the Ispell copyright with permission of Geoff
338 Updated to version 4.1 of VarCon.
340 Added the "british_z" spelling category which is British using the
343 From Revision 4a to 5 (January 3, 2002)
345 Added variants that were not really spelling variants (such as
346 forwards) back into the main list.
348 Fixed a bug which caused variants of words to incorrectly appear in
349 the non-variant lists.
351 Moved rarely used inflections of a word into higher number lists.
353 Added other inflections of a words based on the following criteria
354 If the word is in the base form: only include that word.
355 If the word is in a plural form: include the base word and the plural
356 If the word is a verb form (other than plural): include all verb forms
357 If the word is an ad* form: include all ad* forms
358 If the word is in a possessive form: also include the non-possessive
360 Updated to the latest version of many of the source dictionaries.
362 Removed the DEC Word List due to the questionable licence and
363 because removing it will not seriously decrease the quality of SCOWL
364 (there are a few less proper names).
366 From Revision 4 to 4a (April 4, 2001)
368 Reran the scripts on a never version of AGID (3a) which fixes a bug
369 which caused some common words to be improperly marked as variants.
371 From Revision 3 to 4 (January 28, 2001)
373 Split the variant "spelling category" up into 3 different levels.
375 Added words in the Ispell word list at the 65 level.
377 Other changes due to using more recent versions of various sources
378 included a more accurate version of AGID thanks to the work of
381 From Revision 2 to 3 (August 18, 2000)
383 Renamed special-unix-terms to special-hacker and added a large
384 number of commonly used words within the hacker (not cracker)
387 Added a couple more signature words including "newbie".
389 Minor changes due to changes in the inflection database.
391 From Revision 1 to 2 (August 5, 2000)
393 Moved the male and female name lists from the mwords package and the
394 DEC name lists form the 50 level to the 60 level and moved Alan's
395 name list from the 60 level to the 50 level. Also added the top
396 1000 male, female, and last names from the 1990 Census report to the
397 50 level. This reduced the number of names in the 50 level from
400 Added a large number of Uppercase words to the 50 level.
402 Properly accented the possessive form of some words.
404 Minor other changes due to changes in my raw data files which have
405 not been released yet. Email if you are interested in these files.
407 COPYRIGHT, SOURCES, and CREDITS:
409 The collective work is Copyright 2000-2016 by Kevin Atkinson as well
410 as any of the copyrights mentioned below:
412 Copyright 2000-2016 by Kevin Atkinson
414 Permission to use, copy, modify, distribute and sell these word
415 lists, the associated scripts, the output created from the scripts,
416 and its documentation for any purpose is hereby granted without fee,
417 provided that the above copyright notice appears in all copies and
418 that both that copyright notice and this permission notice appear in
419 supporting documentation. Kevin Atkinson makes no representations
420 about the suitability of this array for any purpose. It is provided
421 "as is" without express or implied warranty.
423 Alan Beale <biljir@pobox.com> also deserves special credit as he has,
424 in addition to providing the 12Dicts package and being a major
425 contributor to the ENABLE word list, given me an incredible amount of
426 feedback and created a number of special lists (those found in the
427 Supplement) in order to help improve the overall quality of SCOWL.
429 The 10 level includes the 1000 most common English words (according to
430 the Moby (TM) Words II [MWords] package), a subset of the 1000 most
431 common words on the Internet (again, according to Moby Words II), and
432 frequently class 16 from Brian Kelk's "UK English Wordlist
433 with Frequency Classification".
435 The MWords package was explicitly placed in the public domain:
437 The Moby lexicon project is complete and has
438 been place into the public domain. Use, sell,
439 rework, excerpt and use in any way on any platform.
441 Placing this material on internal or public servers is
442 also encouraged. The compiler is not aware of any
443 export restrictions so freely distribute world-wide.
445 You can verify the public domain status by contacting
449 Arcata, CA 95521-4884
454 The "UK English Wordlist With Frequency Classification" is also in the
457 Date: Sat, 08 Jul 2000 20:27:21 +0100
458 From: Brian Kelk <Brian.Kelk@cl.cam.ac.uk>
460 > I was wondering what the copyright status of your "UK English
461 > Wordlist With Frequency Classification" word list as it seems to
462 > be lacking any copyright notice.
464 There were many many sources in total, but any text marked
465 "copyright" was avoided. Locally-written documentation was one
466 source. An earlier version of the list resided in a filespace called
467 PUBLIC on the University mainframe, because it was considered public
470 Date: Tue, 11 Jul 2000 19:31:34 +0100
472 > So are you saying your word list is also in the public domain?
474 That is the intention.
476 The 20 level includes frequency classes 7-15 from Brian's word list.
478 The 35 level includes frequency classes 2-6 and words appearing in at
479 least 11 of 12 dictionaries as indicated in the 12Dicts package. All
480 words from the 12Dicts package have had likely inflections added via
481 my inflection database.
483 The 12Dicts package and Supplement is in the Public Domain.
485 The WordNet database, which was used in the creation of the
486 Inflections database, is under the following copyright:
488 This software and database is being provided to you, the LICENSEE,
489 by Princeton University under the following license. By obtaining,
490 using and/or copying this software and database, you agree that you
491 have read, understood, and will comply with these terms and
494 Permission to use, copy, modify and distribute this software and
495 database and its documentation for any purpose and without fee or
496 royalty is hereby granted, provided that you agree to comply with
497 the following copyright notice and statements, including the
498 disclaimer, and that the same appear on ALL copies of the software,
499 database and documentation, including modifications that you make
500 for internal use or for distribution.
502 WordNet 1.6 Copyright 1997 by Princeton University. All rights
505 THIS SOFTWARE AND DATABASE IS PROVIDED "AS IS" AND PRINCETON
506 UNIVERSITY MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR
507 IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, PRINCETON
508 UNIVERSITY MAKES NO REPRESENTATIONS OR WARRANTIES OF MERCHANT-
509 ABILITY OR FITNESS FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF THE
510 LICENSED SOFTWARE, DATABASE OR DOCUMENTATION WILL NOT INFRINGE ANY
511 THIRD PARTY PATENTS, COPYRIGHTS, TRADEMARKS OR OTHER RIGHTS.
513 The name of Princeton University or Princeton may not be used in
514 advertising or publicity pertaining to distribution of the software
515 and/or database. Title to copyright in this software, database and
516 any associated documentation shall at all times remain with
517 Princeton University and LICENSEE agrees to preserve same.
519 The 40 level includes words from Alan's 3esl list found in version 4.0
520 of his 12dicts package. Like his other stuff the 3esl list is also in the
523 The 50 level includes Brian's frequency class 1, words appearing
524 in at least 5 of 12 of the dictionaries as indicated in the 12Dicts
525 package, and uppercase words in at least 4 of the previous 12
526 dictionaries. A decent number of proper names is also included: The
527 top 1000 male, female, and Last names from the 1990 Census report; a
528 list of names sent to me by Alan Beale; and a few names that I added
529 myself. Finally a small list of abbreviations not commonly found in
530 other word lists is included.
532 The name files form the Census report is a government document which I
533 don't think can be copyrighted.
535 The file special-jargon.50 uses common.lst and word.lst from the
536 "Unofficial Jargon File Word Lists" which is derived from "The Jargon
537 File". All of which is in the Public Domain. This file also contain
538 a few extra UNIX terms which are found in the file "unix-terms" in the
541 The 55 level includes words from Alan's 2of4brif list found in version
542 4.0 of his 12dicts package. Like his other stuff the 2of4brif is also
543 in the public domain.
545 The 60 level includes all words appearing in at least 2 of the 12
546 dictionaries as indicated by the 12Dicts package.
548 The 70 level includes Brian's frequency class 0 and the 74,550 common
549 dictionary words from the MWords package. The common dictionary words,
550 like those from the 12Dicts package, have had all likely inflections
551 added. The 70 level also included the 5desk list from version 4.0 of
552 the 12Dics package which is in the public domain.
554 The 80 level includes the ENABLE word list, all the lists in the
555 ENABLE supplement package (except for ABLE), the "UK Advanced Cryptics
556 Dictionary" (UKACD), the list of signature words from the YAWL package,
557 and the 10,196 places list from the MWords package.
559 The ENABLE package, mainted by M\Cooper <thegrendel@theriver.com>,
560 is in the Public Domain:
562 The ENABLE master word list, WORD.LST, is herewith formally released
563 into the Public Domain. Anyone is free to use it or distribute it in
564 any manner they see fit. No fee or registration is required for its
565 use nor are "contributions" solicited (if you feel you absolutely
566 must contribute something for your own peace of mind, the authors of
567 the ENABLE list ask that you make a donation on their behalf to your
568 favorite charity). This word list is our gift to the Scrabble
569 community, as an alternate to "official" word lists. Game designers
570 may feel free to incorporate the WORD.LST into their games. Please
571 mention the source and credit us as originators of the list. Note
572 that if you, as a game designer, use the WORD.LST in your product,
573 you may still copyright and protect your product, but you may *not*
574 legally copyright or in any way restrict redistribution of the
575 WORD.LST portion of your product. This *may* under law restrict your
576 rights to restrict your users' rights, but that is only fair.
578 UKACD, by J Ross Beresford <ross@bryson.demon.co.uk>, is under the
581 Copyright (c) J Ross Beresford 1993-1999. All Rights Reserved.
583 The following restriction is placed on the use of this publication:
584 if The UK Advanced Cryptics Dictionary is used in a software package
585 or redistributed in any form, the copyright notice must be
586 prominently displayed and the text of this document must be included
589 There are no other restrictions: I would like to see the list
590 distributed as widely as possible.
592 The 95 level includes the 354,984 single words, 256,772 compound
593 words, 4,946 female names and the 3,897 male names, and 21,986 names
594 from the MWords package, ABLE.LST from the ENABLE Supplement, and some
595 additional words found in my part-of-speech database that were not
598 Accent information was taken from UKACD.
600 The VarCon package was used to create the American, British, Canadian,
601 and Australian word list. It is under the following copyright:
603 Copyright 2000-2016 by Kevin Atkinson
605 Permission to use, copy, modify, distribute and sell this array, the
606 associated software, and its documentation for any purpose is hereby
607 granted without fee, provided that the above copyright notice appears
608 in all copies and that both that copyright notice and this permission
609 notice appear in supporting documentation. Kevin Atkinson makes no
610 representations about the suitability of this array for any
611 purpose. It is provided "as is" without express or implied warranty.
613 Copyright 2016 by Benjamin Titze
615 Permission to use, copy, modify, distribute and sell this array, the
616 associated software, and its documentation for any purpose is hereby
617 granted without fee, provided that the above copyright notice appears
618 in all copies and that both that copyright notice and this permission
619 notice appear in supporting documentation. Benjamin Titze makes no
620 representations about the suitability of this array for any
621 purpose. It is provided "as is" without express or implied warranty.
623 Since the original words lists come from the Ispell distribution:
625 Copyright 1993, Geoff Kuenning, Granada Hills, CA
628 Redistribution and use in source and binary forms, with or without
629 modification, are permitted provided that the following conditions
632 1. Redistributions of source code must retain the above copyright
633 notice, this list of conditions and the following disclaimer.
634 2. Redistributions in binary form must reproduce the above copyright
635 notice, this list of conditions and the following disclaimer in the
636 documentation and/or other materials provided with the distribution.
637 3. All modifications to the source code must be clearly marked as
638 such. Binary redistributions based on modified source code
639 must be clearly marked as modified versions in the documentation
640 and/or other materials provided with the distribution.
641 (clause 4 removed with permission from Geoff Kuenning)
642 5. The name of Geoff Kuenning may not be used to endorse or promote
643 products derived from this software without specific prior
646 THIS SOFTWARE IS PROVIDED BY GEOFF KUENNING AND CONTRIBUTORS ``AS IS'' AND
647 ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
648 IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
649 ARE DISCLAIMED. IN NO EVENT SHALL GEOFF KUENNING OR CONTRIBUTORS BE LIABLE
650 FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
651 DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
652 OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
653 HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
654 LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
655 OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
659 The variant word lists were created from a list of variants found in
660 the 12dicts supplement package as well as a list of variants I created
663 The Readmes for the various packages used can be found in the
664 appropriate directory under the r/ directory.
668 The process of "sort"s, "comm"s, and Perl scripts to combine the many
669 word lists and separate out the variant information is inexact and
670 error prone. The whole things needs to be rewritten to deal with
671 words in terms of lemmas. When the exact lemma is not known a best
672 guess should be made. I'm not sure what form this should be in. I
673 originally thought this should be some sort of database, but maybe I
674 should just slurp all that data into memory and process it in one
675 giant perl script. With the amount of memory available these days (at
676 least 2 GB, often 4 GB or more) this should not really be a problem.
678 In addition, there is a very nice frequency analyze of the BNC corpus
679 done by Adam Kilgarriff. Unlike Brian's word lists the BNC lists
680 include part of speech information. I plan on somehow using these
681 lists as Adam Kilgarriff has given me the OK to use it in SCOWL.
682 These lists will greatly reduce the problem of inflected forms of a
683 word appearing at different levels due to the part-of-speech
686 There is frequency information for some other corpus such as COCA
687 (Corpus of Contemporary American English) and ANS (American National
688 Corpus) which I might also be able to use. The former will require
689 permission, and the latter is of questionable quality.
691 RECREATING THE WORD LISTS:
693 In order to recreate the word lists you need a modern version of Perl,
694 bash, the traditional set of shell utilities, a system that supports
695 symbolic links, and quite possibly GNU Make. The easiest way to
696 recreate the word lists is to checkout the corresponding Git version
697 (see the version string at the start of the file) and simply type
698 "make" (see http://wordlist.aspell.net). You can try to download all
699 the pieces manually, but this method is not no longer tested nor
702 The src/ directory contains the numerous scripts used in the creation
703 of the final product.
705 The r/ directory contains the raw data used to create the final
706 product. If you checkout from Git this directory should be populated
707 automatically for you. If you insist on doing it the hard way see the
708 README file in the r/ directory for more information.
710 The l/ directory contains symbolic links used by the actual scripts.
712 Finally, the working/ directory is where all the intermittent files go
713 that are not specific to one source.