1 Spell Checking Oriented Word Lists (SCOWL)
3 Sun Oct 6 20:44:03 2019 -0400 [755d6dd]
4 by Kevin Atkinson (kevina@gnu.org)
6 The SCOWL is a collection of word lists split up in various sizes, and
7 other categories, intended to be suitable for use in spell checkers.
8 However, I am sure it will have numerous other uses as well.
10 The latest version can be found at http://wordlist.aspell.net/.
12 The directory final/ contains the actual word lists broken up into
13 various sizes and categories. The r/ directory contains Readmes from
14 the various sources used to create this package.
16 The misc/ contains a small list of taboo words, see the README file
17 for more info. The speller/ directory contains scripts for creating
18 spelling dictionaries for Aspell and Hunspell.
20 The other directories contain the necessary information to recreate the
21 word lists from the raw data. Unless you are interested in improving the
22 words lists you should not need to worry about what's here. See the
23 section on recreating the words lists for more information on what's
26 Except for the special word lists the files follow the following
28 <spelling category>-<sub-category>.<size>
29 Where the spelling category is one of
30 english, american, british, british_z, canadian, australian
31 variant_1, variant_2, variant_3,
32 british_variant_1, british_variant_2,
33 canadian_variant_1, canadian_variant_2,
34 australian_variant_1, australian_variant_2
35 Sub-category is one of
36 abbreviations, contractions, proper-names, upper, words
38 10, 20, 35 (small), 40, 50 (medium), 55, 60, 70 (large),
39 80 (huge), 95 (insane)
40 The special word lists follow are in the following format:
41 special-<description>.<size>
42 Where description is one of:
43 roman-numerals, hacker
45 The perl script "mk-list" can be used to create a word list of the
46 desired size, its usage is:
47 ./mk-list [-f] [-v#] <spelling categories> <size>
48 where <spelling categories> is one of the above spelling categories
49 (the english and special categories are automatically included as well
50 as all sub-categories) and <size> is the desired size. The
51 "-v" option can be used to also include the appropriate
52 variants file up to level '#'. The normal output will be a sorted
53 word list. If you rather see what files will be included, use the
56 When manually combining the words lists the "english" spelling
57 category should be used as well as one of "american", "british",
58 "british_z" (british with ize spelling), "canadian" or "australian".
59 Great care has been taken so that only one spelling for any particular
60 word is included in the main list (with some minor exceptions). When
61 two variants were considered equal I randomly picked one for inclusion
62 in the main word list. Unfortunately this means that my choice in how
63 to spell a word may not match your choice. If this is the case you
64 can try including one of the "variant_1" spelling categories which
65 includes most variants which are considered almost equal. The
66 "variant_1" spelling category corresponds mostly to American variants,
67 while the "british_variant_1", "canadian_variant_1" and
68 "australian_variant_1" are for British, Canadian and Australian
69 variants, respectively. The "variant_2" spelling categories include
70 variants which are also generally considered acceptable, and
71 "variant_3" contains variants which are seldom used and may not even
72 be considered correct. There is no "british_variant_3",
73 "canadian_variant_3" or "australian_variant_3" spelling category since
74 the distinction would be almost meaningless.
76 The "abbreviation" category includes abbreviations and acronyms which
77 are not also normal words. The "contractions" category should be self
78 explanatory. The "upper" category includes upper case words and proper
79 names which are common enough to appear in a typical dictionary. The
80 "proper-names" category includes all the additional uppercase words.
81 Finally the "words" category contains all the normal English words.
83 To give you an idea of what the words in the various sizes look like
84 here is a sample of 25 random words found only in that size:
86 10: anyone arrives asks calculate change compromise cost discussed doubtful
87 encountering external feed images isolate materials necessary owner phase
88 precisely programmer reflected regular sex sound trap
90 20: brave cage commit cooked courier crunches dashes disconnect fantasy
91 fights filter inclination leak noticeably overseas rotating sights
92 socially sole song spit swallowing triumph trousers unwise
94 35: awaking creeping crucifix defacing dome ethically garnish granular hedges
95 hushing impotence jaunt lifeboat militated nearsightedness notations pew
96 rawer repulse sardines scoffs tripping tweaked upholds viability
98 40: alohas badmouths chump clobber cockiness deviants disfigurements fests
99 fuck gassiest geologic gizmo impersonator masseuse monochromes peppy
100 pigsties piss publicists rethinks slushier smooching sweltered
103 50: acquirable aquanauts blinders circlet condoling despoil dormouse
104 emulsification fetishist hansoms interrogative misapply miscounting
105 naysayers ovulation palefaces pasha phoneyed photosensitive
106 significations skylark squiggle supremacist tiresomeness wildfowl
108 55: anglicize aquatically autobahns beanpole bevvies centralism cuboids
109 drapers footballing ghettoizes gorgons hoofer immobilizers magicked
110 neckband neckbands prezzies scorekeepers spymasters syllabubs tinplate
111 treacly uncomprehendingly yellowness yuppified
113 60: activator airbuses beadles chevalier comfortableness consulship dabber
114 daces inexpiable marriageability nondisclosure palatine pantywaists
115 postmeridian preformed rabbeted reedit rezoning satori terrycloth
116 thrombotic tradeswomen unapproved versa whippletree
118 70: adactylous aerometry animalism chalcedonic crownpiece downburst
119 electrocorticogram foreshowed irenicism irresponsibleness jacklighting
120 lewis lippiness naumachias nihil nonobedience normalizer pipage pyas
121 rickettsias secco superrich tetanize thromboembolisms ultramodernism
123 80: burhels convivialists defeudalizing détraquée explosivenesses fies
124 flagrancies fluidifies gratillity houdah indigolite lamaistic multiagency
125 oporice paupered preappointed progressionism radicating reccy sheriffdom
126 sloebushes southeasts steening tourings unpresuming
128 95: acierations comminator coumbite deligated foremisgiving impalmed kerrite
129 laverocked mirandous nearaways nonceremonial nonlyrical pbxes
130 periependymal preinsinuate quistron somatognostic taxodont terebate thisn
131 tracksick transubstantiationalists unresembling unstrategically verquire
134 And here is a count on the number of words in each spelling category
135 (american + english spelling category):
137 Size Words Names Running Total %
138 10 4,425 13 4,438 0.7
139 20 8,128 0 12,566 1.9
140 35 37,259 222 50,047 7.6
141 40 6,853 491 57,391 8.7
142 50 25,238 18,680 101,309 15.4
143 55 6,489 0 107,798 16.4
144 60 14,516 850 123,164 18.7
145 70 35,303 7,897 166,364 25.3
146 80 144,178 33,367 343,909 52.3
147 95 227,641 86,631 658,181 100.0
150 (The "Words" column does not include the name count.)
152 Size 35 is the recommended small size, 50 the medium and 70 the large.
153 Sizes 70 and below contain words found in most dictionaries while the
154 80 size contains all the strange and unusual words people like to use
155 in word games such as Scrabble (TM). While a lot of the words in the
156 80 size are not used very often, they are all generally considered
157 valid words in the English language. The 95 contains just about every
158 English word in existence and then some. Many of the words at the 95
159 level will probably not be considered valid English words by most
162 For spell checking I recommend using size 60. This size is the
163 largest size that I am fairly confident does not contain any
164 misspellings or invalid words. In addition an effort is made to
165 exclude valid yet problematic words (such as "calender") from the 60
166 size that are likely to be a misspelling of a more common word. The
167 70 size is reasonable for those wanting a larger list and don't mind a
168 few errors. The 80 or larger sizes are not reasonable for spell
171 Accents are present on certain words such as café in iso8859-1 format.
175 From Version 2018.04.16 to 2019.10.06
179 Remove compare's and fail's.
181 From Version 2017.08.24 to 2018.04.16
185 Fix build problems on macOS.
187 From Version 2017.01.22 to 2017.08.24
191 From Version 2016.11.20 to 2017.01.22
195 From Version 2016.06.26 to 2016.11.20
197 New Australian spelling category thanks to the work of Benjamin
198 Titze (btitze@protonmail.ch)
202 From Version 2016.01.19 to 2016.06.26
206 Updated to Version 6.0.2 of 12dicts
210 From Version 2015.08.24 to 2016.01.19
214 Clarified README to indicate why the 60 size is the preferred size
217 Remove some very uncommon possessive forms.
219 Change "SET UTF8" to "SET UTF-8" in hunspell affix file.
221 From Version 2015.05.18 to 2015.08.24 (Aug 24, 2015)
225 From Version 2015.04.24 to 2015.05.18 (May 18, 2015)
227 Added some new words found to have a high frequency in the COCA
228 corpus. (http://corpus.byu.edu/coca/).
230 Fix en spelling suggestions for 'alot' and 'exersize' in hunspell
231 dictionary (upstreamed from the changes made in Firefox).
233 From Version 2015.02.15 to 2015.04.24 (April 24, 2015)
235 Added some new words.
237 Convert hunspell dictionary to UTF-8 in order to handle smart
240 From Version 2015.01.28 to 2015.02.15 (February 15, 2015)
242 Added a large number of neologisms (newly invented words)
243 such as "selfie" and "smartwatch" thanks to Alan Beale.
245 Various other new words.
247 Clean up the special-hacker category by removing some words that
248 didn't exist in the Google Book's Corpus (1980 - 2008) and
249 originated from the "Unofficial Jargon File Word Lists".
251 From Version 2014.11.17 to 2015.01.28 (January 28, 2015)
253 Various new words, many from analyzing the Google Book's Corpus
254 (1980 - 2008). See http://app.aspell.net/lookup-freq.
256 Moved some uncommon words that can easily hide a misspelling of a
257 more common word to level 70. (calender, adrenalin and Joesph)
259 Removed several -er and -est forms from adjectives that were so
260 uncommon that they were not found anywhere is the Google Book's
261 Corpus (1980 - 2008).
263 From Version 2014.08.11.1 to 2014.11.17 (November 17, 2014)
267 Fix typo in Hunspell readme.
269 From Version 2014.08.11 to 2014.08.11.1 (August 13, 2014)
271 Forgot to mention this important change from 7.1 to 2014.08.11:
273 Shifted the variant levels up by one: variant_0 is now variant_1,
274 variant_1 is now variant_2, and variant_2 is now variant_3.
276 Other minor fixes in this README.
278 No changes to the contents of the lists.
280 From Revision 7.1 to Version 2014.08.11 (August 11, 2014)
282 Added some missing possessive forms.
284 Added some new words and proper names.
286 Clean up the categories (words, upper, proper-names etc) so that they
289 Convert documentation to UTF-8. For now, the wordlist are still in
290 ISO-8859-1 to prevent compatibility problems.
292 Add schema and scripts for creating a SQLite database from SCOWL.
293 Add some utility and library functions using them. This database is
294 used by the new web app's (http://app.aspell.net/lookup & create).
296 Enhance speller/make-hunspell-dict. The biggest improvement is that
297 it that it now generates several more dictionaries in addition to
298 the official ones. These additional dictionaries are ones for
299 British English and larger dictionaries that include up to SCOWL
302 From Revision 7 to 7.1 (January 6, 2011)
304 Updated to revision 5.1 of Varcon which corrected several errors.
306 Fixed various problems with the variant processing which corrected a
309 Added several now common proper names and some other words now
312 Include misc/ and speller/ directory which were in SVN but left
313 out of the release tarball.
315 Other minor fixes, including some fixes to the taboo word lists.
317 From Revision 6 to 7 (December 27, 2010)
319 Updated to revision 5.0 of Varcon which corrected many errors,
320 especially in the British and Canadian spelling categories. Also
321 added new spelling categories for the British and Canadian spelling
322 variants and separated them out from the main variant_* categories.
324 Moved Moby names lists (3897male.nam 4946fema.len 21986na.mes) to 95
325 level since they contain too many errors and rare names.
327 Moved frequently class 0 from Brian Kelk's Wordlist from
328 level 60 to 70, and also filter it with level 80 due to, too many
331 Many other minor fixes.
333 From Revision 5 to 6 (August 10, 2004)
335 Updated to version 4.0 of the 12dicts package.
337 Included the 3esl, 2of4brif, and 5desk list from the new 12dicts
338 package. The 3esl was included in the 40 size, the 2of4brif in the
339 55 size and the 5desk in the 70 size.
341 Removed the Ispell word list as it was a source of too many errors.
342 This eliminated the 65 size.
344 Removed clause 4 from the Ispell copyright with permission of Geoff
347 Updated to version 4.1 of VarCon.
349 Added the "british_z" spelling category which is British using the
352 From Revision 4a to 5 (January 3, 2002)
354 Added variants that were not really spelling variants (such as
355 forwards) back into the main list.
357 Fixed a bug which caused variants of words to incorrectly appear in
358 the non-variant lists.
360 Moved rarely used inflections of a word into higher number lists.
362 Added other inflections of a words based on the following criteria
363 If the word is in the base form: only include that word.
364 If the word is in a plural form: include the base word and the plural
365 If the word is a verb form (other than plural): include all verb forms
366 If the word is an ad* form: include all ad* forms
367 If the word is in a possessive form: also include the non-possessive
369 Updated to the latest version of many of the source dictionaries.
371 Removed the DEC Word List due to the questionable licence and
372 because removing it will not seriously decrease the quality of SCOWL
373 (there are a few less proper names).
375 From Revision 4 to 4a (April 4, 2001)
377 Reran the scripts on a never version of AGID (3a) which fixes a bug
378 which caused some common words to be improperly marked as variants.
380 From Revision 3 to 4 (January 28, 2001)
382 Split the variant "spelling category" up into 3 different levels.
384 Added words in the Ispell word list at the 65 level.
386 Other changes due to using more recent versions of various sources
387 included a more accurate version of AGID thanks to the work of
390 From Revision 2 to 3 (August 18, 2000)
392 Renamed special-unix-terms to special-hacker and added a large
393 number of commonly used words within the hacker (not cracker)
396 Added a couple more signature words including "newbie".
398 Minor changes due to changes in the inflection database.
400 From Revision 1 to 2 (August 5, 2000)
402 Moved the male and female name lists from the mwords package and the
403 DEC name lists form the 50 level to the 60 level and moved Alan's
404 name list from the 60 level to the 50 level. Also added the top
405 1000 male, female, and last names from the 1990 Census report to the
406 50 level. This reduced the number of names in the 50 level from
409 Added a large number of Uppercase words to the 50 level.
411 Properly accented the possessive form of some words.
413 Minor other changes due to changes in my raw data files which have
414 not been released yet. Email if you are interested in these files.
416 COPYRIGHT, SOURCES, and CREDITS:
418 The collective work is Copyright 2000-2018 by Kevin Atkinson as well
419 as any of the copyrights mentioned below:
421 Copyright 2000-2018 by Kevin Atkinson
423 Permission to use, copy, modify, distribute and sell these word
424 lists, the associated scripts, the output created from the scripts,
425 and its documentation for any purpose is hereby granted without fee,
426 provided that the above copyright notice appears in all copies and
427 that both that copyright notice and this permission notice appear in
428 supporting documentation. Kevin Atkinson makes no representations
429 about the suitability of this array for any purpose. It is provided
430 "as is" without express or implied warranty.
432 Alan Beale <biljir@pobox.com> also deserves special credit as he has,
433 in addition to providing the 12Dicts package and being a major
434 contributor to the ENABLE word list, given me an incredible amount of
435 feedback and created a number of special lists (those found in the
436 Supplement) in order to help improve the overall quality of SCOWL.
438 The 10 level includes the 1000 most common English words (according to
439 the Moby (TM) Words II [MWords] package), a subset of the 1000 most
440 common words on the Internet (again, according to Moby Words II), and
441 frequently class 16 from Brian Kelk's "UK English Wordlist
442 with Frequency Classification".
444 The MWords package was explicitly placed in the public domain:
446 The Moby lexicon project is complete and has
447 been place into the public domain. Use, sell,
448 rework, excerpt and use in any way on any platform.
450 Placing this material on internal or public servers is
451 also encouraged. The compiler is not aware of any
452 export restrictions so freely distribute world-wide.
454 You can verify the public domain status by contacting
458 Arcata, CA 95521-4884
463 The "UK English Wordlist With Frequency Classification" is also in the
466 Date: Sat, 08 Jul 2000 20:27:21 +0100
467 From: Brian Kelk <Brian.Kelk@cl.cam.ac.uk>
469 > I was wondering what the copyright status of your "UK English
470 > Wordlist With Frequency Classification" word list as it seems to
471 > be lacking any copyright notice.
473 There were many many sources in total, but any text marked
474 "copyright" was avoided. Locally-written documentation was one
475 source. An earlier version of the list resided in a filespace called
476 PUBLIC on the University mainframe, because it was considered public
479 Date: Tue, 11 Jul 2000 19:31:34 +0100
481 > So are you saying your word list is also in the public domain?
483 That is the intention.
485 The 20 level includes frequency classes 7-15 from Brian's word list.
487 The 35 level includes frequency classes 2-6 and words appearing in at
488 least 11 of 12 dictionaries as indicated in the 12Dicts package. All
489 words from the 12Dicts package have had likely inflections added via
490 my inflection database.
492 The 12Dicts package and Supplement is in the Public Domain.
494 The WordNet database, which was used in the creation of the
495 Inflections database, is under the following copyright:
497 This software and database is being provided to you, the LICENSEE,
498 by Princeton University under the following license. By obtaining,
499 using and/or copying this software and database, you agree that you
500 have read, understood, and will comply with these terms and
503 Permission to use, copy, modify and distribute this software and
504 database and its documentation for any purpose and without fee or
505 royalty is hereby granted, provided that you agree to comply with
506 the following copyright notice and statements, including the
507 disclaimer, and that the same appear on ALL copies of the software,
508 database and documentation, including modifications that you make
509 for internal use or for distribution.
511 WordNet 1.6 Copyright 1997 by Princeton University. All rights
514 THIS SOFTWARE AND DATABASE IS PROVIDED "AS IS" AND PRINCETON
515 UNIVERSITY MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR
516 IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, PRINCETON
517 UNIVERSITY MAKES NO REPRESENTATIONS OR WARRANTIES OF MERCHANT-
518 ABILITY OR FITNESS FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF THE
519 LICENSED SOFTWARE, DATABASE OR DOCUMENTATION WILL NOT INFRINGE ANY
520 THIRD PARTY PATENTS, COPYRIGHTS, TRADEMARKS OR OTHER RIGHTS.
522 The name of Princeton University or Princeton may not be used in
523 advertising or publicity pertaining to distribution of the software
524 and/or database. Title to copyright in this software, database and
525 any associated documentation shall at all times remain with
526 Princeton University and LICENSEE agrees to preserve same.
528 The 40 level includes words from Alan's 3esl list found in version 4.0
529 of his 12dicts package. Like his other stuff the 3esl list is also in the
532 The 50 level includes Brian's frequency class 1, words appearing
533 in at least 5 of 12 of the dictionaries as indicated in the 12Dicts
534 package, and uppercase words in at least 4 of the previous 12
535 dictionaries. A decent number of proper names is also included: The
536 top 1000 male, female, and Last names from the 1990 Census report; a
537 list of names sent to me by Alan Beale; and a few names that I added
538 myself. Finally a small list of abbreviations not commonly found in
539 other word lists is included.
541 The name files form the Census report is a government document which I
542 don't think can be copyrighted.
544 The file special-jargon.50 uses common.lst and word.lst from the
545 "Unofficial Jargon File Word Lists" which is derived from "The Jargon
546 File". All of which is in the Public Domain. This file also contain
547 a few extra UNIX terms which are found in the file "unix-terms" in the
550 The 55 level includes words from Alan's 2of4brif list found in version
551 4.0 of his 12dicts package. Like his other stuff the 2of4brif is also
552 in the public domain.
554 The 60 level includes all words appearing in at least 2 of the 12
555 dictionaries as indicated by the 12Dicts package.
557 The 70 level includes Brian's frequency class 0 and the 74,550 common
558 dictionary words from the MWords package. The common dictionary words,
559 like those from the 12Dicts package, have had all likely inflections
560 added. The 70 level also included the 5desk list from version 4.0 of
561 the 12Dics package which is in the public domain.
563 The 80 level includes the ENABLE word list, all the lists in the
564 ENABLE supplement package (except for ABLE), the "UK Advanced Cryptics
565 Dictionary" (UKACD), the list of signature words from the YAWL package,
566 and the 10,196 places list from the MWords package.
568 The ENABLE package, mainted by M\Cooper <thegrendel@theriver.com>,
569 is in the Public Domain:
571 The ENABLE master word list, WORD.LST, is herewith formally released
572 into the Public Domain. Anyone is free to use it or distribute it in
573 any manner they see fit. No fee or registration is required for its
574 use nor are "contributions" solicited (if you feel you absolutely
575 must contribute something for your own peace of mind, the authors of
576 the ENABLE list ask that you make a donation on their behalf to your
577 favorite charity). This word list is our gift to the Scrabble
578 community, as an alternate to "official" word lists. Game designers
579 may feel free to incorporate the WORD.LST into their games. Please
580 mention the source and credit us as originators of the list. Note
581 that if you, as a game designer, use the WORD.LST in your product,
582 you may still copyright and protect your product, but you may *not*
583 legally copyright or in any way restrict redistribution of the
584 WORD.LST portion of your product. This *may* under law restrict your
585 rights to restrict your users' rights, but that is only fair.
587 UKACD, by J Ross Beresford <ross@bryson.demon.co.uk>, is under the
590 Copyright (c) J Ross Beresford 1993-1999. All Rights Reserved.
592 The following restriction is placed on the use of this publication:
593 if The UK Advanced Cryptics Dictionary is used in a software package
594 or redistributed in any form, the copyright notice must be
595 prominently displayed and the text of this document must be included
598 There are no other restrictions: I would like to see the list
599 distributed as widely as possible.
601 The 95 level includes the 354,984 single words, 256,772 compound
602 words, 4,946 female names and the 3,897 male names, and 21,986 names
603 from the MWords package, ABLE.LST from the ENABLE Supplement, and some
604 additional words found in my part-of-speech database that were not
607 Accent information was taken from UKACD.
609 The VarCon package was used to create the American, British, Canadian,
610 and Australian word list. It is under the following copyright:
612 Copyright 2000-2016 by Kevin Atkinson
614 Permission to use, copy, modify, distribute and sell this array, the
615 associated software, and its documentation for any purpose is hereby
616 granted without fee, provided that the above copyright notice appears
617 in all copies and that both that copyright notice and this permission
618 notice appear in supporting documentation. Kevin Atkinson makes no
619 representations about the suitability of this array for any
620 purpose. It is provided "as is" without express or implied warranty.
622 Copyright 2016 by Benjamin Titze
624 Permission to use, copy, modify, distribute and sell this array, the
625 associated software, and its documentation for any purpose is hereby
626 granted without fee, provided that the above copyright notice appears
627 in all copies and that both that copyright notice and this permission
628 notice appear in supporting documentation. Benjamin Titze makes no
629 representations about the suitability of this array for any
630 purpose. It is provided "as is" without express or implied warranty.
632 Since the original words lists come from the Ispell distribution:
634 Copyright 1993, Geoff Kuenning, Granada Hills, CA
637 Redistribution and use in source and binary forms, with or without
638 modification, are permitted provided that the following conditions
641 1. Redistributions of source code must retain the above copyright
642 notice, this list of conditions and the following disclaimer.
643 2. Redistributions in binary form must reproduce the above copyright
644 notice, this list of conditions and the following disclaimer in the
645 documentation and/or other materials provided with the distribution.
646 3. All modifications to the source code must be clearly marked as
647 such. Binary redistributions based on modified source code
648 must be clearly marked as modified versions in the documentation
649 and/or other materials provided with the distribution.
650 (clause 4 removed with permission from Geoff Kuenning)
651 5. The name of Geoff Kuenning may not be used to endorse or promote
652 products derived from this software without specific prior
655 THIS SOFTWARE IS PROVIDED BY GEOFF KUENNING AND CONTRIBUTORS ``AS IS'' AND
656 ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
657 IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
658 ARE DISCLAIMED. IN NO EVENT SHALL GEOFF KUENNING OR CONTRIBUTORS BE LIABLE
659 FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
660 DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
661 OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
662 HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
663 LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
664 OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
668 The variant word lists were created from a list of variants found in
669 the 12dicts supplement package as well as a list of variants I created
672 The Readmes for the various packages used can be found in the
673 appropriate directory under the r/ directory.
677 The process of "sort"s, "comm"s, and Perl scripts to combine the many
678 word lists and separate out the variant information is inexact and
679 error prone. The whole things needs to be rewritten to deal with
680 words in terms of lemmas. When the exact lemma is not known a best
681 guess should be made. I'm not sure what form this should be in. I
682 originally thought this should be some sort of database, but maybe I
683 should just slurp all that data into memory and process it in one
684 giant perl script. With the amount of memory available these days (at
685 least 2 GB, often 4 GB or more) this should not really be a problem.
687 In addition, there is a very nice frequency analyze of the BNC corpus
688 done by Adam Kilgarriff. Unlike Brian's word lists the BNC lists
689 include part of speech information. I plan on somehow using these
690 lists as Adam Kilgarriff has given me the OK to use it in SCOWL.
691 These lists will greatly reduce the problem of inflected forms of a
692 word appearing at different levels due to the part-of-speech
695 There is frequency information for some other corpus such as COCA
696 (Corpus of Contemporary American English) and ANS (American National
697 Corpus) which I might also be able to use. The former will require
698 permission, and the latter is of questionable quality.
700 RECREATING THE WORD LISTS:
702 In order to recreate the word lists you need a modern version of Perl,
703 bash, the traditional set of shell utilities, a system that supports
704 symbolic links, and quite possibly GNU Make. The easiest way to
705 recreate the word lists is to checkout the corresponding Git version
706 (see the version string at the start of the file) and simply type
707 "make" (see http://wordlist.aspell.net). You can try to download all
708 the pieces manually, but this method is not no longer tested nor
711 The src/ directory contains the numerous scripts used in the creation
712 of the final product.
714 The r/ directory contains the raw data used to create the final
715 product. If you checkout from Git this directory should be populated
716 automatically for you. If you insist on doing it the hard way see the
717 README file in the r/ directory for more information.
719 The l/ directory contains symbolic links used by the actual scripts.
721 Finally, the working/ directory is where all the intermittent files go
722 that are not specific to one source.