1 Spell Checking Oriented Word Lists (SCOWL)
3 Sun Jan 22 17:39:13 2017 -0500 [fbc7107]
4 by Kevin Atkinson (kevina@gnu.org)
6 The SCOWL is a collection of word lists split up in various sizes, and
7 other categories, intended to be suitable for use in spell checkers.
8 However, I am sure it will have numerous other uses as well.
10 The latest version can be found at http://wordlist.aspell.net/.
12 The directory final/ contains the actual word lists broken up into
13 various sizes and categories. The r/ directory contains Readmes from
14 the various sources used to create this package.
16 The misc/ contains a small list of taboo words, see the README file
17 for more info. The speller/ directory contains scripts for creating
18 spelling dictionaries for Aspell and Hunspell.
20 The other directories contain the necessary information to recreate the
21 word lists from the raw data. Unless you are interested in improving the
22 words lists you should not need to worry about what's here. See the
23 section on recreating the words lists for more information on what's
26 Except for the special word lists the files follow the following
28 <spelling category>-<sub-category>.<size>
29 Where the spelling category is one of
30 english, american, british, british_z, canadian, australian
31 variant_1, variant_2, variant_3,
32 british_variant_1, british_variant_2,
33 canadian_variant_1, canadian_variant_2,
34 australian_variant_1, australian_variant_2
35 Sub-category is one of
36 abbreviations, contractions, proper-names, upper, words
38 10, 20, 35 (small), 40, 50 (medium), 55, 60, 70 (large),
39 80 (huge), 95 (insane)
40 The special word lists follow are in the following format:
41 special-<description>.<size>
42 Where description is one of:
43 roman-numerals, hacker
45 The perl script "mk-list" can be used to create a word list of the
46 desired size, its usage is:
47 ./mk-list [-f] [-v#] <spelling categories> <size>
48 where <spelling categories> is one of the above spelling categories
49 (the english and special categories are automatically included as well
50 as all sub-categories) and <size> is the desired size. The
51 "-v" option can be used to also include the appropriate
52 variants file up to level '#'. The normal output will be a sorted
53 word list. If you rather see what files will be included, use the
56 When manually combining the words lists the "english" spelling
57 category should be used as well as one of "american", "british",
58 "british_z" (british with ize spelling), "canadian" or "australian".
59 Great care has been taken so that only one spelling for any particular
60 word is included in the main list (with some minor exceptions). When
61 two variants were considered equal I randomly picked one for inclusion
62 in the main word list. Unfortunately this means that my choice in how
63 to spell a word may not match your choice. If this is the case you
64 can try including one of the "variant_1" spelling categories which
65 includes most variants which are considered almost equal. The
66 "variant_1" spelling category corresponds mostly to American variants,
67 while the "british_variant_1", "canadian_variant_1" and
68 "australian_variant_1" are for British, Canadian and Australian
69 variants, respectively. The "variant_2" spelling categories include
70 variants which are also generally considered acceptable, and
71 "variant_3" contains variants which are seldom used and may not even
72 be considered correct. There is no "british_variant_3",
73 "canadian_variant_3" or "australian_variant_3" spelling category since
74 the distinction would be almost meaningless.
76 The "abbreviation" category includes abbreviations and acronyms which
77 are not also normal words. The "contractions" category should be self
78 explanatory. The "upper" category includes upper case words and proper
79 names which are common enough to appear in a typical dictionary. The
80 "proper-names" category includes all the additional uppercase words.
81 Finally the "words" category contains all the normal English words.
83 To give you an idea of what the words in the various sizes look like
84 here is a sample of 25 random words found only in that size:
86 10: argue arguing believe center character clear comparing corrects exact
87 extreme get idea irritates kept linking notes observing occurred printers
88 regulations remembered scores show signs unite
90 20: advocates classifying commandment competent culprit cumulative
91 differential earning extensions mature obeys optic orientating overloaded
92 perception prisoners proofs puzzled restrictions retaining rock sister
93 stuffing terrorists unfounded
95 35: aides ascetic braiding clung conquerors dispassionate edicts equalized
96 exposition gardenias glamour godmothers handlebars huffs impudent lunge
97 masterful needled paddocks pots raping shouldered snooker sprawled
100 40: boobed carjackings crapped floozies footsie freethinkers gassiest
101 genuflecting geriatric globetrotter innovate jukebox marinaded menorah
102 nannies neutralization piously premarital rekindling riverbed stilts
103 stonewall swimmer tattletale twerps
105 50: aglitter amazon blinders boggier cerebra coiffuring discernment flintlock
106 interstices japanned katydid lagniappes loganberry lorgnette misdealt
107 monograph peripatetic polliwogs radicalism schoolhouse seismic soppier
108 suffused trisected wastrel
110 55: bedsit candida coaxingly completions contextualization cutaways
111 functionalism handsaw hardboard hyperinflation hypnotherapy inglenooks
112 isotherms jobshare naffer outguns playgroup precariousness remaster
113 ruched seltzers straitlaced trainspotting woad yukky
115 60: basinfuls bastardization blueness charlatanism dater dispassion duper
116 glutinously goofball greeters imitable lacewing misspeak nickering
117 nonbelligerent noneffective nonindependent premix resize retarder
118 southeaster steerable talky tarantellas ungoverned
120 70: apostrophic bioecology celoms chloropicrin choli diapause dithyramb
121 doorsill eluder ergotisms geomagnetically hispid inebriant lobelia
122 meatman osteitides overprescribes plausibleness quadroon quincunx
123 sacculate tache toxics trophoplasm unenthusiastically
125 80: angledozers arrogancies beadledoms centesis cryobiologist entrepreneuses
126 estafettes floriation forgivingness glucosamines hairlocks hoofprint
127 intraventricular keffiyahs keloidal lunateds posttranslational rewinded
128 sandspurs seeable sparlings starstone underarmed unmarriageableness
131 95: amusively anabrus anglophily atrophous augustin bachelry barbarianize
132 coadjacence cothe hemicardia inblowing jardini lindo mallear manx
133 morcellement olericulturist oversliding palsier pertinate proctoptosis
134 recollectednesses rowdyishly tinta unsplattered
137 And here is a count on the number of words in each spelling category
138 (american + english spelling category):
140 Size Words Names Running Total %
141 10 4,426 13 4,439 0.7
142 20 8,124 0 12,563 1.9
143 35 37,258 222 50,043 7.6
144 40 6,853 491 57,387 8.7
145 50 25,224 18,221 100,832 15.3
146 55 6,489 0 107,321 16.3
147 60 14,141 774 122,236 18.6
148 70 35,506 7,912 165,654 25.2
149 80 144,283 33,370 343,307 52.2
150 95 227,674 86,649 657,630 100.0
153 (The "Words" column does not include the name count.)
155 Size 35 is the recommended small size, 50 the medium and 70 the large.
156 Sizes 70 and below contain words found in most dictionaries while the
157 80 size contains all the strange and unusual words people like to use
158 in word games such as Scrabble (TM). While a lot of the words in the
159 80 size are not used very often, they are all generally considered
160 valid words in the English language. The 95 contains just about every
161 English word in existence and then some. Many of the words at the 95
162 level will probably not be considered valid English words by most
165 For spell checking I recommend using size 60. This size is the
166 largest size that I am fairly confident does not contain any
167 misspellings or invalid words. In addition an effort is made to
168 exclude valid yet problematic words (such as "calender") from the 60
169 size that are likely to be a misspelling of a more common word. The
170 70 size is reasonable for those wanting a larger list and don't mind a
171 few errors. The 80 or larger sizes are not reasonable for spell
174 Accents are present on certain words such as café in iso8859-1 format.
178 From Version 2016.11.20 to 2017.01.22
182 From Version 2016.06.26 to 2016.11.20
184 New Australian spelling category thanks to the work of Benjamin
185 Titze (btitze@protonmail.ch)
189 From Version 2016.01.19 to 2016.06.26
193 Updated to Version 6.0.2 of 12dicts
197 From Version 2015.08.24 to 2016.01.19
201 Clarified README to indicate why the 60 size is the preferred size
204 Remove some very uncommon possessive forms.
206 Change "SET UTF8" to "SET UTF-8" in hunspell affix file.
208 From Version 2015.05.18 to 2015.08.24 (Aug 24, 2015)
212 From Version 2015.04.24 to 2015.05.18 (May 18, 2015)
214 Added some new words found to have a high frequency in the COCA
215 corpus. (http://corpus.byu.edu/coca/).
217 Fix en spelling suggestions for 'alot' and 'exersize' in hunspell
218 dictionary (upstreamed from the changes made in Firefox).
220 From Version 2015.02.15 to 2015.04.24 (April 24, 2015)
222 Added some new words.
224 Convert hunspell dictionary to UTF-8 in order to handle smart
227 From Version 2015.01.28 to 2015.02.15 (February 15, 2015)
229 Added a large number of neologisms (newly invented words)
230 such as "selfie" and "smartwatch" thanks to Alan Beale.
232 Various other new words.
234 Clean up the special-hacker category by removing some words that
235 didn't exist in the Google Book's Corpus (1980 - 2008) and
236 originated from the "Unofficial Jargon File Word Lists".
238 From Version 2014.11.17 to 2015.01.28 (January 28, 2015)
240 Various new words, many from analyzing the Google Book's Corpus
241 (1980 - 2008). See http://app.aspell.net/lookup-freq.
243 Moved some uncommon words that can easily hide a misspelling of a
244 more common word to level 70. (calender, adrenalin and Joesph)
246 Removed several -er and -est forms from adjectives that were so
247 uncommon that they were not found anywhere is the Google Book's
248 Corpus (1980 - 2008).
250 From Version 2014.08.11.1 to 2014.11.17 (November 17, 2014)
254 Fix typo in Hunspell readme.
256 From Version 2014.08.11 to 2014.08.11.1 (August 13, 2014)
258 Forgot to mention this important change from 7.1 to 2014.08.11:
260 Shifted the variant levels up by one: variant_0 is now variant_1,
261 variant_1 is now variant_2, and variant_2 is now variant_3.
263 Other minor fixes in this README.
265 No changes to the contents of the lists.
267 From Revision 7.1 to Version 2014.08.11 (August 11, 2014)
269 Added some missing possessive forms.
271 Added some new words and proper names.
273 Clean up the categories (words, upper, proper-names etc) so that they
276 Convert documentation to UTF-8. For now, the wordlist are still in
277 ISO-8859-1 to prevent compatibility problems.
279 Add schema and scripts for creating a SQLite database from SCOWL.
280 Add some utility and library functions using them. This database is
281 used by the new web app's (http://app.aspel.net/lookup & create).
283 Enhance speller/make-hunspell-dict. The biggest improvement is that
284 it that it now generates several more dictionaries in addition to
285 the official ones. These additional dictionaries are ones for
286 British English and larger dictionaries that include up to SCOWL
289 From Revision 7 to 7.1 (January 6, 2011)
291 Updated to revision 5.1 of Varcon which corrected several errors.
293 Fixed various problems with the variant processing which corrected a
296 Added several now common proper names and some other words now
299 Include misc/ and speller/ directory which were in SVN but left
300 out of the release tarball.
302 Other minor fixes, including some fixes to the taboo word lists.
304 From Revision 6 to 7 (December 27, 2010)
306 Updated to revision 5.0 of Varcon which corrected many errors,
307 especially in the British and Canadian spelling categories. Also
308 added new spelling categories for the British and Canadian spelling
309 variants and separated them out from the main variant_* categories.
311 Moved Moby names lists (3897male.nam 4946fema.len 21986na.mes) to 95
312 level since they contain too many errors and rare names.
314 Moved frequently class 0 from Brian Kelk's Wordlist from
315 level 60 to 70, and also filter it with level 80 due to, too many
318 Many other minor fixes.
320 From Revision 5 to 6 (August 10, 2004)
322 Updated to version 4.0 of the 12dicts package.
324 Included the 3esl, 2of4brif, and 5desk list from the new 12dicts
325 package. The 3esl was included in the 40 size, the 2of4brif in the
326 55 size and the 5desk in the 70 size.
328 Removed the Ispell word list as it was a source of too many errors.
329 This eliminated the 65 size.
331 Removed clause 4 from the Ispell copyright with permission of Geoff
334 Updated to version 4.1 of VarCon.
336 Added the "british_z" spelling category which is British using the
339 From Revision 4a to 5 (January 3, 2002)
341 Added variants that were not really spelling variants (such as
342 forwards) back into the main list.
344 Fixed a bug which caused variants of words to incorrectly appear in
345 the non-variant lists.
347 Moved rarely used inflections of a word into higher number lists.
349 Added other inflections of a words based on the following criteria
350 If the word is in the base form: only include that word.
351 If the word is in a plural form: include the base word and the plural
352 If the word is a verb form (other than plural): include all verb forms
353 If the word is an ad* form: include all ad* forms
354 If the word is in a possessive form: also include the non-possessive
356 Updated to the latest version of many of the source dictionaries.
358 Removed the DEC Word List due to the questionable licence and
359 because removing it will not seriously decrease the quality of SCOWL
360 (there are a few less proper names).
362 From Revision 4 to 4a (April 4, 2001)
364 Reran the scripts on a never version of AGID (3a) which fixes a bug
365 which caused some common words to be improperly marked as variants.
367 From Revision 3 to 4 (January 28, 2001)
369 Split the variant "spelling category" up into 3 different levels.
371 Added words in the Ispell word list at the 65 level.
373 Other changes due to using more recent versions of various sources
374 included a more accurate version of AGID thanks to the work of
377 From Revision 2 to 3 (August 18, 2000)
379 Renamed special-unix-terms to special-hacker and added a large
380 number of commonly used words within the hacker (not cracker)
383 Added a couple more signature words including "newbie".
385 Minor changes due to changes in the inflection database.
387 From Revision 1 to 2 (August 5, 2000)
389 Moved the male and female name lists from the mwords package and the
390 DEC name lists form the 50 level to the 60 level and moved Alan's
391 name list from the 60 level to the 50 level. Also added the top
392 1000 male, female, and last names from the 1990 Census report to the
393 50 level. This reduced the number of names in the 50 level from
396 Added a large number of Uppercase words to the 50 level.
398 Properly accented the possessive form of some words.
400 Minor other changes due to changes in my raw data files which have
401 not been released yet. Email if you are interested in these files.
403 COPYRIGHT, SOURCES, and CREDITS:
405 The collective work is Copyright 2000-2016 by Kevin Atkinson as well
406 as any of the copyrights mentioned below:
408 Copyright 2000-2016 by Kevin Atkinson
410 Permission to use, copy, modify, distribute and sell these word
411 lists, the associated scripts, the output created from the scripts,
412 and its documentation for any purpose is hereby granted without fee,
413 provided that the above copyright notice appears in all copies and
414 that both that copyright notice and this permission notice appear in
415 supporting documentation. Kevin Atkinson makes no representations
416 about the suitability of this array for any purpose. It is provided
417 "as is" without express or implied warranty.
419 Alan Beale <biljir@pobox.com> also deserves special credit as he has,
420 in addition to providing the 12Dicts package and being a major
421 contributor to the ENABLE word list, given me an incredible amount of
422 feedback and created a number of special lists (those found in the
423 Supplement) in order to help improve the overall quality of SCOWL.
425 The 10 level includes the 1000 most common English words (according to
426 the Moby (TM) Words II [MWords] package), a subset of the 1000 most
427 common words on the Internet (again, according to Moby Words II), and
428 frequently class 16 from Brian Kelk's "UK English Wordlist
429 with Frequency Classification".
431 The MWords package was explicitly placed in the public domain:
433 The Moby lexicon project is complete and has
434 been place into the public domain. Use, sell,
435 rework, excerpt and use in any way on any platform.
437 Placing this material on internal or public servers is
438 also encouraged. The compiler is not aware of any
439 export restrictions so freely distribute world-wide.
441 You can verify the public domain status by contacting
445 Arcata, CA 95521-4884
450 The "UK English Wordlist With Frequency Classification" is also in the
453 Date: Sat, 08 Jul 2000 20:27:21 +0100
454 From: Brian Kelk <Brian.Kelk@cl.cam.ac.uk>
456 > I was wondering what the copyright status of your "UK English
457 > Wordlist With Frequency Classification" word list as it seems to
458 > be lacking any copyright notice.
460 There were many many sources in total, but any text marked
461 "copyright" was avoided. Locally-written documentation was one
462 source. An earlier version of the list resided in a filespace called
463 PUBLIC on the University mainframe, because it was considered public
466 Date: Tue, 11 Jul 2000 19:31:34 +0100
468 > So are you saying your word list is also in the public domain?
470 That is the intention.
472 The 20 level includes frequency classes 7-15 from Brian's word list.
474 The 35 level includes frequency classes 2-6 and words appearing in at
475 least 11 of 12 dictionaries as indicated in the 12Dicts package. All
476 words from the 12Dicts package have had likely inflections added via
477 my inflection database.
479 The 12Dicts package and Supplement is in the Public Domain.
481 The WordNet database, which was used in the creation of the
482 Inflections database, is under the following copyright:
484 This software and database is being provided to you, the LICENSEE,
485 by Princeton University under the following license. By obtaining,
486 using and/or copying this software and database, you agree that you
487 have read, understood, and will comply with these terms and
490 Permission to use, copy, modify and distribute this software and
491 database and its documentation for any purpose and without fee or
492 royalty is hereby granted, provided that you agree to comply with
493 the following copyright notice and statements, including the
494 disclaimer, and that the same appear on ALL copies of the software,
495 database and documentation, including modifications that you make
496 for internal use or for distribution.
498 WordNet 1.6 Copyright 1997 by Princeton University. All rights
501 THIS SOFTWARE AND DATABASE IS PROVIDED "AS IS" AND PRINCETON
502 UNIVERSITY MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR
503 IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, PRINCETON
504 UNIVERSITY MAKES NO REPRESENTATIONS OR WARRANTIES OF MERCHANT-
505 ABILITY OR FITNESS FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF THE
506 LICENSED SOFTWARE, DATABASE OR DOCUMENTATION WILL NOT INFRINGE ANY
507 THIRD PARTY PATENTS, COPYRIGHTS, TRADEMARKS OR OTHER RIGHTS.
509 The name of Princeton University or Princeton may not be used in
510 advertising or publicity pertaining to distribution of the software
511 and/or database. Title to copyright in this software, database and
512 any associated documentation shall at all times remain with
513 Princeton University and LICENSEE agrees to preserve same.
515 The 40 level includes words from Alan's 3esl list found in version 4.0
516 of his 12dicts package. Like his other stuff the 3esl list is also in the
519 The 50 level includes Brian's frequency class 1, words appearing
520 in at least 5 of 12 of the dictionaries as indicated in the 12Dicts
521 package, and uppercase words in at least 4 of the previous 12
522 dictionaries. A decent number of proper names is also included: The
523 top 1000 male, female, and Last names from the 1990 Census report; a
524 list of names sent to me by Alan Beale; and a few names that I added
525 myself. Finally a small list of abbreviations not commonly found in
526 other word lists is included.
528 The name files form the Census report is a government document which I
529 don't think can be copyrighted.
531 The file special-jargon.50 uses common.lst and word.lst from the
532 "Unofficial Jargon File Word Lists" which is derived from "The Jargon
533 File". All of which is in the Public Domain. This file also contain
534 a few extra UNIX terms which are found in the file "unix-terms" in the
537 The 55 level includes words from Alan's 2of4brif list found in version
538 4.0 of his 12dicts package. Like his other stuff the 2of4brif is also
539 in the public domain.
541 The 60 level includes all words appearing in at least 2 of the 12
542 dictionaries as indicated by the 12Dicts package.
544 The 70 level includes Brian's frequency class 0 and the 74,550 common
545 dictionary words from the MWords package. The common dictionary words,
546 like those from the 12Dicts package, have had all likely inflections
547 added. The 70 level also included the 5desk list from version 4.0 of
548 the 12Dics package which is in the public domain.
550 The 80 level includes the ENABLE word list, all the lists in the
551 ENABLE supplement package (except for ABLE), the "UK Advanced Cryptics
552 Dictionary" (UKACD), the list of signature words from the YAWL package,
553 and the 10,196 places list from the MWords package.
555 The ENABLE package, mainted by M\Cooper <thegrendel@theriver.com>,
556 is in the Public Domain:
558 The ENABLE master word list, WORD.LST, is herewith formally released
559 into the Public Domain. Anyone is free to use it or distribute it in
560 any manner they see fit. No fee or registration is required for its
561 use nor are "contributions" solicited (if you feel you absolutely
562 must contribute something for your own peace of mind, the authors of
563 the ENABLE list ask that you make a donation on their behalf to your
564 favorite charity). This word list is our gift to the Scrabble
565 community, as an alternate to "official" word lists. Game designers
566 may feel free to incorporate the WORD.LST into their games. Please
567 mention the source and credit us as originators of the list. Note
568 that if you, as a game designer, use the WORD.LST in your product,
569 you may still copyright and protect your product, but you may *not*
570 legally copyright or in any way restrict redistribution of the
571 WORD.LST portion of your product. This *may* under law restrict your
572 rights to restrict your users' rights, but that is only fair.
574 UKACD, by J Ross Beresford <ross@bryson.demon.co.uk>, is under the
577 Copyright (c) J Ross Beresford 1993-1999. All Rights Reserved.
579 The following restriction is placed on the use of this publication:
580 if The UK Advanced Cryptics Dictionary is used in a software package
581 or redistributed in any form, the copyright notice must be
582 prominently displayed and the text of this document must be included
585 There are no other restrictions: I would like to see the list
586 distributed as widely as possible.
588 The 95 level includes the 354,984 single words, 256,772 compound
589 words, 4,946 female names and the 3,897 male names, and 21,986 names
590 from the MWords package, ABLE.LST from the ENABLE Supplement, and some
591 additional words found in my part-of-speech database that were not
594 Accent information was taken from UKACD.
596 The VarCon package was used to create the American, British, Canadian,
597 and Australian word list. It is under the following copyright:
599 Copyright 2000-2016 by Kevin Atkinson
601 Permission to use, copy, modify, distribute and sell this array, the
602 associated software, and its documentation for any purpose is hereby
603 granted without fee, provided that the above copyright notice appears
604 in all copies and that both that copyright notice and this permission
605 notice appear in supporting documentation. Kevin Atkinson makes no
606 representations about the suitability of this array for any
607 purpose. It is provided "as is" without express or implied warranty.
609 Copyright 2016 by Benjamin Titze
611 Permission to use, copy, modify, distribute and sell this array, the
612 associated software, and its documentation for any purpose is hereby
613 granted without fee, provided that the above copyright notice appears
614 in all copies and that both that copyright notice and this permission
615 notice appear in supporting documentation. Benjamin Titze makes no
616 representations about the suitability of this array for any
617 purpose. It is provided "as is" without express or implied warranty.
619 Since the original words lists come from the Ispell distribution:
621 Copyright 1993, Geoff Kuenning, Granada Hills, CA
624 Redistribution and use in source and binary forms, with or without
625 modification, are permitted provided that the following conditions
628 1. Redistributions of source code must retain the above copyright
629 notice, this list of conditions and the following disclaimer.
630 2. Redistributions in binary form must reproduce the above copyright
631 notice, this list of conditions and the following disclaimer in the
632 documentation and/or other materials provided with the distribution.
633 3. All modifications to the source code must be clearly marked as
634 such. Binary redistributions based on modified source code
635 must be clearly marked as modified versions in the documentation
636 and/or other materials provided with the distribution.
637 (clause 4 removed with permission from Geoff Kuenning)
638 5. The name of Geoff Kuenning may not be used to endorse or promote
639 products derived from this software without specific prior
642 THIS SOFTWARE IS PROVIDED BY GEOFF KUENNING AND CONTRIBUTORS ``AS IS'' AND
643 ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
644 IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
645 ARE DISCLAIMED. IN NO EVENT SHALL GEOFF KUENNING OR CONTRIBUTORS BE LIABLE
646 FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
647 DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
648 OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
649 HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
650 LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
651 OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
655 The variant word lists were created from a list of variants found in
656 the 12dicts supplement package as well as a list of variants I created
659 The Readmes for the various packages used can be found in the
660 appropriate directory under the r/ directory.
664 The process of "sort"s, "comm"s, and Perl scripts to combine the many
665 word lists and separate out the variant information is inexact and
666 error prone. The whole things needs to be rewritten to deal with
667 words in terms of lemmas. When the exact lemma is not known a best
668 guess should be made. I'm not sure what form this should be in. I
669 originally thought this should be some sort of database, but maybe I
670 should just slurp all that data into memory and process it in one
671 giant perl script. With the amount of memory available these days (at
672 least 2 GB, often 4 GB or more) this should not really be a problem.
674 In addition, there is a very nice frequency analyze of the BNC corpus
675 done by Adam Kilgarriff. Unlike Brian's word lists the BNC lists
676 include part of speech information. I plan on somehow using these
677 lists as Adam Kilgarriff has given me the OK to use it in SCOWL.
678 These lists will greatly reduce the problem of inflected forms of a
679 word appearing at different levels due to the part-of-speech
682 There is frequency information for some other corpus such as COCA
683 (Corpus of Contemporary American English) and ANS (American National
684 Corpus) which I might also be able to use. The former will require
685 permission, and the latter is of questionable quality.
687 RECREATING THE WORD LISTS:
689 In order to recreate the word lists you need a modern version of Perl,
690 bash, the traditional set of shell utilities, a system that supports
691 symbolic links, and quite possibly GNU Make. The easiest way to
692 recreate the word lists is to checkout the corresponding Git version
693 (see the version string at the start of the file) and simply type
694 "make" (see http://wordlist.aspell.net). You can try to download all
695 the pieces manually, but this method is not no longer tested nor
698 The src/ directory contains the numerous scripts used in the creation
699 of the final product.
701 The r/ directory contains the raw data used to create the final
702 product. If you checkout from Git this directory should be populated
703 automatically for you. If you insist on doing it the hard way see the
704 README file in the r/ directory for more information.
706 The l/ directory contains symbolic links used by the actual scripts.
708 Finally, the working/ directory is where all the intermittent files go
709 that are not specific to one source.