1 Spell Checking Oriented Word Lists (SCOWL)
3 Mon Dec 7 20:14:35 2020 -0500 [5ef55f9]
4 by Kevin Atkinson (kevina@gnu.org)
6 The SCOWL is a collection of word lists split up in various sizes, and
7 other categories, intended to be suitable for use in spell checkers.
8 However, I am sure it will have numerous other uses as well.
10 The latest version can be found at http://wordlist.aspell.net/.
12 The directory final/ contains the actual word lists broken up into
13 various sizes and categories. The r/ directory contains Readmes from
14 the various sources used to create this package.
16 The misc/ contains a small list of taboo words, see the README file
17 for more info. The speller/ directory contains scripts for creating
18 spelling dictionaries for Aspell and Hunspell.
20 The other directories contain the necessary information to recreate the
21 word lists from the raw data. Unless you are interested in improving the
22 words lists you should not need to worry about what's here. See the
23 section on recreating the words lists for more information on what's
26 Except for the special word lists the files follow the following
28 <spelling category>-<sub-category>.<size>
29 Where the spelling category is one of
30 english, american, british, british_z, canadian, australian
31 variant_1, variant_2, variant_3,
32 british_variant_1, british_variant_2,
33 canadian_variant_1, canadian_variant_2,
34 australian_variant_1, australian_variant_2
35 Sub-category is one of
36 abbreviations, contractions, proper-names, upper, words
38 10, 20, 35 (small), 40, 50 (medium), 55, 60, 70 (large),
39 80 (huge), 95 (insane)
40 The special word lists follow are in the following format:
41 special-<description>.<size>
42 Where description is one of:
43 roman-numerals, hacker
45 The perl script "mk-list" can be used to create a word list of the
46 desired size, its usage is:
47 ./mk-list [-f] [-v#] <spelling categories> <size>
48 where <spelling categories> is one of the above spelling categories
49 (the english and special categories are automatically included as well
50 as all sub-categories) and <size> is the desired size. The
51 "-v" option can be used to also include the appropriate
52 variants file up to level '#'. The normal output will be a sorted
53 word list. If you rather see what files will be included, use the
56 When manually combining the words lists the "english" spelling
57 category should be used as well as one of "american", "british",
58 "british_z" (british with ize spelling), "canadian" or "australian".
59 Great care has been taken so that only one spelling for any particular
60 word is included in the main list (with some minor exceptions). When
61 two variants were considered equal I randomly picked one for inclusion
62 in the main word list. Unfortunately this means that my choice in how
63 to spell a word may not match your choice. If this is the case you
64 can try including one of the "variant_1" spelling categories which
65 includes most variants which are considered almost equal. The
66 "variant_1" spelling category corresponds mostly to American variants,
67 while the "british_variant_1", "canadian_variant_1" and
68 "australian_variant_1" are for British, Canadian and Australian
69 variants, respectively. The "variant_2" spelling categories include
70 variants which are also generally considered acceptable, and
71 "variant_3" contains variants which are seldom used and may not even
72 be considered correct. There is no "british_variant_3",
73 "canadian_variant_3" or "australian_variant_3" spelling category since
74 the distinction would be almost meaningless.
76 The "abbreviation" category includes abbreviations and acronyms which
77 are not also normal words. The "contractions" category should be self
78 explanatory. The "upper" category includes upper case words and proper
79 names which are common enough to appear in a typical dictionary. The
80 "proper-names" category includes all the additional uppercase words.
81 Finally the "words" category contains all the normal English words.
83 To give you an idea of what the words in the various sizes look like
84 here is a sample of 25 random words found only in that size:
86 10: attempt base borrows clever cold concerned contribution decide deletes
87 easiest inclined mine natural obviously opportunity organized pain
88 potential signed significance standing survey this training trick
90 20: brave comma confining conviction delicious embedding enlarging equations
91 era farmer flip frustrates keystrokes officers peoples personalities
92 principally restarts revert risks singular sneaky stealing sweep
95 35: bantered barrens bronzing chisel debtors doorstep earache elaborating
96 expressly glistened humping joyfully leashes lofting logician obsessions
97 paralytics pillowed portrayals pruned rarities reconfigured scrupulous
100 40: astrologer bestsellers busboys childproofed clapboarded crispiest
101 embroiling enfranchises enthused exorcists firebrand gringo irresponsibly
102 matchstick missteps oinks pocketfuls reinventing scorecard streetlights
103 temped turncoat voyeur warmongering wimps
105 50: apologias assay biochemists brashness brattier councilman detainees
106 discontentedly ethnology evincing excoriation halberd housemothers
107 humdinger moraines permutes pilaf purebred putsch quadrature
108 secularization skyjacking snowsuit transmuted zeppelins
110 55: articulacy bookbinders chapati faffing gunge hotpots hurtfulness innit
111 kaleidoscopically leching megastars ockers paperclips pedestrianization
112 peeler plainsong rand righto stationmasters sundecks tossers triathletes
113 turbocharges twitchiest yobs
115 60: allurements bespangle centripetally dashers eclogue estoppel ethologist
116 gleaners gratingly imputable jobholder mendicancy minnesingers muscats
117 nontransparent nosher obtrusion parasympathetics patroons
118 phosphorescently reforging reintegrate stringiness transecting vixenishly
119 70: animalisms bestializing blague chlorpromazine decury dolmans ecclesiology
120 hymnody incommutable listers lucubrator methodic mizenmasts monochord
121 natality ninepence pyrogenic rath sabayons serenata shitwork superlunary
122 talapoin unresigned whickered
124 80: batatas diapente discipled doofuses faintheartednesses geophagous gooky
125 grandeurs hypesthesic kagouls mandataries minimalized operettists
126 pseudoephedrine readvertizing rumblegumption sabermetrics scritches
127 sextonship simuliums superspectaculars thickoes tripersonalism unmoneyed
130 95: adalat afdecho basirhinal crossopodia decalomania earthmaker gaudeamuses
131 guanayes haemodoraceous hardsalt heterostrophies kadikane mastoidale
132 misconceited osteoarthrotomy perpetuant photolyte querulation
133 splenonephric storymaker thrangity turgider unquailingly unthriftlike
137 And here is a count on the number of words in each spelling category
138 (american + english spelling category):
140 Size Words Names Running Total %
141 10 4,425 13 4,438 0.7
142 20 8,126 0 12,564 1.9
143 35 37,260 220 50,044 7.6
144 40 6,858 489 57,391 8.7
145 50 25,289 18,683 101,363 15.4
146 55 6,487 0 107,850 16.4
147 60 14,551 850 123,251 18.7
148 70 35,294 7,897 166,442 25.3
149 80 144,158 33,368 343,968 52.3
150 95 227,633 86,630 658,231 100.0
153 (The "Words" column does not include the name count.)
155 Size 35 is the recommended small size, 50 the medium and 70 the large.
156 Sizes 70 and below contain words found in most dictionaries while the
157 80 size contains all the strange and unusual words people like to use
158 in word games such as Scrabble (TM). While a lot of the words in the
159 80 size are not used very often, they are all generally considered
160 valid words in the English language. The 95 contains just about every
161 English word in existence and then some. Many of the words at the 95
162 level will probably not be considered valid English words by most
165 For spell checking I recommend using size 60. This size is the
166 largest size that I am fairly confident does not contain any
167 misspellings or invalid words. In addition an effort is made to
168 exclude valid yet problematic words (such as "calender") from the 60
169 size that are likely to be a misspelling of a more common word. The
170 70 size is reasonable for those wanting a larger list and don't mind a
171 few errors. The 80 or larger sizes are not reasonable for spell
174 Accents are present on certain words such as café in iso8859-1 format.
178 From Version 2019.10.06 to 2020.12.07
184 Bump irregardless, froward (+ derivatives) and perpend to level 70.
186 From Version 2018.04.16 to 2019.10.06
190 Remove compare's and fail's.
192 From Version 2017.08.24 to 2018.04.16
196 Fix build problems on macOS.
198 From Version 2017.01.22 to 2017.08.24
202 From Version 2016.11.20 to 2017.01.22
206 From Version 2016.06.26 to 2016.11.20
208 New Australian spelling category thanks to the work of Benjamin
209 Titze (btitze@protonmail.ch)
213 From Version 2016.01.19 to 2016.06.26
217 Updated to Version 6.0.2 of 12dicts
221 From Version 2015.08.24 to 2016.01.19
225 Clarified README to indicate why the 60 size is the preferred size
228 Remove some very uncommon possessive forms.
230 Change "SET UTF8" to "SET UTF-8" in hunspell affix file.
232 From Version 2015.05.18 to 2015.08.24 (Aug 24, 2015)
236 From Version 2015.04.24 to 2015.05.18 (May 18, 2015)
238 Added some new words found to have a high frequency in the COCA
239 corpus. (http://corpus.byu.edu/coca/).
241 Fix en spelling suggestions for 'alot' and 'exersize' in hunspell
242 dictionary (upstreamed from the changes made in Firefox).
244 From Version 2015.02.15 to 2015.04.24 (April 24, 2015)
246 Added some new words.
248 Convert hunspell dictionary to UTF-8 in order to handle smart
251 From Version 2015.01.28 to 2015.02.15 (February 15, 2015)
253 Added a large number of neologisms (newly invented words)
254 such as "selfie" and "smartwatch" thanks to Alan Beale.
256 Various other new words.
258 Clean up the special-hacker category by removing some words that
259 didn't exist in the Google Book's Corpus (1980 - 2008) and
260 originated from the "Unofficial Jargon File Word Lists".
262 From Version 2014.11.17 to 2015.01.28 (January 28, 2015)
264 Various new words, many from analyzing the Google Book's Corpus
265 (1980 - 2008). See http://app.aspell.net/lookup-freq.
267 Moved some uncommon words that can easily hide a misspelling of a
268 more common word to level 70. (calender, adrenalin and Joesph)
270 Removed several -er and -est forms from adjectives that were so
271 uncommon that they were not found anywhere is the Google Book's
272 Corpus (1980 - 2008).
274 From Version 2014.08.11.1 to 2014.11.17 (November 17, 2014)
278 Fix typo in Hunspell readme.
280 From Version 2014.08.11 to 2014.08.11.1 (August 13, 2014)
282 Forgot to mention this important change from 7.1 to 2014.08.11:
284 Shifted the variant levels up by one: variant_0 is now variant_1,
285 variant_1 is now variant_2, and variant_2 is now variant_3.
287 Other minor fixes in this README.
289 No changes to the contents of the lists.
291 From Revision 7.1 to Version 2014.08.11 (August 11, 2014)
293 Added some missing possessive forms.
295 Added some new words and proper names.
297 Clean up the categories (words, upper, proper-names etc) so that they
300 Convert documentation to UTF-8. For now, the wordlist are still in
301 ISO-8859-1 to prevent compatibility problems.
303 Add schema and scripts for creating a SQLite database from SCOWL.
304 Add some utility and library functions using them. This database is
305 used by the new web app's (http://app.aspell.net/lookup & create).
307 Enhance speller/make-hunspell-dict. The biggest improvement is that
308 it that it now generates several more dictionaries in addition to
309 the official ones. These additional dictionaries are ones for
310 British English and larger dictionaries that include up to SCOWL
313 From Revision 7 to 7.1 (January 6, 2011)
315 Updated to revision 5.1 of Varcon which corrected several errors.
317 Fixed various problems with the variant processing which corrected a
320 Added several now common proper names and some other words now
323 Include misc/ and speller/ directory which were in SVN but left
324 out of the release tarball.
326 Other minor fixes, including some fixes to the taboo word lists.
328 From Revision 6 to 7 (December 27, 2010)
330 Updated to revision 5.0 of Varcon which corrected many errors,
331 especially in the British and Canadian spelling categories. Also
332 added new spelling categories for the British and Canadian spelling
333 variants and separated them out from the main variant_* categories.
335 Moved Moby names lists (3897male.nam 4946fema.len 21986na.mes) to 95
336 level since they contain too many errors and rare names.
338 Moved frequently class 0 from Brian Kelk's Wordlist from
339 level 60 to 70, and also filter it with level 80 due to, too many
342 Many other minor fixes.
344 From Revision 5 to 6 (August 10, 2004)
346 Updated to version 4.0 of the 12dicts package.
348 Included the 3esl, 2of4brif, and 5desk list from the new 12dicts
349 package. The 3esl was included in the 40 size, the 2of4brif in the
350 55 size and the 5desk in the 70 size.
352 Removed the Ispell word list as it was a source of too many errors.
353 This eliminated the 65 size.
355 Removed clause 4 from the Ispell copyright with permission of Geoff
358 Updated to version 4.1 of VarCon.
360 Added the "british_z" spelling category which is British using the
363 From Revision 4a to 5 (January 3, 2002)
365 Added variants that were not really spelling variants (such as
366 forwards) back into the main list.
368 Fixed a bug which caused variants of words to incorrectly appear in
369 the non-variant lists.
371 Moved rarely used inflections of a word into higher number lists.
373 Added other inflections of a words based on the following criteria
374 If the word is in the base form: only include that word.
375 If the word is in a plural form: include the base word and the plural
376 If the word is a verb form (other than plural): include all verb forms
377 If the word is an ad* form: include all ad* forms
378 If the word is in a possessive form: also include the non-possessive
380 Updated to the latest version of many of the source dictionaries.
382 Removed the DEC Word List due to the questionable licence and
383 because removing it will not seriously decrease the quality of SCOWL
384 (there are a few less proper names).
386 From Revision 4 to 4a (April 4, 2001)
388 Reran the scripts on a never version of AGID (3a) which fixes a bug
389 which caused some common words to be improperly marked as variants.
391 From Revision 3 to 4 (January 28, 2001)
393 Split the variant "spelling category" up into 3 different levels.
395 Added words in the Ispell word list at the 65 level.
397 Other changes due to using more recent versions of various sources
398 included a more accurate version of AGID thanks to the work of
401 From Revision 2 to 3 (August 18, 2000)
403 Renamed special-unix-terms to special-hacker and added a large
404 number of commonly used words within the hacker (not cracker)
407 Added a couple more signature words including "newbie".
409 Minor changes due to changes in the inflection database.
411 From Revision 1 to 2 (August 5, 2000)
413 Moved the male and female name lists from the mwords package and the
414 DEC name lists form the 50 level to the 60 level and moved Alan's
415 name list from the 60 level to the 50 level. Also added the top
416 1000 male, female, and last names from the 1990 Census report to the
417 50 level. This reduced the number of names in the 50 level from
420 Added a large number of Uppercase words to the 50 level.
422 Properly accented the possessive form of some words.
424 Minor other changes due to changes in my raw data files which have
425 not been released yet. Email if you are interested in these files.
427 COPYRIGHT, SOURCES, and CREDITS:
429 The collective work is Copyright 2000-2018 by Kevin Atkinson as well
430 as any of the copyrights mentioned below:
432 Copyright 2000-2018 by Kevin Atkinson
434 Permission to use, copy, modify, distribute and sell these word
435 lists, the associated scripts, the output created from the scripts,
436 and its documentation for any purpose is hereby granted without fee,
437 provided that the above copyright notice appears in all copies and
438 that both that copyright notice and this permission notice appear in
439 supporting documentation. Kevin Atkinson makes no representations
440 about the suitability of this array for any purpose. It is provided
441 "as is" without express or implied warranty.
443 Alan Beale <biljir@pobox.com> also deserves special credit as he has,
444 in addition to providing the 12Dicts package and being a major
445 contributor to the ENABLE word list, given me an incredible amount of
446 feedback and created a number of special lists (those found in the
447 Supplement) in order to help improve the overall quality of SCOWL.
449 The 10 level includes the 1000 most common English words (according to
450 the Moby (TM) Words II [MWords] package), a subset of the 1000 most
451 common words on the Internet (again, according to Moby Words II), and
452 frequently class 16 from Brian Kelk's "UK English Wordlist
453 with Frequency Classification".
455 The MWords package was explicitly placed in the public domain:
457 The Moby lexicon project is complete and has
458 been place into the public domain. Use, sell,
459 rework, excerpt and use in any way on any platform.
461 Placing this material on internal or public servers is
462 also encouraged. The compiler is not aware of any
463 export restrictions so freely distribute world-wide.
465 You can verify the public domain status by contacting
469 Arcata, CA 95521-4884
474 The "UK English Wordlist With Frequency Classification" is also in the
477 Date: Sat, 08 Jul 2000 20:27:21 +0100
478 From: Brian Kelk <Brian.Kelk@cl.cam.ac.uk>
480 > I was wondering what the copyright status of your "UK English
481 > Wordlist With Frequency Classification" word list as it seems to
482 > be lacking any copyright notice.
484 There were many many sources in total, but any text marked
485 "copyright" was avoided. Locally-written documentation was one
486 source. An earlier version of the list resided in a filespace called
487 PUBLIC on the University mainframe, because it was considered public
490 Date: Tue, 11 Jul 2000 19:31:34 +0100
492 > So are you saying your word list is also in the public domain?
494 That is the intention.
496 The 20 level includes frequency classes 7-15 from Brian's word list.
498 The 35 level includes frequency classes 2-6 and words appearing in at
499 least 11 of 12 dictionaries as indicated in the 12Dicts package. All
500 words from the 12Dicts package have had likely inflections added via
501 my inflection database.
503 The 12Dicts package and Supplement is in the Public Domain.
505 The WordNet database, which was used in the creation of the
506 Inflections database, is under the following copyright:
508 This software and database is being provided to you, the LICENSEE,
509 by Princeton University under the following license. By obtaining,
510 using and/or copying this software and database, you agree that you
511 have read, understood, and will comply with these terms and
514 Permission to use, copy, modify and distribute this software and
515 database and its documentation for any purpose and without fee or
516 royalty is hereby granted, provided that you agree to comply with
517 the following copyright notice and statements, including the
518 disclaimer, and that the same appear on ALL copies of the software,
519 database and documentation, including modifications that you make
520 for internal use or for distribution.
522 WordNet 1.6 Copyright 1997 by Princeton University. All rights
525 THIS SOFTWARE AND DATABASE IS PROVIDED "AS IS" AND PRINCETON
526 UNIVERSITY MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR
527 IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, PRINCETON
528 UNIVERSITY MAKES NO REPRESENTATIONS OR WARRANTIES OF MERCHANT-
529 ABILITY OR FITNESS FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF THE
530 LICENSED SOFTWARE, DATABASE OR DOCUMENTATION WILL NOT INFRINGE ANY
531 THIRD PARTY PATENTS, COPYRIGHTS, TRADEMARKS OR OTHER RIGHTS.
533 The name of Princeton University or Princeton may not be used in
534 advertising or publicity pertaining to distribution of the software
535 and/or database. Title to copyright in this software, database and
536 any associated documentation shall at all times remain with
537 Princeton University and LICENSEE agrees to preserve same.
539 The 40 level includes words from Alan's 3esl list found in version 4.0
540 of his 12dicts package. Like his other stuff the 3esl list is also in the
543 The 50 level includes Brian's frequency class 1, words appearing
544 in at least 5 of 12 of the dictionaries as indicated in the 12Dicts
545 package, and uppercase words in at least 4 of the previous 12
546 dictionaries. A decent number of proper names is also included: The
547 top 1000 male, female, and Last names from the 1990 Census report; a
548 list of names sent to me by Alan Beale; and a few names that I added
549 myself. Finally a small list of abbreviations not commonly found in
550 other word lists is included.
552 The name files form the Census report is a government document which I
553 don't think can be copyrighted.
555 The file special-jargon.50 uses common.lst and word.lst from the
556 "Unofficial Jargon File Word Lists" which is derived from "The Jargon
557 File". All of which is in the Public Domain. This file also contain
558 a few extra UNIX terms which are found in the file "unix-terms" in the
561 The 55 level includes words from Alan's 2of4brif list found in version
562 4.0 of his 12dicts package. Like his other stuff the 2of4brif is also
563 in the public domain.
565 The 60 level includes all words appearing in at least 2 of the 12
566 dictionaries as indicated by the 12Dicts package.
568 The 70 level includes Brian's frequency class 0 and the 74,550 common
569 dictionary words from the MWords package. The common dictionary words,
570 like those from the 12Dicts package, have had all likely inflections
571 added. The 70 level also included the 5desk list from version 4.0 of
572 the 12Dics package which is in the public domain.
574 The 80 level includes the ENABLE word list, all the lists in the
575 ENABLE supplement package (except for ABLE), the "UK Advanced Cryptics
576 Dictionary" (UKACD), the list of signature words from the YAWL package,
577 and the 10,196 places list from the MWords package.
579 The ENABLE package, mainted by M\Cooper <thegrendel@theriver.com>,
580 is in the Public Domain:
582 The ENABLE master word list, WORD.LST, is herewith formally released
583 into the Public Domain. Anyone is free to use it or distribute it in
584 any manner they see fit. No fee or registration is required for its
585 use nor are "contributions" solicited (if you feel you absolutely
586 must contribute something for your own peace of mind, the authors of
587 the ENABLE list ask that you make a donation on their behalf to your
588 favorite charity). This word list is our gift to the Scrabble
589 community, as an alternate to "official" word lists. Game designers
590 may feel free to incorporate the WORD.LST into their games. Please
591 mention the source and credit us as originators of the list. Note
592 that if you, as a game designer, use the WORD.LST in your product,
593 you may still copyright and protect your product, but you may *not*
594 legally copyright or in any way restrict redistribution of the
595 WORD.LST portion of your product. This *may* under law restrict your
596 rights to restrict your users' rights, but that is only fair.
598 UKACD, by J Ross Beresford <ross@bryson.demon.co.uk>, is under the
601 Copyright (c) J Ross Beresford 1993-1999. All Rights Reserved.
603 The following restriction is placed on the use of this publication:
604 if The UK Advanced Cryptics Dictionary is used in a software package
605 or redistributed in any form, the copyright notice must be
606 prominently displayed and the text of this document must be included
609 There are no other restrictions: I would like to see the list
610 distributed as widely as possible.
612 The 95 level includes the 354,984 single words, 256,772 compound
613 words, 4,946 female names and the 3,897 male names, and 21,986 names
614 from the MWords package, ABLE.LST from the ENABLE Supplement, and some
615 additional words found in my part-of-speech database that were not
618 Accent information was taken from UKACD.
620 The VarCon package was used to create the American, British, Canadian,
621 and Australian word list. It is under the following copyright:
623 Copyright 2000-2016 by Kevin Atkinson
625 Permission to use, copy, modify, distribute and sell this array, the
626 associated software, and its documentation for any purpose is hereby
627 granted without fee, provided that the above copyright notice appears
628 in all copies and that both that copyright notice and this permission
629 notice appear in supporting documentation. Kevin Atkinson makes no
630 representations about the suitability of this array for any
631 purpose. It is provided "as is" without express or implied warranty.
633 Copyright 2016 by Benjamin Titze
635 Permission to use, copy, modify, distribute and sell this array, the
636 associated software, and its documentation for any purpose is hereby
637 granted without fee, provided that the above copyright notice appears
638 in all copies and that both that copyright notice and this permission
639 notice appear in supporting documentation. Benjamin Titze makes no
640 representations about the suitability of this array for any
641 purpose. It is provided "as is" without express or implied warranty.
643 Since the original words lists come from the Ispell distribution:
645 Copyright 1993, Geoff Kuenning, Granada Hills, CA
648 Redistribution and use in source and binary forms, with or without
649 modification, are permitted provided that the following conditions
652 1. Redistributions of source code must retain the above copyright
653 notice, this list of conditions and the following disclaimer.
654 2. Redistributions in binary form must reproduce the above copyright
655 notice, this list of conditions and the following disclaimer in the
656 documentation and/or other materials provided with the distribution.
657 3. All modifications to the source code must be clearly marked as
658 such. Binary redistributions based on modified source code
659 must be clearly marked as modified versions in the documentation
660 and/or other materials provided with the distribution.
661 (clause 4 removed with permission from Geoff Kuenning)
662 5. The name of Geoff Kuenning may not be used to endorse or promote
663 products derived from this software without specific prior
666 THIS SOFTWARE IS PROVIDED BY GEOFF KUENNING AND CONTRIBUTORS ``AS IS'' AND
667 ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
668 IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
669 ARE DISCLAIMED. IN NO EVENT SHALL GEOFF KUENNING OR CONTRIBUTORS BE LIABLE
670 FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
671 DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
672 OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
673 HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
674 LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
675 OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
679 The variant word lists were created from a list of variants found in
680 the 12dicts supplement package as well as a list of variants I created
683 The Readmes for the various packages used can be found in the
684 appropriate directory under the r/ directory.
688 The process of "sort"s, "comm"s, and Perl scripts to combine the many
689 word lists and separate out the variant information is inexact and
690 error prone. The whole things needs to be rewritten to deal with
691 words in terms of lemmas. When the exact lemma is not known a best
692 guess should be made. I'm not sure what form this should be in. I
693 originally thought this should be some sort of database, but maybe I
694 should just slurp all that data into memory and process it in one
695 giant perl script. With the amount of memory available these days (at
696 least 2 GB, often 4 GB or more) this should not really be a problem.
698 In addition, there is a very nice frequency analyze of the BNC corpus
699 done by Adam Kilgarriff. Unlike Brian's word lists the BNC lists
700 include part of speech information. I plan on somehow using these
701 lists as Adam Kilgarriff has given me the OK to use it in SCOWL.
702 These lists will greatly reduce the problem of inflected forms of a
703 word appearing at different levels due to the part-of-speech
706 There is frequency information for some other corpus such as COCA
707 (Corpus of Contemporary American English) and ANS (American National
708 Corpus) which I might also be able to use. The former will require
709 permission, and the latter is of questionable quality.
711 RECREATING THE WORD LISTS:
713 In order to recreate the word lists you need a modern version of Perl,
714 bash, the traditional set of shell utilities, a system that supports
715 symbolic links, and quite possibly GNU Make. The easiest way to
716 recreate the word lists is to checkout the corresponding Git version
717 (see the version string at the start of the file) and simply type
718 "make" (see http://wordlist.aspell.net). You can try to download all
719 the pieces manually, but this method is not no longer tested nor
722 The src/ directory contains the numerous scripts used in the creation
723 of the final product.
725 The r/ directory contains the raw data used to create the final
726 product. If you checkout from Git this directory should be populated
727 automatically for you. If you insist on doing it the hard way see the
728 README file in the r/ directory for more information.
730 The l/ directory contains symbolic links used by the actual scripts.
732 Finally, the working/ directory is where all the intermittent files go
733 that are not specific to one source.