3 commits

Author	SHA1	Message	Date
nicolasconnault	b7064779f5	Updated copyright notice in header	2007-10-10 05:25:14 +00:00
stronk7	783fbd2234	Now the tokeniser is using the new textlib->entities_to_utf8() ... own function (PHP4 and PHP compatible) Merged from MOODLE_18_STABLE	2007-06-13 23:22:57 +00:00
stronk7	7bb11c76d1	First cut of the tokeniser library. Modified from drupal search.module ... code (see copyrights). The library converts any text/html into an array of tokens with their score (weight). Supports stop_words, cjk basic tokeniser (for indexers) and different modes of handling numbers. Merged from MOODLE_18_STABLE	2007-06-12 18:15:55 +00:00