LittleWhite One

A new LittleWhite implemented by heavily modifying the chatterbot framework~

Introduction

• core: core framework, a modded version of chatterbot
• corpus: English and Chinese training corpora
• nltk_data: nltk datasets that are needed

Text Similarity Algorithms

• LevenshteinDistance
• SpacySimilarity
• JaccardSimilarity

Levenshtein Distance Algorithm

Chinese name: Levenshtein distance Reference: https://en.wikipedia.org/wiki/Levenshtein_distance

Levenshtein distance is a concept that characterizes or measures the degree of difference between two strings. Taking the words kitten and sitting as an example, three standard edit operations are defined: substitution, insertion (insert), and deletion (delete) to eliminate the differences between these two words. Each time a standard edit operation is applied, the Levenshtein distance is incremented by one. kitten requires 2 substitutions and 1 insertion of a new character to obtain the word sitting, so the so-called Levenshtein distance should be 3. By definition, this distance is proportional to the degree of difference between the strings.

Definition: The Levenshtein distance between two strings a and b

Edit distance is a basic NLP algorithm for measuring text similarity. It can serve as one of the important features for text similarity tasks, and can be applied to spelling checking, paper plagiarism detection, gene sequence analysis, and many other areas. However, its shortcomings are also obvious: the algorithm computes based on the structure of the text itself and has no way to obtain information at the semantic level.

SpacySimilarity Algorithm

SpacySimilarity belongs to semantic similarity computation methods.

Reference: https://spacy.io/api/doc/#similarity

First average the word vectors of the sentence to obtain the sentence's semantic representation, then compute the cosine similarity of the two sentences' semantic representations.

JaccardSimilarity Algorithm

Reference: https://en.wikipedia.org/wiki/Jaccard_index

Given two sets A and B, the Jaccard coefficient is defined as the ratio of the size of the intersection of A and B to the size of the union of A and B, defined as follows:

When both sets A and B are empty, J(A,B) is defined as 1.

A related metric to the Jaccard coefficient is called the Jaccard distance, used to describe the dissimilarity between sets. The larger the Jaccard distance, the lower the sample similarity. The formula is defined as follows:

todo

Algorithm Optimization

• Try several other word-vector-based methods for computing text similarity:
  • Compute similarity by averaging word vectors
  • Compute similarity by tfidf-weighted averaging of word vectors
  • Compute similarity with word-vector weighting-PCA
• Compute sentence semantic similarity based on deep learning

Training Optimization

• Train with a new large-capacity corpus
• Connect tai to extract chat corpus
• Combine with LittleWhite v3's dual engines to learn from user conversations