LLM Exploration: The Vector Magic Behind Large Models, A Primer on Embedding Technology

This article was originally written in 2022, but it was never finalized 😂. Since the rise of LLMs, the technology in the NLP field has been updating rapidly. This article only briefly introduces the basic knowledge related to Embedding, and specific applications will be updated and published on the blog later.

Preface

It's been quite a while since I last updated my blog. This summer was very busy, but I managed to achieve a little progress. I'll continue to share my findings on the blog going forward~

Large models and related applications have been very popular this year, and one such application is called "Vector Databases." Vector databases are primarily used to store vectors and provide vector query functionality. How are the vectors they use generated? This naturally brings us to the Embedding technique.

Embedding originated from Word Embedding and has made significant progress over the years. Horizontally, it has evolved from simple Word Embedding to various forms such as Item Embedding, Entity Embedding, Graph Embedding, Position Embedding, Segment Embedding, etc. Vertically, it has advanced from static Word Embedding to dynamic pre-trained models like ELMo, BERT, GPT, GPT-2, GPT-3, ALBERT, XLNet, etc. These pre-trained models can be fine-tuned to serve downstream tasks. Embedding is no longer fixed and unchangeable; instead, these pre-trained models can learn semantics in new semantic environments, efficiently completing various downstream tasks such as classification, question answering, summarization, reading comprehension, etc. For many of these tasks, the completion efficiency has surpassed the average level achieved by humans.

Embedding

In recent years, in the fields of machine learning and deep learning, embedding technology has developed rapidly and become widespread.

Simply put, embedding is the representation of an object using a vector. This object can be a word, a sentence, a sequence, a product, an action, a book, a movie, etc. It can be said that embedding involves the vast majority of objects in machine learning and deep learning. These objects are the most fundamental, commonly used, and most important in machine learning and deep learning. Therefore, how to effectively represent and learn these objects becomes critically important. The widespread application of Word Embeddings, such as word2vec, has particularly brought about broader extensions and expansions. Embedding technology has extended from its initial field of natural language processing (NLP) into traditional machine learning, search ranking, recommendation systems, knowledge graphs, and other areas. This is specifically manifested in the progression from Word Embeddings to Item Embeddings, Graph Embeddings, Categorical Variables Embeddings, and more.

Embedding itself is also continuously evolving, expanding from initially singular, static representations to richer, dynamic ones. This is exemplified by the transition from static Word Embeddings to dynamic pre-trained models like ELMo, Transformer, GPT, BERT, XLNet, ALBERT, etc.

Embedding can accomplish many things, mainly involving the following areas:

• Word Embedding
• Item Embedding
• Using Embedding to Handle Categorical Features
• Graph Embedding
• Contextual Word Embedding
• Implementing Chinese Automatic Summarization Using Word Embeddings

However, due to space constraints, this article only selects and introduces content related to NLP and knowledge base question answering.

General Steps for Handling Sequence Problems

Sequence problems are extremely common—fields such as natural language processing, web browsing, and time series analysis are all closely tied to sequences. Therefore, understanding how to handle sequence problems and uncover the hidden rules and logic within sequences is very important.

Take natural language processing (NLP) as an example. Suppose you receive a lengthy article or news report as language material and are tasked with using NLP methods to extract a summary from it. How should you approach this? What factors should you consider? What steps are involved? Where do you start?

Upon receiving language material—whether in Chinese or English—you first need to perform some necessary cleaning tasks, such as removing special symbols, converting formats, and filtering out stopwords. Then, you tokenize the text and index it. Next, you vectorize the tokens (words, etc.) using relevant models or algorithms. Finally, the processed data is output for downstream tasks.

As shown in the figure, word embeddings or pre-trained models are key—their quality directly impacts the effectiveness of downstream tasks. The word embedding and model training stages involve a variety of algorithms and models. In recent years, significant progress has been made with methods like word2vec, Transformer, BERT, and ALBERT, which have set new performance records in tasks such as natural language processing, language recognition, recommendation systems, and search ranking.

Word Embedding

Machines cannot directly receive identifiers (tokens) such as words, terms, or characters. The initial approach was to represent each identifier using integers. This method is simple but not flexible enough.

Later, One-Hot Encoding was adopted for representation. While convenient, this method is highly sparse, constitutes hard coding, and cannot encode additional information.

Subsequently, researchers came up with using numerical vectors or Token Embedding for representation, which is commonly referred to as Word Embedding, also known as distributed representation.

The differences between these methods can be illustrated directly with the following diagram:

One-Hot Encoding is a sparse, high-dimensional hard coding. If a corpus contains ten thousand distinct words, each word would require a ten-thousand-dimensional one-hot vector for representation. In contrast, if represented using vectors or word embeddings, these vectors are low-dimensional and dense. Moreover, the values of these vectors are learned through training rather than being explicitly assigned.

Methods for Learning Word Embeddings

There are mainly two approaches:

• Learning Word Embeddings through a Platform's Embedding Layer: Learn word embeddings while completing tasks. For example, use an Embedding layer as the first layer, initialize the word vectors randomly, and then continuously learn using a platform (like PyTorch, TensorFlow, etc.), including both forward and backward learning, to finally obtain the required word vectors.
• Using Pre-trained Models: There are many word embeddings or pre-trained models that have been pre-trained on larger corpora and can be used directly. Before having enough quality data, using a pre-trained model may yield better results than training your own.

I will introduce the specific code implementation in my later articles.

Item Embedding

Embedding is a powerful concept that is not limited to the field of natural language processing but can also be applied to many other areas.

A team from Microsoft wrote a paper titled "Item2Vec: Neural Item Embedding for Collaborative Filtering," which is highly practical and significantly expands the application scope of word2vec, extending it from the NLP domain directly to any field where sequences can be generated, such as recommendation, advertising, and search ranking.

Processing Categorical Features

In traditional machine learning, input data often contains categorical features, and processing these categorical features is one of the key aspects of feature engineering. Categorical features, also known as discrete features, typically have a data type of object. However, most machine learning models can only process numerical data, so it is necessary to convert categorical data into numeric data.

Categorical features can be divided into two types: ordinal and nominal.

These two types of data can be converted into numbers using different methods. For nominal data, One-Hot Encoding is commonly used for conversion. However, when dealing with large-scale data, such as a feature with hundreds, thousands, or even more categories, using one-hot encoding can lead to a dramatic increase in feature dimensions, resulting in a massive feature matrix. Additionally, one-hot encoding only converts categorical data into 0s or 1s and cannot accurately express the potential rules or distribution characteristics between categories. For example, if there is a feature representing addresses, including places like Beijing, Shanghai, Hangzhou, New York, Washington, Los Angeles, Tokyo, and Osaka, there are certain spatial relationships between these geographic locations—Beijing, Shanghai, and Hangzhou are relatively close to each other, while Shanghai and New York are far apart. However, one-hot encoding cannot reflect these geographic distribution patterns.

In recent years, neural networks and deep learning have become increasingly prevalent in fields such as computer vision, natural language processing, and time series forecasting. In deep learning applications, Embedding, as a technique that converts discrete variables into continuous vectors, has greatly facilitated the development of traditional machine learning and neural networks. Currently, there are two main applications of Embedding technology: one is Word Embedding in natural language processing, and the other is Entity Embedding for categorical data. Simply put, Embedding uses low-dimensional vectors to represent an object, which can be a word, a categorical feature (such as a product, movie, item, etc.), or time series features. Through learning, Embedding vectors can more accurately capture the intrinsic meaning of features, allowing objects represented by vectors that are geometrically close to each other to have similar semantics.

Graph Embedding

In the previous content, we explored techniques derived from Word Embedding, such as Item Embedding, which rely on their inherent sequential characteristics. However, the scope of applicable domains extends far beyond that. At first glance, some scenarios may seem unrelated to sequences, but with appropriate adjustments and transformations, these techniques can still be applied.

Graph Embedding, similar to Word Embedding, aims to represent nodes in a network using low-dimensional, dense, and real-valued vectors. Today, Graph Embedding is widely popular in fields such as recommendation systems, search ranking, and ad delivery, demonstrating excellent performance and achieving significant results in practical applications.

A graph is a form of expressing "two-dimensional" relationships, while a sequence represents a "one-dimensional" relationship. Therefore, when converting a graph into a graph embedding (Graph Embedding), it is typically necessary to first transform the graph structure into a sequence form using specific algorithms. Subsequently, related models or algorithms are employed to further convert these sequences into embedding vectors.

Common methods include the following (due to space constraints, we will not elaborate further here):

• DeepWalk
• LINE
• node2vec

Application Example: Recommendation Systems

In recent years, many researchers have been dedicated to integrating knowledge graphs into recommendation systems to effectively address the sparsity and cold-start challenges faced by traditional recommendation systems. Currently, the methods for applying knowledge graph feature learning to recommendation systems can be broadly categorized into three types: One-by-One Learning, which involves independently learning the knowledge graph and the recommendation model; Joint Learning, which optimizes the objective functions of both the knowledge graph and the recommendation system simultaneously; and Alternate Learning, which alternately and iteratively optimizes the knowledge graph and the recommendation model.

Contextual Word Embedding

The Word Embedding we mentioned earlier gained widespread attention due to the popularity of word2vec. Compared to discrete one-hot encoding, it has significant advantages: on one hand, it reduces data dimensionality, and on the other, it can capture linear relationships in the semantic space—for example, semantic similarities like "king - queen ≈ man - woman." Therefore, word2vec and similar methods became almost standard in all deep learning models. However, this representation method also has limitations. It is based on a fixed dictionary generated from a corpus, where each word corresponds to a vector of fixed length. When encountering polysemy (一词多义), it fails to accurately distinguish between different meanings, appearing inadequate.

How to address the polysemy problem?

Since the word embeddings generated by word2vec are fixed and unchanging, they are referred to as static word embeddings. This embedding method does not consider the context; regardless of the context in which a word appears, its vector representation remains consistent. Therefore, if contextual influence needs to be considered, one cannot rely on static word embeddings. Instead, dynamic word embedding methods should be adopted, or a combination of pre-training and fine-tuning approaches should be used. For example, models like ELMo, GPT, GPT-2, BERT, ERNIE, XLNet, and ALBERT all fall into the category of dynamic word embeddings. These methods have greatly enhanced the performance of natural language processing and are still rapidly evolving.

Summary

Embedding is almost everywhere—it is involved in traditional machine learning, recommendation systems, natural language processing in deep learning, and even image processing. In a sense, getting Embedding right means overcoming the key challenges of the entire project.