Django Full-Text Search Implementation: HayStack + Whoosh + Jieba

Tools Involved

• haystack is Django's open-source search framework. This framework supports Solr, Elasticsearch, Whoosh, **Xapian search engines; you can switch engines directly without changing code, reducing the amount of code.

• The search engine used is Whoosh, a full-text search engine implemented in pure Python. It has no binary files and the like, is relatively compact, and fairly simple to configure, though of course its performance is naturally a bit lower.

• Chinese word segmentation with Jieba: since Whoosh comes with English tokenization and does not support Chinese word segmentation very well, jieba is used to replace whoosh's tokenizer component.

Model Configuration

class Post(models.Model):
    """post 文章页面"""
    id = models.AutoField(primary_key=True, max_length=20)
    name = models.CharField(max_length=200, help_text="Post文章的URL链接名称")
    post_category = models.ForeignKey(PostCategory, blank=True, null=True, default=None, on_delete=models.SET_NULL)
    post_author = models.ForeignKey(PostAuthor, blank=True, null=True, default=None, on_delete=models.SET_NULL)
    post_date = models.DateTimeField(default=timezone.now, help_text="创建日期")
    post_modified = models.DateTimeField(default=timezone.now, help_text="修改日期")
    post_content = models.TextField(help_text="html格式的页面内容，仅在page类型才可用")
    post_title = models.CharField(max_length=200)
    post_summary = models.TextField(max_length=300, blank=True)
    post_status = models.CharField(choices=(('publish', '已发布'), ('draft', '草稿')), default='publish', max_length=20,
                                   help_text="页面状态")

    # post_slug = models.CharField(max_length=200, help_text="URL链接名称")

    def __str__(self):
        return self.post_title

Installing Python Modules

• whoosh
• django-haystack
• jieba

Modifying Whoosh's Analyzer

Copy the file whoosh_backend.py (the file path is python路径/lib/python版本/site-packages/haystack/backends/whoosh_backend.py) into the app folder, and modify it as follows Add from jieba.analyse import ChineseAnalyzer

Change it to the following

schema_fields[field_class.index_fieldname] =
    TEXT(stored=True, analyzer=ChineseAnalyzer(),
            field_boost=field_class.boost)

Modifying settings.py

• Add Haystack to Django's INSTALLED_APPS

HAYSTACK_CONNECTIONS = {
    'default': {
        'ENGINE': 'Core.whoosh_backend.WhooshEngine',
        'PATH': os.path.join(BASE_DIR, 'Resources', 'index', 'whoosh'),
    },
}

Note:

• Core is my App name; please replace it with your own App folder name as appropriate
• PATH is the folder for storing Whoosh index files.
• For configurations of other engines, please consult the official documentation.

Creating the Index

Create a search_indexes.py file in the App folder that needs search functionality, used to create the index.

The contents are as follows:

import datetime
from haystack import indexes
from Core import models


class PostIndex(indexes.SearchIndex, indexes.Indexable):
    text = indexes.CharField(document=True, use_template=True)

    post_author = indexes.CharField(model_attr='post_author')
    post_date = indexes.DateTimeField(model_attr='post_date')

    def get_model(self):
        return models.Post

    def index_queryset(self, using=None):
        """Used when the entire index for model is updated."""
        return self.get_model().objects.filter(post_date__lte=datetime.datetime.now())

Each index must have one and only one field with document=True, which means haystack and the search engine will use this field's content as the index for retrieval (primary field). Other fields are merely auxiliary attributes for convenient access and are not used as retrieval data.

Note: If a field is set with document=True, it is generally convention that this field is named text. This is the consistent naming in the SearchIndex class to prevent backend confusion. Of course you can change the name as you like, but it is not recommended.

Additionally, haystack provides use_template=True on the text field, which allows us to use a data template to build the search engine index file. It is convenient to use (officially recommended; of course there are other more complex ways to build index files, which I don't know yet). The data template path is yourapp/templates/search/indexes/yourapp/classname_text.txt, for example in this case Resources/templates/search/indexes/Core/post_text.txt The filename must be 类名_text.txt of the class to be indexed, and its contents are:

{{ object.post_title }}
{{ object.post_author }}
{{ object.post_content }}
{{ object.post_summary }}

The purpose of this data template is to build indexes for

• Post.post_title
• Post.post_author
• Post.post_content
• Post.post_summary

these four fields. When searching, full-text search matching will be performed on these three fields.

Configuring URLs

Add the following to the URL configuration

    path('search/', include('haystack.urls')),

Writing search.html

Add a search.html file under the templates/search/ folder:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Title</title>
</head>
<body>
<h2>Search</h2>

<form method="get" action=".">
    <table>
        {{ form.as_table }}
        <tr>
            <td> </td>
            <td>
                <input type="submit" value="Search">
            </td>
        </tr>
    </table>

    {% if query %}
        <h3>Results</h3>

        {% for result in page.object_list %}
            <p>
                <a href="{{ result.object.get_absolute_url }}">{{ result.object.post_title }}</a>
            </p>
        {% empty %}
            <p>No results found.</p>
        {% endfor %}

        {% if page.has_previous or page.has_next %}
            <div>
                {% if page.has_previous %}<a href="?q={{ query }}&page={{ page.previous_page_number }}">{% endif %}«
                Previous{% if page.has_previous %}</a>{% endif %}
                |
                {% if page.has_next %}<a href="?q={{ query }}&page={{ page.next_page_number }}">{% endif %}Next »
                {% if page.has_next %}</a>{% endif %}
            </div>
        {% endif %}
    {% else %}
        {# Show some example queries to run, maybe query syntax, something else? #}
    {% endif %}
</form>
</body>
</html>

Rebuilding the Index

Use python manage.py rebuild_index or use the update_index command.

Add the following configuration in settings.py to automatically refresh the index.

HAYSTACK_SIGNAL_PROCESSOR = "haystack.signals.RealtimeSignalProcessor"

That's it! You can open 网站/search and try out the search feature~

A First Look at SearchQuerySet

I found that none of the materials online covered this, so I looked it up on the official site. Documentation URL: https://django-haystack.readthedocs.io/en/v2.8.1/searchqueryset_api.html#ref-searchqueryset-api

This SearchQuerySet is similar to the QuerySet in the Django ORM framework. Those familiar with Django should get the hang of it quickly, though handling the results can be rather troublesome. Using SearchQuerySet is generally for Ajax APIs. I found that serializers previously used for QuerySet cannot be used for SearchQuerySet; some modifications would still be needed to make it work. I'll continue working on this when I have time. Or I'll look at the official docs again and study it.

The usage is pretty simple; just look at the official documentation, it's already written very clearly~