Controlling the Data Import Process with the Django-Import-Export Plugin

Introduction

I've previously written two articles related to this plugin, which you can review:

• Using django-import-export, a Django Data Import/Export Powerhouse
• Handling Foreign Keys with the Django-Import-Export Plugin

Recently a friend left a comment asking me a question about the django-import-export plugin

To illustrate this problem more vividly, I'll use a book management example

Database table

id	name	price
1	book1	10
2	book2	20
3	book3	30

Excel table to import

id	name	price	tax
4	book4	40	5
5	book5	50	6
6	book6	60	7

As you can see, each book in the Excel file has two attributes, price and tax, but the database only has the price attribute

When importing, you need to add each book's price + tax; that is the final price to store in the database

In the past, for this kind of scenario I would suggest using pandas to process the data and then import it, and only using the django-import-export plugin for data export, because its documentation is quite sparse, and the examples it provides can hardly solve real-world problems. Often a requirement that takes very little time to handle manually with pandas would require digging through the source code and sparse documentation with this plugin, which is far too inefficient.

But in the spirit of tinkering, let's still look into whether django-import-export can actually implement this feature. (Of course it can, otherwise this article wouldn't exist)

Analysis

First, looking at the official documentation, there's a section called import data workflow

Address: https://django-import-export.readthedocs.io/en/latest/import_workflow.html

import_data(dataset, dry_run=False, raise_errors=False)

The import_data() method of Resource is responsible for importing data from a given dataset.

dataset is required and expected to be a tablib.Dataset with a header row.

dry_run is a Boolean which determines if changes to the database are made or if the import is only simulated. It defaults to False.

raise_errors is a Boolean. If True, import should raise errors. The default is False, which means that eventual errors and traceback will be saved in Result instance.

According to the documentation, when importing data, we can process the data to be imported through the import_data hook

Then this hook has a parameter, dataset, which is a tablib thing

I hadn't used tablib before. After looking it up, it's a library made by the author of requests, so it shouldn't be bad

Official documentation: https://tablib.readthedocs.io/en/stable/tutorial.html

Writing the Code

Let's just use the official code example

Code repository: https://github.com/django-import-export/django-import-export

It's also about book management

Models Code

Let's look at its model design

class Book(models.Model):
    name = models.CharField('Book name', max_length=100)
    author = models.ForeignKey(Author, blank=True, null=True, on_delete=models.CASCADE)
    author_email = models.EmailField('Author email', max_length=75, blank=True)
    imported = models.BooleanField(default=False)
    published = models.DateField('Published', blank=True, null=True)
    published_time = models.TimeField('Time published', blank=True, null=True)
    price = models.DecimalField(max_digits=10, decimal_places=2, null=True, blank=True)
    added = models.DateTimeField(blank=True, null=True)

    categories = models.ManyToManyField(Category, blank=True)

    def __str__(self):
        return self.name

Lots of fields

Data to Import

Still provided by the official project, available in various formats. I chose CSV, which is more convenient

id,name,author_email
1,Some book,test@example.com

Converted to a table it looks like this

As you can see, there are far fewer fields than defined in the model

We need to add a price attribute to the dataset when importing

Resources Code

This is the original code

class BookResource(ModelResource):
    class Meta:
        model = Book

    def for_delete(self, row, instance):
        return self.fields['name'].clean(row) == ''

        return super(BookResource, self).import_data(
            dataset, dry_run, raise_errors, use_transactions,
            collect_failed_rows, rollback_on_validation_errors, **kwargs

Now we need to add a hook to process the imported data

The code is as follows

def import_data(self, dataset: tablib.Dataset, dry_run=False, raise_errors=False,
                use_transactions=None, collect_failed_rows=False,
                rollback_on_validation_errors=False, **kwargs):
    cols = []
    for item in dataset['id']:
      cols.append(int(item) * 99)

    dataset.append_col(cols, header='price')

    return super(BookResource, self).import_data(
      dataset, dry_run, raise_errors, use_transactions,
      collect_failed_rows, rollback_on_validation_errors, **kwargs
    )

Use the append_col method of DataSet to add a new column

For more operations on this DataSet, please refer to the Tablib documentation

The specific operations in this part can be modified according to actual needs. Here I simply and crudely multiply the ID by 99

After processing the DataSet, remember to call the parent class's import_data to complete the data import operation.

Results

After performing the import in the admin backend, you can get the following result

You can see that the price attribute has become 99

That's it, done~

It's actually quite simple, it's just that the official documentation is too sparse, without even an example, so you have to figure it out yourself

References

Although the links are already given above, let me summarize them here

• Plugin documentation: https://django-import-export.readthedocs.io/en/latest/import_workflow.html
• Plugin project homepage: https://github.com/django-import-export/django-import-export
• Tablib documentation: https://tablib.readthedocs.io/en/stable/tutorial.html