How-To Guides

Bulk Import Data

How to import large datasets from CSV or JSON files

This guide explains how to import large datasets of zines, agents, holdings, and repositories from CSV or JSON files.

Prerequisites

✅ ZineCore2 server installed
✅ Database migrated
✅ Vocabularies loaded
✅ Virtual environment activated
✅ Data files prepared (CSV or JSON)

Import Methods

Method	Best For	Complexity
Web-based JSON Import	Quick admin imports (<1,000 records)	Very Low
API (POST requests)	Small datasets (<100 records)	Low
Django Management Command	Medium datasets (100-10,000 records)	Medium
Direct SQL COPY	Large datasets (>10,000 records)	High

Method 1: Web-based JSON Import (Easiest)

The Django Admin provides a web-based JSON import tool for quick imports without writing code.

Access the Import Tool

Log in to Django Admin at http://localhost:8000/admin/
Navigate to "Import JSON" in the sidebar
Or visit directly: http://localhost:8000/admin/import-json/

Features

✅ Auto-detection of schema types

Automatically detects ZineCore2, AgentCore2, HoldingCore2, or RepoCore2 based on record structure

✅ Optional ID fields

Omit id, zine_id, agent_id, repo_id, or holding_id fields to auto-generate them
System generates unique IDs following naming conventions

✅ Vocabulary codes instead of primary keys

Use vocabulary codes like "kind": "person" instead of database primary keys
Applies to: AgentCore2.kind, RepositoryCore2.kind, HoldingCore2.access_status

✅ Two-phase validation

Phase 1: JSON Schema validation (structure and required fields)
Phase 2: Django REST Framework serializer validation (business logic and foreign keys)

✅ Import modes

Atomic mode (default): All records succeed or all fail
Non-atomic mode: Continue importing valid records even if some fail

Example: Import Agents

Paste this JSON:

[
  {
    "kind": "person",
    "display_name": "Judith Arcana",
    "public": true,
    "biography": "Writer and activist focused on reproductive rights.",
    "pronouns": ["she/her"]
  },
  {
    "agent_id": "agent_sara_ahmed",
    "kind": "person",
    "display_name": "Sara Ahmed",
    "public": true,
    "biography": "Feminist scholar and author.",
    "pronouns": ["she/her"]
  }
]

Note:

First record has no agent_id — will be auto-generated
Second record provides explicit agent_id
kind uses vocabulary code "person" (not a primary key)

Example: Import Zines

[
  {
    "title": "Mutate Zine #3",
    "creator": ["agent_judith_arcana"],
    "subject": ["feminism", "reproductive-rights"],
    "genre": ["personal-zine"],
    "date": ["2024"],
    "language": ["en"],
    "rights": ["cc-by-4.0"],
    "abstract": "Personal narratives about reproductive rights"
  },
  {
    "zine_id": "zine_feminist_killjoy_1",
    "title": "Feminist Killjoy #1",
    "creator": ["agent_sara_ahmed"],
    "subject": ["feminism"],
    "genre": ["personal-zine"],
    "date": ["2024"],
    "language": ["en"],
    "rights": ["cc-by-4.0"]
  }
]

Note:

First zine has no zine_id — will be auto-generated
Second zine provides explicit zine_id

Example: Import Repositories

[
  {
    "repository_name": "Barnard Zine Library",
    "kind": "zine-library",
    "city": "New York",
    "region": "NY",
    "country": "US",
    "website": "https://zines.barnard.edu",
    "description": "Academic zine collection focused on feminist and activist publications"
  },
  {
    "repo_id": "repo_abc_no_rio",
    "repository_name": "ABC No Rio Zine Library",
    "kind": "distro",
    "city": "New York",
    "region": "NY",
    "country": "US"
  }
]

Note:

kind uses vocabulary codes: "zine-library", "distro", "library", "archive"

Validation Errors

If validation fails, you'll see detailed error messages:

Example error:

Agent with agent_id 'agent_unknown' does not exist
Invalid subject codes: invalid-subject
Field 'title' is required

Fix the errors in your JSON and re-submit.

Best Practices

✅ Start small

Test with 2-3 records first
Verify import worked correctly
Then import full dataset

✅ Use atomic mode for critical data

Ensures data integrity (all-or-nothing)
Prevents partial imports

✅ Use non-atomic mode for large datasets

Allows continuing past errors
Import valid records, fix errors later

✅ Validate data before importing

Check required fields are present
Verify vocabulary codes are valid
Ensure foreign key references exist

Pros:

No coding required
Built into Django Admin
Auto-generates IDs
Accepts vocabulary codes (human-friendly)
Two-phase validation catches errors early
Visual feedback in browser

Cons:

Limited to ~1,000 records (browser memory constraints)
Slower than management commands for large datasets
Requires admin access

Method 2: API Import (Small Datasets)

Use API POST requests for small datasets.

Python Script

import requests
import json

API_BASE = "http://localhost:8000/api"
TOKEN = "YOUR_TOKEN_HERE"

headers = {
    "Authorization": f"Token {TOKEN}",
    "Content-Type": "application/json"
}

# Load data from JSON file
with open('zines.json') as f:
    zines = json.load(f)

# Import each zine
for zine_data in zines:
    response = requests.post(
        f"{API_BASE}/zines/",
        headers=headers,
        json=zine_data
    )

    if response.status_code == 201:
        print(f"✓ Created: {zine_data['title']}")
    else:
        print(f"✗ Failed: {zine_data['zine_id']}")
        print(f"  Error: {response.json()}")

Input Format (JSON)

[
  {
    "zine_id": "zine_mutate_3_1st",
    "title": "Mutate Zine #3",
    "creator": ["agent_judith_arcana"],
    "subject": ["feminism"],
    "genre": ["personal-zine"],
    "date": ["2024"],
    "language": ["en"],
    "rights": ["cc-by-4.0"]
  },
  {
    "zine_id": "zine_feminist_killjoy_1",
    "title": "Feminist Killjoy #1",
    "creator": ["agent_sara_ahmed"],
    "subject": ["feminism"],
    "genre": ["personal-zine"],
    "date": ["2024"],
    "language": ["en"],
    "rights": ["cc-by-4.0"]
  }
]

Pros:

Validates data via serializers
Provides detailed error messages
Works with existing API authentication

Cons:

Slow for large datasets (one HTTP request per record)
Rate limiting may apply

Note about optional IDs: Since all schema types now support optional ID fields, you can omit them in API requests:

{
  "title": "My Zine",
  "creator": ["agent_001"],
  "subject": ["feminism"],
  "genre": ["personal-zine"],
  "date": ["2024"],
  "language": ["en"],
  "rights": ["cc-by-4.0"]
}

The API will auto-generate a unique zine_id.

Method 3: Django Management Command (Recommended)

Create a custom management command for bulk imports.

Step 1: Create Management Command

# Create directory structure
mkdir -p backend/catalog/management/commands

# Create command file
touch backend/catalog/management/commands/import_zines.py

Step 2: Write Command

# catalog/management/commands/import_zines.py
import csv
import json
from django.core.management.base import BaseCommand
from catalog.models import Zine
from catalog.serializers import ZineWriteSerializer

class Command(BaseCommand):
    help = 'Import zines from CSV or JSON file'

    def add_arguments(self, parser):
        parser.add_argument('file', type=str, help='Path to CSV or JSON file')
        parser.add_argument(
            '--format',
            type=str,
            default='json',
            choices=['json', 'csv'],
            help='File format (default: json)'
        )

    def handle(self, *args, **options):
        file_path = options['file']
        file_format = options['format']

        if file_format == 'json':
            self.import_from_json(file_path)
        elif file_format == 'csv':
            self.import_from_csv(file_path)

    def import_from_json(self, file_path):
        with open(file_path, 'r') as f:
            data = json.load(f)

        total = len(data)
        created = 0
        errors = 0

        for item in data:
            serializer = ZineWriteSerializer(data=item)
            if serializer.is_valid():
                serializer.save()
                created += 1
                self.stdout.write(self.style.SUCCESS(f'✓ Created: {item["title"]}'))
            else:
                errors += 1
                self.stdout.write(self.style.ERROR(f'✗ Failed: {item.get("zine_id", "unknown")}'))
                self.stdout.write(self.style.ERROR(f'  Errors: {serializer.errors}'))

        self.stdout.write(self.style.SUCCESS(f'\nImport complete: {created}/{total} created, {errors} errors'))

    def import_from_csv(self, file_path):
        with open(file_path, 'r') as f:
            reader = csv.DictReader(f)

            created = 0
            errors = 0

            for row in reader:
                # Transform CSV row to API format
                zine_data = {
                    'zine_id': row['zine_id'],
                    'title': row['title'],
                    'creator': row['creator'].split('|'),  # Split on pipe
                    'subject': row['subject'].split('|'),
                    'genre': row['genre'].split('|'),
                    'date': row['date'].split('|'),
                    'language': row['language'].split('|'),
                    'rights': row['rights'].split('|'),
                    'abstract': row.get('abstract', ''),
                }

                serializer = ZineWriteSerializer(data=zine_data)
                if serializer.is_valid():
                    serializer.save()
                    created += 1
                    self.stdout.write(self.style.SUCCESS(f'✓ Created: {row["title"]}'))
                else:
                    errors += 1
                    self.stdout.write(self.style.ERROR(f'✗ Failed: {row["zine_id"]}'))

            self.stdout.write(self.style.SUCCESS(f'\nImport complete: {created} created, {errors} errors'))

Step 3: Prepare CSV Data

zine_id,title,creator,subject,genre,date,language,rights,abstract
zine_mutate_3_1st,"Mutate Zine #3","agent_judith_arcana","feminism|reproductive-rights","personal-zine","2024","en","cc-by-4.0","Personal narratives about reproductive rights"
zine_feminist_killjoy_1,"Feminist Killjoy #1","agent_sara_ahmed","feminism","personal-zine","2024","en","cc-by-4.0","Reflections on feminist theory"

Array fields: Use | (pipe) as delimiter.

Step 4: Run Import

cd backend
DJANGO_SETTINGS_MODULE=zinecore.settings.development \
  ../.venv/bin/python manage.py import_zines ../data/zines.csv --format=csv

Output:

✓ Created: Mutate Zine #3
✓ Created: Feminist Killjoy #1

Import complete: 2/2 created, 0 errors

Pros:

Fast (direct database access)
Validates via serializers
Progress reporting
Error handling

Cons:

Requires custom code
Less flexible than API

Method 4: Direct SQL Import (Large Datasets)

For very large datasets (>10,000 records), use PostgreSQL's COPY command.

Step 1: Prepare CSV

Format: Match exact database schema.

id,zine_id,title,creator,subject,genre,date,language,rights,created_at,updated_at
1,zine_mutate_3_1st,"Mutate Zine #3","{agent_judith_arcana}","{feminism,reproductive-rights}","{personal-zine}","{2024}","{en}","{cc-by-4.0}","2024-02-25 10:30:00","2024-02-25 10:30:00"

Note: Array fields use PostgreSQL array syntax: {value1,value2}

Step 2: Import via COPY

psql -U zinecore2 -d zinecore2 -c "\COPY catalog_zine (id, zine_id, title, creator, subject, genre, date, language, rights, created_at, updated_at) FROM 'zines.csv' CSV HEADER;"

Step 3: Update Sequence

After bulk insert, update the ID sequence:

SELECT setval('catalog_zine_id_seq', (SELECT MAX(id) FROM catalog_zine));

Pros:

Extremely fast (millions of records per minute)
Minimal overhead

Cons:

No validation (can insert invalid data)
Requires exact schema knowledge
Bypasses Django ORM
PostgreSQL-specific

Import Order

Import records in this order to satisfy dependencies:

1. Vocabularies

DJANGO_SETTINGS_MODULE=zinecore.settings.development \
  python manage.py load_vocabularies

2. Agents

python manage.py import_agents data/agents.csv --format=csv

3. Repositories

python manage.py import_repositories data/repositories.csv --format=csv

4. Zines

python manage.py import_zines data/zines.csv --format=csv

5. Holdings

python manage.py import_holdings data/holdings.csv --format=csv

Why this order?

Zines reference Agents (via creator field)
Holdings reference Zines and Repositories (via foreign keys)

Data Preparation

Clean Your Data

Before importing:

✅ Validate required fields

All required fields must be present:

Zines: zine_id, title, creator, subject, genre, date, language, rights
Agents: agent_id, kind, display_name, public

✅ Check vocabulary codes

Ensure subject/genre/etc. codes match vocabulary:

# Get valid subject codes
curl http://localhost:8000/api/vocabularies/subjects/ | jq '.[].code'

✅ Verify agent IDs exist

Before importing zines, ensure referenced agents exist:

from agents.models import Agent
Agent.objects.filter(agent_id='agent_judith_arcana').exists()
# Should return True

✅ Remove duplicates

# Check for duplicate IDs in CSV
cut -d',' -f1 zines.csv | sort | uniq -d

✅ Encode special characters

Use UTF-8 encoding for all files.

CSV Format Tips

Quotes: Enclose fields with commas in quotes:

"Title with, comma","Normal field"

Newlines: Escape newlines in multi-line fields:

"Title","Abstract with\nmultiple lines"

Arrays: Use pipe delimiter:

"zine_001","agent1|agent2|agent3"

Handling Errors

Validation Errors

Error: "Agent 'agent_unknown' does not exist"

Solution:

Import agents first
Or create placeholder agent
Or fix reference in data file

Duplicate IDs

Error: "Zine with this zine_id already exists"

Solution:

Option 1: Update instead of create:

# In management command
zine = Zine.objects.filter(zine_id=data['zine_id']).first()
if zine:
    serializer = ZineWriteSerializer(zine, data=data)
else:
    serializer = ZineWriteSerializer(data=data)

if serializer.is_valid():
    serializer.save()

Option 2: Delete existing and re-import:

# Delete all zines (WARNING: destructive!)
python manage.py shell
>>> from catalog.models import Zine
>>> Zine.objects.all().delete()

Character Encoding

Error: "UnicodeDecodeError"

Solution: Ensure UTF-8 encoding:

with open(file_path, 'r', encoding='utf-8') as f:
    reader = csv.DictReader(f)

Progress Monitoring

Large Imports

For imports >1000 records, add progress reporting:

from tqdm import tqdm

for item in tqdm(data, desc="Importing zines"):
    # Import logic
    pass

Install tqdm:

pip install tqdm

Batch Processing

Process in batches to avoid memory issues:

def import_in_batches(data, batch_size=100):
    for i in range(0, len(data), batch_size):
        batch = data[i:i+batch_size]

        # Process batch
        for item in batch:
            # Import logic
            pass

        print(f"Processed {min(i+batch_size, len(data))}/{len(data)}")

Rollback Strategy

Before Import

Backup database:

pg_dump -U zinecore2 -d zinecore2 > backup-pre-import.sql

After Failed Import

Restore from backup:

# Drop database
psql -U postgres -c "DROP DATABASE zinecore2;"

# Recreate database
psql -U postgres -c "CREATE DATABASE zinecore2 OWNER zinecore2;"

# Restore backup
psql -U zinecore2 -d zinecore2 < backup-pre-import.sql

Or delete imported records:

# Delete zines created in last hour
from datetime import timedelta
from django.utils import timezone
from catalog.models import Zine

cutoff = timezone.now() - timedelta(hours=1)
Zine.objects.filter(created_at__gte=cutoff).delete()