How to Migrate Large Data in WordPress Without Blocking Requests

The naive approach—looping through thousands of records in a single request—is a guaranteed failure at scale. The real solution is not “making it faster”, but breaking it into small, asynchronous steps. This article presents a simple yet highly effective pattern: self-rescheduling cron-based migration that runs in the background, avoids request blocking, and scales safely.

The Core Idea

Instead of processing everything at once, split the workload into small batches. Each run:

• Processes a limited number of records (e.g. 100–200)
• Checks if more data remains
• If yes → schedules the next run

This loop continues until the migration is complete. Each execution is lightweight, so it does not impact user-facing requests.

Why Not Run Migration in admin_init?

At first glance, hooking into admin_init might seem convenient. In reality, it is a trap.

admin_init runs on every admin request: dashboard loads, AJAX calls, even background heartbeats. If your migration takes several seconds, every admin page load becomes blocked.

Worse, when users refresh the page, multiple migration processes can overlap, leading to CPU spikes, database contention, and system instability.

This leads to a key realization:

Heavy tasks must not run inside the request lifecycle.

Implementing a Background Migration with WP-Cron

<?php
if ( ! defined( 'ABSPATH' ) ) exit;

add_action( 'init_html_migration_event', 'init_html_migration_runner' );

function init_html_migration_runner() {
    // Prevent concurrent execution
    if ( get_transient( 'init_html_migration_lock' ) ) return;
    set_transient( 'init_html_migration_lock', 1, 60 );

    $has_more = init_html_maybe_migrate_batch();

    if ( $has_more ) {
        // Schedule next batch after 30 seconds
        wp_schedule_single_event( time() + 30, 'init_html_migration_event' );
    }

    delete_transient( 'init_html_migration_lock' );
}

Triggering the First Run

<?php
register_activation_hook( __FILE__, function () {
    if ( ! wp_next_scheduled( 'init_html_migration_event' ) ) {
        wp_schedule_single_event( time() + 30, 'init_html_migration_event' );
    }
});

Example: Migrating user_meta to a Custom Table

Assume you store structured data in user_meta using a key pattern like:

_init_html_data_{post_id}_{device}

Now you want to migrate it into a dedicated table for better performance and query flexibility.

Batch Processing Function

<?php function init_html_maybe_migrate_batch() { global $wpdb; $meta_pattern = '_init_html_data_%'; $user_ids = $wpdb->get_col(
        $wpdb->prepare(
            "SELECT DISTINCT user_id FROM {$wpdb->usermeta}
             WHERE meta_key LIKE %s
             ORDER BY user_id ASC
             LIMIT 200",
            $meta_pattern
        )
    );

    if ( empty( $user_ids ) ) {
        update_option( 'init_html_migration_done', 1 );
        return false;
    }

    foreach ( $user_ids as $user_id ) {
        init_html_migrate_user( (int) $user_id );
    }

    return true;
}

Migrating Per User

<?php function init_html_migrate_user( $user_id ) { global $wpdb; $rows = $wpdb->get_results(
        $wpdb->prepare(
            "SELECT meta_key, meta_value FROM {$wpdb->usermeta}
             WHERE user_id = %d AND meta_key LIKE %s",
            $user_id,
            '_init_html_data_%'
        ),
        ARRAY_A
    );

    if ( empty( $rows ) ) return;

    foreach ( $rows as $row ) {
        $meta_key   = $row['meta_key'];
        $meta_value = maybe_unserialize( $row['meta_value'] );

        if ( ! is_array( $meta_value ) ) {
            delete_user_meta( $user_id, $meta_key );
            continue;
        }

        if ( ! preg_match( '/^_init_html_data_(\d+)_(.+)$/', $meta_key, $m ) ) {
            delete_user_meta( $user_id, $meta_key );
            continue;
        }

        $post_id = (int) $m[1];
        $device  = sanitize_key( $m[2]);

        $value = sanitize_text_field( $meta_value['value'] ?? '' );
        $updated_at = current_time( 'mysql', true );

        init_html_upsert( $user_id, $post_id, $device, $value, $updated_at );

        delete_user_meta( $user_id, $meta_key );
    }
}

Upsert into Custom Table

<?php function init_html_upsert( $user_id, $post_id, $device, $value, $updated_at ) { global $wpdb; $table = $wpdb->prefix . 'init_html_data';

    $wpdb->query(
        $wpdb->prepare(
            "INSERT INTO $table (user_id, post_id, device, value, updated_at)
             VALUES (%d, %d, %s, %s, %s)
             ON DUPLICATE KEY UPDATE
               value = VALUES(value),
               updated_at = VALUES(updated_at)",
            $user_id,
            $post_id,
            $device,
            $value,
            $updated_at
        )
    );
}

Key Design Principles

• Avoid OFFSET when data is being mutated
• Delete old data after migration to ensure idempotency
• Keep batch size small and predictable
• Always guard against concurrent execution

Why This Approach Works

• Non-blocking: runs outside user requests
• Safe: avoids timeouts and memory spikes
• Resumable: continues automatically until completion
• Simple: no external queue or worker required

Conclusion

WordPress does not provide a true background job system out of the box. But with a small amount of engineering, you can build a reliable migration pipeline using nothing more than WP-Cron and disciplined batching.

Instead of forcing everything into a single request, let the system work incrementally. It is slower per step—but infinitely more stable in production.