Speed up media migration



  • Hi, I'm evaluating this plugin and realise that the migration is a bit slow (25-30 images/min). This will take a tremendous amount of time, and I was hoping that there would a be a possibility to manually upload the images to s3, having the plugin assume the image is in the remote bucket and only rewrite the postmeta entries to map to the s3 bucket of choice.

    This would also help for people who can't or don't want to install this plugin on the source server if migrating from a standalone server to a clustered setup (which is my case, actually running in AWS Fargate). Which means I have a backup of all files readily available and uploading a few GB of images takes merely minutes (compared to days for this plugin).



  • @Jimmy-Shimizu

    It'd be an interesting feature but it's more complicated than it appears. How do you deal with thumbnails and image sizes considering the generated filenames for those include the image dimensions? That would require Media Cloud to calculate the dimensions which, in turn, would require knowing what the dimensions of the master image are, which means having to download it. And then what if the thumbnails or image sizes are missing?

    One thing, with Amazon S3, is making sure transfer acceleration is on and that you are uploading to a bucket geographically close to you.

    Another thing you can do is run the importer from the command line in parallel using the paging option:

    wp mediacloud import --limit=1000 --page=1 &
    wp mediacloud import --limit=1000 --page=2 &
    wp mediacloud import --limit=1000 --page=3 &
    wp mediacloud import --limit=1000 --page=4 &
    

    I'd recommend using GNU Parallel instead of & though.

    I will look into adding the feature, though likely it would be added as a command line tool.


Log in to reply