How to migrate 2.6 million images



  • I have a wordpress install (on WPEngine) that has 2.6 million image files using 270GB in an S3 bucket. What's the best way to get these into MediaCloud? Note that the actual number of images is probably about 100-150k, which is all I need, and the rest are stupid wordpress thumbnails that I am hoping I won't need any longer now that I'm using MediaCloud.



  • Holy sh-

    The best method would be to use the command line if you have SSH access. There is a wp-cli command that ships with media cloud:

    wp mediacloud import
    

    But I would break it up into relatively small batches:

    wp mediacloud import --limit=5000 --page=1
    

    You would increment the page to do the next 5000 items.

    If you don't have SSH access, you'll have to use the Migrate to Cloud batch tool:

    Media Cloud Tools ‹ Media Cloud — WordPress 2019-07-20 13-22-24.png



  • Is there any feasible way to delete all the useless thumbnails and only migrate the original image? My goal is to only migrate the originals and then use imgix to supply all the required sizes on the fly.



  • If you give me a day, I'll push a fix that will allow you to toggle on an option to skip uploading thumbnail sizes.

    You will have to use Imgix or Dynamic Images though, otherwise you will get a lot of broken images on your site because those thumbnails and other image sizes will need to exist somehow. If you using Imgix or Dynamic Images they are generated on the fly.



  • That would be amazing! Thank you!



  • @jasondiff

    Ok so 2 days 😉

    3.1.0 has this feature. You can read more about it here:

    https://talk.mediacloud.press/topic/40/3-1-0-release-notes


Log in to reply