transferring a LOT of files



  • I have a website I've been struggling to manage; I have almost exactly one terabyte of images on my wordpress site (I KNOW! It's 12 years of extensive archives that we'd like to keep online).

    If I install Media Cloud and sign up for cloud storage, am I correct in assuming it will just transfer them over automatically? I assume it will take a while. Can I then delete them from my current server? Does it do that automatically as well?

    From my understanding it seems to be the case, I just want to verify! It's a lot of data to deal with so I want to make sure that's correct.

    Thanks!

    (sorry, I had this posted in "technical questions" but it should have been here!)



  • I should note that these are all in NextGen Gallery.



  • @gigi

    Unfortunately, the way the NextGen Gallery integration works is that images are dynamically uploaded when a gallery is first viewed. It basically works like this:

    • Install Media Cloud and get working
    • When a user or you views a nextgen gallery for the first time, Media Cloud looks up the cached url for the image
    • If a cached url for the image doesn't exist, then Media Cloud will upload the image and cache the url

    Because NextGen, for whatever inexplicable reason, doesn't manage their media through WordPress, there are no tools in Media Cloud to do a bulk transfer.

    Do you have database access? Can you run this query and tell me how many actual images it is?

    select count(pid) from wp_ngg_pictures

    Thanks,

    Jon.



  • 453,111!

    I could just go visit each post and it would move them over? We have about 2500 galleries total.

    That's not the worst as it's a lot but not too difficult, if I'm understanding correctly.



  • @gigi

    Holy crap that's a lot.

    Listen, why don't you give me a week to come up with a better solution. My fear is that with that many images, the current solution would actually end up being a hinderance.

    Do you have shell access and is WP-CLI installed?



  • @jong Ha! This is why I've been super excited about Media Cloud maybe being a solution for me to move them all over to be served from the cloud and not from my struggling little VPS (which isn't so little, but the photos are a lot).

    It's been not great for a while so if the PROCESS takes a while I'm ok with that, but if you mean it wouldn't work well once they're all moved to cloud storage?

    I'm on dreamhost and I do have shell access and it appears that WP-CLI is installed, according to their documentation.



  • @gigi

    We've just released the new premium version that has very extensive support for NGG.

    What you'll want to do is:

    1. Backup your database
    2. In Media Cloud, go to Settings and then select "Integrations" from the pull down in the very upper right of the screen
    3. Scrolll down to the NGG section
    4. Make sure that Delete Uploaded Files is turned OFF
    5. Click Save

    After you've done that, in the admin will be an entry called Migrate NextGen Galleries. Running this tool will migrate your existing galleries to cloud storage.

    However, due to the size of your library, I'd urge you to do it from the command line instead.

    To do that, you'll need to SSH into your server and change to your wordpress directory. Then you'll type:

    wp mediacloud migrateNGG
    

    And then have a coffee or fifteen because it's going to take awhile.

    After that you should be good to go.

    Once it's done, you should never have to do that again as Media Cloud will automatically upload for you. Also, once you've verified everything is working, you can delete the gallery images from your server. But always verify first.



  • !!! Amazing !!!

    I'm making sure everything is backed up (this might take a while) and have set up a cloud storage account via google; I'll update you when I start to run it! I want to now but I'll wait for all the backups to finish, I guess.



  • Screenshot 2019-11-17 10.14.00 copy.jpg

    Ok, backups are finished and I'm trying it, but I'm getting an error with jetpack, it seems?

    1. there was an error with that module so I tried deleting it via ftp, then it said it was missing that module so I put it back

    2. we had site accelerator and lazy loading for images turned on in jetpack so I tried turning that off, and it just gave me different errors.

    Site backups are running through jetpack so I hesitate to uninstall but maybe I should do that temporarily?



  • (ok I'm both upgrading php from 7.2 to 7.3 as recommended by the mediacloud testing and also trying to increase the php memory limit, will update when I do that)



  • Ok. It's working now!

    I de-activated any caching, for the record from jetpack, yoast seo, and w3 total cache (we've tried all sorts of caching to help this struggling overloaded site).

    It is indeed going slowly, but that's ok. Uh, it does estimate 7 months as a finish time.

    My connection in terminal was "closed by the remote host" but it appears to still be slowly going in the task manager of mediacloud? So I assume it will just keep moving things?

    Thanks so much for your help.



  • also for the record, the task manager seems to be frozen and not updating, but it still says it's running and when I head to the media browser it looks like things are actually moving over fairly quickly.


Log in to reply