Announcement Regarding Support

Hello everyone, Charles here. Please read my post on changes to Media Cloud support and updates regarding Media Cloud in general.

Thanks!

Uploaded images duplicating and not being deleted



  • I noticed the size of my wp-content/uploads directory getting bigger and was puzzled to find that many of the folders still contain images that should have been uploaded and deleted!

    Now, the odd thing is that the WP media library images actually show a cloud icon and the URL is in fact pointing to our cloud storage solution which is as expected, however, the concern being why there are additional directories being created with the same images uploaded and not deleted.

    I can confirm that the media cloud settings are configured to perform the upload and delete the original file and this previously worked fine. I am a little suspicious since these new "Big size threshold" and "Upload Original" feature were included in the plugin and I'm a little uncertain whether that has anything to do with this issue.

    I have included the attached screenshot of the settings that I have enabled just as an FYI.

    Really need some help understanding why this is happening!

    Many thanks in advance.

    Screen Shot 2020-01-27 at 1.41.24 pm.png



  • One note to add, I use the @versioning parameter to create unique folder names for uploaded images. Testing the upload of a recent image, I can see that the image was originally created in a folder named wp-content/uploads/27140233 and the image files were uploaded to cloud storage and no longer exist in this folder. However, there was also another folder created named 27140232 (appearing to be 1 second earlier) and the image file including all resized variants EXIST within this folder.

    There's something not quote right happening here and it appears as though files are being duplicated and saved in different versioning folders. Dev, can you please help take a look on your end and validate whether this is a bug? This will eventually blow our HDD capacity (thankfully now discovered) and clearly incorrect behaviour with how the plugin should work.

    Please advise.



  • Can I get some help here please, this appears to be a bug.



  • Can the plugin author please reply with something, anything. This is a major problem for us and it looks like it’s effecting others too.



  • @Dcipher said in Uploaded images duplicating and not being deleted:

    have included the attached screenshot of the settings that I have enabled just as an FYI.
    Really need some help understanding why this is happening!
    Many thanks in advance.

    Hey @Dcipher-
    We are experiencing this issue as well.



  • @wholesalesolar Good to know, this is clearly a major problem.

    @jong are you able to please respond?



  • It looks like he has been MIA for at least a week now.



  • @columbusgeek Not good...



  • It's plausible he/she might be tangled up in the coronavirus lockdown.



  • Thanks @charles for the update.

    We get that paid plugin support gets priority but this particular case is urgent and affecting the masses. We need this issue looked into and resolved as a matter of priority please.

    Awaiting an update on:

    1. a fix;
    2. how do we now remove the duplicated images that have bogged down our local disk...


  • We've tried to replicate this issue in a variety of ways, and the only way we could replicate it was:

    • Queue deletes is ENABLED in Cloud Storage Settings
    • WP-Cron is misconfigured, ie. it is disabled in WordPress but a crontab on your server is not making WP-Cron run.

    For WP-Cron, we recommend:

    • Disable WP-Cron via define('DISABLE_WP_CRON', true); in your functions.php
    • Setup a cron job on your server that looks something like: */1 * * * * curl https://YOURSERVER/wp/wp-cron.php?doing_wp_cron > /dev/null 2>&1

    However, the easiest answer is to just disable Queued Deletes in Cloud Storage Settings. If you have a plugin that depends on the files being local for a short amount of time after an upload to do whatever processing, your only option is to make sure that you have wp-cron setup right.

    Finally, the "Clean Uploads" task deletes uploaded media from your local system, but only media associated with items in your media library.

    Hope this helps!

    Thank You,

    Charles



  • I'd like to add, if doing the above doesn't work, then your problem is likely outside of Media Cloud. Maybe a permissions issue, maybe the unlink() function is unavailable due to security, or you're running on Windows (why?).

    Because we can't replicate it, the only option for us to investigate further is to look at your server, which is something we are happy to do, but only for Pro and Agency customers.

    Thank you,

    Charles



  • Hi @charles,

    Many thanks for your detailed reply. I can confirm the following:

    1. Queue deletes is enabled (for no specific reason)
    2. Wp-Cron has been disabled via wp-config.php and a system cron job was setup (done within Plesk) using the following syntax: wget -q -O - https://mydomain/wp-cron.php?doing_wp_cron */15 * * * *

    Testing the cron job appears to work without producing any errors so I don't believe that it's a cron related issue.

    Finally, based on your comment regarding the Clean Uploads feature, I don't believe that this will help considering the duplicate files don't appear within the media library. Short of having to go through each folder manually (up to 25GB's worth now), is there another way to remove those duplicate files? Once again, the versioning folder for each set of duplicates is versioned 1 second prior to the set that appears in the media library.

    Many thanks.



  • @Dcipher

    Well let's figure out why they are happening in the first place.

    Does this happen every time you upload to the media library?

    Are you using some kind of import plugin?

    Thank you,

    Charles



  • @Dcipher
    So, I've been playing around with settings related to another odd issue... and may have found something here.

    1. We have "Organize my uploads into month- and year-based folders" setting enabled. (/wp/wp-admin/options-media.php)
    2. We have a custom Upload Path set on the cloud storage settings set to "media". (/wp/wp-admin/admin.php?page=media-cloud-settings-storage)

    In this scenario - we would see folders created like 2020/02 (but no files in them), and the uploaded file would end up in the "media" folder - and stay there. Even when Queued Deletes was off.

    Just now, I disabled the " "Organize my uploads into month- and year-based folders"" setting, uploaded a file, and the file was removed from the media folder after upload and no extraneous directories were created.

    !! EDIT !!
    I dunno - I'm still seeing files lingering around...



  • Hey @charles ,

    In response to your earlier post, this absolutely happens on every media upload (was never an issue with earlier versions of Media Cloud). Furthermore, we're not using any additional importer style plugins. Again, earlier versions weren't an issue, it would have started since one of the last few MC updates released.

    Regarding your later post, so we do have that option enabled however I believe this gets overridden by the Media Cloud 'Upload Path' setting, which in our case is configured to use @{versioning} (see screenshot).

    So again, with the above configuration, an image would be uploaded, WP does its thing where it creates various sizes of it, uploads them to a folder such as /11021234 (which appear in the media library AND get removed from this folder) THEN upon inspection, there's another folder named 11021233 (once second earlier) which has all of those images within it and these are nowhere to be seen inside the WP media library.

    (head explosion)

    Screen Shot 2020-02-11 at 8.36.48 am.png



  • I would agree that this started happening "at some point" - not sure when.



  • Just confirming that even after disabling the "Queue deletes" option, the same issue persists.



  • Ok, just to add, I've also just confirmed that the duplicated images also get uploaded to S3!

    As per my earlier example, both /11021234 and /11021233 folders and their contents get uploaded to S3, with the former being handled the way it should be, and the latter keeping a copy on the local server.

    In short, that second folder should not exist, period. So in addition to duplicates being left behind on the local server, we're also uploading duplicates to S3, obviously adding to storage charges.

    Very sinister issue.



  • @Dcipher

    Can you try downgrading to a previous version of Media Cloud to see if it still happens?

    Can you email Charles the system report from the debug log in Media Cloud? [email protected]


Log in to reply