Ilab consuming lot of cpu time according to new relic



  • The Ilab media plugin is the second most time consuming plugin, right above elementor. Anything we can do about this?

    Screenshot_091019_084113_PM.jpg



  • I mean what does that metric mean? Front end or admin? Are you doing uploads or an import?

    Uploads are pretty CPU intensive. Migrations are pretty CPU intensive too.

    If it's just front end that's causing that spike, that'd be news to me because the only thing Media Cloud does on the front end is some simple post content filtering and you can mitigate that with caching.



  • If you have Assets enabled, that can be pretty intensive too the first time the asset cache is primed.

    If you are using Dynamic Images, that's super CPU intensive the first time it needs to generate an image.

    Again, both mitigated by caching and CDN usage.



  • I've installed newrelic on a couple of sites I have control of and will report back in a few days.

    I can already see on https://mediacloud.press (a site we don't do uploads on, that is all done on staging) that it bounces between 130 to 203ms, which is way acceptable imho:

    Screenshot of Safari (9-11-19, 7-17-31 PM).png

    That number will jack up, of course, if you are doing uploads or migrations or any other batch processing, but that's to be expected. Also, in WordPress admin the plugin is doing a lot of things that it doesn't do on the front end in terms of initialization, etc.

    The https://mediacloud.press/ test will be a good one for just front end performance because we rarely use the admin on the production site, we do that all on staging or local dev.

    I will update in a few days though.



  • So in the last 24 hours, these are the results:

    Screenshot of Safari (9-12-19, 3-33-33 PM).png

    It's very important to note that the time value (in this case 5.23 sec) is the cumulative time over the given period. So over 24 hours, media cloud took 5.23 seconds to do it's thing. It's not 5.23 seconds per request, it's 5.23 seconds for all of the requests in the last 24 hours.

    I'm not sure what time period your image is because it's obscured.

    Also new relic for sure is including any admin tasks, such as uploads, migrations, etc.



  • @jong said in Ilab consuming lot of cpu time according to new relic:

    It's very important to note that the time value (in this case 5.23 sec) is the cumulative time over the given period. So over 24 hours, media cloud took 5.23 seconds to do it's thing. It's not 5.23 seconds per request, it's 5.23 seconds for all of the requests in the last 24 hours.

    Hey @jong, thanks for the reply and sorry for the late response. The team have been debugging a client site for a bit and have been looking at other metrics. You're right, the metric I looked at was cumulative and probably not the only one I should be looking at. Based on known client usage, they upload maybe 8 images per post at a time - I don't know how much processing goes into that.

    However, when I look at the Function Call Count within the past 30 minutes, I'm getting a large number. In comparison, Yoast is doing around the same, but I'm trying to understand if this metric helps debug further or if it is normal.

    Screenshot_091719_075440_PM.jpg

    I'm obviously going to turn off Simple History and Project Huddle

    I'm including a benchmark of the "most time consuming" as well - not sure if that helps or proves media cloud is totally fine:

    Screenshot_091719_075629_PM.jpg



  • @jong bumping this post to see if you had any opinions about it



  • @jong

    Some more logs

    Screenshot_100219_063920_PM.jpg



  • @Tanner-Chung

    I'm not sure what you want me to say? 🙂

    Media Cloud is as complicated as, if not more so, than Yoast. So that 1.28m number doesn't really mean that much to me.

    I have been doing some work on reducing when some of the database calls on page loads but that's not really about performance as much as it is about being a good neighbor.



  • I guess my objective is to debug what it is crashing the site. I can only assume what is crashing the site is initiating a lot of CPU execution because it happens when the processor is jumping in utilization. So naturally, I'm looking at the plugins and functions that have the highest count.

    If you're confirming that Media Cloud is not the issue even with this function call count then I suppose I have nothing to worry about, but what I'm hoping is if you can share insight as to what it might be calling and if those calls are all necessary or if something is causing those spikes artificially.



  • This post is deleted!


  • @Tanner-Chung

    Your site is crashing? Do you have specifics on that, like the web server is crashing, PHP-FPM is constantly restarting, the actual server needs to be rebooted, etc?

    I was doing some support with a fellow who was having the admin slow to a crawl when Media Cloud was enabled. I replicated his environment almost exactly software wise but wasn't able to replicate the crawl. It was the damndest thing. But it ended up being a hardware issue.

    I profile using https://blackfire.io/ and I'd wager the bulk of those calls aren't anything Media Cloud is doing specifically, but are things like autoload or WordPress functions that do a bazillion different things.

    Not sure about the max execution time stuff or that it's related to Media Cloud as the version of Media Cloud you are using (if it's < 3.1.8) uses Guzzle and not the WordPress HTTP library.

    But I'd be interested in more information about the crashing.



  • I don't believe the server is fully crashing, but the CPU usage spikes, and New Relic shows large amounts of function calls. When that happens, the site itself is impossible to reach. Typically, the cache will take care of site visits since it's on the Nginx level even if WordPress is unavailable, but when it gets bad, the server can't even serve the cache.

    To try and combat it, I left PHP-FPM static with maxed workers with some memory to spare, but yes, I have to reboot the entire server. There are times where I can't reach a computer fast enough after the client reports a crash and if I wait long enough, PHP-FPM is able to finally finish whatever the heck it's doing and the site is back up. Today was an example, they reported a crash at 2:05. I took a look around 2:30, and it was back up but sometimes it is longer than that.

    I did manage to read that thread in the forum but didn't know if it was related. Would be great to just finally figure out what the heck is going on. What I could do is disable Media Cloud and see if they experience this issue any furuther?

    I don't know what else to provide in terms of the crash, I can't definitely say it's media cloud. The wordpress logs showed that WP Rocket's cache preloading would trigger whenever the client deleted or updated tags. They would delete maybe 30, then the preload would go nuts - the CPU would spike etc. Then I disabled preload thinking I finally figured it out, but then it still crashed today.

    I'm reducing the plugins as much as I can one by one, but in the meantime, the high function calls/average function times are the only thing I can go by. What other items do you suggest?



  • Does this help @jong

    Screenshot_100719_055915_PM.jpg

    Screenshot_100719_055757_PM.jpg

    Screenshot_100719_055833_PM.jpg

    Screenshot_100719_055842_PM.jpg

    Looks like it has something to do with the final cdn call?
    and Any idea what CurlMultiHandler does? It seems to appear in every slow transaction



  • @Tanner-Chung

    Curl is "sleeping" until it gets a response. So if it's taking a long time, it means your connectivity with DO is bad or they are having issues.

    Though it seems the chain is unnecessary because DO doesn't have regions, so I'm not sure why it's trying to figure out what it is.

    I'll look into it later this week as I'm swamped with client work and a support back log that's frankly pretty depressing.



  • What's XML RPC doing in a transaction call with digital ocean?
    Screenshot_100919_011533_AM.jpg


Log in to reply