Project

General

Profile

Forums » Appion and Leginon Web tools »

Appion image display delays?

Added by Patrick Goetz almost 6 years ago

Users have complained that it often takes 5-20 minutes to open a project in the Appion web interface and display the first image. I observed this myself today, with about a 5-8 minute delay from the time the project was clicked until the image loaded. I ran top to see if it provided any clues to what was going on. Someone in the lab was collecting images at the same time (could this be the cause?), so the top processes fluctuated between:

mysqld
reduxd
start-leginon.py
python

(roughly speaking). Could this delay be an issue with the mysql server? We are running everything on a single machine, but it is a very beefy machine, with dual Intel Xeon 6-core E5-2667 processors running at 3.5Ghz, 256G RAM, 4 Nvidia GTX1080 cards, and OS + database running from RAIDed SSD disks.

I should add that the machine load was quite light at the time, too; about 1.8 average processor load on a machine with 12 physical cores.


Replies (29)

RE: Appion image display delays? - Added by Patrick Goetz almost 6 years ago

@Anchi Cheng: You're right! Now I feel brain-damaged, as this doesn't make sense at all. Since I was attempting to figure out the source of the bottleneck, I thought I was running

mysql> SHOW full processlist\G

prior to each and every SQL query, but perhaps I forgot to check the one time the result took 20 minutes to deliver. Anyway, I just tried both loading the problematic session through the web interface and re-running the query, and both are now fast.

RE: Appion image display delays? - Added by Patrick Goetz almost 6 years ago

I just reverted the change to myamiweb/imageviewer.php, and uncommented $ptcl = (require_once "inc/particledata.inc") ? true : false;. Both the sessions which were known to consistently load slowly are still loading quickly now. Arrrgh. I'm back to having no idea what is going on. What I'm fairly certain of is that the slowness of the reference SQL query is what was causing the related session to load slowly in the web interface. In particular, I could consistently click on a slow loading session and then check

mysql> show full processlist\G

and the afore mentioned SQL command was always the only thing showing up in the query process list.

Of course I have no idea why it was loading slowly before and now suddenly isn't.

RE: Appion image display delays? - Added by Sargis Dallakyan almost 6 years ago

Since you increased query_cache_size it might be caching this query.

RE: Appion image display delays? - Added by Patrick Goetz almost 6 years ago

@Sargis: But I didn't:

mysql> SHOW VARIABLES LIKE '%query_cache%';
+------------------------------+-----------+
| Variable_name                | Value     |
+------------------------------+-----------+
| have_query_cache             | YES       |
| query_cache_limit            | 104857600 |
| query_cache_min_res_unit     | 4096      |
| query_cache_size             | 104857600 |
| query_cache_type             | ON        |
| query_cache_wlock_invalidate | OFF       |
+------------------------------+-----------+
6 rows in set (0.02 sec)

100MB is the default. Also, there's this: https://haydenjames.io/mysql-query-cache-size-performance/

(26-29/29)