Bug #7812
openDownload particle data takes too much memory
0%
Description
Downloading particle data as csv file takes about 30X as much memory as the final file size. For example, 1 GB RAM was not enough for a 527,000 particle download.
Made changes to downloadparticledata.php to use standard php functions to create csv file.
Added inc/datatocsv.inc which includes this function.
Downloading particle data now can be done with standard 1 GB RAM allocation to php webserver.
One consequence is that I eliminated the special tratment for "ef" presets. Likely this is not used anymore.
Other csv downloads could be adjusted to use this file, but they are not so large so perhaps it is not worth doing,
Updated by Anchi Cheng about 5 years ago
- Status changed from New to In Code Review
- Assignee changed from William Rice to Gabriel Lander
- Target version set to Appion/Leginon 3.5
Gabe,
I think this change impacts your helical picker. The change means all fields are turned into csv. Please check if this will work for you or we need to make it a separate option. We could also handle this in the query side if needed.
Updated by Gabriel Lander about 5 years ago
Does anyone use the helical picker? I don't think anyone does at this end.
Updated by Anchi Cheng about 5 years ago
Last person asked about this was Rui Zhang