Bug #1331
openImages downloaded using dbemwiz are empty files
Added by Neil Voss over 13 years ago. Updated over 10 years ago.
0%
get an amiguest scripps account and use rsync
Description
I was trying to download some images using dbemwiz. It is my first time using it and I was able to get it to download, but instead of downloading any images it was downloading blank files. Actually the files all contained the plain text:
string(2) "50"
I was trying with sessions, 11jan12b and 11jan06b. Amber/Anchi any ideas?
Files
| dbemwizsrc1_1.tgz (56.1 KB) dbemwizsrc1_1.tgz | Clint Potter, 06/17/2011 02:48 PM |
Updated by Amber Herold over 13 years ago
- Status changed from New to Assigned
- Assignee set to Amber Herold
Jim, Christopher and I have been unable to locate the source code for dbemwiz. Checked source control, and Denis and Eric's home directories. Do you have any idea where it might be?
Neil, we can put the images on a drive and send them to you if we don't get a solution today...probably easiest/fastest that way.
Updated by Amber Herold over 13 years ago
Neil, there may be an issue with those data sets because they are on the external data drive /ami/exdata. You can try one of the other sets to see if they work for you. If you need those specific sets, we'll send them to you.
Updated by Jim Pulokas over 13 years ago
I looked everywhere I could think of. All I found was an email dated May 15, 2010 from Denis where he says he is going to send us the source code. I do not have any follow-up email where the source code is actually sent, so maybe this only went directly to Eric or someone else. Can everyone check there old emails around that date to see if they have an attachment.
Updated by Neil Voss over 13 years ago
Thanks for looking at this. Sorry, I did not try another set until now, but sets on the /ami/dataXX drives work fine, so it is an issue with /ami/exdata.
I know Eric made several updates to the dbemwiz code, so it is unfortunate that you could not find it in his home directory. But it does work now.
I will get back to you on Monday on how to get my data, I may just ship you a drive.
Thanks guys!
Updated by Clint Potter over 13 years ago
- File dbemwizsrc1_1.tgz dbemwizsrc1_1.tgz added
Here is source code from Denis email of 5/17/10
From: "Denis F." <denis@nanoimagingservices.com>
Date: May 17, 2010 12:39:56 AM PDT
To: Bridget Carragher <bcarr@scripps.edu>, erichou@scripps.edu
Cc: Clint Potter <cpotter@nanoimagingservices.com>, Jim Pulokas <pulokas@nanoimagingservices.com>
Subject: Re: Downloader
Please find attached the java source code. testwiz describes how to compile and run dbem wizard. The idea of this wizard is to use php to feed/stream data via http requests; no need to share directories /ami/dataXX online via httpd.conf, since PHP can already access the file system. This was a prototype, no heavy usage; most users ended up bringing their own hard-drives. You might want to use a servlet to talk to the java client for heavy usage, but I found handy to use the public website for authentication, etc, since the users can already view their data online.
Hope this helps
-Denis.
The files you want to look into are:
ami/JgetURL.java
ami/DownloadTask.java
ami/LoginWizardPanel.java
The http request descriptions:
- get summary info, su.php:
String link = "http://emg.nysbc.org/experiment/su.php?exp="+expId;
- get a list of filenames, list.php:
String link = "http://emg.nysbc.org/experiment/list.php?exp="+expId;
- get a list of filenames queried on preset, lq.php:
String link = "http://emg.nysbc.org/experiment/lq.php?exp="+expId+querystr;
- get a list of experiments, le.php:
URL urlexp = new URL;
- download data, dl.php:
in = jg.download(new URL);
- login, lg.php:
URL urlexp = new URL;
Updated by Amber Herold over 13 years ago
Thanks Clint!
Neil,
Christopher mounted exdata on the web server, so go ahead and give those datasets another try!
Updated by Neil Voss over 13 years ago
So, I downloaded 124GB over the weekend and all of them are coming up in EMAN v2 as invalid files. After investigating further I found using a hex editor that all the files started with string(2) "50"\n then followed by the MRC file.
Using the command:
tail -c +16 11jan06b_00020gr_00019sq_00012hl_v01_00002en2.mrc > edit.mrc
should remove the header and make it a normal mrc file and it worked, but what a pain. Using Jim's hack from my Yahoo! email (not posted here for security reasons) it works fine with out hack, so there is a problem in dbemwiz.
System: Scientific Linux 6, Java 1.6, x86_64
Updated by Neil Voss over 13 years ago
Another bug is that the last four pixels at the end of the image file were not downloaded, so I had to edit them and included four blank pixels.
Nonetheless, I downloaded all 3925 images this way and despite the removal of the 'string(2) "50"\n' at the beginning and the missing pixels all the data is fine. But it is probably time to kill dbemwiz and come up with a new solution.
Updated by Amber Herold over 10 years ago
- Status changed from Assigned to New
- Assignee deleted (
Amber Herold) - Affected Version changed from Appion/Leginon 2.2.0 (trunk) to Appion/Leginon 2.2.0