Project

General

Profile

Actions
Copy link

Bug #1331

open

Images downloaded using dbemwiz are empty files

Added by Neil Voss over 13 years ago. Updated about 10 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
Start date:
06/14/2011
Due date:
% Done:

0%

Estimated time:
Affected Version:
Appion/Leginon 2.2.0
Show in known bugs:
No
Workaround:

get an amiguest scripps account and use rsync

Description

I was trying to download some images using dbemwiz. It is my first time using it and I was able to get it to download, but instead of downloading any images it was downloading blank files. Actually the files all contained the plain text:

string(2) "50"

I was trying with sessions, 11jan12b and 11jan06b. Amber/Anchi any ideas?

Files

dbemwizsrc1_1.tgz (56.1 KB) dbemwizsrc1_1.tgz Clint Potter, 06/17/2011 02:48 PM
Actions
Copy link
 #1

Updated by Amber Herold over 13 years ago

  • Status changed from New to Assigned
  • Assignee set to Amber Herold

Jim, Christopher and I have been unable to locate the source code for dbemwiz. Checked source control, and Denis and Eric's home directories. Do you have any idea where it might be?

Neil, we can put the images on a drive and send them to you if we don't get a solution today...probably easiest/fastest that way.

Actions
Copy link
 #2

Updated by Amber Herold over 13 years ago

Neil, there may be an issue with those data sets because they are on the external data drive /ami/exdata. You can try one of the other sets to see if they work for you. If you need those specific sets, we'll send them to you.

Actions
Copy link
 #3

Updated by Jim Pulokas over 13 years ago

I looked everywhere I could think of. All I found was an email dated May 15, 2010 from Denis where he says he is going to send us the source code. I do not have any follow-up email where the source code is actually sent, so maybe this only went directly to Eric or someone else. Can everyone check there old emails around that date to see if they have an attachment.

Actions
Copy link
 #4

Updated by Neil Voss over 13 years ago

Thanks for looking at this. Sorry, I did not try another set until now, but sets on the /ami/dataXX drives work fine, so it is an issue with /ami/exdata.

I know Eric made several updates to the dbemwiz code, so it is unfortunate that you could not find it in his home directory. But it does work now.

I will get back to you on Monday on how to get my data, I may just ship you a drive.

Thanks guys!

Actions
Copy link
 #5

Updated by Clint Potter over 13 years ago

Here is source code from Denis email of 5/17/10

From: "Denis F." <>
Date: May 17, 2010 12:39:56 AM PDT
To: Bridget Carragher <>,
Cc: Clint Potter <>, Jim Pulokas <>
Subject: Re: Downloader

Please find attached the java source code. testwiz describes how to compile and run dbem wizard. The idea of this wizard is to use php to feed/stream data via http requests; no need to share directories /ami/dataXX online via httpd.conf, since PHP can already access the file system. This was a prototype, no heavy usage; most users ended up bringing their own hard-drives. You might want to use a servlet to talk to the java client for heavy usage, but I found handy to use the public website for authentication, etc, since the users can already view their data online.

Hope this helps

-Denis.

The files you want to look into are:
ami/JgetURL.java
ami/DownloadTask.java
ami/LoginWizardPanel.java

The http request descriptions:

  1. get summary info, su.php:
    String link = "http://emg.nysbc.org/experiment/su.php?exp="+expId;
  1. get a list of filenames, list.php:
    String link = "http://emg.nysbc.org/experiment/list.php?exp="+expId;
  1. get a list of filenames queried on preset, lq.php:
    String link = "http://emg.nysbc.org/experiment/lq.php?exp="+expId+querystr;
  1. get a list of experiments, le.php:
    URL urlexp = new URL;
  1. download data, dl.php:
    in = jg.download(new URL);
  1. login, lg.php:
    URL urlexp = new URL;
Actions
Copy link
 #6

Updated by Amber Herold over 13 years ago

Thanks Clint!

Neil,
Christopher mounted exdata on the web server, so go ahead and give those datasets another try!

Actions
Copy link
 #7

Updated by Neil Voss over 13 years ago

So, I downloaded 124GB over the weekend and all of them are coming up in EMAN v2 as invalid files. After investigating further I found using a hex editor that all the files started with string(2) "50"\n then followed by the MRC file.

Using the command:

tail -c +16 11jan06b_00020gr_00019sq_00012hl_v01_00002en2.mrc > edit.mrc

should remove the header and make it a normal mrc file and it worked, but what a pain. Using Jim's hack from my Yahoo! email (not posted here for security reasons) it works fine with out hack, so there is a problem in dbemwiz.

System: Scientific Linux 6, Java 1.6, x86_64

Actions
Copy link
 #8

Updated by Neil Voss over 13 years ago

Another bug is that the last four pixels at the end of the image file were not downloaded, so I had to edit them and included four blank pixels.

Nonetheless, I downloaded all 3925 images this way and despite the removal of the 'string(2) "50"\n' at the beginning and the missing pixels all the data is fine. But it is probably time to kill dbemwiz and come up with a new solution.

Actions
Copy link
 #9

Updated by Amber Herold about 10 years ago

  • Status changed from Assigned to New
  • Assignee deleted (Amber Herold)
  • Affected Version changed from Appion/Leginon 2.2.0 (trunk) to Appion/Leginon 2.2.0
Actions
Copy link

Also available in: Atom PDF