Project

General

Profile

Actions

Feature #1128

closed

import/export/copy data between two databases

Added by Jim Pulokas almost 14 years ago. Updated about 11 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Sinedon
Target version:
-
Start date:
01/10/2011
Due date:
% Done:

0%

Estimated time:
Deliverable:

Description

Need ability to export from one database and import into another. This should recursively preserve all associated references between objects and preserve timestamps. For example, assuming some global DB config was done in sinedon.cfg:

import sinedon
import sinedon.transfer   # a hypothetical module for doing the transfer
import leginon.leginondata

# export 10 most recent images from first database
sinedon.setConfig('leginondata', db='leginondb1')
imagequery = leginon.leginondata.AcquisitionImageData()
images = imagequery.query(results=10)

# import them into the second database
sinedon.setConfig('leginondata', db='leginondb2')
siedon.transfer.transfer(images)

The result would be that leginondb2 will now contain copies of the image data plus any references to SessionData (create new unique session), AcquisitionImageTargetData, etc...


Related issues 1 (0 open1 closed)

Blocked by Leginon - Bug #1173: appion can not find project for experiment using branch myami-dbcopyWon't Fix or Won't Do Anchi Cheng02/05/2011

Actions
Actions #1

Updated by Jim Pulokas almost 14 years ago

r15245 adds the ability to set the timestamp before insert. For any object, you can set the timestamp attribute to a datetime object to override the default NOW.

Actions #2

Updated by Jim Pulokas almost 14 years ago

branched at r15304

Actions #3

Updated by Jim Pulokas almost 14 years ago

  • Status changed from Assigned to In Test
  • Assignee changed from Jim Pulokas to Anchi Cheng

commited r15305 (branches/myami-dbcopy) with all the changes

Actions #4

Updated by Anchi Cheng over 13 years ago

  • Status changed from In Test to Assigned
  • Assignee changed from Anchi Cheng to Jim Pulokas
  • Priority changed from Normal to Urgent

The function copyImportMapping works. However, the way dbdatakeeper is keeping all mapping in the memory ate up too much memory that the import could not be completed. Memory usage went up for each target and never came down and the number of new targets it could get through before crashing became smaller as the mapping accumulates.

Actions #5

Updated by Anchi Cheng over 13 years ago

import does not filling up free memory after r15334, r15335.
r15337 makes it possible to insert sessiondata to projectdata.projectexperimentch.

Now the import works right but Leginon runs can't insert projectexperiment in projectdata. sinedon created projectexperiment in leginondata instead....

Actions #6

Updated by Jim Pulokas over 13 years ago

  • Status changed from Assigned to In Code Review
  • Assignee changed from Jim Pulokas to Anchi Cheng

Just commited r15342 which may help, but I am not sure because I don't see how it ever worked right considering what I discovered. Every node (and Manager) was holding a static reference to the DBDataKeeper that is connected to the leginondata database. This was being used in the publish and research methods. If you use publish() to save projectdata objects, they would go into the leginondata db. Now the DBDataKeeper is always dynamically selected based on the class and what module it belongs to.

Actions #7

Updated by Anchi Cheng over 13 years ago

  • Status changed from In Code Review to In Test
  • Priority changed from Urgent to Normal

import data and leginon both work fine now. Will try to do it for real on Monday.

Actions #8

Updated by Anchi Cheng over 13 years ago

  • Tracker changed from Bug to Feature
  • Assignee changed from Anchi Cheng to Jim Pulokas

The import was successful. This branch will need to be clear of appion bug before it can be merged back to the trunk.

Issue that need to be clear first: #1173

Actions #9

Updated by Anchi Cheng about 11 years ago

  • Status changed from In Test to Closed

in its own branch, it works

Actions

Also available in: Atom PDF