Feature #1128
closed
import/export/copy data between two databases
Added by Jim Pulokas almost 14 years ago.
Updated about 11 years ago.
Description
Need ability to export from one database and import into another. This should recursively preserve all associated references between objects and preserve timestamps. For example, assuming some global DB config was done in sinedon.cfg:
import sinedon
import sinedon.transfer # a hypothetical module for doing the transfer
import leginon.leginondata
# export 10 most recent images from first database
sinedon.setConfig('leginondata', db='leginondb1')
imagequery = leginon.leginondata.AcquisitionImageData()
images = imagequery.query(results=10)
# import them into the second database
sinedon.setConfig('leginondata', db='leginondb2')
siedon.transfer.transfer(images)
The result would be that leginondb2 will now contain copies of the image data plus any references to SessionData (create new unique session), AcquisitionImageTargetData, etc...
r15245 adds the ability to set the timestamp before insert. For any object, you can set the timestamp attribute to a datetime object to override the default NOW.
- Status changed from Assigned to In Test
- Assignee changed from Jim Pulokas to Anchi Cheng
commited r15305 (branches/myami-dbcopy) with all the changes
- Status changed from In Test to Assigned
- Assignee changed from Anchi Cheng to Jim Pulokas
- Priority changed from Normal to Urgent
The function copyImportMapping works. However, the way dbdatakeeper is keeping all mapping in the memory ate up too much memory that the import could not be completed. Memory usage went up for each target and never came down and the number of new targets it could get through before crashing became smaller as the mapping accumulates.
import does not filling up free memory after r15334, r15335.
r15337 makes it possible to insert sessiondata to projectdata.projectexperimentch.
Now the import works right but Leginon runs can't insert projectexperiment in projectdata. sinedon created projectexperiment in leginondata instead....
- Status changed from Assigned to In Code Review
- Assignee changed from Jim Pulokas to Anchi Cheng
Just commited r15342 which may help, but I am not sure because I don't see how it ever worked right considering what I discovered. Every node (and Manager) was holding a static reference to the DBDataKeeper that is connected to the leginondata database. This was being used in the publish and research methods. If you use publish() to save projectdata objects, they would go into the leginondata db. Now the DBDataKeeper is always dynamically selected based on the class and what module it belongs to.
- Status changed from In Code Review to In Test
- Priority changed from Urgent to Normal
import data and leginon both work fine now. Will try to do it for real on Monday.
- Tracker changed from Bug to Feature
- Assignee changed from Anchi Cheng to Jim Pulokas
The import was successful. This branch will need to be clear of appion bug before it can be merged back to the trunk.
Issue that need to be clear first: #1173
- Status changed from In Test to Closed
in its own branch, it works
Also available in: Atom
PDF