Appion

Docker is fine. This is going to be an issue, I do miss the revision numbers. I guess, we could use git to check to see if a certain commit (e.g., 30ac218) has been applied.

Not sure if I should start a new issue or add to this one, but "appionlib/StackClass" is not copied to the lib directory as part of the installation.

committed 0e91e55d for StackClass under Issue #4014

Yeah, I will take a look I knew this would be a problem.

Hi Anchi, I cannot get you command to work, but as I understand it, you want to check if commits commit:3af740e7 commit:5cb057f7 are present in the current branch.

If I understand I think this is the cleanest command line test

git branch -q --contains 3af740e7 5cb057f7 2> /dev/null; echo $?

return 0 if applied and 129 if not.

No, for finding out which dbschema need to be applied, we need to know if the current hash is at later date than the other. It is like having a version number. I would be happy enough if there is a push/commit date comparison to replace the number sorting mechanism in dbschema/udatelib.py

Yeah, I am just skipping the current hash check. You can apply hashes out of order via cherry-pick, so is-ancestor is not always going to work.

Also, that code did not work:

git checkout myami-2.0
Switched to branch 'myami-2.0'
Your branch is up-to-date with 'origin/myami-2.0'.

git branch -q --contains 3af740e7 5cb057f7 2> /dev/null; echo $?
0

git checkout trunk
Switched to branch 'trunk'
Your branch is up-to-date with 'origin/trunk'.

git branch -q --contains 3af740e7 5cb057f7 2> /dev/null; echo $?
0

Happy to do it, but I need some info on what the new schema changes are and what commit starts those changes, I assume commit:3af740e7

Or I could just create a sample case, where nothing in the database is changed.

Making an example would be a good ideal any way.

The one I need starts with the change in ecd31e30. If created the Issue #4655 and fixed in 661ad086. I need to set instrument before the fix to hidden=False.

Thanks.

Hi Anchi, I spent a good chunk of the day reading all of the code in the dbschema folder and it is quite difficult to follow. Frankly, I want to clean it up, because a lot of changes need to happen for git, but I need to know the use cases.

How to determine whether to update the database:

1. check the branch (using git branch | grep '\*' | cut -d' ' -f2-)

2. check last update number or code in the leginon database

3. for each available update in folder:
a. does update apply to current branch
b. is update newer than last update in database
c. perform update
d. log update to leginon database

Is it more complicated than this, are there special cases that this would miss.

Here are more:

1. Determine if an update is required if appion is not used so that they don't need to be applied.
2. Back-track to reset revision for incremental update when the branch is switched. See updatelib.py getBranchResetRevision. It has not happened recently.
3. Most updates should not be repeated such as the change of signs of ctf astigmatism angle. See schemabase.py commitUpdate.
4. Others must be allowed to repeat because they were found too late in history and therefore newer update may have been committed to database. Or if a special situation arrive that it needs to be repaired again. For example, people failed to give right Cs to their scope in instruments.cfg to match database and repeatly adding new instrument that breaks the InstrumentData.
5. The order of update is important sometimes, in other words, there are dependencies.

Go ahead and rewrite but please make sure that it is usable for any upgrade from 3.0 and up. We just had a case with an upgrade starting from 2.1.

After some extensive reading about git, I outlined a method for the schema updates. It largely will follow the current system with a few changes: (1) use 'git tag' to denote commits that require a schema update and (2) create a new database table in projectdb that will keep track of when and what schema updates have been applied.

Now, that I have a system that I think will work, I am going to begin implementing it. A prototype should be ready by Friday and barring any major bugs should be ready for use this Monday.

Can you tag an older revision ? update is needed normally because of bug so that the revision has gone on.

Yes, I tagged some old revisions, so I knew where the old subversion revisions were

git tag r18417 a0aba4d909906084c30eb7327baeed7c42923bcb
git tag r19936 b6671a2c1

I plan to use this command to denote a commit that requires a schema update

git tag schema19721 <commit id>

This command will determine which tags are available in the current branch, this would not work if the schema commit is cherry-picked, it must be merged.

git tag --merged

Hi Anchi, do you have an SQL dump of an older Leginon/Appion database that needs upgrading. I want to test the new system.

Ran into a major bug 'git tag --merged' which determine which tags are available in the current branch is not available in git 1.7 that ships with CentOS, the latest version is git 2.11.1.

I forgot to push the tags to the main repo using git push --tags so the schema update was not finding any tags it is working now.

• create the python script to update the database
• create a git tag (schema19470) to mark the commit id (c727825) with the python script, git tag -a schema19470 c727825
• push the tag to the main git trunk git push --tags

Neil,

Found today on a new installation of trunk the hidden field on InstrumentData does not exist. Fixed it in commit:850004f
Also noticed that install table in projectdb still has the old format on this new db.
The new db creates a situation that the dbschema update is required. There is always one row of data if
auto-installer is used. You can use that to test.

So, you want me to create a schema update to add the boolean field in InstrumentData at commit commit:850004f?

I know I originally wrote the auto-installer, but I have not used it in over 7 years. So, it reads from the XML file to get table values? I guess that is what the myamiweb/setup script would do too.

No, I am just giving you a way of generate an example broken db for testing. You said you wanted one.

Sorry for being so dense, if I create a new install and new database with the auto-installer it should be ready for a schema update.

I think so. I have not read your code how it should look like with a new install in the install table but database needs a place to log its current dbschema version.