Globus

h1. Globus Data Transfer from SEMC

{{toc}}

h2. Introduction

Globus is an application that lets you transfer large amounts of research data efficiently and securely to your personal computer, institution’s storage systems or a cloud provider. It eliminates the use of big portable external hard drives that need to be mounted to a local storage to transfer data. The New York Structural Biology Center primarily uses Globus to transfer data that is collected on-site from electron microscopes. To get started with, follow the instructions below. 

h2. Logging in to Globus

Typically, you can log into Globus with your institutional credentials.  To determine if this is possible, check if your institution is listed under the dropdown menu located on the "Globus login page":https://app.globus.org (https://app.globus.org).  If it is, you can skip to the next section (_Logging in with Institutional Credentials_).

!{width:33%}globus_cilogon_1.png!

h3. Creating an ORCID Account and Linking it with Globus

If you do not have access to Globus via your institution, you can use your ORCID to sign in to Globus.  If you do not already have an ORCID, you can sign up for one "here":https://orcid.org/register.  To sign in with your ORCID, you will need to click the green ORCID button on the login page:

!{width:33%}globus_orcid_1.png!

You will then be prompted to connect your ORCID to Globus.  Click the _Authorize access_ button when prompted. 

!{width:33%}globus_orcid_2.png!

If your ORCID account is successfully linked with Globus, you will be presented with the following screen.  Click _Continue_.

!{width:33%}globus_orcid_3.png!

Fill out the subsequent form.  Enter in your email address and organizational affiliation when prompted.

!{width:33%}globus_orcid_4.png!

Finally, grant Globus the permissions it needs from your ORCID account by clicking on _Allow_ on the last screen:

!{width:33%}globus_orcid_5.png!

You should now be logged into Globus.  For future sessions, you can click the green ORCID button on the main page to log in again.

h3. Logging in with Institutional Credentials

Select your institution from the dropdown on the Globus login page and click _Continue_.

!{width:33%}globus_cilogon_1.png!

You will now be prompted with the dialogue you typically use to log in to your institution's services (email, etc).  Enter your institutional credentials (e.g., NetID, UNI) and sign in as normal.

!{width:33%}globus_cilogon_2.png!

h3. Globus File Manager

Once you have successfully signed in (via ORCID or institutional credentials) you will be presented with the Globus file manager.  In the next section, we will describe how you can use this web interface to initiate data transfers to your home institution.

!{width:33%}globus_filemanager.png!

h2. Transferring Data

h3. Transferring files from NYSBC to your personal endpoint 

1. Click on the Collections tab and type ‘NYSBC#SEMC’ This will act as your source endpoint, where you will be transferring files from 

!{width:25%}image6.png!

2. You will see a login widget below which will ask for your username and password to authenticate.  Please take note that you need to enter the username which was provided by SEMC when you first registered for your project. Also, note that this username is all lowercase 

If you are already logged into another NYSBC account. Please go to Settings> Manage Identities > NYSBC OIDC Server  

!{width:25%}image7.png!

3. After you authenticate, you will see your default path which is currently set at /h1/<username> 

!{width:25%}image8.png!

4. You can transfer raw frames and references by selecting the ‘frames’ folder  

 * Select your username 

 * Then, select the session you want to transfer frames from eg: (22mar18g) 

 * Under the rawdata folder, you will see all your raw frames 

5. You can also transfer aligned images by selecting the ‘leginon’ folder 

 * Select your username  

 * Then select the session you want to transfer frames from eg: (22mar18g) 

 * Under the rawdata folder you will see all your aligned images 

!{width:25%}image9.png!

6. Now, you need to select the destination endpoint. For transferring files to your personal computer, you need to install Globus Connect Personal. Click on ‘Endpoints’ on the left navigation bar and select ‘⊕ Create a personal endpoint’ on the top right of the screen 

!{width:25%}image10.png!

You will see the following screen 

!{width:25%}image11.png!

 * Download and install the Globus Connect Personal as per your OS distribution  

For specific installation instructions, 

Mac installation: https://docs.globus.org/how-to/globus-connect-personal-mac/ Windows: https://docs.globus.org/how-to/globus-connect-personal-windows/ Linux: https://docs.globus.org/how-to/globus-connect-personal-linux/  

* Log In, then press allow, this will take you to the Globus Connect Personal Setup 

    * For Owner Identity, select your Globus ID 

    * Enter your endpoint collection name. eg: Personal Computer 

    * Press ‘Save’ 

!{width:25%}image12.png!

8. Next, select your personal endpoint as your destination endpoint. 

 * Go back to the file manager and select NYSBC#SEMC in the Collection tab  

 * Then select, transfer and sync to option.  

 * On the right side, select your personal endpoint as the destination endpoint. Please see the screenshot 

!{width:25%}image13.png!

You will now see the folders on your laptop.  

Once you select the appropriate source and destination folder, then click on Start to initiate the transfer process 

!{width:25%}image13.png! 

To monitor the transfer process, click on the Activity tab on the left sidebar 

You will see the transfer logs and the message when the transfer is completed.  

Note: Sometimes, your personal computer will be under a firewall which will not allow transferring files. To configure your firewall settings, please follow the link https://docs.globus.org/how-to/configure-firewall-gcp/  

For any issues related to Globus please check the following guides: 

 * https://docs.globus.org/how-to/ 

 * https://docs.globus.org/faq/ 

h3. Transferring files from NYSBC to your home institution endpoint  

* Follow steps 1 - 5 as described above. 

 * Then select your home institution endpoint as the destination endpoint. Please contact your organization’s IT team to acquire the name of the endpoint authorized by your institution and for the login credentials and queries related to your directory permissions. 

 * Follow step 11 as described above. 

 * You can keep track of data transfer status by clicking on the Activity tab. When the transfer is completed, you will see a green check mark and “Condition” will say SUCCEEDED.