Globus is the most efficient way to transfer files – large files, in particular – between NCAR file systems such as Campaign Storage, GLADE, and non-NCAR resources and storage systems. NCAR and UCAR researchers can also use the Globus for Google Drive connector service to transfer files to the NCAR Google Drive collection on Globus for sharing and storage purposes as described below.
Globus has both a web interface and a command line interface (CLI), which are described below. To use either interface, the first step is to create a free personal account and log in using a Globus ID or a Google account. (UCAR and NCAR staff: Do not use the NCAR RDA organizational login.)
Several mapped collections, which provide access to different file system locations on a public endpoint, are set up on the Globus system for transferring files to and from NCAR storage systems. These include:
Users can also create their own guest collections as described here in order to facilitate data sharing with colleagues and to accommodate unattended workflows.
Globus also offers a feature called Globus Connect Personal for moving files to and from a laptop or desktop computer and other endpoints.
When transferring files between systems, keep in mind that your username might not be the same on each system.
Follow these steps to transfer files. See the image below for reference.
You can check the status of your transfers any time through the web interface and will be notified when they are complete.
The Globus CLI application, an installable Python package, can be used to make both manual and unattended file transfers.
The application is:
To begin, log in as shown in this example for using the data-access nodes. (If your UCAR username and your username on your local computer are different, follow the alternative example.)
ssh data-access.ucar.edu (alternative: ssh username@data-access.ucar.edu)
Run globus login and follow the on-screen instructions.
globus login
Output example (if you are not already logged in):
Please authenticate with Globus here: ------------------------------------ URL to copy and paste into your browser. ------------------------------------ Enter the resulting Authorization Code here:
Copy the lengthy URL and paste it into your browser. It will ask you to choose an identity – as above, use either a Globus ID or Google account – and then it will take you to a consent form that looks like this:
Click Allow to give the CLI app the necessary access and you will receive an authorization code.
After entering the code at the terminal prompt, you will be logged in to the Globus CLI and in your /glade/u/home directory. You can use this authentication credential for 30 days before you will need to reactivate a mapped collection. Workflows that need longer collection access – to facilitate unattended file transfers, for example – can instead use guest collections.
Using the CLI, it is possible to query your current mapped collection activation and also force Globus to reactivate it (thus extending your activation lifetime):
gc_glade=$(globus endpoint search "ncar@globusid.org NCAR GLADE" --jq "DATA[0].id" --format UNIX) globus endpoint activate $gc_glade Endpoint is already activated. Activation expires at 2022-11-22 20:47:46+00:00 globus endpoint activate --force $gc_glade Autoactivation succeeded with message: Endpoint activated successfully using Globus Online credentials.
Your default shell on the data-access nodes is tcsh. To change your current shell, just enter bash or another preferred shell.
For details regarding how to make batch transfers and single-item transfers, manage endpoints, and more, see these resources:
NCAR and UCAR researchers can use the Globus for Google Drive connector service to transfer files to the NCAR Google Drive collection on Globus for sharing and storage purposes by following the instructions below. (To transfer files to a personal Google Drive, see Transferring files to Google Drive or DropBox instead.)
The Globus Google Drive connector is not designed to transfer Google apps products – such as Google Docs, Sheets, or Slides – between Google accounts or beyond Google Drive. While such files might appear to be visible in your Google Drive as .gsheet or .gdoc files, those are just pointers and downloading them will not download the data stored in the files.
It also is not intended for backing up your scratch space or other GLADE files. It is intended for sharing smaller files such as plots. If you need to transfer many small files, compress or archive them into a smaller number of files.
To make a transfer, follow these steps after logging in to your personal Globus account:
Your Google Drive endpoint will originate in your /My Drive/ path by default. To access other Google Drive folders such as Shared with me or Starred, simply navigate up one level.
Keep these storage and file size limitations in mind when using Globus to transfer data to the NCAR Google Drive:
The Globus Google Drive connector can streamline data sharing and collaboration with team members who don't have access to Globus.
For example, a meteorologist who performs daily operational simulations on an NCAR system as part of a multi-organizational field campaign and generates plots for flight planning can share them with the operational team and collaborators via a shared drive.
Instead of manually transferring the plots to a personal device and then uploading them to the shared drive, the meteorologist can leverage the connector to automate the workflow and move the plots directly from GLADE to Google Drive.
To set up your laptop or desktop computer to use Globus Connect Personal:
Your local system should now appear as an endpoint that can be used for transferring files.