Working with data

On your computer you have files and directories.
In iRODS you have data objects and collections.
This analogy is useful for now, although later tutorials will show that collections and data objects have additional behaviour beyond a normal filesystem.

To work with data you need a connected session and an IrodsPath pointing to a location inside iRODS.
We assume that a session named session already exists from the previous module and you already created a collection demo.

Upload a file

Ensure that a collection demo under your home exists:

home = IrodsPath(session, "~")
coll = home / "demo"
coll.create_collection()
print("Collection exists:", coll.collection_exists())

Check that a local file exists.
In this tutorial the file is called hello.txt and is located on your Desktop:

from pathlib import Path
local_file = Path.home() / "Desktop" / "hello.txt"

if not local_file.is_file():
    print("Please create:", local_file)

Upload the file:

from ibridges import upload
ops = upload(local_file, coll)

If you took part in the CLI and GUI training you already have a file hello.txt under demo. You will see an error.

Inspect the error and see why the upload is failing.

Adjust your code to:

ops = upload(local_file, coll, overwrite=True)

Inspect the collection:

print("Subcollections:", coll.collection.subcollections)
print("Data objects:", coll.collection.data_objects)

Downloading data

Choose a local directory:

local_dir = Path.home() / "Downloads"
assert local_dir.is_dir()

Preview changes:

from ibridges import download
ops = download(coll / "hello.txt", local_dir, dry_run=True, overwrite=True)
ops.print_summary()

Download:

download(coll / "hello.txt", local_dir)

Again, if the file already exists in the Download directory, you will see an error.

Downloading again without overwrite raises FileExistsError. With the overwrite flag you can force iBridges to trigger the download. iBridges compares checksums and skips transfers when source and destination match.

Synchronising data

Synchronisation keeps two locations aligned: a local directory on your computer and a collection in iRODS.
The function sync compares checksums and transfers only what has changed.
It creates new files and overwrites outdated ones, but it does not delete files from the target if they were removed from the source.

The direction is determined automatically:

• local Path → IrodsPath means uploading
  
• IrodsPath → local Path means downloading

Both locations must exist.

Example setup:

from pathlib import Path
from ibridges.path import IrodsPath

source = IrodsPath(session, "~", "demo_sync")
source.create_collection()

target = Path.home() / "Downloads" / "Sync"
print("Local target exists:", target.is_dir())
print("iRODS source exists:", source.collection_exists())

You can control how deeply the directory tree is synchronised:

• max_level=None synchronises everything
  
• max_level=1 synchronises only the top level
  
• max_level=2 includes the first level of subcollections

Other options:

• copy_empty_folders controls whether empty folders are included
  
• dry_run lists planned changes without transferring data

Dry run:

from ibridges import sync

ops = sync(
    source=source,
    target=target,
    max_level=None,
    dry_run=True,
    copy_empty_folders=True
)
ops.print_summary()

Actual synchronisation:

ops = sync(
    source=source,
    target=target,
    max_level=None,
    dry_run=False,
    copy_empty_folders=True
)

The returned ops object summarises all created folders and transferred files.

Note, that the iBridges sync function will never delete files or data objects!

System metadata

Collections and data objects carry system metadata such as creation time, modification time, owner, size, and checksum.

We can use the IrodsPath and extend it with .collection or .dataobject to see the real data behind that path. Note that this only works when the path exists.

Collection metadata:

print(coll.collection.create_time)
print(coll.collection.modify_time)
print(coll.collection.owner_name)

Data object metadata:

obj = IrodsPath(session, coll / "hello.txt")

print(obj.dataobject.create_time)
print(obj.dataobject.modify_time)
print(obj.dataobject.owner_name)
print("Size:", obj.size)
print("Checksum:", obj.checksum)

Checksums allow you to verify that the contents match exactly.

Replicas

Some iRODS systems store multiple copies of a data object on different storage resources.
These are called replicas and they belong to the same data object.
You always access the data object through a single path; iRODS decides which replica to use.

Inspect replicas:

from ibridges.util import obj_replicas
replicas = obj_replicas(obj.dataobject)
print(replicas)

Each entry contains the replica index, storage resource, checksum, size, and status.
A status of good means the replica is verified.

Exercises (15 minutes)

Exercise 1: Uploading

• Use your collection ~/demo
• Inspect the parameters of the upload function
  
• Change the content of hello.txt on your computer
• Use dry-run to inspect what would change in a new upload
• Which other options look useful for updating the data on iRODS?
• Optional: list the collection’s data objects using IrodsPath.walk

Exercise 2: Downloading and uploading a Collection

• You have “read” access to /tempZone/home/training/my_books
• Download the whole collection to your Downloads folder
• Upload the collecion again to your iRODS home

Exercise 3: Metadata

• Print creation time, owner, size, and checksum of your data object
• Reupload a new version of your hello.txt
• Compare checksum before and after downloading

Exercise 4 (optional): Replicas

• Retrieve replica information for your data object
  
• Print the status of each replica

Exercise 5 (optional): Synchronisation

• Create a new local folder and a new iRODS collection
  
• Perform a dry-run synchronisation
  
• Change one file locally and synchronise again
  
• Inspect which files were transferred and why