Introduction

Why this training exists

  • Many research data platforms use iRODS
  • You interact with iRODS when uploading or organising data
  • This training introduces iRODS and iBridges

What is iRODS

iRODS:

  • Virtual data layer
  • Unified way to organise and access data
  • Works on top of storage systems

Difference to normal file systems:

  • Collections are virtual
  • Data Objects can have multiple replicas
  • Physical storage is hidden

Virtual view vs physical storage

  • Data in iRODS is virtualised: it lives in storage resources
  • Resources have logical names and physical locations
  • Replicas provide safety and performance
  • You see Data Objects and Collections, tha actual file is stored elsewhere
  • iRODS manages storage: you do not need to worry about that.

irods_replication cluster_zone iRODS Zone Collection Collection /my_project DataObject Data Object report.csv Collection->DataObject ResourceA Resource Active Host: storage1 Path: /data/report.csv DataObject->ResourceA Replica 1 ResourceB Resource Archive Host: storage2 Path: /data_backup/report.csv DataObject->ResourceB Replica 2

Metadata

  • Metadata describes data
  • Metadata enables discovery
  • Metadata supports automation

  • System metadata
    • Set by the system
    • Time stamps, identifiers, checksums
  • User metadata
    • Used by developers for automation
    • Used by scientists to annotate data

iBridges client suite

  • Python API (layer to access (meta)data using Python)
  • Command line interface
  • Graphical user interface

Made for:

  • Researchers
  • Workflow developers
  • Lab technicians
  • Data managers

iBridges capabilities

  • Connect to iRODS
  • Upload and download
  • Edit metadata
  • Synchronise folders
  • Create tickets
  • Search metadata

Python API

  • Convenience layer around python irods client
  • Creates an irods session automatically
  • Suitable for scripts and workflows

Command line interface

  • Simple commands
  • Auto completion
  • No Python code needed

Graphical user interface

  • Browse collections
  • Upload / Download files and metadata
  • Edit metadata
  • Find data by their metadata

Architecture overview

  • iBridges uses python-irodsclient internally
  • CLI and GUI are based on the iBridges API

ibridges_prc cluster_api_features iBridges Python API cluster_gui_features iBridges Graphical User Interface cluster_cli_features iBridges Command Line Interface cluster_prc Python iRODS client (PRC) Session iBridges session instantiates irods_session PrcPut put Session->PrcPut calls DataOps Data operations DataOps->Session uses IrodsPath IrodsPath IrodsPath->Session uses Permissions Permissions Permissions->Session uses GuiDataOps Data operations GuiDataOps->IrodsPath calls GuiIrodspath IrodsPath GuiMetadata Metadata operations GuiPermissions Permission operations CliDataOps Data operations CliNavigation Navigation CliMetadata Metadata operations CliPermissions Permission operations CliPermissions->IrodsPath calls PrcSession iRODSSession PrcDataObjects data_objects PrcCollections collections PrcGet get PrcACLs Permissions PrcRescs Resources

Questions