1. NAME

feelpp_remotedata - manage remotedata for Feel++

2. SYNOPSIS

feelpp_remotedata is a simple application which can download data, upload data and print remote data listing from multiple platforms including GitHub, Girder, and CKAN.

3. DESCRIPTION

The feelpp_remotedata tool provides a unified interface for managing data across different remote storage platforms. It supports downloading, uploading, and browsing data with comprehensive progress reporting and multiple verbosity levels.

4. SUPPORTED PLATFORMS

feelpp_remotedata supports three major remote data platforms:

Table 1. Platform Support Matrix
Platform Download Upload Contents List Organizations Authentication Progress Support

GitHub

✓

✗

✗

✗

Token

✓ Enhanced

Girder

✓

✓

✓

✗

API Key

✓ Full

CKAN

✓

✓

✓

✓

API Key

✓ Full

URL/HTTP

✓

✗

✗

✗

None

✓ Enhanced

5. OPTIONS

5.1. Common Options

feelpp_remotedata requires some options.

Table 2. Table of command-line feelpp_remotedata options
Name Description Default value

upload

specify the remote server and the location of uploaded files on remote server

download

specify remote data to download

data

  • with upload mode : datas path (file or dir) to upload,

  • with download mode : the path of downloaded datas

contents

specify data contents to print in the terminal

list-organizations

list available organizations (CKAN only)

5.2. Progress and Verbosity Options

Table 3. Progress Control Options
Option Description

--quiet, -q

Suppress progress output, show only errors

--verbose, -v

Enable verbose progress output

--progress

Show detailed progress information with file sizes

--debug

Enable debug output with full API details and responses

6. GITHUB

GitHub integration allows downloading files and folders from public and private repositories with comprehensive progress reporting.

6.1. Enhanced Progress Support

GitHub downloads now feature enhanced progress reporting similar to CKAN and Girder:

  • Repository-aware progress: Shows repository information and branch details

  • File-by-file tracking: Individual file progress for folders with item counts

  • Real-time status updates: Live progress updates with file sizes after download

  • Directory processing: Shows progress when processing subdirectories recursively

  • Debug information: Full API request details and response debugging in debug mode

Progress reporting works seamlessly for both single files and entire repository folders, providing consistent user experience across all download types.

6.2. Configuration Options

Option Default value Description

owner

feelpp

the github organization

repo

feelpp

the github repository in organization

branch

<default in github>

the branch in the git repository

path

<root of repository>

the path in the git repository

token

<no default value>

an authentication token

6.3. Authentication

GitHub authentication is required for private repositories or to increase API rate limits.

  • Environment variable:

    export FEELPP_GITHUB_TOKEN=xxxxx

  • Inline token:

    github:{repo:feelpp,path:README.adoc,token:xxxx}

6.4. Examples

  • Download file from GitHub:

    feelpp_remotedata --download "github:{repo:feelpp,path:README.adoc}"

  • Download with progress reporting:

    feelpp_remotedata --download "github:{repo:feelpp,path:toolboxes/}" --progress

  • Download from private repository with debug output:

    feelpp_remotedata --download "github:{owner:myorg,repo:myrepo,path:data/,token:xxxx}" --debug

  • Download specific branch with verbose progress:

    feelpp_remotedata --download "github:{repo:feelpp,branch:develop,path:examples/}" --verbose

7. URL/HTTP DOWNLOADS

Direct URL downloads support various protocols (HTTP, HTTPS, FTP) with enhanced progress reporting.

7.1. Enhanced Progress Support

URL downloads now feature comprehensive progress reporting:

  • Real-time progress bars: Live download progress with transfer rates and completion percentages

  • Filename-aware progress: Shows the actual filename being downloaded

  • Size reporting: Displays file sizes in human-readable format during and after download

  • Error handling: Clear error messages for network issues, HTTP status codes, and file creation problems

  • Adaptive progress: Automatically switches between progress-enabled and standard downloads based on verbosity level

Progress reporting provides the same detailed information as Girder and CKAN downloads, ensuring consistent user experience across all platforms.

7.2. Examples

  • Basic URL download:

    feelpp_remotedata --download https://raw.githubusercontent.com/feelpp/feelpp/develop/README.adoc

  • URL download with progress reporting:

    feelpp_remotedata --download https://example.com/largefile.zip --progress

  • URL download with custom directory and verbose output:

    feelpp_remotedata --download https://example.com/data.tar.gz --data $HOME/downloads --verbose

  • URL download with debug information:

    feelpp_remotedata --download https://example.com/dataset.csv --debug

8. GIRDER

Girder is a data management platform that provides a web-based interface for storing, managing, and sharing scientific data.

8.1. Configuration Options

Option Default value Description

url

girder.math.unistra.fr

url of a Girder platform

file

<no default value>

one or several file id(s) or path(s)

folder

<no default value>

one or several folder id(s) or path(s)

item

<no default value>

one or several item id(s) or path(s)

path

<no default value>

generic resource path (alternative to file/folder/item)

api_key

<no default value>

an authentication key

token

<no default value>

a direct authentication token

8.2. Resource Identification: IDs vs Collection Paths

Girder resources can be identified in two ways:

8.2.1. Using Resource IDs

Resource IDs are unique identifiers assigned by Girder:

  • File ID: girder:{file:5b1f8707b0e9570499f66bd0}

  • Folder ID: girder:{folder:6743a47bb0e95728eb010c47}

  • Item ID: girder:{item:5ac72871b0e9574027047895}

8.2.2. Using Collection-Based Resource Paths

Resource paths provide a human-readable way to identify resources using their hierarchical location within collections:

  • Folder collection path: girder:{folder:/collection/feelpp/testsuite/feelcore/feelpp_test_remotedata}

  • File collection path: girder:{file:/collection/feelpp/data/simulation-results.csv}

  • Item collection path: girder:{item:/collection/feelpp/testsuite/feelcore/feelpp_test_remotedata/dataset}

  • Generic collection path: girder:{path:/collection/feelpp/testsuite/feelcore/feelpp_test_remotedata}

Collection paths must start with /collection/ followed by the collection name and resource path. The system automatically converts collection paths to IDs using Girder’s resource lookup API.
Table 4. Collection Path Format Examples
Path Type Example Description

Collection folder

/collection/feelpp/testsuite/feelcore/feelpp_test_remotedata

Folder within Feel++ collection

Collection file

/collection/feelpp/data/simulation-results.csv

File within Feel++ collection

Nested resource

/collection/feelpp/datasets/cfd/results

Deeply nested resource

Item with files

/collection/feelpp/testsuite/feelcore/feelpp_test_remotedata

Item containing multiple files

8.2.3. Advantages of Using Paths

  • Human-readable: Easier to understand and remember

  • Hierarchical: Shows the organizational structure

  • Stable: Less likely to change than auto-generated IDs

  • Self-documenting: Path describes the content location

8.3. Automatic ZIP Extraction

When downloading folders or items from Girder, the tool automatically handles ZIP archives:

  • Automatic extraction: ZIP files are automatically extracted without manual intervention

  • Individual file listing: Shows progress and details for each extracted file rather than just the ZIP

  • Clean file access: Extracted files are immediately available for use

  • Progress tracking: Displays extraction progress and lists all extracted files in verbose mode

This enhancement provides a seamless experience similar to direct file downloads, eliminating the need for manual ZIP extraction when working with Girder folders and items.

8.4. Authentication

Girder authentication is required for private data access and upload operations.

  • Environment variable:

    export FEELPP_GIRDER_API_KEY=xxxxx

  • Environment variable (recommended):

    export FEELPP_GIRDER_API_KEY=your-api-key-here
girder:{folder:5ac72871b0e9574027047894}

  • Inline API key (deprecated):

    girder:{folder:5ac72871b0e9574027047894,api_key:xxxx}
Using FEELPP_GIRDER_API_KEY environment variable is recommended over inline api_key as it’s more secure and the remotedata app automatically manages token lifecycle.

8.5. Examples

  • Download file from Girder using ID:

    feelpp_remotedata --download "girder:{file:5b1f8707b0e9570499f66bd0}" --progress

  • Download file from Girder using collection path:

    feelpp_remotedata --download "girder:{file:/collection/feelpp/data/simulation-results.csv}" --progress

  • Download folder with automatic ZIP extraction using ID:

    feelpp_remotedata --download "girder:{folder:5ac72871b0e9574027047894}" --verbose

    This automatically extracts the folder contents and shows individual file progress.

  • Download folder using collection path:

    feelpp_remotedata --download "girder:{folder:/collection/feelpp/testsuite/feelcore/feelpp_test_remotedata}" --verbose

    Downloads entire folder using human-readable collection path.

  • Download item with automatic extraction using collection path:

    feelpp_remotedata --download "girder:{item:/collection/feelpp/testsuite/feelcore/feelpp_test_remotedata/dataset}" --progress

    Items containing multiple files are automatically extracted and listed.

  • Upload data to Girder using ID:

    feelpp_remotedata --upload "girder:{folder:5ac72871b0e9574027047894}" --data $HOME/mydata

  • Upload data to Girder using collection path:

    feelpp_remotedata --upload "girder:{folder:/collection/feelpp/uploads}" --data $HOME/mydata --verbose

  • Browse Girder contents using IDs:

    feelpp_remotedata --contents "girder:{folder:5ac72871b0e9574027047894}"
feelpp_remotedata --contents "girder:{file:5b1f8707b0e9570499f66bd0}"

  • Browse Girder contents using collection paths:

    feelpp_remotedata --contents "girder:{folder:/collection/feelpp/testsuite/feelcore/feelpp_test_remotedata}"
feelpp_remotedata --contents "girder:{path:/collection/feelpp/data}"
feelpp_remotedata --contents "girder:{item:/collection/feelpp/testsuite/feelcore/feelpp_test_remotedata/dataset}"

  • Mixed ID and collection path usage:

    # Upload to collection path-specified folder
feelpp_remotedata --upload "girder:{folder:/collection/feelpp/incoming}" --data results.txt

# Download using generic collection path
feelpp_remotedata --download "girder:{path:/collection/feelpp/datasets/cfd-simulations}" --data ./downloads

9. CKAN

CKAN (Comprehensive Knowledge Archive Network) is an open-source data management system used for publishing, sharing and finding data.

9.1. Configuration Options

Option Default value Description

url

<from environment>

URL of a CKAN instance

api_key

<from environment>

API key for authentication

organization

<from environment>

organization name

dataset

<no default value>

dataset name or ID

resource

<no default value>

resource ID for specific file operations

9.2. Enhanced Progress Reporting

CKAN downloads now feature enhanced progress reporting:

  • Filename-aware progress: Shows the actual filename being downloaded instead of just generic progress

  • Real-time updates: Live progress bars with transfer rates and completion percentages

  • Timeout support: Configurable timeout handling for robust downloads

  • Detailed progress format: Progress bars show format like filename.ext: [====⇒] 42.3% (1.81 MB/1.81 MB)

This provides the same comprehensive progress reporting as Girder downloads, ensuring consistent user experience across platforms.

9.3. Environment Variables

CKAN operations can be configured using environment variables:

export FEELPP_CKAN_URL="https://ckan.example.com"
export FEELPP_CKAN_API_KEY="your-api-key"
export FEELPP_CKAN_ORGANIZATION="your-org"

9.4. Authentication

CKAN authentication is required for private datasets and upload operations.

  • Environment variables (recommended):

    export FEELPP_CKAN_API_KEY=xxxxx
export FEELPP_CKAN_ORGANIZATION=myorg

  • Inline credentials:

    ckan:{url:https://ckan.example.com,api_key:xxxx,organization:myorg}

9.5. Examples

  • List available organizations:

    feelpp_remotedata --list-organizations="ckan:{}"
feelpp_remotedata --list-organizations="ckan:{url:https://demo.ckan.org}"

  • Download entire dataset:

    feelpp_remotedata --download "ckan:{dataset:my-dataset}" --progress

  • Download specific resource:

    feelpp_remotedata --download "ckan:{resource:resource-id}" --debug

  • Upload data to CKAN:

    feelpp_remotedata --upload "ckan:{dataset:my-dataset}" --data $HOME/mydata

  • Browse dataset contents:

    feelpp_remotedata --contents "ckan:{dataset:my-dataset}"

10. GENERAL EXAMPLES

10.1. Download Examples

  • Download file from URL with progress:

    feelpp_remotedata --download https://raw.githubusercontent.com/feelpp/feelpp/develop/README.adoc --progress

  • Download GitHub repository with progress:

    feelpp_remotedata --download "github:{repo:feelpp,path:examples/}" --verbose

  • Download with custom directory:

    feelpp_remotedata --download "girder:{file:5b1f8707b0e9570499f66bd0}" --data $HOME/mydir

  • Download with different verbosity levels:

    # Quiet mode (errors only)
feelpp_remotedata --download "ckan:{dataset:sample}" --quiet

# Normal mode (default) - shows operation status and file names
feelpp_remotedata --download "github:{repo:feelpp,path:README.adoc}"

# Progress mode (detailed progress) - shows file sizes, transfer rates
feelpp_remotedata --download "ckan:{dataset:sample}" --progress

# Debug mode (full API details) - shows URLs, API requests/responses
feelpp_remotedata --download "ckan:{dataset:sample}" --debug

10.2. Upload Examples

  • Upload to Girder:

    feelpp_remotedata --upload "girder:{folder:folder-id}" --data $HOME/mydata --verbose

  • Upload to CKAN:

    feelpp_remotedata --upload "ckan:{dataset:my-dataset}" --data $HOME/mydata --progress

10.3. Browse Examples

  • Browse with different platforms:

    # Girder folder contents
feelpp_remotedata --contents "girder:{folder:5ac72871b0e9574027047894}"

# CKAN dataset contents
feelpp_remotedata --contents "ckan:{dataset:my-dataset}"

# Multiple Girder items
feelpp_remotedata --contents "girder:{folder:[id1,id2],file:id3}"

11. PROGRESS REPORTING

The feelpp_remotedata tool provides comprehensive progress reporting with four verbosity levels across all supported platforms (GitHub, Girder, CKAN, and URL/HTTP):

Table 5. Progress Levels
Level Option Description Output

Quiet

--quiet, -q

Minimal output

Errors only

Normal

(default)

Standard progress

Operation status, file names, completion messages

Verbose

--verbose, -v or --progress

Detailed progress

File sizes, transfer rates, detailed status, progress bars

Debug

--debug

Full diagnostic

API requests/responses, URLs, technical details, all verbose output

11.1. Universal Progress Features

All platforms now support consistent progress reporting:

  • File-by-file progress: Shows individual file download/upload progress with transfer rates

  • Size reporting: Displays file sizes in human-readable format (KB, MB, GB)

  • Platform-specific information:

    • GitHub: Repository details, branch information, directory structure

    • Girder: Item/folder metadata, automatic ZIP extraction progress

    • CKAN: Dataset information, resource details, organization context

    • URL/HTTP: Direct download progress with HTTP status information

  • Error handling: Clear error messages with specific failure reasons and status codes

  • Operation summaries: Final status with success/failure counts and timing information

  • API debugging: Full request/response logging in debug mode with URLs and headers

11.2. Platform-Specific Enhancements

GitHub Progress Features

  • Repository and branch information display

  • Recursive directory download progress with item counts

  • File size reporting after download completion

  • API rate limit and authentication status

Girder Progress Features

  • Automatic ZIP extraction with individual file progress

  • Folder/item structure visualization

  • Upload progress with chunk transfer status

  • Authentication token management

CKAN Progress Features

  • Dataset and resource metadata display

  • Organization context information

  • Enhanced filename-aware progress bars

  • Resource listing with detailed information

URL/HTTP Progress Features

  • Real-time download progress bars

  • HTTP status code reporting

  • Network error handling and retry information

  • Protocol-agnostic support (HTTP, HTTPS, FTP)

12. ERROR HANDLING

The tool provides robust error handling:

  • Network errors: HTTP status codes and connection issues

  • Authentication errors: Clear messages for invalid credentials

  • Resource not found: Specific error messages for missing files/datasets

  • Permission errors: Access denied messages with suggestions

  • Data validation: JSON parsing errors and malformed responses

13. COPYING

Copyright (C) 2017-2025 Feel++ Consortium.
Free use of this software is granted under the terms of the GPLv3 License.