Using the MSO4SC Portal

This document will provide basic information:

  • to connect to the Portal,

  • to register an HiFiMagnet application,

  • to purchase an HiFiMagnet application,

  • to run an HiFiMagnet application.

and it defines the role of

An overview of the main portal functionalities is given in this section. This is followed by a brief description of the HiFiMagnet apps available and on the over-whole workflow of actually running apps within the portal.

The user that have already an account on Portal may skip the next section and go directly to the section related to his/her role.

1. Get an account on MSO4SC Portal

Connect to MSOPortal

login page MSOPortal
  • Click on login

You will be forwarded to IDM component to the portal.

IDM component of MSOPortal
  • Sign up to get an account

IDM component of MSOPortal
do not check gravatar bug as it is buggy right now.
  • Create an account there providing a valid email and a password.

Then you have to validate your account by following the instructions send by the portal to your email address.

Right now the portal is not using https protocol. As a consequence, the credentials user/password are sent in clear. So be careful.

In the sequel, each time you start a new service that relies on the IDM component responsible for the authentification you will be notify by the system and will have to authorize this service like in the image below:

Register new compnent to IDM MSOPortal

2. Overview of the Portal

Main page for MSOPortal

the portal is split into several services:

  • Marketplace, where you can add and/or purchase applications

  • Data Catalogue, where you can add and/or retrieve data

  • Experiments, where the app is setup and the simulation is launched

  • Visualization, to access a Remote desktop to pre and/or post-processes your simulations

more details on:

For smooth operations of portal, the following ports have to open:

  • 3000: IDM, the service responsible for authentification

  • 5000: MarkePlace

  • 8000: Data Catalogue

To check, you can use nmap -p <port> portal.mso4sc.eu.

You may need to contact your system admin to check and eventually open these ports.

For DevUser using the portal dev version …​.

  • 5050, 8080: …​

3. HiFiMagnet Offerings in Marketplace

Currently the apps available in the Marketplace are given in the table bellow. They are grouped into 4 categories:

  • MagnetTools:

  • Preprocessing:

  • Simulations:

  • Postprocessing:

  • Bundles:

Table 1. Available HiFiMagnet apps
Offering Description Category Notes

HiFiMagnet Offering

Basic demonstrator

Simulations

For details about settings up these apps please refer to these docs. For the new user we recommend to read this guide. The impatient may go directly to the EndUser and try to run the basic demonstrator.

4. Synopsis of the HiFiMagnet Offering

The HiFiMagnet offering is created by DevUser as described in this section. Any EndUser may purchase the offering following the generic procedure. Then, he or she can run the purchased offering provided that some fields are filled properly as in this example. Details for setup each offering will be giving in the next section.

Even if each offering is specific, what is going on under the hood is described here:

  • deploy:

  • running:

  • undeploy:

For app that requires some additional services like:

  • Sregistry: a registry holding singularity images,

  • Girder: a data management system

that requires additional settings (like credentials).

It is important to notify the EndUser that he has to get an account on these services.

For Sregistry see this manual

For managing the account and this note to setup the Sregistry client.

As for Girder …​

5. DevUser role

5.1. Becoming DevUser

Once registered in the portal, you are a EndUser. To be granted DevUser rights, just email to Portal Admins (actually Victor or Javi).

5.2. Register an HiFiMagnet application

To register an application you need to be DevUSer The process is split into several part:

  • add the app and create an offer in the Marketplace:

    • go to My stock/Catalogue

    • go to My stock/Product

    • go to My stock/Offering

Then you need to perform the following steps in Experiments:

  • go to RegisterApp

    • select your Offering in the sliding menu New Products

    • give the app a meaningful name

    • Choose a package file to load.

The package file is an gzipped archive holding the formal description of the app. It’s actually a directory which contains :

  • a blueprint.yml TOSCA file that describes the workflow

  • a scripts sub-directory holding the scripts required to deploy and undeploy the app

In this git repository you will find the actual application corresponding to the HiFiMagnet basic demonstrator.

To actually build the app, use: ./deploy pkg

An app can only be registered once. It means that once you have gone througth the registration process your only choice is to RemoveApp.

As a consequence, to upgrade your app package you have first to remove the app and go back to re-register your app.

5.3. Validate an HiFiMagnet application purchase order

Once the app has been registered in the Marketplace (actually it means that an offering has been defined) you will be owner of this app. Any EndUser may purchase your app. To check if you have any order:

  • go to My Inventory

  • go to My Inventory/Product Orders

  • check Recibibas menu

  • if any new message, you need to:

    • click to validate the order

    • click again to finalize the order

The EndUser may now proceed to perform Experiments with you product

Do not forget to notify the user that he/she has to connect once to:

and eventually to :

This steps are mandatory for smooth operation of the app since the user has to be included into HiFiMagnet user group. Fortunately this need to be done only once for any HiFiMagnet app.

Unfortunately the management of user has to be carried out manually by the DevUser in charge of the app and cannot be performed from the portal.

5.4. Adding an EndUser to Sregistry team

For app like HiFiMagnet apps that require an private singularity image, you must grant EndUser access to the Sregistry HiFiMagnet team. To do so follow these instructions:

ex: HiFiMagnet Sregistry team
  • As in the image above, click on Invite Users

  • Copy the link (display in blue) and send it to your new EndUser

Once the EndUser will validate the invitation, his/her name will be display in the list of members. You can also make him/her owner of the team by clicking on the Make Owner button.

To grant the EndUser access to the images in the HiFiMagnet collection, you need to proceed as follows:

  • click on the Collections menu on the top left

  • click on the link to the HiFiMagnet collections in the collections list

  • once the list of images contained in the HiFiMagnet collection is display, you have to click settings button

A list of owners and contributorsllaborators appears in the top right part of the page. Bellow are lists of user that are nor part of owners nor contributors.

  • To grant owner or contributor role to a given user select his/her name

  • finaly simply click on the red save button

The previously selected user shall not be show in the appropriate list according to your choice.

5.5. Adding an EndUser to Girder

5.6. Data Catalogue

As a DevUser you can create Organization in Data Catalogue

  • click on Organizations

  • click on Add Organization

You, then, need to fill the form and finalize the creation by clicking on the create organization button.

You have then to manage user that are allowed to be part of your organization. They may the following role:

  • admin : can view/add dataset, add user and manage role

  • editor : can add dataset

  • member : can view dataset

EndUser has to connect at least once to Data Catalogue before you can grant them a role in your organization.

6. EndUser role

when running into some strange portal behavior, first try to logout/login from the portal before reporting an issue.
make sure that you have access to sregistry holding all the singularity images. Right now its the only such service (see key sregistry_url in blueprints) available in the frame of the project.

6.1. Purchase an application

To purchase an application from the Marketplace you can either be DevUser or a simple EndUser. The process is the following:

  • connect to the Marketplace, in Home (top left Marketplace page) you will find all available apps (All categories).

  • Click on Add to Cart to purchase the app you want.

  • Click on Shopping Cart and Checkout to go on

    • fill the required forms for shipping address

    • provide a shipping address if not already defined

    • provide a billing address if not already defined

    • Finalize your order by clicking on Checkout

To use the app, you will have to wait that the owner/seller of the app validate you order. To check the status of your order:

  • go to My Inventory

  • go to Product Orders

You shall be to see your products and their status in each product form.

To finalize your purchase DO NOT FORGET to connect to :

Then notify the owner/seller that you request to be part of HiFiMagnet user group. This will allow you to access {himagnet} singularity images and datasets.

Girder is optional.

6.2. Upload Data to Data Catalogue

  • Connect to Data Catalogue

  • Select an organization

  • Add a dataset if you are assign the role of editor

you need to ask the owner/seller of HiFiMagnet app to be granted the role of editor

6.3. Upload Data to Girder service

6.4. Setting up an HPC resource in Experiments

  • Go to Experiments

  • Select Settings

  • Define a HPC resource

Name: cesga (1)
Host: ft2.cesga.es (2)
User: [Your cesga user]
Password: [Your cesga password]
timezone: (3)
Workload Manager: SLURM (4)

6.5. Connecting to Sregistry service

Before proceeding to the setup of the app, you need to make sure, at least for HiFiMagnet app (or more generally for apps requiring access to a private singularity image) that you have been granted the proper right on HiFiMagnet Sregistry team.

One more step consists in retrieving the TOKEN and storing it into your home directory on the HPC resource. This TOKEN is needed to download programmatically the Singularity image :

  • connect to Sregistry using FiWare authentification

  • go to your account (top right button)

  • select Token

ex: {Get Sregistry token
ex: {Display Sregistry token
  • copy the token into a file .sregistry

  • copy .sregistry on the HPC resource, see previously defined, into your home directory

To copy the file to the HPC resource you can use scp .sregistry username@host:. or whatever method you prefer. username and host are key defined in the HPC resource setting

You may connect to your account on HPC resource

ssh -Y username@host

If you cannot connect directly to the HPC resource, you may have to use alternative. On cesga, you can either:

  • connect to cesga user portal with your credentials,

  • click on Create a new Desktop to get a remote desktop

  • in a terminal try to scp user@yourmachine/.sregistry $HOME

an alternative is to use: * a VPN solution: for cesga use fortisvpn client:

/opt/forticlient-sslvpn/64bit/forticlientsslvpn_cli --server gateway.cesga.es:443 --vpnuser user_email

6.6. Running an HiFiMagnet application in Experiments

  • Select Create App Instance

    • define an ID

    • select a registered app in the menu

At this point you should have a page with all the options that need to be filled before moving to Deploy Instance

ex: Main HiFiMagnet app

The fields to fill may be grouped into categories:

  • specs for running simulations

  • HPC settings

  • specs retrieving singularity images

  • specs for singularity image

The EndUser shall only need to fill the fields related to the 1st and 2nd category. The fields that are to be filled by EndUser are marked in bold.

The others fields are designed for more advanced users.

Table 2. List of keys to be defined for being notified by email (only available in HiFiMagnet)
Key Description Default Notes

email_type

type of notifications

workload_manager dependant (for slurm: ALL,END,FAIL)

email_user

email address

Table 3. List of keys specs for running simulations
Key Description Default Notes

cfgfile

configuration file

/usr/share/doc/hifimagnet/ThermoElectricModel/quarter-turn3D.cfg

cfg file need to be in the container, or in a mounted dir or in the dataset

Dataset resource: input_url

url to retrieve for cfg file and input data

None

URL pointing to an archive of data files (comes from the data Catalog)

execfile

executable file

feelpp_hfm_thermoelectric_model_3D_V1T1_N1

any exec feelpp_hfm…​

Table 4. List of keys to be defined for HPC settings
Key Description Default Notes

max_time

maximum allowed time for run (minutes and seconds)

'00:30:00'

shall depends on batch system and selected partition

monitor_entrypoint

Monitor entrypoint IP

"193.144.35.146"

for this release it should be empty

job_prefix

Job name prefix in HPCs

"mso4sc"

HPC: primary

HPC settings

"cesga"

hpc_modules

modules to load depending on the targeted machine

[gcc/6.3.0, openmpi/2.0.2, singularity/2.4.2]

shall depends on HPC and singularity image

hpc_volumes

volumes to be mounted on the targeted machine

[/scratch,/mnt,${LUSTRE}/feel:/feel]

depends on HPC and singularity image

hpc_partition

select partition queue

'thin-shared'

shall depends on batch system and HPC (TODO)

hpc_reservation

select reservation queue

``

is optional

parallel_tasks

number of tasks/processes to run in parallel

2

shall depends on batch system and selected partition

Table 5. List of keys specs for retrieving singularity images
Key Description Default Notes

sregistry_image

URI pointing to the sregistry-cli image

mso4sc/sregistry

sregistry_client

define default sregistry client

registry

sregistry_client_secrets

path to file where sregistry secret are stored

$HOME/.sregistry

sregistry_storage

path to container directory

${LUSTRE}/singularity_images

sregistry_url

URI pointing to the sregistry

sregistry.srv.cesga.es

Table 6. List of keys specs for singularity image
Key Description Default Notes

singularity_image_cleanup

force remove of singularity image

false

singularity_image_uri

URI pointing to the singularity image

hifimagnet/hifimagnet:stretch

singularity_image_filename

Filename of the singularity image

hifimagnet-stretch.simg

Once filled, click on Create Instance. This will result in the executions on the selected HPC resource of the bootstrap script included in the app definition.

Any error will result in a message at the top of the page, bellow the menus.

The next step is to Deploy Instance:

  • select the ID, you defined in the previous step, in the *App instance * menu

  • click on Deploy, wait until the button in blue turns green

To Run Instance, the procedure is similar:

  • select the ID, you defined in the previous step, in the App instance menu

  • click on RUN

These step may take some minutes to be performed. A red button will indicates an error. Check the message in the frame bellow the status button. To have more details about the actual error is more tricky; we recommend to:

  • make sure you have unchecked the cleanup option

  • connect to the HPC resource

  • move to the selected base_dir/working_prefix<id>

  • check the logs

base_dir, working_prefix are filled within the portal Experiments. The <id> is a random character generated by the deployment.

An other option is to run the app using the orchestrator CLI. see this file for details

You may cleanup your working by

  • UnDeploy your Instance

    • select the ID, you defined in the previous step, in the *App instance * menu

    • click on UnDeploy

  • Destroy your Instance

    • select the ID, you defined in the previous step, in the *App instance * menu

    • click on Destroy

6.7. Running an HiFiMagnet application in Experiments using datasets from DataCatalogue

Basically the procedure is the same as above. The only difference here is that you need:

  • to select a dataset in the DataSet sliding menu

  • to validate your choice

  • to enter the name of the `<cfgfile> you will use

You should at the end have something like in the following image.

Selecting a dataset

The dataset, at least those from HiFiMagnet organization in the Data Catalogue are gzipped archive that holds all the files needed to run:

  • a mesh or cad file

  • a <cfgfile>

  • a list of json files for physical properties and/or boundary conditions

  • eventually some more files when using some MagnetTools features (eg …​)

The format of the files in a dataset are respectively described in:

  • for HiFiMagnet simulation

  • for mesh and cad format

  • for {magnettoools}

6.8. Setting up an Remote Desktop resource in Visualization

  • Go to Visualization

  • Select Settings

  • Define a Remote Desktop resource

Name: cesga
Host: vis.lan.cesga.es
User: [Your cesga user]
Password: [Your cesga password]
Remote Desktop Tools: [select item in the sliding menu]
List command: /opt/cesga/vis/bin/desktops
Create command: /opt/cesga/vis/bin/getdesktop

6.9. Running an Remote Desktop resource in Visualization

  • Go to Visualization

  • Click Desktop Available

    • Select your Infrastructure in the sliding menu

if no Remote desktop is defined, you will receive a warning message. Create a Setting, then go back to this step.
  • Click on Create

  • Click either on Desktop or View Only

it should start a web page within your browser. In this page you should be able to:

  • start a 'terminal' and check the output of your simulation

The outputs of the job (ie slurm jobs) are stored in the <hpc_basedir>/<hpc_workdir_prefix>_<date>_<id>. The actual results for the simulation would be stored in <hpc_basedir>/feel directory following Feel++ usage.

The result are stored in ${LUSTRE}/feel/…​ if you have chosen to use the <hpc_volumes> bindmount option: eg ${LUSTRE}/feel:/feel on cesga.

Check the output of the script (aka slurm-<id>.out) in <basedir>/<workdir_prefix>_<date>_<id> to see the exact name of this directory.

In slurm-<id>.out you should see lines like that at the beginning of the files:

Reading /usr/share/doc/hifimagnet/ThermoElectricModel/quarter-turn3D.cfg...
[ Starting Feel++ ] application thermoelectric version 0.1 date 2018-Jun-07
 . thermoelectric files are stored in /feel/thermoelectric/np_2
 .. exports :/feel/thermoelectric/np_2/exports
 .. logfiles :/feel/thermoelectric/np_2/logs
[modelProperties] Loading Model Properties : "/usr/share/doc/hifimagnet/ThermoElectricModel/quarter-turn3D.json"
 . thermoelectric files are stored in /feel/hifimagnet/ThermoElectricModel/quarter-turn3D.geo/thermoelectric-linear_V1T1_N1/np_2
 .. exports :/feel/hifimagnet/ThermoElectricModel/quarter-turn3D.geo/thermoelectric-linear_V1T1_N1/np_2/exports
 .. logfiles :/feel/hifimagnet/ThermoElectricModel/quarter-turn3D.geo/thermoelectric-linear_V1T1_N1/np_2/logs

It means that the actual simulation results are store in ${LUSTRE}/feel/hifimagnet/ThermoElectricModel/quarter-turn3D.geo/thermoelectric-linear_V1T1_N1/np_2/exports/ensightgold/

The <hpc_basedir>/<hpc_workdir_prefix>_<date>_<id> directory remains present only if you have set <do_not_delete_workdir> to true. The <id> field is a random number …​

This directory holds:

  • *.script

  • slurm-*.out

  • mso*.err

  • mso*.out

and the files coming from the dataset.

From a computer that is allowed to access HPC resources (if not use a VPN client):

  • start a terminal,

  • connect to the HPC resource with ssh -Y <username>@<hpc_host>,

  • run ls -lrt <hpc_basedir> to find the actual name of the working directory which may appears as the last output of this command.

  • start a post-processing GUI:

  • to quit just click on Menu/logout (top left)

For more details on using ParaView or EnSight please check XXXX.

In the remote desktop to get ParaView running:

  • open a terminal,

  • type: module load gcc/6.4.0 openmpi/2.1.1 paraview/5.4.0

For EnSight you would need to have:

  • a valid license stored in you account on the HPC resource

  • access to EnSight singularity image from HiFiMagnet Sregistry private collection.

In the remote desktop to get EnSight running:

  • open a terminal

  • use the following script (see EnSight doc in HiFiMagnet docs).

7. To go further

  • MagnetTools:

  • Preprocessing:

  • Simulations:

    • direct models

    • CRB

  • Postprocessing: