Table Of Contents
- Guided Tour of Functionalities
The Youpi web interface is splitted in tabs. Each tab corresponds to a specific task and, this section, will describe the functionnality related to the tab header.
Before diving into Youpi, let’s first introduce to the basic functionnalities of the pipeline.
New in version 0.6.
Ingesting images is a required step before being able to process FITS images within Youpi. During ingestion, Youpi reads information in the FITS header and store (ingest) part of the data into the database. Along with those FITS keywords, important information such as filename, image path or md5 checksum is saved to the database. This way, further image processing with Youpi will use data stored at ingestion step.
Youpi supports generic FITS image ingestion through its ingestion translation table (ITT) feature. Means that almost any FITS image can be ingested as long as you provide a special translation table which is a specific ingestion configuration file.
For the moment, the Youpi software package is bundled with 3 predefined ITTs that allow ingestion of FITS images from the MEGACAM, WIRCAM and VIRCAM instruments. Other intruments can be supported as well by defining an ITT that tells Youpi which FITS keyword it should look for in the FITS image header during ingestion.
The ingestion translation table is a plain text file. Lines starting with an “#” will be treated as comments; blank lines are ignored. Every line of data must be a list of 2 or 3 semicolon-separated fields of the form:
Youpi keyword; Keyword mapping in source image[; keyword mapping for .head]
Here is the megacam.conf file that adds support for MEGACAM image ingestion (and processing) into Youpi:
#--------- Youpi Parameters for MEGACAM Instrument ------------
YRUN; RUNID
YINSTRUMENT; DETECTOR
YTELESCOP; TELESCOP
YDETECTOR; DETECTOR
YOBJECT; OBJECT
YAIRMASS; AIRMASS
YEXPTIME; EXPTIME
YDATEOBS; DATE-OBS
YEQUINOX; EQUINOX
YFILTER; FILTER
YFLAT; IMRED_FF
YMASK; IMRED_MK
YRA; RA_DEG
YDEC; DEC_DEG
This config file is rather simple. There is no third column because those header keywords are supported by all Terapix software packages. We don’t need any keyword remapping since Youpi don’t need to generate any .head file.
Here is the wircam.conf file that adds support for WIRCAM image ingestion (and processing) into Youpi. It’s almost the same as the megacam.conf file except for the YFLAT and YMASK source keyword mappings (second column) that will force Youpi to take the values from the FLATNAME and BPIXNAME FITS keywords, respectively:
#--------- Youpi Parameters for WIRCAM Instrument ------------
YRUN; RUNID
YINSTRUMENT; DETECTOR
YTELESCOP; TELESCOP
YDETECTOR; DETECTOR
YOBJECT; OBJECT
YAIRMASS; AIRMASS
YEXPTIME; EXPTIME
YDATEOBS; DATE-OBS
YEQUINOX; EQUINOX
YFILTER; FILTER
YFLAT; FLATNAME
YMASK; BPIXNAME
YRA; RA_DEG
YDEC; DEC_DEG
Now, let’s have a look at the vircam.conf config file for supporting the VISTA instrument:
#--------- Youpi Parameters for VIRCAM Instrument ------------
YRUN; HIERARCH ESO OBS PROG ID; RUNID
YINSTRUMENT; "VISTA"; INSTRUMENT
YTELESCOP; TELESCOP
YDETECTOR; "VISTA"; DETECTOR
YOBJECT; OBJECT
YAIRMASS; HIERARCH ESO TEL AIRM START; AIRMASS
YEXPTIME; EXPTIME
YDATEOBS; DATE-OBS
YEQUINOX; EQUINOX
YFILTER; HIERARCH ESO INS FILT1 NAME; FILTER
YFLAT; FLATCOR; IMRED_FF
YMASK; "badpix.fits"; IMRED_MK
YRA; RA; RA_DEG
YDEC; DEC; DEC_DEG
With this config file, Youpi is able to read the ESO FITS keywords in VISTA images (using the 2nd column). The keyword mapping provided in the third column allows Youpi to generate .head files compatible with the low-level tools used for image processing.
If you want to add support for a different FITS format header, just edit a new ITT file and save it under <instrument_name>.conf in the $YOUPI_INSTALL_DIR/terapix/lib/itt/conf/ directory. Now run the checksetup command:
$ python manage.py checksetup
from the $YOUPI_INSTALL_DIR/terapix directory. Youpi will find your configuration file and update the database accordingly. That’s it, the ITT config should be available in Youpi from the ingestion page.
New in version 0.6.1.
As explained in the previous section, ingesting any FITS image is possible when using appropriate translation tables (ITTs). Now, in order to be able to process those ingested images, Youpi and the other low-level tools it relies on must be able to communicate properly: low-level tools expect reading images with a mandatory set of FITS keywords in their header.
In order to be able to override or define new FITS keywords without modifying an image header (keeping the precious source image untouched), low-level tools support reading external files with a .head extension. When processing an image, if a file with the same base name and the .head extension is found in the same directory, then the FITS keywords in this .head file will be used to add information to the original image’s header.
Since Youpi’s goal is to allow generic FITS image ingestion and processing, it comes with support for automatic .head file generation. Before processing images, it will generate .head files on-the-fly when needed. This is the purpose of the third column of the ITT file where a new FITS keyword can be defined. It will then be used during .head file generation along with the value of the keyword defined in the second column.
Let’s illustrate this feature with a concrete example applied to a set of FITS keywords. Let’s say that the v20091023_00238.fits file is a VISTA image whose header keywords are among:
HIERARCH ESO INS FILT1 NAME = 'Y'
FLATCOR = 'Y_flat_20091016.fit[16].fits'
...
The translation table for the VIRCAM instrument used for this ingestion has the following parameters:
YFILTER; HIERARCH ESO INS FILT1 NAME; FILTER
YFLAT; FLATCOR; IMRED_FF
YMASK; "badpix.fits"; IMRED_MK
...
During image processing, Youpi will generate a v20091023_00238.head file in the same working directory as the source image with the following content:
FILTER = 'Y'
IMRED_FF = 'Y_flat_20091016.fit[16].fits'
IMRED_MK = 'badpix.fits'
...
END
(Repeated for all HDUs)
Finally, the low-level tools will be able to process that image since all required keywords are supplied.
Note
Youpi will not generate any .head file if the ITT used for ingesting an image does not contain any keyword mapping definition (third column empty). Only lines with a keyword mapping definition will be parsed and added to the final .head file. Also note that the set of FITS keywords is repeated for all extensions (Header Data Units). Finally, for Scamp processings, Youpi generates a .ahead (note the “a” before “head”) file. This is because Scamp reads .ahead files as input and produces final .head files with photometric data.
Once your cluster path is defined in your configuration files, you can easily browse your cluster to find out which directories contain FITS images from the cluster path browser.
It is a real time tree view of your cluster architecture. Powerful and user friendly, here is a graphical way to navigate through folders and have a quick view of the number of contained FITS images. It counts automatically the number of FITS images, and, display it after the name of the folder. No number means no fits image.
Several paths can be selected by clicking on folders from the browser and your selections will appear in the box below. From the selected paths box you can clear all selections you’ve done by clicking the “clear selection” button, or, if you only want to delete one of your selected paths, click on the red cross button ahead of the path.
Before clicking on the “run ingestion” button, let’s see some others features of the ingestion tab process:
A set of options is proposed to add a fine-grain control during ingestion:
For each ingestion done under Youpi, you can specify a flag “validated” or “observed” which will be applied for all ingested images, and let you determine a first step of classification and validation. Flag images as validated means that future ingested images are usable for science issue. Flag images as observed is the opposite, and permit to exclude automatically from future processing.
Fitsverify is integrated to the ingestion at its low level checking, it will verify that the structure of the FITS image is not corrupted and will not affect the futur processing of it.
The image will have the same name but the generated checksum, computed and linked to the image in the youpi database, will be different. Youpi allows to keep data integrity, uniqueness, and tracks of the ingestion process.
Note
Server side, for each image a database entry is created and filled with relevant informations contained in the FITS header (run, instrument, Ra, Dec, observation date, channel...) and others computed on the fly of the ingestion (center of the field, sky footprint, checksum).
It adds more flexibility for searching, classifying, sorting operations which are needed during the image lifetime and through the pipeline.
The condor Setup tab will show in real time your cluster status and let customize basic or advanced policies related to your cluster functionalities. Condor is a specialized workload management system for compute-intensive jobs. Like other full-featured batch systems, Condor provides a job queueing mechanism, scheduling policy, priority scheme, resource monitoring and resource management. Linked to Youpi, it automatically probes your cluster, list all computation nodes(processors) in the pool, and display availability and status for all of them. Youpi also provides high level condor controls on its condor set up interface.
There are 2 different ways to dispatch resources from this interface:
Policies are automatic selections based on the characteristics of each machine of your cluster. It may be interesting to select automatically all machines with available memory greater than 8 GB of RAM (for a scamp processing), or to exclude from the pool, machines with a specified name. So you can divide your cluster and save those sub nodes selection as policies in order to use predefined sub pool for a specific processing.
If the characteristics of your cluster have no secrets for you, and you decide to make selections of nodes manually, you have to use the “Available Cluster Nodes” box and save them as custom selections.
Note
Adding a default policy to a kind of processing will be treated in the Set up Your Preferences part. Adding a special policy to a kind of processing will be treated in the Processing Your Data part.
The preference tab in the Youpi interface is as important as the condor setup tab. It handles global options for youpi and it is user dependant. Each user can set his preferences for processing data, manage his style themes and define his own permissions.
You can define a default processing cart item behaviour, means if you intend to use policies or selections, for each processing plugin of youpi. Once this done, you can opt for the selections and policies you have created in the Condor setup, and affect to plugins.
Youpi authorize to fully custom your data permissions as a user/group UNIX like permissions.You can change the default one (set to 640) or change the visibility for a group or anyone.
To improve readability, Youpi provides themes that can be changed anytime, during your navigation.It makes Youpi interface more accessible, for example concerning visually impaired users.
Tags are keywords or terms associated with or assigned to a piece of information. In the Youpi case, let’s say that this is a metadata (a text field) that you could assigned to one or a set of images to “labelize” them. Then it will help you find your “foo” set or your “bar” group of images.
The tagging tab interface is divided in two parts:
it shows all tags created in Youpi. You can add a new tag by completing the drop down box with:
- a mandatory name (the one which is displayed)
- a comment (not mandatory)
- a style color for text, background and border
Tags can be edited, and shared by changing visibility permissions, like your data. Then restrictions to a group or to the rest of the world, the graphic elements of tags can be set up through this part by double clicking on the tag.
As explained in the image selector section, once your selection is done, youpi has a drag and drop solution that is simply to grab the tag you want to apply to images, drag it on the empty light blue box of the image selector and mark (or unmark ) your selection.
Note
A tag is a selection parameter that you can use in the image selector in order to find all images with a specific tag, for example untag a tagged selection.
For each plugin implemented, as a computing process which could be run on FITS images through Youpi, there are severals steps to observe, represented as tabs as shown in the figure. To process images and for each plugin, you must pass through each tab of the plugin from the left to the right.
The first is the selection of a subset of ingested images in Youpi. So, a powerfull selection tool has been made, which allows to sort, group images by specific fields: the image selector.
Most of plugins require specific input data, as weight maps, flat maps ahead files etc. The select data path tab helps you inputting such files when they don’t appear to be an output product of an other plugin, or simply at the beginning of the pipeline process.
Every plugins need a configuration file to run, and, they all have a default configuration file for generic processing. Youpi provides the default one for each implemented plugin. If you need to finely tune your processing, you can also create your own by editing in place the default one in the configFile editor, or upload it from your computer. Once your tuned file has been saved, select it and you can go to the next tab.
Warning
A plugin configuration file obeys to a formal syntax, changing keyword value would be sufficient ...
Every plugin outputs are written by default in a preseted path:
/processing_output/user/plugin/
The processing output directory is defined at the first step of Youpi configuration. Then youpi completes this path with the Youpi’s user name and the name of the plugin concerned by the processing.You can add one more custom level directory in the output directory tab:
/processing_output/user/plugin/mySpecialProcess/
Each plugin implemented in Youpi has its own input/output data. In a pipeline context, output data from one could be the input of another, and every IN/OUT dependancies between them, are handled. The input data validation tab has its purpose, take care of ouput-input bridges between plugins. when all previous tabs have been treated, the only thing you have to do is click in the input data validation button to see if things are going well.If it’s not, you might assume that previous tabs have not been fully carried or just missed.
If you pass through all tabs of your plugin, it means that you are now about to add your processing to the cart. By clicking on the ADD TO CART button (in the top right of the interface) a message will appears to confirm that. If something is wrong an error message will guide you to the related tab which needed to be fixed.
New in version 0.7.
The processing cart really acts like shopping carts you’re likely to find on commercial websites. Once processing items are created with a suitable image selection along with a configuration file and various options depending on the kind of processing, they must be added to the processing cart which is the last step before submitting them to the cluster.
This is the place to select which items - in which order - you want to submit first. You can also specify the cluster policy to use with the current selection of items you are about to submit.
Since version 0.7, the processing cart has been reworked to offer better UI design and experience, and comes with more functionalities. The remaining material in this section describes the features of version 0.7 or later.
You can access the processing cart page by clicking on the “Processing Cart” link in the upper right corner of the window (also available at the /youpi/cart/ URL). It comes with the following features:
The processing cart’s page is made of two sections or menus. The “Cart items” default menu shows all cart items related to your current user session (Youpi handles sessions using cookies). That means that if you log out, all items currently available on this page will be lost since your user session will be deleted. To address this issue, Youpi provides a “Saved items” menu where can you save items into the database for permanent storage then reload them back later to the current user session at any time in order to submit them to the cluster. Let’s see how they work.
This menu holds the items you recently added from the top-level processing menu. The following figure shows a processing cart with six items of three different kinds: 3 Swarp items, 2 Scamps and 1 QualityFits:
As you can see, cart items are grouped by kind, one item per line:
Every inline item comes with:
Items can be selected individually, one at a time, or they can all be selected at once by clicking on the All link. You can also click on the None link to unselected all selected items, or click on the Inverse link to inverse the current item selection.
Once items are selected, the following actions can be performed:
Runtime options provide a way to associate a custom cluster configuration for the current item selections. User-defined cluster policies and selections are very useful to define a subset of cluster nodes to be used as targets for running the current items you’re about to submit to the cluster.
Clicking on the Runtime options will raise a draggable box. The default behaviour is to use your default Condor configuration setup, but you can also select a custom configuration for the current item selection. In the following example, the MIX_ALL custom policy is selected and will apply to the 2 Swarp items selected. This will ensure that those 2 jobs run on all cluster nodes matching the MIX_ALL policy, which maps to all nodes with hostname matching the mix* wildcard pattern in our private network:
Once the jobs are submitted to the cluster by hitting the Run button, a progress bar will appear and will track item submission progression in real time. Please note that depending on the number of images involved in the process and all the runtime checks made by Youpi, item submission can take a long time. When every selected items have been processed (the progress bar is at 100%), items that are effectively running or queued on the cluster will be highlighted in green:
As you can see, the SWARP2 and SWARP3 items were successfully submitted. Note that their respective checkboxes have been removed to prevent duplicate item submission. They are not part of the processing cart anymore. They remain in the processing cart for convenience only, so you can easily keep track of which items have already been submitted and which not. If your reload the page in your browser, you will see that all successful items have disappeared.
Moreover, the Condor job ID has been appended to the item name. This number can be useful for tracking the job on the Active Monitoring page. In our example, the Swarp jobs got respectively numbers 12761 and 12762.
Sometimes, instead of getting the cluster job ID, you will get a NOP flag next to the item’s name. That means that Youpi did not submit the item to the cluster since your runtime options control panel has the “Do not reprocess already successful processings” option checked. The following example shows that item QF3 - which is a QualityFITS job for 1 image - has already been processed by Youpi, and indicates that there is no need to submit it one more time (since you don’t want to, according to the runtime option):
If any error occured during item submission, the item will be highlighted in red and the Job Error flag will be appended. Just move the mouse cursor on the flag to get the error message in a tooltip. In our example, the cluster seems to be down for maintenance:
Finally, please note that several item flags can be displayed at a time. Suppose you get an error on the first try because the cluster is unavailable at this time. Then you try submitting this item a second time but it appears it already has been processed by Youpi. By forcing the item submission (by unchecking the related option in the Runtime options), the item would finally be submitted to the cluster. In this case, 3 flags will be appended:
The QualityFITS job has been successfully submitted and has ID 12763.
If you want to process items in a different order, just grab the line
... and move it to another place.
The SWARP1 item is now second in the list:
Note that line items are inserted before the target line entry and can only be moved to positions belonging to the same plugin kind.
After having launched jobs, from the processing cart to the Condor management system, Youpi provides a realtime instant view of all proccessed jobs, with related nodes in which jobs are running, the management status, the owner. You will be able to kill your own jobs and assuming that you give data permissions to your team, they could do the same.The interface for live monitoring is decomposed in pages referencing 10 running jobs per pages (pages are represented by the top red numbers above jobs).
The processing history tab is separated for the moment, in 2 parts. The first one is a processing selector under the history tab, and proposes a custom search of processings by plugin: filtering by user, status, plugin and displaying jobs results in the right panel on click on the desired result. Results in red are the one which failed, green are successful.
Once you click on the results you want to display, you can see a full description of it in the right panel. For every plugins some of element results are the same:
The second part is the statistics tab and permit, by browsing your related output directory processing, to resume processing images status. It is a quick way to know if 100% of your images group has been successfully processed.
Every actions, leaded by the user during his session on youpi, have database impacts. It means that, at anytime, youpi could generate from your account, global or very specific reports. The reports engine works with a graphic library, and can generate statistic images including histogramms, plots and many others features that could enhance graphically your data results from the pipeline.