2.2. Guided Tour of Functionalities

The Youpi web interface is splitted in tabs. Each tab corresponds to a specific task and, this section, will describe the functionnality related to the tab header.

Before diving into Youpi, let’s first introduce to the basic functionnalities of the pipeline.

2.2.1. Generic FITS Image Ingestion

New in version 0.6.

Ingesting images is a required step before being able to process FITS images within Youpi. During ingestion, Youpi reads information in the FITS header and store (ingest) part of the data into the database. Along with those FITS keywords, important information such as filename, image path or md5 checksum is saved to the database. This way, further image processing with Youpi will use data stored at ingestion step.

Youpi supports generic FITS image ingestion through its ingestion translation table (ITT) feature. Means that almost any FITS image can be ingested as long as you provide a special translation table which is a specific ingestion configuration file.

For the moment, the Youpi software package is bundled with 3 predefined ITTs that allow ingestion of FITS images from the MEGACAM, WIRCAM and VIRCAM instruments. Other intruments can be supported as well by defining an ITT that tells Youpi which FITS keyword it should look for in the FITS image header during ingestion.

ITT File Format

The ingestion translation table is a plain text file. Lines starting with an “#” will be treated as comments; blank lines are ignored. Every line of data must be a list of 2 or 3 semicolon-separated fields of the form:

Youpi keyword; Keyword mapping in source image[; keyword mapping for .head]

Youpi Keyword

The first column is an internal keyword required by Youpi and always starts with an “Y”. During ingestion, Youpi will look for this keyword to identify data read in the source image’s header by using the second column’s value. The ingestion process will fail if you modify or remove Youpi keywords in the first column.

Source Image Keyword Mapping

The second column must be an uppercase FITS header keyword, as defined in the source image header you plan to ingest. If you don’t want to use a FITS keyword but want to provide a static value, define a custom string value encapsulated by double quotes. The current keyword value will be read in the FITS header and linked to the internal Youpi keyword defined in the first column.

Keyword Mapping for .head

The third (optional) column must be an uppercase FITS header keyword that will be used to generate .head files. Use it if you want to map a source image FITS keyword to another keyword name. Generated .head files will be passed to all low-level tools during processing. This column is especially useful to make processing of mostly any FITS image possible using Youpi and the low-level tools it relies on.

Translation Table for MEGACAM

Here is the megacam.conf file that adds support for MEGACAM image ingestion (and processing) into Youpi:

#--------- Youpi Parameters for MEGACAM Instrument ------------

YRUN;         RUNID
YINSTRUMENT;  DETECTOR
YTELESCOP;    TELESCOP
YDETECTOR;    DETECTOR
YOBJECT;      OBJECT
YAIRMASS;     AIRMASS
YEXPTIME;     EXPTIME
YDATEOBS;     DATE-OBS
YEQUINOX;     EQUINOX
YFILTER;      FILTER
YFLAT;        IMRED_FF
YMASK;        IMRED_MK
YRA;          RA_DEG
YDEC;         DEC_DEG

This config file is rather simple. There is no third column because those header keywords are supported by all Terapix software packages. We don’t need any keyword remapping since Youpi don’t need to generate any .head file.

Translation Table for WIRCAM

Here is the wircam.conf file that adds support for WIRCAM image ingestion (and processing) into Youpi. It’s almost the same as the megacam.conf file except for the YFLAT and YMASK source keyword mappings (second column) that will force Youpi to take the values from the FLATNAME and BPIXNAME FITS keywords, respectively:

#--------- Youpi Parameters for WIRCAM Instrument ------------

YRUN;         RUNID
YINSTRUMENT;  DETECTOR
YTELESCOP;    TELESCOP
YDETECTOR;    DETECTOR
YOBJECT;      OBJECT
YAIRMASS;     AIRMASS
YEXPTIME;     EXPTIME
YDATEOBS;     DATE-OBS
YEQUINOX;     EQUINOX
YFILTER;      FILTER
YFLAT;        FLATNAME
YMASK;        BPIXNAME
YRA;          RA_DEG
YDEC;         DEC_DEG

Translation Table for VIRCAM

Now, let’s have a look at the vircam.conf config file for supporting the VISTA instrument:

#--------- Youpi Parameters for VIRCAM Instrument ------------

YRUN;        HIERARCH ESO OBS PROG ID;      RUNID
YINSTRUMENT; "VISTA";                       INSTRUMENT
YTELESCOP;   TELESCOP
YDETECTOR;   "VISTA";                       DETECTOR
YOBJECT;     OBJECT
YAIRMASS;    HIERARCH ESO TEL AIRM START;   AIRMASS
YEXPTIME;    EXPTIME
YDATEOBS;    DATE-OBS
YEQUINOX;    EQUINOX
YFILTER;     HIERARCH ESO INS FILT1 NAME;   FILTER
YFLAT;       FLATCOR;                       IMRED_FF
YMASK;       "badpix.fits";                 IMRED_MK
YRA;         RA;                            RA_DEG
YDEC;        DEC;                           DEC_DEG

With this config file, Youpi is able to read the ESO FITS keywords in VISTA images (using the 2nd column). The keyword mapping provided in the third column allows Youpi to generate .head files compatible with the low-level tools used for image processing.

Adding a new ITT file

If you want to add support for a different FITS format header, just edit a new ITT file and save it under <instrument_name>.conf in the $YOUPI_INSTALL_DIR/terapix/lib/itt/conf/ directory. Now run the checksetup command:

$ python manage.py checksetup

from the $YOUPI_INSTALL_DIR/terapix directory. Youpi will find your configuration file and update the database accordingly. That’s it, the ITT config should be available in Youpi from the ingestion page.

2.2.2. Automatic .head File Generation

New in version 0.6.1.

As explained in the previous section, ingesting any FITS image is possible when using appropriate translation tables (ITTs). Now, in order to be able to process those ingested images, Youpi and the other low-level tools it relies on must be able to communicate properly: low-level tools expect reading images with a mandatory set of FITS keywords in their header.

In order to be able to override or define new FITS keywords without modifying an image header (keeping the precious source image untouched), low-level tools support reading external files with a .head extension. When processing an image, if a file with the same base name and the .head extension is found in the same directory, then the FITS keywords in this .head file will be used to add information to the original image’s header.

Since Youpi’s goal is to allow generic FITS image ingestion and processing, it comes with support for automatic .head file generation. Before processing images, it will generate .head files on-the-fly when needed. This is the purpose of the third column of the ITT file where a new FITS keyword can be defined. It will then be used during .head file generation along with the value of the keyword defined in the second column.

Let’s illustrate this feature with a concrete example applied to a set of FITS keywords. Let’s say that the v20091023_00238.fits file is a VISTA image whose header keywords are among:

HIERARCH ESO INS FILT1 NAME = 'Y'
FLATCOR = 'Y_flat_20091016.fit[16].fits'
...

The translation table for the VIRCAM instrument used for this ingestion has the following parameters:

YFILTER;     HIERARCH ESO INS FILT1 NAME;   FILTER
YFLAT;       FLATCOR;                       IMRED_FF
YMASK;       "badpix.fits";                 IMRED_MK
...

During image processing, Youpi will generate a v20091023_00238.head file in the same working directory as the source image with the following content:

FILTER = 'Y'
IMRED_FF = 'Y_flat_20091016.fit[16].fits'
IMRED_MK = 'badpix.fits'
...
END
(Repeated for all HDUs)

Finally, the low-level tools will be able to process that image since all required keywords are supplied.

Note

Youpi will not generate any .head file if the ITT used for ingesting an image does not contain any keyword mapping definition (third column empty). Only lines with a keyword mapping definition will be parsed and added to the final .head file. Also note that the set of FITS keywords is repeated for all extensions (Header Data Units). Finally, for Scamp processings, Youpi generates a .ahead (note the “a” before “head”) file. This is because Scamp reads .ahead files as input and produces final .head files with photometric data.

2.2.3. Ingesting Images in Youpi

Once your cluster path is defined in your configuration files, you can easily browse your cluster to find out which directories contain FITS images from the cluster path browser.

It is a real time tree view of your cluster architecture. Powerful and user friendly, here is a graphical way to navigate through folders and have a quick view of the number of contained FITS images. It counts automatically the number of FITS images, and, display it after the name of the folder. No number means no fits image.

Several paths can be selected by clicking on folders from the browser and your selections will appear in the box below. From the selected paths box you can clear all selections you’ve done by clicking the “clear selection” button, or, if you only want to delete one of your selected paths, click on the red cross button ahead of the path.

Before clicking on the “run ingestion” button, let’s see some others features of the ingestion tab process:

• You can give an identification text field for your ingestion in order to distinguish it from others, or to find easily your set of ingested images.
• A log file is generated each time an ingestion has run and finished. This log file can be sent as email at the end of the ingestion, by quoting the check box before running ingestions.
• You can also see previous ingestions in the history tab of the ingestion page, and their relative options.

A set of options is proposed to add a fine-grain control during ingestion:

• Observed vs validated data

For each ingestion done under Youpi, you can specify a flag “validated” or “observed” which will be applied for all ingested images, and let you determine a first step of classification and validation. Flag images as validated means that future ingested images are usable for science issue. Flag images as observed is the opposite, and permit to exclude automatically from future processing.

• Data integrity control

Fitsverify is integrated to the ingestion at its low level checking, it will verify that the structure of the FITS image is not corrupted and will not affect the futur processing of it.

• Youpi allows the possibility to ingest the same image many times

The image will have the same name but the generated checksum, computed and linked to the image in the youpi database, will be different. Youpi allows to keep data integrity, uniqueness, and tracks of the ingestion process.

Note

Server side, for each image a database entry is created and filled with relevant informations contained in the FITS header (run, instrument, Ra, Dec, observation date, channel...) and others computed on the fly of the ingestion (center of the field, sky footprint, checksum).

It adds more flexibility for searching, classifying, sorting operations which are needed during the image lifetime and through the pipeline.

2.2.4. Deal With Condor Setup

The condor Setup tab will show in real time your cluster status and let customize basic or advanced policies related to your cluster functionalities. Condor is a specialized workload management system for compute-intensive jobs. Like other full-featured batch systems, Condor provides a job queueing mechanism, scheduling policy, priority scheme, resource monitoring and resource management. Linked to Youpi, it automatically probes your cluster, list all computation nodes(processors) in the pool, and display availability and status for all of them. Youpi also provides high level condor controls on its condor set up interface.

There are 2 different ways to dispatch resources from this interface:

• Automatic custom by setting policies:

Policies are automatic selections based on the characteristics of each machine of your cluster. It may be interesting to select automatically all machines with available memory greater than 8 GB of RAM (for a scamp processing), or to exclude from the pool, machines with a specified name. So you can divide your cluster and save those sub nodes selection as policies in order to use predefined sub pool for a specific processing.

• Manual custom by setting custom selections:

If the characteristics of your cluster have no secrets for you, and you decide to make selections of nodes manually, you have to use the “Available Cluster Nodes” box and save them as custom selections.

Note

Adding a default policy to a kind of processing will be treated in the Set up Your Preferences part. Adding a special policy to a kind of processing will be treated in the Processing Your Data part.

2.2.5. Preferences Setup

The preference tab in the Youpi interface is as important as the condor setup tab. It handles global options for youpi and it is user dependant. Each user can set his preferences for processing data, manage his style themes and define his own permissions.

• Default processing cart item behaviour

You can define a default processing cart item behaviour, means if you intend to use policies or selections, for each processing plugin of youpi. Once this done, you can opt for the selections and policies you have created in the Condor setup, and affect to plugins.

• Default permissions

Youpi authorize to fully custom your data permissions as a user/group UNIX like permissions.You can change the default one (set to 640) or change the visibility for a group or anyone.

• Themes

To improve readability, Youpi provides themes that can be changed anytime, during your navigation.It makes Youpi interface more accessible, for example concerning visually impaired users.

2.2.6. Organize Data With Tags

Tags are keywords or terms associated with or assigned to a piece of information. In the Youpi case, let’s say that this is a metadata (a text field) that you could assigned to one or a set of images to “labelize” them. Then it will help you find your “foo” set or your “bar” group of images.

The tagging tab interface is divided in two parts:

• Available tags

it shows all tags created in Youpi. You can add a new tag by completing the drop down box with:

• a mandatory name (the one which is displayed)
• a comment (not mandatory)
• a style color for text, background and border

Tags can be edited, and shared by changing visibility permissions, like your data. Then restrictions to a group or to the rest of the world, the graphic elements of tags can be set up through this part by double clicking on the tag.

• Select images

As explained in the image selector section, once your selection is done, youpi has a drag and drop solution that is simply to grab the tag you want to apply to images, drag it on the empty light blue box of the image selector and mark (or unmark ) your selection.

Note

A tag is a selection parameter that you can use in the image selector in order to find all images with a specific tag, for example untag a tagged selection.

2.2.7. Processing Data

For each plugin implemented, as a computing process which could be run on FITS images through Youpi, there are severals steps to observe, represented as tabs as shown in the figure. To process images and for each plugin, you must pass through each tab of the plugin from the left to the right.

The first is the selection of a subset of ingested images in Youpi. So, a powerfull selection tool has been made, which allows to sort, group images by specific fields: the image selector.

2.2.8. The Image Selector

An instance of the image selector is provided for each plugin under the “select images” tab. If you want to select all images in Youpi, observed through “i.MP9701” channel, and among them, select all images contained in the D1 field of the sky, this tool will automatically do the selection for you.

There are two kind of selections:

• Single selection

In single selection you will be able to constuct basics selections of images, based on:

• several FITS header fields and others types of criteria
• on a list of images in plain-text separated by semicolons

You can even import n plain-text files in order to construct n basic selections.

• Batch selection

Batch selection grant to raise multiple selections based on coordinates on the celestial sphere. You can define circles with a center and a radius and each image which have his center field in the circle, will be selected. XML files with a special DTD are the means to do such selections in Youpi.

Here is the DTD:

<batch>
       <selection>
               <label>SEL1</label>
               <alpha>33.1</alpha>
               <delta>-6.3</delta>
               <radius>0.1</radius>
       </selection>
       <selection>
               <label>SEL2</label>
               <alpha>22.0</alpha>
               <delta>-6.3</delta>
               <radius>0.1</radius>
       </selection>
       ...
</batch>

Alpha, delta are the circle coordinates center. Each selection node is a proper one. With this other method of selection, you will have one list of as many selection nodes as contained in the XML file.A sky visualisation of your selections in batch mode, is generated (even the center of each image selected).To see this visualisation, check the corresponding checkbox on the image selector in Batch selection.

Note

For all selections you have done under single or batch mode, it’s possible to save, edit or delete selections from the top left panel of the image selector.

Note

** saved selections matching mode **

2.2.9. Processing Data Considering Plugins

Most of plugins require specific input data, as weight maps, flat maps ahead files etc. The select data path tab helps you inputting such files when they don’t appear to be an output product of an other plugin, or simply at the beginning of the pipeline process.

2.2.9.1. Custom configuration

Every plugins need a configuration file to run, and, they all have a default configuration file for generic processing. Youpi provides the default one for each implemented plugin. If you need to finely tune your processing, you can also create your own by editing in place the default one in the configFile editor, or upload it from your computer. Once your tuned file has been saved, select it and you can go to the next tab.

Warning

A plugin configuration file obeys to a formal syntax, changing keyword value would be sufficient ...

2.2.9.2. Set output directory

Every plugin outputs are written by default in a preseted path:

/processing_output/user/plugin/

The processing output directory is defined at the first step of Youpi configuration. Then youpi completes this path with the Youpi’s user name and the name of the plugin concerned by the processing.You can add one more custom level directory in the output directory tab:

/processing_output/user/plugin/mySpecialProcess/

2.2.9.3. Input versus Output validation

Each plugin implemented in Youpi has its own input/output data. In a pipeline context, output data from one could be the input of another, and every IN/OUT dependancies between them, are handled. The input data validation tab has its purpose, take care of ouput-input bridges between plugins. when all previous tabs have been treated, the only thing you have to do is click in the input data validation button to see if things are going well.If it’s not, you might assume that previous tabs have not been fully carried or just missed.

2.2.9.4. Adding to the cart

If you pass through all tabs of your plugin, it means that you are now about to add your processing to the cart. By clicking on the ADD TO CART button (in the top right of the interface) a message will appears to confirm that. If something is wrong an error message will guide you to the related tab which needed to be fixed.

2.2.10. The Processing Cart

The processing cart comes naturally, after adding to cart one or many processings, elaborated from plugins of the processing your data tab. It represents a visual synthesis of processings, sorted by plugin and ready to be run. A summary of your parameters are shown as a quick view. Then you can decide to delete, save for a later use, or run the corresponding processing as jobs, handled by Condor. The saved items tab will summarize your saved processings with elementary informations, and an add to cart click will push them back to the cart, ready to be computed.

Note

During your logged session, if cookies are activated, you will keep your cart with your items added, even if you continue browsing the Youpi interface and without saving them. But it is recommended to save for later your items in the cart, not to be punish by a simple network disconnection.

Runtime options are about management of jobs through Condor. If you have set a default condor configuration (i.e Condor Setup), or if you intend to provide a custom configuration for a specific process, it can be managed through this dropdown box.

2.2.11. Live Monitoring of Processings

After having launched jobs, from the processing cart to the Condor management system, Youpi provides a realtime instant view of all proccessed jobs, with related nodes in which jobs are running, the management status, the owner. You will be able to kill your own jobs and assuming that you give data permissions to your team, they could do the same.The interface for live monitoring is decomposed in pages referencing 10 running jobs per pages (pages are represented by the top red numbers above jobs).

2.2.12. Processing Results

The processing history tab is separated for the moment, in 2 parts. The first one is a processing selector under the history tab, and proposes a custom search of processings by plugin: filtering by user, status, plugin and displaying jobs results in the right panel on click on the desired result. Results in red are the one which failed, green are successful.

Once you click on the results you want to display, you can see a full description of it in the right panel. For every plugins some of element results are the same:

• Images, as produced png files from the processing, can be viewed by clicking on their displayed thumb version
• permissions that have been defined for this processing
• A set of Condor LOG files(output log, main log, error log) are available in one click
• the running parameters and the related configuration file you set to launch the job are viewable(dropdown box)
• the plugin output information

The second part is the statistics tab and permit, by browsing your related output directory processing, to resume processing images status. It is a quick way to know if 100% of your images group has been successfully processed.

2.2.13. Generating Reports

Every actions, leaded by the user during his session on youpi, have database impacts. It means that, at anytime, youpi could generate from your account, global or very specific reports. The reports engine works with a graphic library, and can generate statistic images including histogramms, plots and many others features that could enhance graphically your data results from the pipeline.