CrossGRID tests, statistics and utilities

CrossGrid is developing a suite of  test software to validate and help to manage the testbed. The software is being developed in the context of task 4.4 Testbed quality assurance. 

Release 1.2.0 alpha
This is the first release of the package and is intended for internal use namely as a reference for the development of further test programs.

Contains:

• Library routines to be used in the development of the test programs.
• A RB statistics package.
  • HTML interface
  • Graphical interface in PHP (available in a separate tarball "crossgridstats" )
• A CE statistics package.
  • HTML interface
  • Graphical interface in PHP (available in a separate tarball "crossgridstats" )
• A RB information index generation program that generates the II file based on a set of MDS site verifications.
• Host check utility checks the installation and configuration of CE and SE systems.
• Downtime of testbed nodes.

There are three distribution packages:

• The "crossgridstats" package that contains the PHP scripts for the statistics graphical web interfaces such as the ones available at the crossgrid Mapcenter web server.
• The "cg-stats" package that contains PERL scripts to produce web tables with statistics totals.
• The "cg-tests" package that contains PERL scripts to perform tests on the crossgrid testbed.

Installation
The software tree assumes that will be installed below /opt/cg hence the top installation directory should be /opt/cg/cg-tests. To install the package just cd into /opt and extract the files with tar:

cd /opt
tar xzvf cg-tests.tgz

    or if using the non graphical statistics distribution:

cd /opt
tar xzvf cg-stats.tgz

If you install the package under other directory then you must also change the environment variables for SH in the configuration file $TOPDIR/etc/env.sh.

.

Usage
These are usage guidelines and examples for the test and statistics programs.

CE_STATS
This is a Computing Element job submission statistics package. It collects information from the gatekeeper log file usually located at /var/log/globus-gatekeeper.log and produces some statistics. CE_STATS is necessary to produce the database used by the PHP graphical interface available in the "crossgridstats" tarball.

Files:
ce_stats.pl	The Perl script that processes the log file and produces HTML or ascii reports.
ce_remote_stats_cron.sh	BASH script to be run from a UI system. Retrieves a remote globus-gatekeeper.log file Invokes ce_stats.pl, produces web pages with: a total summary, by IP address, by user DN, and lost jobs due to CRL expiration. The script also generates  an index page for the generate web pages and a general index by date with links for the current and previous executions of the script. Finally the pages are published into a web server.
ce_remote_stats_cron_all.sh	BASH script to be run from a UI system. Invokes ce_remote_stats_cron.sh for a set of systems defined in the grid_nodes.conf file.
ce_stats_total.pl	Produces a table with the usage summary for a set of gatekeepers.
ce_stats_total.sh	BASH script to be run from a UI system. Invokes ce_stats_total.pl to produce and publish the page into a web server.

 

RB_STATS
This is a Resource Broker job submission statistics package. It collects information from the logging and bookkeeping database inside the RB system and produces some statistics. The script is meant to be run inside the RB.

Files:
rb_stats.pl	The Perl script that processes the LB database and produces HTML or ascii reports.
rb_stats_cron.sh	BASH script to be run from a RB system. Invokes rb_stats.pl and publishes the web page into a web server with an index page containing links by date for the current and previous executions of the script.
rb_stats_20.pl	The Perl script that processes the LB database and produces HTML or ascii reports. For EDG 2.x.
rb_stats_cron_20.sh	BASH script to be run from a RB system. Invokes rb_stats.pl and publishes the web page into a web server with an index page containing links by date for the current and previous executions of the script. For EDG 2.x.

DOWNTIME
This is a Resource Broker job submission statistics package. It collects information from the logging and bookkeeping database inside the RB system and produces some statistics. The script is meant to be run inside the RB.

Files:
mc_downtime.pl	A PErl script that processes Mapcenter historical information to produce a table with the systems downtime.
mc_downtime.sh	BASH script that invokes mc_downtime.pl.

RB_CONF
This is a Resource Broker script that checks the remote MDS GRIS information published by CE and SE systems, when the MDS GRISes aren't correctly configured or behaving badly the script removes the corresponding site from the RB Information Index, the site is automatically added when the problem is corrected. The script is meant to be run inside the RB.

Files:
rb_conf.pl	The Perl script that check the MDS information for issues that might affect the RB.

HOST_CHECK
This is a package to verify the installation and configuration of CE and SE systems. The script is meant to be run from a user interface (UI) it collects information through LDAP searches and by submitting jobs to the remote systems.

Files:
ce_remote_check.sh	The BASH script that performs the checks and produces HTML reports. The scripts contains the CE and SE check functionality. The script behaviour depends on the invocation name.
se_remote_check.sh	A link to the ce_remote_check.sh script.
remote_check_cron_all.sh	A BASH script that checks several nodes in sequence. The node characteristics are obtained from a configuration file. The script also produces and publishes an HTML index page.

LIB
The lib directory contains Perl functions used by the scripts. 

Download
The current version can be downloaded as a compressed tarball:

Files:
RB and CE statistics web graphical interface in PHP. Used in the CrossGrid Mapcenter.	crossgridstats.zip
Statistics non-graphical interface and support for gathering CE statistics:	cg-stats.tar.gz
Tests release:	cg-tests.tar.gz

 

grid.support@lip.pt