Opal: The Application Wrapper Web Service (Version 1.5.1)

Overview

Opal, the Application Wrapper Web service, provides a way to wrap up existing scientific applications rapidly as Web services, and expose them to various clients. The implementation provides features such as scheduling (e.g. using Condor/SGE via Globus or DRMAA) and security (using GSI-based certificates), Furthermore, the service provides job and data management (by executing every job in a separate working directory), and state management (by storing the service state in a PostgreSQL database). The application developer specifies a configuration for a scientific application and deploys the application as a service following a small sequence of steps. End-users can now access this application remotely using the WSDL of the service.

The WSDL for the service is defined inside wsdl/opal.wsdl. Stub generators provided by Web service toolkits are then used to generate the client and server-side bindings for the services. In our case, the services are implemented in Java using the Apache Axis toolkit, while the clients can be in any language.

The WSDL defines operations to do the following:

Get application metadata (getAppMetadata): This returns metadata about the application. This includes usage information, along with any number of arbitrary application-specific metadata specified as an array of info elements, e.g. description of the various options that are passed to the application binary. Ideally, these elements should be arbitrary XML's (specified by xsd:any tags). However, due to the limitations of certains toolkits in other languages (e.g. ZSI in Python), we specify them as Strings for now. You may embed XML tags inside them using CDATA sections.
Launch job (launchJob): This operation requires the list of arguments as a string, and an array of structures representing the input files. The structure contains the name of the input file and the contents in Base64 encoded binary form. The operation returns a Job ID that can be used to retrieve job status and outputs.
Query job status (queryStatus): This operation expects a Job ID to query the status of a running job. A status code, message, and URL of the working directory for the job are returned.
Get job outputs (getOutputs): This operation returns the outputs from a job that is identified by a Job ID. The output consists of the urls for the standard output and error, and an array of structures representing the output files. The structure contains the name of the output file and the url from where it can be downloaded.
Get output file by name (getOutputAsBase64ByName): This operation returns the contents of an output file as Base64 binary. The input is a data structure that contains the Job ID for a particular job, and the name of the file to be retrieved.
Destroy job (destroy): This operation destroys a running job identified by a Job ID.
Launch blocking job (launchJobBlocking): This operation requires the list of arguments as a string, and an array of structures representing the input files. The structure contains the name of the input file and the contents in Base64 encoded binary form. The operation blocks until the remote execution is complete, and returns job outputs (as described above) as the response. This operation may only be appropriate for jobs that are not long running.

Prerequisites

The source distribution can be used to install both the server and the client sides on Unix and Windows platforms. The prerequisites for the client and server installations are as follows.

Client

Java 1.5.x or higher: Download and install J2SE from http://java.sun.com/j2se/downloads, if you don't have it installed already. Add the Java bin directory to the environment variable PATH. Also set the environment variable JAVA_HOME to point to the top level directory of the JDK installation.

Ant 1.6.1 or higher: Ant is a make-like utility for compiling Java programs. Install Ant from http://ant.apache.org. Set the environment variable ANT_HOME to point to your Ant installation.

Server

Apart from the above packages required for the client, the server also requires the following packages:

Axis 1.2.1: Axis is a SOAP and Web services toolkit for Java. Install the binary or source version of Axis 1.2.1 from http://ws.apache.org/axis/. Alternatively, you could also download a cached version. Henceforth, we will refer to the location of Axis installation as AXIS_HOME.

Tomcat 5.0.30: Tomcat is a servlet container than can be used to host our Web service implementation. Install the binary version of Tomcat 5.0.30 from http://jakarta.apache.org/tomcat/. A cached version may also be downloaded alternatively. Henceforth, we will refer to the location of the Tomcat installation as CATALINA_HOME.

The following packages are optional:

MPI: If you plan to run your application in parallel, you will need a version of MPI. You will also need to make sure that your SSH keys are set up to be able to run MPI jobs without prompting for a password.

Postgres 7.4.6 (or higher): If you plan to persist the state of the service in a database, you will need to install Postgres from http://www.postgresql.org/.

Scheduler Tools: One way to enable access to schedulers is via the Globus Toolkit. The primary requirement is that clients must be able to launch jobs on your resource using GRAM via the Globus gatekeeper. In addition, jobmanager plugins must be available for the particular scheduler that you have installed. In other words, GRAM clients must be able to submit jobs to your scheduler via the Globus gatekeeper. We have used Globus version 3.2.1 for our development and testing, but you may use any other version as long as the above requirements are met.
We have used Condor for serial jobs only, and SGE for serial as well as parallel jobs. Globus jobmanager plugins are available for both of the above, and must be installed if you plan to use either of them.
Another alternative to access schedulers is via the DRMAA API. The Web services can submit jobs to schedulers as long as they support the DRMAA API. We have only tested job submits to SGE via DRMAA.

Installation

Download the source distribution for installation from here, if you don't have the appropriate version already.

Extract the downloaded tarball using the GNU tar utility (or other similar utilities for Windows), as follows:

    tar zxvf opal-ws-1.5.1.tar.gz

This should create a new directory called opal-ws-1.5.1/ where all the sources are expanded. Henceforth, we refer to this directory as OPAL_HOME.

Server

1. Install Axis inside Tomcat by copying over the $AXIS_HOME/webapps/axis directory to $CATALINA_HOME/webapps. On Windows, you can use Windows Explorer to do the same, while on Unix you can execute the following command:

    cp -r $AXIS_HOME/webapps/axis $CATALINA_HOME/webapps

2. Edit $OPAL_HOME/etc/opal.properties to configure the static container properties correctly. It should look something like the following:

    # the base URL for the tomcat installation 
    # this is required since Java can't figure out the IP 
    # address if there are multiple network interfaces
    tomcat.url=http://localhost:8080

    # parallel parameters
    num.procs=1
    mpi.run=/Users/sriramkrishnan/Misc/mpich-1.2.7/bin/mpirun

    # location of working directory relative to $CATALINA_HOME/webapps.
    # this could be a symbolic link to another location (which should be
    # NFS mounted if this is on a cluster). if this is a symlink, copy
    # etc/opal.xml to $CATALINA_HOME/conf/Catalina/localhost/opal.xml. if
    # the name of the symlink is changed to something other than "opal", 
    # modify the opal.xml accordingly
    # working.dir=opal

    # database information
    database.use=false
    database.url=jdbc:postgresql://localhost/app_db
    database.user=app_user
    database.passwd=app_passwd

    # drmma information
    drmaa.use=false
    drmaa.pe=mpich

    # globus information
    globus.use=false
    globus.gatekeeper=localhost:2119/jobmanager-sge
    globus.service_cert=/Users/sriramkrishnan/certs/app_service.cert.pem
    globus.service_privkey=/Users/sriramkrishnan/certs/app_service.privkey

Set the tomcat.url to the correct http://ip-address:port of the server. If the service will wrap a parallel application, set the num.procs to the number of processors available, and the mpi.run to the location of the mpirun on your host.

By default, all new working directories for job executions are created inside the $CATALINA_HOME/webapps/ROOT directory. You may wish to change this to another location on your system (which should be NFS mounted, if you are installing on a cluster). If you would like to do so for any reason, uncomment the property working.dir. Then run the following commands:

    cp $OPAL_HOME/etc/opal.xml $CATALINA_HOME/conf/Catalina/localhost/ 
    cd $CATALINA_HOME/webapps 
    ln -s /path/to/working_dir/on/nfs opal

Note that the above use of symbolic links will only work on Unix systems.

If you wish to install a database to persist service state, please refer to database.html. If you would like to set up scheduler support, refer to scheduler.html. You can also optionally set up secure access to your services by consulting security.html.

3. Edit the $OPAL_HOME/build-opal.properties file to ensure that the build properties are set correctly. Set catalina.home to point to the location of your Tomcat installation (i.e. $CATALINA_HOME), axis.webapp to the name used by the Axis web application (typically, it is just axis), and tomcat.port to the port number that the Tomcat server is running on.

4. Install the Opal jars into the Axis webapp, by executing the following command:

    ant -f build-opal.xml install

5. Run the Tomcat server by changing to the $CATALINA_HOME/bin directory, and running the appropriate command (bat on Windows, sh on Unix):

    ./startup.bat|sh

6. Validate that Axis has been deployed correctly by clicking on http://localhost:8080/axis/happyaxis.jsp. If you have deployed Tomcat on another port, you will have to change the port number above. If all the Needed Components are found, Axis has been deployed fine. You can ignore the warnings about the Optional Components.

7. Create a configuration file for your application. For example, you can download the Babel program, which is designed to support molecular modeling, chemistry, and many related areas, including interconversion of file formats and data. A sample configuration file for the same, located in $OPAL_HOME/etc/babel_config.xml, is shown below.

<appConfig xmlns="http://nbcr.sdsc.edu/opal/types"
           xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <metadata>
    <usage><![CDATA[./babel [-i<input-type>] <name> [-o<output-type>] <name>]]></usage>
    <info xsd:type="xsd:string">
    <![CDATA[
    ...
    Currently supported input types
	alc -- Alchemy file
	prep -- Amber PREP file
	...
    Currently supported output types
    ...
	caccrt -- Cacao Cartesian file
	cacint -- Cacao Internal file
    ...
    Additional options : 
    ...
    ]]>
    </info>
  </metadata>
  <binaryLocation>/Users/sriramkrishnan/bin/babel</binaryLocation>
  <defaultArgs></defaultArgs>
  <parallel>false</parallel>
</appConfig>

The configuration consists of a top level element appConfig, which contains metadata, binaryLocation, defaultArgs, and parallel elements. The metadata consists of usage, which is a string specifying how the application is to be invoked, and an array of optional application information specified as info elements. In the above file, the various options that can be used with the application are specified within the metadata. The binaryLocation specifies the location of the application binary - note that this has to be the location of a single executable. No arguments may be specified here. However, default arguments that need to be used for every run can be specified inside the defaultArgs element. The parallel element specifies if an application is parallel or not. Use the babel_config.xml as a guideline to write configurations for your particular applications.

8. Deploy the services inside Tomcat, by changing to the $OPAL_HOME directory, and running the following command:

    $ANT_HOME/bin/ant -f build-opal.xml deploy -DserviceName=<serviceName> -DappConfig=<appConfig>

For example, to deploy the Babel service, you would type the following command:

    $ANT_HOME/bin/ant -f build-opal.xml deploy -DserviceName=BabelServicePort -DappConfig=etc/babel_config.xml

Note that you can undeploy your service at any time by running the following command:

    $ANT_HOME/bin/ant -f build-opal.xml undeploy -DserviceName=<serviceName>

If you plan on using the service with large inputs and outputs, it is a good idea to increase the heap size being used by the JVM. This can be done by setting the JAVA_OPTS environment variable to -Xmx1024m, and restarting Tomcat. This increases the heap size to 1GB. If all went well until this step, the services are running and ready to be used. Test them by running the client, described in the following section.

The service creates new working directories for every execution. These working directories are not deleted automatically, so they must be periodically cleaned up. You may use the script $OPAL_HOME/etc/cleanup.sh to do so - you will have to modify it to point to your Tomcat installation (and optionally modify the number of days to retain scratch directories).

Client

A generic Java client is implemented by $OPAL_HOME/src/edu/sdsc/nbcr/opal/GenericServiceClient.java. It lets you run any application exposed as an Opal service, and retrieve its status and outputs.

To compile the client, perform the following step from inside the $OPAL_HOME directory.

    $ANT_HOME/bin/ant -f build-opal.xml jar

Before running the client, set your classpath using the classpath.bat|(c)sh script depending on the OS/shell you use - e.g. if you use tcsh on Unix, set your classpath by typing the following command:

    source classpath.csh

To launch a job using the Babel service which was described above, you could use the following command, which displays the resulting job id, along with the preliminary status.

    java edu.sdsc.nbcr.opal.GenericServiceClient -l http://localhost:8080/axis/services/BabelServicePort 
	                                         -r launchJob 
						 -a "-ipdb sample.pdb -h -opdb output.pdb" 
						 -f etc/sample.pdb

You can retrieve job status by running the following command:

    java edu.sdsc.nbcr.opal.GenericServiceClient -l http://localhost:8080/axis/services/BabelServicePort 
	                                         -r queryStatus
						 -j <job_id>

Once the job has finished executing, you can also retrieve output metadata using:

    java edu.sdsc.nbcr.opal.GenericServiceClient -l http://localhost:8080/axis/services/BabelServicePort 
	                                         -r getOutputs
						 -j <job_id>

You may need to change the above URL if you used a different port, or are running the client from another machine. Note that you can get the complete usage information for the client by using the following command:

   java edu.sdsc.nbcr.opal.GenericServiceClient

Feedback

To report bugs and feature requests, please use the NBCR Web Services User Forum (WSUF).