Yu(Carol) Deng CGL Report

Status Report for 04/23/2008 -- 04/30/2008(updating...)

2008-04-30T10:08:00.000-04:00

Status Report for 04/09/2008 -- 04/23/2008(updating...)

2008-04-30T10:07:00.000-04:00

Status Report for 03/26/2008 -- 04/09/2008(updating...)

2008-04-30T10:06:00.000-04:00

Status Report for 03/12/2008 -- 03/26/2008(updating...)

2008-03-31T15:15:00.000-04:00

Status Report for 02/27/2008 -- 03/12/2008

2008-03-25T15:14:00.001-04:00

Progress

Implementation of one user registering and monitoring one sensor WITH displaying history data is done.

1. User interface

Based on the previous version that allows user to register and monitor any one of IUMSC sensors with displaying only most recent data, the current version can display history data and current data in real time.

2. Implementation

The previous version was improved in following aspects:

(1) Server side:

(a) Using ArrayList to store parcel data instead of File

Read parcel (a parcel is in XML format) data into DOM;
Parse the parcel, and get values of SensorName,TimeStamp and DoubleData;
Encapsulate the parcel data to a user-defined Parcel class, and write it to an ArrayList.

Note: this implementation is synchronized.

(b) Using JSON to represent parcel data instead of XML

Definition of Parcel class:

public class Parcel {

public String sensorName;

public String timeStamp;

public String doubleData;

public int parcelID;

public Parcel(String sensorName, String timeStamp, String doubleData, int parcelID) {

this.sensorName = sensorName;

this.timeStamp = timeStamp;

this.doubleData = doubleData;

this.parcelID = parcelID;

}

public JSONObject toJSON() throws JSONException {

JSONObject JObj = new JSONObject();

JObj.put("sensorName", sensorName);

JObj.put("timeStamp", timeStamp);

JObj.put("doubleData", doubleData);

JObj.put("parcelID", parcelID);

return JObj;

}

getparcels.jsp is responsible for:

Handle HTTP request from the client;
Each parcel data(element in the ArrayList) is converted to a JSON object, and added to a JSONArray which is used to hold a JavaScript representation all the parcels we’ll return;
Writes representation to client.

Note: Choosing JSON as means of transferring data in our application[4]

Disadvantage of XML:

We must ensure the XML is sent with the correct header if it is sent from the server. But if the XML is being generated on the fly by a server-side programming language, it is easy to miss this requirement, then responseXML property would be empty;
It often takes quite a few lines of JavaScript just to generate a small chunk of markup even if the DOM is eminently suited to parsing XML once it reaches the browser.

JSON is designed to store data, and it is lightweight alternative to XML. Anything that can be stored in XML can also be stored in JSON. Both are text-based representations of data, but while XML requires opening and closing tags, JSON just uses colons, commas, and curly braces.

JSON isn’t a data format that needs to be interpreted by JavaScript: JSON is JavaScript.

(2) Client side

The client side is responsible for querying the server for parcels and displaying them on a page. We define a ParcelManager class object-oriented JavaScript with Prototype to handle the fetching, caching and displaying of the parcels.

new Ajax.PeriodicalUpdater({failure:'error'}, "fetchparcels.jsp?start=" + start + "&pageSize=" + pageSize,

{

method: "get",

frequency: monitorRate,

decay: 2,

onSuccess: function(response) {

cacheParcelData(response);

drawParcelDIV(start, pageSize, div, false);

on404: function(response) {

alert('Server Status: 404: ' + response.statusText);

onFailure: function(response) {

alert('Server Status: ' + response.status + ' - ' + response.statusText);

}

);

The core part of the client side is Prototype Ajax.PeriodicalUpdater class which schedules the automatic updates without having to code any scheduling logic or writing timeout handlers.[3]

(a) When the client wants to monitor the parcel data, an Ajax.PeriodicalUpdater object is created, and getparcels.jsp is employed to fetch data on the server side;

(b) onSuccess event handler takes resposiblility to cache the parcel data and display them.

cacheParcelData()function is invoked only after we’ve fetched parcels from the server. It gets the JSON text we’ve received from the server and creates a JavaScript array out of it;
drawParcelDIV()function takes the cached parcel data and creates a list in the
that we specify.

Discussion & Future work

1. One user can register and monitor multiple sensors;

The hard part is how to tell which data (parcel) belongs to which registration request when the user is allowed to register the same sensor with different data interval.

2. The problem with the interface of unregistering a sensor

Reference

[1] JSP : the complete reference. Hanna, Phil.

[2] Core servlets and JavaServer pages. Hall, Marty; Brown, Larry

[3] Ajax in practice Crane, Dave.

[4] Bulletproof Ajax, Keith, Jeremy

Status Report for 02/13/2008 -- 02/27/2008

2008-02-29T16:06:00.004-05:00

Progress

Implementation of one user registering and monitoring one sensor is done.

1. User interface

Now user can register and monitor any one of IUMSC sensors via application “CIMA Sensor Monitor” .

(1) Choose a sensor name, which is a CIMA service data source name, e.g “Bay1 Temperature”, from one of twelve IUMSC sensors;

(2) Type a data interval, which is CIMA service data reporting interval in seconds;

(3) Click button “Register”;

(4) After getting response from server “Successfully registered some sensor”, type a Monitor Rate, which is the frequency in seconds the user wants to send the polling requests to the server, and then click button “Monitor”, the most recent data will be shown on the screen continuously.

2. Implementation

(1) Client side[3]

Prototype provides a number of easy ways to make asynchronous requests via Ajax. We are using Ajax.Request class to register a sensor and Ajax.PeriodicalUpdater class to poll a periodical request to get the new real time data.

new Ajax.Request(

"httpcimagateway.jsp?type="+encodeURI(type)+"&"+"source="+encodeURI(source)+"&"+"dataInterval="+encodeURI(dataInterval),

{

method:"get",

onComplete:function(xhr){

document.getElementById('fileUrl').value=xhr.responseText;

}

onFailure: function(xhr){

throw new Error('Registration failed: ' + xhr.statusText);

}

);

The asynchronous request itself is triggered by constructing a new instance of Ajax.Request, passing two parameters: the URL for the request, and a hash object containing properties that specify the options of the request. The URL specifies a JSP file that we will start CIMA service and register a sensor, and it passed the value of parcel type, CIMA service data source and CIMA service data reporting interval. In the options parameters object, we specify the HTTP method as a GET with the method property, and provide function reference for completion and failure handlers with onComplete and onFailure, respectively.

The completion handler is just to get the URL of the file which stores the data of the registed sensor. The failure handler, which is passed a reference to the XHR instance, throws an error depicting the failure status.

new Ajax.PeriodicalUpdater({success: 'details'}, fileUrl,

{

method: 'get',

frequency: monitorRate,

decay: 2,

onSuccess:function(xhr) {

eval(xhr.responseText);

}

);

This class can satisfy our needs to obtain and display data from the server at periodic intervals. First, we create an instance of Ajax.PeriodicalUpdater that refers to a new JSP page in the request URL(Note: for simplicity, currently this URL is just a file containing sensor data and this will be improved in next step). Second, frequency property in the parameters object controls the number of seconds between updates, and decay property controls the rate at which the request interval grows when the response is unchanged.

(2) Server side

httpcimagateway.jsp

a. Handle HTTP request from the client;

b. If the parameter check passes, then do the service in the request, e.g. Register/Unregister/……

c. Return the URL of the file which contains the data

do_Register

Set up the embedded axis server and client listener;
Set up the call to the CIMA service;
Invoke the service

do_Unregister

The procedure is similar with do_Register.

Discussion & Future work

Change the storage of the data from file to queue;
One user can register and monitor multiple sensors;

According to the requirement, users are allowed to

(a) register the different sensors, and/or

(b) register the same sensor with different data interval.

For one user, all registered sensors’ data will come to the same port, so

for (a), we only need to filter different data to different requests based on different sensors’ name, and this can be done easily; But

for (b), the current web service just outputs the “pure” data without any extra information to web service client. So the client cannot tell which data (parcel) belongs to which registration request.

The problem with the interface of unregistering a sensor

Reference

[1] JSP : the complete reference. Hanna, Phil.

[2] Core servlets and JavaServer pages. Hall, Marty; Brown, Larry

[3] Ajax in practice Crane, Dave.

Status Report for 01/30/2008 -- 02/13/2008

2008-02-29T16:04:00.001-05:00

Progress

Based on my previous post, I found that invoking/starting a web service by using Axis1.4 API is not type safe, so I turned my work to newly introduced Axis2 for the project.

1. Brief introduction

Apache Axis2 is built on Apache AXIOM a new high performance, pull-based XML object model. Axis2 comes with many new features, enhancements and industry specification implementations. Axis 2 has five primary strengths: better performance, messaging support, synchronous support, better support for WS-extensions, and better deployment support. The strength of Axis 2 lies in the flexibility and the functionality that Axis 2 provides; migration can be justified for the cases where few of those functionalities are vital. Future articles will provide examples on using Axis 2.

2. A test Web Services via Apache Axis2 in Eclipse

We create a bottom up JAVA bean Web service and Web service client using Axis2 WTP Tools. It shows how to create a simple Web service and Web service client from a JAVA class. The JAVA class in this scenario converts between the Celsius and Fahrenheit temperature scales and its the same class that used in the Axis web services tutorials.

(1) Download the latest Axis2 runtime and extract it;

(2) Point Eclipse WTP to downloaded Axis2 Runtime;

(3) Create a project with the support of Axis2 feature;

(4) Select the Axis2 Web service facet;

(5) Import the wtp/Converter.java class into Axis2WSTest/src (be sure to preserve the package);

package wtp;

public class Converter {

  public float celsiusToFarenheit ( float celsius ){

    return (celsius * 9 / 5) + 32;

  public float farenheitToCelsius ( float farenheit ){

    return (farenheit - 32) * 5 / 9;

(6) Create a web service based on the above java class;

(7) Generate the client for the newly created service by referring the ?wsdl generated by the Axis2 Server;

(8) The Clients stubs will be generated to above Dynamic Web project;

(9) Write Java main program to invoke the client stub. Import the ConverterClient.java file to the workspace into the wtp package in the src folder of the web service client;

package wtp;

import java.rmi.RemoteException;

import org.apache.axis2.AxisFault;

import wtp.ConverterConverterSOAP11Port_httpStub.CelsiusToFarenheit;

import wtp.ConverterConverterSOAP11Port_httpStub.CelsiusToFarenheitResponse;

public class ConverterClient {

  public static void main(String[] args) {

    try {

      float celsiusValue = 100;

      ConverterConverterSOAP11Port_httpStub stub = new ConverterConverterSOAP11Port_httpStub();

      CelsiusToFarenheit c2f = new CelsiusToFarenheit();

      c2f.setCelsius(celsiusValue);

      CelsiusToFarenheitResponse res = stub.celsiusToFarenheit(c2f);

      System.out.println("Celsius : "+celsiusValue+" = "+"Farenheit : "+res.get_return());

    } catch (AxisFault e) {

      e.printStackTrace();

    } catch (RemoteException e) {

      e.printStackTrace();

Discussion

Now I am working on how to start a web service dynamically based on the feather of the project and the existing web service. I will state it in more detail later.

Reference

·         http://ws.apache.org/axis2/

·         http://www.developer.com/services/article.php/3525481

·         http://www.eclipse.org/webtools/jst/components/ws/1.0/tutorials/WebServiceExplorer/WebServiceExplorer.html

·         http://www.eclipse.org/webtools/community/tutorials/BottomUpAxis2WebService/bu_tutorial.html

·         http://www.eclipse.org/webtools/jst/components/ws/1.5/tutorials/BottomUpWebService/BottomUpWebService.html

·         http://www.eclipse.org/webtools/jst/components/ws/1.5/tutorials/TopDownWebService/TopDownWebService.html

·         http://www.eclipse.org/webtools/jst/components/ws/1.5/tutorials/WebServiceClient/WebServiceClient.html

Status Report for 01/23/2008 -- 01/30/2008

2008-01-31T19:21:00.002-05:00

Progress

Based on the discussion in my previous posts, I did some work on Axis 1.4 for the project.

1. Install and Deploy Apache Axis 1.4

(1) Make sure that we have

· J2SE SDK 1.5

· A Servlet Container: Tomcat 5.5.17

(2) Download axis-bin-1.4.zip from http://apache.oc1.mirrors.redwire.net/ws/axis/1_4/

(3) Unzip it and look at the dir. tree. Note that Axis runs as a Servlet.

(4) Deploy Axis.

· Copy webapps/axis tree to webapps directory of Tomcat.

· Alternatively, modify server.xml of Tomcat.

(5) Run Tomcat: issue bin/startup from Tomcat home.

Direcotry Structure:

axis-1_0

docs

lib

samples

webapps

axis

WEB-INF

lib

classes

web.xml

(6) Test the Deployment

(a) Point my browser to http://localhost:8080/axis

(b) Click the link "Validate", an error message and two warnings were shown on the next page.

Error: could not find class javax.activation.DataHandler from file activation.jar

Axis will not work

See http://java.sun.com/products/javabeans/glasgow/jaf.html

Optional Components

Warning: could not find class javax.mail.internet.MimeMessage from file mail.jar

Attachments will not work

See http://java.sun.com/products/javamail/

Warning: could not find class org.apache.xml.security.Init from file xmlsec.jar

XML Security is not supported

See http://xml.apache.org/security/

The core axis libraries are present. 2 optional axis libraries are missing

(7) Shutdown Tomcat, and follow the instructions above to

(a) download the zip files;

(b) unzip these files;

(d) copy them to $CATALINA_HOME/webapps/axis/WEB-INF/lib

activation.jar, mail.jar, xmlsec-1.4.1.jar are in jaf-1_1_1.zip, javamail-1_4_1.zip, xml-security-bin-1_4_1.zip respectively.

(8) Startup Tomcat, and run http://localhost:8080/axis

2. A test Web Services Using Apache Axis

The easiest way is to simply copy any independent Java class into our Axis web application directory and access it via SOAP remotely.

(a) developing the Java class,

(b) deploying it,

(1) Developing a Java class

Try out a Hello World -style class which has one method, called greet() that takes someone's name as a parameter and returns a nice greeting message with that person's name.

(a) Locate the $AXIS_HOME/samples directory;

(b) Create a new subdirectory called "hello";

public class Hello {

public String greet(String a_name) {

return "Nice to meet you, "+a_name;

}

(d) Copy it into $TOMCAT_HOME/webapps/axis and name it Hello.jws

(2) Building a client to access our service.

Develop the client code in the file HelloClient.java, which we will store at $AXIS_HOME/samples/hello. HelloClient.java needs to do the following:

· Connect to the our Hello service (a SOAP endpoint)

· Invoke the service method remotely, marshalling parameters in and out.

· Print out the reply from the service.

Below is code for HelloClient.java that enables this:

package samples.hello;

import org.apache.axis.client.Call;

import org.apache.axis.client.Service;

import org.apache.axis.encoding.XMLType;

import org.apache.axis.utils.Options;

import javax.xml.rpc.ParameterMode;

public class HelloClient

{

public static void main(String [] args) throws Exception {

Options options = new Options(args);

String endpoint = "http://localhost:" + options.getPort() + "/axis/Hello.jws";

args = options.getRemainingArgs();

if (args == null || args.length != 1) {

System.err.println("Usage: Hello ");

return;

}

Service service = new Service();

Call call = (Call) service.createCall();

call.setTargetEndpointAddress(new java.net.URL(endpoint));

call.setOperationName("greet");

call.addParameter("name", XMLType.XSD_STRING, ParameterMode.IN);

call.setReturnType(XMLType.XSD_STRING);

String msg = (String)call.invoke(new Object[] {args[0]});

System.out.println("Reply: " + msg);

}

Compile this client: javac samples/hello/HelloClient.java

Run the client: java samples.hello.HelloClient -p8080 Greg

which returns the message:

Nice to meet you, Greg!

Discussion

Based on the previous discussion, it is necessary to remove SimpleAxisServer and just use Axis as an engine to invoke the web service.

http://ws.apache.org/axis/java/apiDocs/org/apache/axis/transport/http/SimpleAxisServer.html

Reference

·         http://www.javaranch.com/journal/2002/05/axis.html
http://ws.apache.org/axis/java/user-guide.html#CustomDeploymentIntroducingWS

Status Report for 01/09/2008 -- 01/23/2008

2008-01-31T19:18:00.000-05:00

Progress

1. Design for server side (JSP/Java servlet)

Based on the discussion on my previous blog, remove the embedded Jetty and make everything run in Tomcat.

On server side:

(1) JSP/servlet gets the parameters in the request from client;

(2) Start SimpleAxisServer(SAS) to call a CIMA web service;

(3) Receive the data at a listener of SAS;

(4) Write the data to a file;

(5) Send the URL of file back to client

On client side:

(1) Send the request to server;

(2) Receive the URL of the file;

(3) Parse the file content and display them on the browser

2. Implementation

(1) Server side

(a) Using Java servlet

Override protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException

in class

public class Main extends javax.servlet.http.HttpServlet implements javax.servlet.Servlet

(b) Using JSP

In a JSP page,

Get the parameters from request;
Call a Java method to invoke the CIMA web service and another method to write data to a file;
Return the URL of the file to client

(2) Client side

In Ajax, make a request either to the Java servlet or the JSP. Parse the file content and show the data on the browser via an Ajax function.

Discussion

(1) Comparison between Java servlets and JSP in this project.

Both Java servlets and JSP have an extensive infrastructure for automatically parsing and decoding HTML form data, reading and setting HTTP headers, handling cookies, tracking sessions, and many other such high-level utilities. But as for outputting the result to the client, we need to write a zillion println statements to generate HTML. Plus, by separating the presentation from the content, we can separate the tasks on Web page design and servlet programming. In this project, it involves some static HTML content, and the client only needs the URL of the file, so use of a combination of JSP and servlets would simplifies the creation and maintenance of the HTML.

(2) Currently, we invoke CIMA web service by start a SimpleAxisServer, but in SimpleAxisServer API, there is a note -- This is a simple implementation of an HTTP server for processing SOAP requests via Apache's xml-axis. This is not intended for production use. Its intended uses are for demos, debugging, and performance profiling. Note this classes uses static objects to provide a thread pool, so you should not use multiple instances of this class in the same JVM/classloader unless you want bad things to happen at shutdown.

So it is necessary to remove SimpleAxisServer and just use Axis as an engine to invoke the web service

http://ws.apache.org/axis/java/apiDocs/org/apache/axis/transport/http/SimpleAxisServer.html

Reference

JSP : the complete reference. Hanna, Phil.
Core servlets and JavaServer pages. Hall, Marty; Brown, Larry

Status Report for 12/26/2007 -- 01/09/2008

2008-01-31T12:58:00.000-05:00

Progress

1. Design for server side (Embedded Jetty)

In this design, we embedded Jetty into our Java applications and used it as the core HttpServer, an HTTP server with the ability to serve static content and servlets.

The basic idea is to implement a handler in the servlet and start a jetty server with it. Once the jetty server gets a request from the jetty port, the handler will get the parameters and start SimpleAxisServer(SAS) to call a CIMA web service and then get the data at a listener of SAS.

2. Implementation

(1) Create an instance of the Jetty server (org.mortbay.jetty.Server) listening at some port, but does not start the server.

Server server = new Server();

Connector connector=new SelectChannelConnector();

connector.setPort(some port);

server.setConnectors(new connector[]{connector});

(2) Add resource handlers which will get the parameters from the client’s request and start the web service.

Handler param = new ParamHandler();
Handler rqst = new RqstHandler();
HandlerCollection handlers = new HandlerCollection();
handlers.setHandlers(new Handler[]{rqst, param});

server.setHandler(handlers);

(3) Start the Jetty server.

server.start();

server.join();

(4) Set up the embedded axis server with deployment descriptor generated by WSDL2Java

SimpleAxisServer server = new SimpleAxisServer();

ServerSocket WSserverSocket = new ServerSocket(someport);

WSserverSocketPort = WSserverSocket.getLocalPort();

server.setServerSocket(WSserverSocket);

server.setMyConfig(wsdddoc.getDeployment());

server.start();

set up the call to the CIMA service

String endpointURL = new String(some IP address);
Call call = new Call(endpointURL);
call.setTargetEndpointAddress( new java.net.URL(endpointURL) );
call.setOperationName(somename);
call.addParameter(……);

call.invokeOneWay(……);

Discussion

On the client side, Ajax is being used for this web application. In a standard way, making a request to a servlet/JSP page on the server in Tomcat would be much easier and convenient. So instead of using embedded Jetty, we can just use servlet/JSP page to get the parameter and call a web service and send the response to client. In that case, everything on the server is running in the Tomcat, and just one port is necessary.

Status Report for 11/28/2007 -- 12/26/2007

2007-12-30T13:39:00.000-05:00

Progress

1. Design for server side (updated)

To do it step by step, we first design the client side for one user who can register/monitor one variable/sensor.

The basic idea is that write new data to an XML file when it is available one the server side and return the file’s URL to the client.

2. Implementation

At current stage, logic in server side is not difficult. A program written by JSP, which is called by client, would do the following steps:

Register a variable based on the parameters from client
Write the new data to an XML file
Return the URL of the file to the client

Discussion

1. Register multiple sensors for one user

The user first registers variable called A-var, the server can get the new coming data from some callback function. After that, he wants to register another variable B-var, is that true that we can obtain both data from the same function? There is a problem in register/unregister.

2. Register multiple sensors for multiple users

If multiple users register the same variable, which is better, the arrary or file? How to maintain the session?

Reference

Bulletproof Ajax, Keith, Jeremy
Ajax in practice, Crane, Dave
JSP : the complete reference, Hanna, Phil
JavaScript bible, Goodman, Danny
Ajax for web application developers, Hadlock, Kris
Ajax design patterns, Mahemoff, Michael
Ajax : creating Web pages with asynchronous JavaScript and XML, Woychowsky, Edmond
Ajax bible, Holzner, Steven

Status Report for 10/31/2007 -- 11/28/2007

2007-11-30T09:45:00.000-05:00

Progress

1. Design for Ajax-based client side (updated)

To do it step by step, we first design the client side for one user who can register/monitor one variable/sensor.

The basic idea is that fetch an XML file when it is available one the server side, parse its contents, and generate markup to display the content details in the Web page.

2. Implementation

In the previous implementation, we set up the communication between client and server by calling the basic functions provided by JavaScript.

(1) Creating a new instance of an XMLHttpRequest object (xhr = new XMLHttpRequest()) is used to make a Http request from client to server. In most browsers, it is very straight-forward, but in IE, we need to create a new instance of an ActiveX object. If xhr equals to false, it means that the browser executing the script does not have Ajax Capabilities. Otherwise, you will get an instance of XMLHttpRequest or ActiveXObject.

In addition, in the callback function which is an attched function to the onreadystatchange event handler, we need to check the status properly to make sure that the response was successfully returned from the server before we process the data from the server.

In a word, we have to take care of a lot of details if we do it from sctatch. So there are a few of open source toolkits that can make our lives a bit easier.

In Ajax.Request class provided by Prototype, all we have to provide is a URL, the HTTP method, and a call back function. Internally, Ajax.Request works out how to create the XHR object and fills in the blanks when shaping the request. It also simplifies the callback semantics considerably, allowing us to supply a funtion that would only be called once on completion of the response, and that therefore only have to deal with the application logic.

(2) In this application, we wish to obtain and display data from the server when they are available, so we need to keep the data constantly up-to-date. The Prototype Ajax.PeriodicalUpdater class can automatically keep the display of the server-provided data up-to-date at a frequency that best suits our page. So it is easy for us to schedule the automatic updates without having to code any scheduling logic or write timeout handlers.

(3) For simplicity, currently on the server side we just write the new data to a file while the client side keeps polling the request to read the file and process it.

Discussion

In using Prototype Ajax.PeriodicalUpdater class, we have to take the following factors into account :

Choosing what to automatically update and at what interval
The expected number of simultaneous visitors
The size of the update response
The process required to generate that response
The load-handling capability of the server configuration

Reference

Bulletproof Ajax, Keith, Jeremy
Ajax in practice, Crane, Dave
JSP : the complete reference, Hanna, Phil
JavaScript bible, Goodman, Danny
Ajax for web application developers, Hadlock, Kris
Ajax design patterns, Mahemoff, Michael
Ajax : creating Web pages with asynchronous JavaScript and XML, Woychowsky, Edmond
Ajax bible, Holzner, Steven

Status Report for 10/17/2007 -- 10/31/2007

2007-10-31T11:45:00.000-04:00

Progress

Prototype architecture for new CIMAClient project

1. Main Frame

2. Implementation of some parts

On server side, main steps for (1), (2) and (3) shown in above figure:

(1) Start the server;

(2) Start handling CIMA http/REST requests

interact with a CIMA service
get data from a handle (registration)
Status information about gateway

(3) Get stream data in "myHandler" method in src/edu/indiana/extreme/www/cima/channel/sink/Channel_sinkImpl.java

On client side, main steps for (4) shown in above figure:

(1) Create a new instance of XMLHttpRequest;

(2) If successful, attach a function to the onreadystatchange event handler, that function will be executed EVERY TIME the server pings the client with an update. The function will be executed every time the readystatechange event is triggered;

In this function,

(a) Check the status properly to make sure that the response was successfully returned from the server;

(b) Extract data from parcel(an XML file);

(d) Create the text for SensorName, TimeStamp and DoubleData to go inside the paragraph;(e) Join these nodes together, and insert the markup just created;

(3) Send the request;

Reference

Bulletproof Ajax, Keith, Jeremy
Ajax in practice, Crane, Dave
JSP : the complete reference, Hanna, Phil
JavaScript bible, Goodman, Danny
Ajax for web application developers, Hadlock, Kris
Ajax design patterns, Mahemoff, Michael
Ajax : creating Web pages with asynchronous JavaScript and XML, Woychowsky, Edmond
Ajax bible, Holzner, Steven

Status Report for 09/12/2007 -- 10/17/2007

2007-10-31T11:44:00.000-04:00

Progress

Prototype architecture for Ajax-based CIMAClient project

Summary of the comparison with different main frames

Figure 1

Figure 1 shows the traditional model of the web application. For the CIMA web services architecture, once the CIMA sensors have been registered to the client machine, the data stream will be kept sending to the web service in CIMA Client, but the data cannot reach client machine automatically. So the client needs to send the HTTP request to retrieve the data manually. In this case, the method of sending data with respect to the user’s actions and the method of sending data with respect to the browser’s actions are synchronous, so we cannot get the real time data. On the other hand, the server will send back the entire page. Rinse and repeat.

Figure 2

Figure 2 shows the Ajax model based on polling. For the CIMA web services architecture, first the client needs to send register request to ask the CIMA web service to send the variable data just registered to the web service in CIMA Client, then a periodic polling will be done by the Ajax layer, and the client can get the updated data. In this model, the method of sending data with respect to the user’s actions is asynchronous while the method of sending data with respect to the browser’s actions is still synchronous. The advantage is that user interface is never blocked and the bandwidth usage is potentially lower than the traditional model even if it is still high.

In our research work, we found that there were some “smarter” Ajax models which have the features of high update frequency, true real time, low bandwidth usage and low load on the network infrastructure. For example, Ajax-Comet model based on smart polling which is asynchronous polling and the variable polling frequency is controlled by server; another example is Ajax-Comet model based on streaming which implements fully asynchronous push. We might employ all the Ajax models to our application and find out which is the best according to our data features.

Status Report for 08/29/2007 -- 09/12/2007

2007-09-16T14:06:00.000-04:00

Progress

Prototype architecture for new CIMAClient project

1. Main Frame

2. Requirements

Get data from CIMA instruments/CIMA web services, and show them on an Ajax-based browser. The main difference between this project and the Atom Feed for IUMSC is as follows:

(1) In “Atom Feed” project, the users can browse the data they requested at anytime at anywhere as long as they have a standard Atom Feed Reader and the URL for the feed(s), and CIMAClientPlatform acts as an adaptor. Since we cannot set the frequency of incoming data in almost all Atom Feed reader at browser (as far as I know), the main frame is called “push model”. So it is not a real-time monitor, and there might have some data missed;

(2) In this project, the users need to monitor near real-time data, so they need to send http requests at some rate to get data instead of getting date passively. So the main frame is called "pull model". Additionally, Ajex can make user interface more friendly and fancy.

3. Work Flow

(1) Startup CIMAClientPlatform to send register requests for all variables/sensors to CIMA web service no matter if the users need them or not;

(2) CIMA web service will send stream data() back including all the registered data;

(3) Users send Http requests to Web service interface at anytime. A http request includes session ID, sensor data and how many data the user-side wants to get(the most recent one or all);

(4) Web service interface sends the data back to the browser.

4. Problems

(1) It is supposed to be used by multiple users, so is there any limitation for the number of users because we cannot assume the data column is small enough for server to process ( The user might request some ccd image and the transfer rate is in the range of 4K to 8M/sec);

(2) The problem in garbage collection in Javascript;

(3) Data format:

The format of return data should be XML or JSON or both. Ajax-based client application can accept JSON much easier than XML.

Discussion

In “Problems” section

Status Report for 08/08/2007 -- 08/29/2007

2007-08-31T14:23:00.000-04:00

Progress

Draft of prototype architecture for new project

1. Main Frame

2. Requirements

Get data from CIMA instruments/CIMA web services, and show them on an Ajax-based browser.

(Difference between this with Atom Feed for IUMS? any special requirements or restrictions for that?)

3. Work Flow

(1) Users send HTTP requests to register a variable(Bay1Temp, for example) to be shown in the browser(for example, http://shim.addr/register?Name=[cima_service]%Source=[variable_name])

(2) Users get a reply with the user ID(for example, http://shim.addr/UUID). This UUID will be used to identify different users and get data from the web service;

(3) Web service interface will create a thread for that request/user;

(4) Web service interface(or The thread?) then send the register request to CIMA web service (with UUID?)(Socket);

(5) CIMA web service will return stream data() back(Socket);

(6) The parcel will be stored in a ring buffer in SVC(???);

(7) Users send Poll URL to Web service interface(or The thread?) (when will it happen?);

(8) Web service interface(or The thread?) sends the data back to the browser.

4. Problems

(1) It is supposed to be used by multiple users, so is there any limitation for the number of users because we cannot assume the data column is small enough for server to process( The user might request some ccd image and the transfer rate is in the range of 4K to 8M/sec);

(2) Compare the following two cases in how to distinguish clients:

(a) Create one thread for every registration;

(b) One service with IDs to distinguish clients;

- Distinguish which parcel goes to which client

- Overhead matching IDs to parcels(?)

(3) Data format:

The format of return data should be XML or JSON or both. Ajax-based client application can accept JSON much easier then XML.

Discussion

In “Problems” section

Status Report for 08/01/2007 -- 08/08/2007

2007-08-09T12:56:00.000-04:00

Progress

Summary of data structure design for IUMSC 12 sensors Atom Feed

1. Requirements

Create Atom Feed for IUMSC 12 sensors

All sensors’ data in one feed
Each sensor data in one feed
Out-of-Range data from all sensors in one feed

2. Work Flow

Implementation is in Method void myHandler(String parcel) in Class Channel_sinkImpl in Package edu.indiana.extreme.www.cima.channel.sink.

(1) Read and parse the parcel and get SensorName, TimeStamp and DoubleData for the incoming sensor data;

(2) Find the SensorStorage for this parcel and update the according SensorStorage with the current parcel;

(3) Update the all-in-one SensorStorage with the current parcel;

(4) Update the SensorStorage for out-of-range data with the current parcel;

3. Data Structure Design

Super Class

abstract class CachedSensorStorage {

private Feed createDefaultFeed; // Create "feed" part of a feed -- Author, Id, Title, Link

private Entry createDefaultEntry // Create "entry" part of a feed -- Author, Title, Id, Updated

protected abstract void updateCachedEntryContent;

protected abstract String generateContentString;

public boolean updateData {

Call Method createDefaultFeed to create basic feed part

Call Method createDefaultEntry to add entry to the feed;

Call Method updateCachedEntryContent which is synchronized to update/append content based on the incoming parcel in cached Entry Content

Call Method FeedDoc.writeFeedDoc provided by AtomSphere to write feed. The method is synchronized too.

}

Sub Classes

(1) class IndividualCachedSensorStorage extends CachedSensorStorage {

ArrayList cachedEntryContent; // An In-Memory data structure to store latest parcel;

protected String generateContentString; // Generate the parcel content string from the cachedEntryContent. It overrides the method in its superclass.

protected void updateCachedEntryContent // Update data in the In-Memory data structure based on incoming parcel

}

(2) class AllInOneCachedSensorStorage extends CachedSensorStorage;

(3) class AbnormalCachedSensorStorage extends CachedSensorStorage;

Class IndividualCachedSensorStorage, Class AllInOneCachedSensorStorage and AbnormalCachedSensorStorage extend their super class and override two methods generateContentString and updateCachedEntryContent based on their different ways to process the In-Memory data.

4. URLs for all Atom Feeds

For all sensors' data :

http://hagar.cs.indiana.edu:8181/IUMSCAtomFeed.xml

For each sensor's data :

http://hagar.cs.indiana.edu:8181/IUMSCAtomFeed_Bay1Humid.xml

http://hagar.cs.indiana.edu:8181/IUMSCAtomFeed_Bay1Temp.xml

http://hagar.cs.indiana.edu:8181/IUMSCAtomFeed_CampCWInTemp.xml

http://hagar.cs.indiana.edu:8181/IUMSCAtomFeed_CrystalTemp.xml

http://hagar.cs.indiana.edu:8181/IUMSCAtomFeed_DEHumid.xml

http://hagar.cs.indiana.edu:8181/IUMSCAtomFeed_DETemp.xml

http://hagar.cs.indiana.edu:8181/IUMSCAtomFeed_FrameBuffer_D_Usage.xml

http://hagar.cs.indiana.edu:8181/IUMSCAtomFeed_FrameBuffer_E_Usage.xml

http://hagar.cs.indiana.edu:8181/IUMSCAtomFeed_FrameBuffer_F_Usage.xml

http://hagar.cs.indiana.edu:8181/IUMSCAtomFeed_LabCWInTemp.xml

http://hagar.cs.indiana.edu:8181/IUMSCAtomFeed_LabCWOutTemp.xml

http://hagar.cs.indiana.edu:8181/IUMSCAtomFeed_LN2Levl.xml

For abnormal data from all sensors:

http://hagar.cs.indiana.edu:8181/IUMSCAtomFeed_Abnormal.xml

Future work

1. SSO;

2. Netvibes Widgets.

Status Report for 07/18/2007 -- 08/01/2007

2007-07-31T16:20:00.000-04:00

Progress

Summary of problems and solutions in Multithreaded Programming for IUMSC 12 sensors Atom Feed

1. Exception in thread occurred for the first time

Background & Problem

CIMA Client can receive parcels from all 12 sensors. When a parcel arrives, the in-memory cache which stores parcels for each sensor will be updated. Then the content of cache will be written to a Feed.

After CIMA Client ran for about 15 to 20 minutes, there was an exception in Thread and no parcel came afterward.

The error message is shown below:

Exception in thread "Thread-3" java.lang.IllegalStateException:

IllegalStateException:

at org.apache.axis.components.threadpool.ThreadPool.addWorker(ThreadPool.java:103)

at org.apache.axis.transport.http.SimpleAxisServer.run(SimpleAxisServer.java:243)

at java.lang.Thread.run(Thread.java:595)

Solution

(1) It seems that the program ran out of threads. It was addWorker that throws this when the thread pool runs out of threads.

(2) Check that the default maximum thread count (org.apache.axis.components.threadpool.ThreadPool, public static final int DEFAULT_MAX_THREADS) is set to 100 when the instance of ThreadPool is created.

(3) Change SimpleAxisServer() to SimpleAxisServer(int maxPoolSize, int

maxSessions). Both of the constructor parameters default to 100. Now we make them 1000 instead.

2. Random occurred exception in thread

Problem

No thread exceptions occurred for 1 hour after changes described above, but the problems were found by observing the output window in eclipse. That is, if only a few parcels came at the same time, the program had enough time to write them to the Feed and receive new incoming parcels. But sometimes many parcels came at the same time, and parcels should be written to the Feed one by one, thus during this period the program could not receive any parcel and it seemed the program had been dead since then.

Restart the CIMA Client, exception in thread arose randomly.

Solution – Step 1

Variable “maxPoolSize” and “maxSessions” were kept to 1000 and 1000, and set a "threshold" in myHandler() to 990, which meant that once there were 990 or more parcels were waiting for being processed the program would skip the latest one. Doing so was aiming to relieve the possible threads burst. As a result, there was no exception in thread for a longer time, but unfortunately the threadpool was full eventurally.

Solution – Step 2

Removed the writing to Feed part from myHandler(), and added/activated a thread which was just responsible for writing parcels to Feed. The result was that the speed of getting parcels and writing them to a remote disk did not match forever no matter how large the buffer we set and no matter whether the thread was added or not. So the burst of threadpool occurred at last.

Solution – Step 3

In myHandler(), the "writing to feed" part was commented and set variable maxPoolSize and maxSessions to only 5 and 5, and set a "threshold" in myHanlder() to 0. The result was that and it worked very well -- it could receive all the parcels and had never been dead. Then I wrote the Feed to /tmp/foo.xml(local disk, local file), and set variable maxPoolSize and maxSessions still to 5 and 5, and it still worked very well!

So I drew a conclusion that the bottleneck might be the "writing to feed" part which was not local operation and the speed of writing to feed did not match the speed of receiving the parcels, so resulted in the burst of the threadpool.

Solution – Step 4

Wrote the Feed to local disk and set a symbolic link to the remote one. – It worked very well and never had been dead for 3 days( Since I need to add one more functionality to CIMA Client, so I stopped it).

Discussion

1. Functionalities for users to select different sensors and/or just show the sensors with abnormal data.

Future work

1. Get sensor values based on the user’s requirement;

2. Show only values not in normal range;

3. The issues described in Discussion section

Status Report for 07/11/2007 -- 07/18/2007

2007-07-20T17:01:00.000-04:00

Progress

Some problems and solutions in detail design for IUMSC 12 sensors Atom Feed

1. Synchronization in writing data to Atom Feed

Problem

It is possible that data from 12 sensors could come simultaneously, but each sensor data should be written to the Feed sequentially. In that case, if the data is being written to Feed, the other data should not be written to the Feed. On the other hand, the incoming data is updated in memory first, then written to the Feed. So the updating should be synchronized too.

Solution

Put the Feed updating statements in a synchronized block and this synchronized block should be as small as possible.

2. Exception in thread

Problem

After CIMA Client ran for about 15 to 20 minutes, there was an exception in Thread and no parcel comes afterward.

Error message is shown below:

Exception in thread "Thread-3" java.lang.IllegalStateException:

IllegalStateException:

at org.apache.axis.components.threadpool.ThreadPool.addWorker(ThreadPool.java:103)

at org.apache.axis.transport.http.SimpleAxisServer.run(SimpleAxisServer.java:243)

at java.lang.Thread.run(Thread.java:595)

Solution

(1) It seems that the program ran out of threads. It was addWorker that throws this when the thread pool runs out of threads.

(3) Change SimpleAxisServer() to SimpleAxisServer(int maxPoolSize, int

maxSessions). Both of the constructor parameters default to 100. Now we make them 1000 instead.

3. A problem caused by Synchronization

Problem

It has been no thread exceptions for 1 hour, but I found another problem by observing the output window in eclipse. That is, if only a few sensor data come at the same time, the program has enough time to write the data to the Feed and receive new incoming data after that. If sometimes many sensor data come at the same time, the program needs to write these data one by one, during this period the program cannot receive any parcel and it seems the program has been dead for twenty or more minutes.

Possible solution

Instead of the critical sections using "synchronized" around the whole feed writing process we need some other way to serialize the writing of the feed. Maybe we should add a queue data structure into which each change is added, then the writer reads everything available in the queue, writes the feed, and reads the next batch of changes. This would limit the critical section to updating and reading an in-memory data structure rather than reading and writing a file.

Discussion

1. Functionalities for users to select different sensors and/or just show the sensors with abnormal data.

Future work

1. Get sensor values based on the user’s requirement;

2. Show only values not in normal range;

3. The issues described in Discussion section

Status Report for 05/16/2007 -- 05/30/2007

2007-05-30T15:27:00.000-04:00

Progress

Detail design for IUMSC 12 sensors Atom Feed

1. Register IUMSC 12 sensors

Before reading all the sensor values from the real proxy box in the IUMSC lab, send requests to register all the sensors with CIMA service endpoint (http://129.79.85.21:7710). Sensor names and meanings are shown below.

LabCWOutTemp Lab Chill water OUT
LabCWInTemp Lab Chill water IN
CampCWInTemp Campus Chill water IN
LN2Levl Liquid Nitrogen level indicator
Bay1Temp Bay1 Temperature
Bay1Humid Bay1 Humidity
DETemp Diffractometer Enclosure Temperature
DEHumid Diffractometer Enclosure Humidity
CrystalTemp Crystal Temperature
FrameBuffer_D_Usage Disk Usage Disk D
FrameBuffer_E_Usage Disk Usage Disk E
FrameBuffer_F_Usage Disk Usage Disk F

Note: The sensor value used before was from test-proxy box (CIMA service endpoint (http://129.79.85.20:7710)) with only two probes (Bay1Temp & LabCWOutTemp) connected to it.

2. Write data to the feed

Part1: Read the parcel into DOM

Get a Parser and parse the parcel

Part2: Update/Append feed

If no feed exists, create the "feed" part of the feed including the following elements: Author, Id, Title and Link and Updated;

Otherwise add an Entry to the feed

If no entry in the feed, create basic elements for the entry including the following elements : Author, Title, Id and Updated

Otherwise get the existing entry handle(there should be only one entry) in the feed

Get the current content in the Content

Set the receiving data to the Content

3. Read feed

Open http://www.netvibes.com/, add the URL of IUMSC Atom Feed(http://hagar.cs.indiana.edu:8181//IUMSCAtomFeed.xml) to netvibes

4. IUMSC Atom Feed in netvibes

The screen shot below shows the Atom Feed in netvibes.

Discussion & Questions

1. The data from 12 sensors come in at different times, that is to say, sometimes nothing comes during 10 to 15 seconds while sometimes data from some sensors come almost at the same time. This might cause that some data might be left out

-- I need to provide file locking function by using the FileLock class before updating, then unlock. Or make the routine that writes the feed "synchronized" or put the feed updating statements in a synchronized block(not as desirable as a synchronized routine).

2. In netvibes, after refreshing the current webpage, new data don’t come, only the old data were shown.

-- I kept clicking the fresh button in the feed area, but it did not work

Future work

1. Get sensor values based on the user’s requirement;

2. Show only values not in normal range;

3. Adjust feed format;

4. The issues described in Discussion section

Status Report for 04/25/2007 -- 05/16/2007

2007-05-17T01:03:00.000-04:00

Progress

Detail design for Bay1Temp Atom Feed (method1 shown in the last report is used here)

1. Create an atom feed document

New a feed

Feed theFeed = new Feed();

Add "xmlns" attribute to the feed

theFeed.addAttribute(new Attribute("xmlns","http://www.w3.org/2005/Atom"));

Add Author

theFeed.addAuthor(new Author("Yu(Carol) Deng"));

Set a universally unique Id for the feed

theFeed.setId(new Id("urn:uuid:" + new UID().toString()));

Add Title

Title feedTitle = new Title("text"); // Set title type for the feed

feedTitle.setText("Bay1Temp Atom Feed"); // Set title content

theFeed.setTitle(feedTitle); // Set the title

Add Link

Link feedLink = new Link(); // New a Link in the feed

feedLink.setRel(new Attribute("rel", "self")); // Set "rel" attribute of the link

feedLink.setType(new Attribute("type", "application/atom+xml")); //Set "type" attribute of the link

feedLink.setHref(new Attribute("href", FeedHref)); //Set "href" attribute of the link

theFeed.addLink(feedLink); //Add the link

Set Updated to the entry

theFeed.setUpdated(new Updated(new Date()));

2. Add an entry document to the feed

New an Entry

parcelEntry = new Entry();

Add Author

parcelEntry.addAuthor(new Author("Yu(Carol) Deng"));

Add Title

Title parcelTitle = new Title("text"); // Set title type for the feed

parcelTitle.setText("SensorName, TimeStamp, DoubleData"); // Set title content

parcelEntry.setTitle(parcelTitle); // Set the title

Set a universally unique Id for the entry

parcelEntry.setId(new Id("urn:uuid:" + new UID().toString()));

Set Updated to the entry

Calendar cal = new GregorianCalendar();

parcelEntry.setUpdated(new Updated(cal.getTime()));

Set the current data to the Content

parcelEntry.setContent(nodeSensorName + nodeTimeStamp + nodeDoubleData);

Add the Entry to the feed

currentFeed.addEntry(parcelEntry);

3. Atom Feed for Bay1Temp

- <feed xmlns="http://www.w3.org/2005/Atom">

<updated>2007-05-17T01:15:29.205-05:00updated>

<title type="text">Bay1Temp Atom Feedtitle>

- <author>

<name>Yu(Carol) Dengname>

author>

- <entry>

<id>urn:uuid:-7012692:11296d2cefc:-7fffid>

<updated>2007-05-16T17:37:17.66-05:00updated>

<title type="text">SensorName, TimeStamp, DoubleDatatitle>

- <author>

<name>Yu(Carol) Dengname>

author>

<content type="html">Bay1Temp 2007-05-17 05:14:48Z 25.5
Bay1Temp 2007-05-17 05:15:08Z 25.5
Bay1Temp 2007-05-17 05:15:28Z 25.5content>

entry>

feed>

4. Bay1Temp Atom Feed in iGoogle

Add the url of Bay1Temp Atom Feed to iGoogle, the screen shot below shows the Atom Feed in iGoogle.

Discussion & Questions

1. In iGoogle, it shows 3 links for Bay1Temp in the feed area

-- I tried to clean some cookies or cache, but they are still there.

2. In iGoogle, after refreshing the current webpage, click the content of feed which had content, is not available for the data.

-- I kept clicking the "+" button in the feed area, data is still not available.

Future work

1. Get all sensors from a lab going to individual feeds;

2. The issues described in Discussion section

Status Report for 04/04/2007 -- 04/25/2007

2007-04-28T01:40:00.000-04:00

Progress

Detail design for Atom Feed for CIMA Web 2.0

1. Method I

Basic idea:

On the server side:

(1) A program keeps running to get Parcel from CIMA Service every several seconds;

(2) Every time the program get Parcel it will call XML Parser to wrap the data into format of item in the Atom Feed;

(3) Write the new-come item to the Atom Feed, and delete the oldest one at the same time;

(4) Once there is any feed request, it will read the Atom feed (file)

On the client side:

(1) User sends feed request to the server;

(2) Show feed content to the user

2. Method II

Basic idea:

On the server side:

(1) Once there is any feed request, a serverlet will be started to listen to the Parcel from CIMA service;

(2) If no data comes, return a message to the user; otherwise, get Parcel and call XML Parser. This step keeps doing this way until it gets enough Parcels based on users requirements(For example, user wants to watch the latest 10 data each time) and to wrap the them into format of item in the Atom Feed

(3) Write these items to the Atom Feed;

(4) Once there is any feed request, it will read the Atom feed (file)

On the client side:

(1) User sends feed request to the server;

(2) Show feed content to the user

3. Comparison
(1) Method I is simple and it can update information at the same rate they receive information, but it needs to run the program all the time even if no feed request comes;

(2) Method II saves resource and it gets the data just upon the user request, but it is a little bit complicated and the user may need to wait for some time for server get enough data

Discussion

1. Comparison of two implementations

2. Request frequency settings in Sage

Future work

1. The issues described in Discussion section

Status Report for 03/07/2007 -- 04/04/2007

2007-04-06T17:55:00.000-04:00

Progress

1. Workflow for Atom Feed for CIMA Web 2.0

(1) CIMA persons subscribe all the available parameters by Sage on the browser side;

(2) CIMA Client gets the http request and registers a CIMA service;

(3) CIMA Service sends back a parcel with the service information users required;

(4) XML Parser converts the parcel format into Atom Syndication format and writes to Atom Feed;

(5) Send results back to users

2. Detail Design

(1) Install Sage at http://sage.mozdev.org/ (Sage is a lightweight RSS and Atom feed reader extension for Mozilla Firefox);

(2) Make CIMAClientPlatform application run. This application a skeleton application to register with a CIMA service and get data (Parcel) back. In a Parcel, it contains many data, currently, the user needs “SensorName”, “TimeStamp” and “DoubleData”;

(3) In the class src/edu/indiana/extreme/www/cima/channel/sink/Channel_sinkImpl.java,

put some code for parsing Parcel to Atom Syndication format and writes to Atom Feed in the "myHandler" method. The basic steps are shown below:

(a) Read XML data into a DOM;

(b) Configure the Factory;

(d) Get a parser and parse the parcel;

(e) Write the data required to Atom Feed;

(4) Copy the Atom Feed to an available URL manually;

(5) On browser side, create a new bookmark in Sage for the Atom Feed just created and Check the feed;

(6) The content/data in the Atom Feed will show on the right panel

Discussion

1. Currently, I manually copied the Atom Feed from hagar.cs.indiana.edu/Web2.0/...... to

http://www.cs.indiana.edu/~ydeng/atomfeedcima.xml, so I am wondering if I need to use some TOMCAT programming to move these files automatically.

2. How often the Sage retrieves the data from the Atom Feed.

Future work

1. The issues described in Discussion section

Status Report for 02/14/2007 -- 03/07/2007

2007-03-07T16:23:00.000-05:00

Progress

1. Atom Feed for CIMA Web 2.0

(1) CIMA persons click the link to feeds which will pop up on all the parameters for them to subscribe;

(2) CIMA Client gets the http request and registers a CIMA service;

(3) CIMA Service sends a parcel with the service information users required;

(4) Atom Feed parses the parcel and convert service information into Atom Syndication format and send them back to users

2. Research methods

(1) Create a simple atom feed (xml file) sample by hand

(2) Make a service with and Create an atom feed from step(1) by using atmosphere Java library;

A brief instruction to create an atom feed:

(a) requirements: JDK 1.4.2 or 1.5

(b) download the following jar and war files at http://www.colorfulsoftware.com/projects/atomsphere

atmosphere which includes atomsphere-1.0.1.0.jar, atomsphere-1.0.1.0-javadoc.jar, atomsphere-1.0.1.0-sources.jar
atmosphere-taglib which includes atomsphere-taglib-1.0.1.0.jar, atomsphere-taglib-1.0.1.0-sources.jar
atmosphere-webapp which includes atomsphere-webapp-1.0.1.0.war
atomphere-weblib which includes atomsphere-weblib-1.0.1.0.jar, atomsphere-weblib-1.0.1.0-sources.jar
jsr173_1.0_api.jar, sjsxp_101.class, sjsxp_fcs_1_0.class, StAX Utilities Project

(d) API(JavaDoc) http://www.colorfulsoftware.com/projects/atomsphere/api/

(e) create an atom feed by atmosphere library (sample code is shown below)

Feed feed = new Feed();
feed.addAuthor(new Author("Yu(Carol) Deng"));
feed.setId(new Id("tag:www.xxxxxxxx.com"));
feed.setTitle(new Title("Cool Atom Feed"));
feed.setUpdated(new Updated(new Date(2005,3,2)));

Entry firstEntry = new Entry();
firstEntry.setTitle(new Title("Some interesting stuff"));
firstEntry.setId(new Id("tag:www.XMML.com/2004/12/08.html"));
firstEntry.setUpdated(new Updated(new Date(2006,3,1)));
feed.addEntry(firstEntry);

FeedDoc.writeFeedDoc("foo.xml",feed,FeedDoc.encoding,FeedDoc.xml_version);

(3) Repeat step(2), use xml beans (apache project) instead;

I tried to write a program in XML beans (http://xmlbeans.apache.org/) to create an atom feed. But I found that the schema of Atom 1.0 feed follows RELAX NG Compact Schema while xml beans supports XML Schema not RELAX NG Compact Schema. I tried to find the conversion between these two, but failed. I am wondering if there is some kind of conversion between RELAX NG Compact Schema and XML Schema.

Discussion

1. The format of parcel from CIMA Service

2. The storage of the Atom feed

Future work

1. The issues described in Discussion section

Status Report for 01/31/2007 -- 02/14/2007

2007-02-14T22:09:00.000-05:00

Progress

1. RSS/Atom Feed for CIMA Web 2.0

2. Research methods

(1) Create an atom feed (atom.xml) sample by hand;

(2) Make a service with atomsphere Java library and produce atom.xml from step(1), that is to say, make a service that responds with the atom feed in the HTTP response;

(3) Repeat step(2), use xml beans (apache project) instead;

(4) Compare the two methods above.

3. Research steps

(1) make feeds;

(2) do flicke;

(3) do pipe

4. Read some papers and books related to RSS/Atom feed

(1) Web sites

http://en.wikipedia.org/wiki/RSS_(file_format)#Example_-_RSS_2.0

http://www.colorfulsoftware.com/projects/atomsphere

http://xmlbeans.apache.org/

http://www.timfanelli.com/item/howto_create_atom_feed

http://www.russellbeattie.com/notebook/1007853.html

(2) Books

Developing Feeds with Rss and Atom by Ben Hammersley
Beginning RSS and Atom Programming by Danny Ayers and Andrew Watt
Hacking RSS and Atom by Leslie M. Orchard

Discussion

1. RSS for CIMA

2. XML Bean for CIMA

Future work

1. The issues described in Discussion section

Status Report for 01/17/2007 -- 01/31/2007

2007-01-31T22:52:00.000-05:00

Progress

1. CIMA ideas for Web 2.0

(1) RSS feeds and XPath subscription

It can allow CIMA users to subscribe their feeds and retrieve the new lab data.

(2) Nagios data in Google map

It can allow CIMA users to monitor CIMA services and show the trends.

(3) Google Gadgets for data form

It can allow CIMA users to use some gadgets offered by Google Desktop.

(4) AJax Tool kits vs. plain JS

o Roll your own

o Evaluation higher level tools

It can allow CIMA users to exchange small amount of data with the server without

reloading the entire web page each time.

(5) Jsow

o “Javascript” “beans” for data equivalent to xml

(6) GUI(Widgets)

o YUI

o Google

o Data Tables

(7) Social CIMA

This intent is to make CIMA portal look like “youTube” Model

(8) Toolbar

o Sidebar for Google

(9) What is the “best” RSS/ATOM rendors

(10) Mobile CIMA

(11) Real-time collaboration portal (Flex)

2. Fix bugs in the build system

3. Finish document for CIMA one-click build system, it includes:

system description
system requirements
system directory structure
user guide(install and uninstall instructions)

4. Read some papers related to Web 2.0

etc. etc.

Discussion

1. Web 2.0 for CIMA

2. CIMA CVS/SVN

Future work

1. The issues described in Discussion section

2. Contact Daphne for a press release

Status Report for 01/10/2007 -- 01/17/2007

2007-01-17T19:56:00.000-05:00

Progress

1. Add “Delete lab” option for one-step multiple-lab installation process

(1) Requirements

Remove portlets for a lab with the short lab name (e.g. RLAB)

(2) Design & implement

(a) Set environnent variables

(b) Get the lab parameters in its property file created by the installation process

(d) Remove xml paragraph for the lab in portlet.xml

(e) Delete 3 portlets ( cimadminprj, XRaySampleData, and showmysamples ) and SampleDataInfo for the lab

· Directories corresponding to the portlet names for the lab under directory $CATALINA_HOME/webapps

· Directories corresponding to the portlet names for the lab under directory $CATALINA_HOME/work/Catalina/localhost

· Files corresponding to the portlet names for the lab that are under $CATALINA_HOME/webapps/gridsphere/WEB-INF/CustomPortal/portlets

· Files corresponding to the portlet names for the lab that are under $CATALINA_HOME/webapps/gridsphere/WEB-INF/CustomPortal/layouts/groups

(f) Delete the lab’s property file

2. Write document for installation instructions, a description of how the software directory structure is laid out and a description of the build system, etc.

Discussion

1. Web 2.0 for CIMA

2. CIMA CVS/SVN

Future work

1. The issues described in Discussion section

2. Fix bugs in the installation process

3. Contact Daphne for a press release

Status Report for 12/27/2006 -- 01/10/2007

2007-01-11T16:08:00.000-05:00

Progress

1. Finish the 5^th version of one-step multiple-lab installation (based on requirements of Dr. Mcmullen) and showed a demo

(1) Requirements

Implement a one-step installation process with the following parameters provided by the user

endpoint of web service (e.g. http://156.56.94.164/cgi-bin/CIMAWebServices/TESTWS.cgi)
short lab name (e.g. RLAB)
lab name for display (e.g. "Rick Lab")
url of laboverview(e.g. http://diffraction.chem.indiana.edu/cgi-bin/test1-cgi/overview4)
url of lababout(e.g. http://www.iumsc.indiana.edu/aboutIUMSC/index.html)

(2) Design & implement

Based on the 4^th version, the 5^th version can add any lab data into CIMA portal at any time with lab parameters shown above. The whole installation is divided into two parts: base part which includes a framework of CIMA portal without any lab data and lab part which includes lab data. The main idea of the installation process is if it is the 1^st time installation, we will install both base part and the lab part for a specific lab, otherwise just install the lab part for a specific lab.

(a) Set environnent variables

(b) If ANT is not installed, install ANT ; otherwise go to step (c)

(d) If it is the 1^st time installation, install base part first, otherwise go to step (e) to install the lab part for a specific lab.

· Unpack and install Ant and Tomcat and Gridsphere

· Build IFrame portlet

· Copy cima portlets

· Build guest user layout screen

(e) Lab part

· Create build.properties for the lab

· Create cima.properties for the lab

· Get SampleDataInfo for the lab

· Deploying 3 portlets ( cimadminprj, XRaySampleData, and showmysamples ) for the lab

· Construct xml paragraph for the lab in GuestUserLayout.xml

· Construct xml paragraph for the lab in portlet.xml

(f) Start up Tomcat

2. Finish some improves based on demo of the 5^th version and submit the latest version

(1) Get SampleDataInfo for each lab

(2) Solve the conflicts with the one-step installation process and Hao Yin’s portlets

(3) Uniform the naming in GuestUserLayout.xml and portlet.xml so that it is easy for a general target to operate

Discussion

1. Add “delete lab” option for the build process

2. CIMA CVS/SVN

Future work

1. The issues described in Discussion section

2. Fix bugs in the installation process

3. Submit a user guide and a package of the one-step process to Kia

4. Write document for installation instructions, a description of how the software directory structure is laid out and a description of the build system, etc.

5. Contact Daphne for a press release

Status Report for 12/13/2006 -- 12/27/2006

2006-12-28T13:26:00.000-05:00

Progress of these 2 weeks

1. Finish the 4^th version of one-step multiple-lab installation (based on requirements of Dr. Mcmullen) and showed a demo

(1) Requirements

Implement a general installation process with the following parameters provided by the user

endpoint of web service (e.g. http://156.56.94.164/cgi-bin/CIMAWebServices/TESTWS.cgi)
short lab name (e.g. RLAB)
lab name for display (e.g. "Rick Lab")

(2) Design & implement

The general process for deploying 3 portlets ( cimadminprj, XRaySampleData, and showmysamples ) includes following general targets:

(a) A general target for modifying portlet.xml to deploy iframe portlet

Construct a template xml file for the specific lab for portlet.xml
Replace short lab name, default page(iframe url) and about page(iframe about url )for the lab in the xml file
Insert the xml file into portlet.xml under $CATALINA_HOME/webapps/iframe-portlet/WEB-INF/portlet.xml

(b) A general target for modifying build.properties in 3 portlets to deploy 3 portlets

Construct 3 template build.properties files for each portlet of the specific lab
Replace short lab name and webservice endpoint with the lab’s information
Deploy 3 portlets

Construct 3 template cima.properties files for each portlet of the specific lab
Make webservice endpoint and database address point to correct address

(d) A general target for modifying GuestUserLayout.xml

Construct a template xml file for the specific lab for GuestUserLayout.xml

Replace short lab name and lab name for display for the lab in the xml file
Insert the xml file into GuestUserLayout.xml under $CATALINA_HOME/webapps/gridsphere/WEB-INF/CustomPortal/layouts

2. Finish some improves based on demo of the 4^th version and submit the latest version

(1) Endpoint of webservice

User provides the URL of the webservice endpoint so that any webservice can run on CIMA portal

(2) Directory of CIMA installed

User can choose or name directory for CIMA to install so that the installation directory is separated from the project directory

Discussion

1. Issues on multi-lab installation based on the general one-lab installation process

2. CIMA CVS/SVN

Future work

1. Move on to multi-lab installation process based on the general one-lab installation process

2. Fix bugs in the installation process

3. Submit an updated user guide and a package of the one-step process to Kia

Status Report for 11/29/2006 -- 12/13/2006

2006-12-13T13:33:00.000-05:00

Progress

1. Finish 3^rd version of one-step multiple-lab installation

(1) Show public information (including Current Status, Public Samples, and About this lab) for each lab on Guest User Layout screen.

Call a general target to add new tabs for new labs in GuestUserLayout.xml
Modify content of in portlet.xml for each portlet

(2) Make lab name used in portlets consistent and make lab name used for display consistent for each lab installation so that it is easy to call a general target to implement

(3) Check cima.properties for cimadminprj, XRaySampleData, and showmysamples portlets and make the endpoint and database point to the correct address

(4) Change background color of CIMA banner

Modify layout.css, page.css, and tab.css files under directory gridsphere/themes/swamps/css

2. Finish a general target for installing user-defined web service

(1) It has two parameters which are the URL of the web service and address of database

(2) Install it to the portal

Files related : installcima.sh, env.sh, cima.properties for each portlet, build.properties for each portlet

Discussion

1. Installation of user-defined web service

2. CIMA CVS/SVN

3. Group for iframe portlet for each lab

Future work

1. Get feedback from Kia to improve the installation process

2. Fix bugs in the installation process

3. Submit an updated user guide and a package of the one-step process to Kia

Status Report for 11/22/2006 -- 11/29/2006

2006-11-29T16:23:00.000-05:00

Progress

1. Finish one-step installation with IUMSC according to requirement from Kia

2. Submit a final user guide and a package of the one-step process to Kia

3. Back to work on multiple lab installation again

(1) When login as root, the user should see all the portlets under the Administration Group

(2) Add public sample portlet for each lab on Guest User Layout Screen

Discussion

1. Issues related to multiple lab installation during last week

2. CIMA CVS/SVN

3. Group for iframe portlet for each lab

Future work

1. Continue working on multiple lab installation process

2. Improve the multiple lab installation process according to Kia’s feedback

3. Try to solve the problems described above in Discussion section

Status Report for 11/15/2006 -- 11/22/2006

2006-11-22T14:36:00.000-05:00

Progress

1. Finish improves made on one-step installation with IUMSC according to feedbacks from Kia

(1) Add public sample portlet for each lab on Guest User Layout

Call a general target to add new tabs for new labs in GuestUserLayout.xml

(2) Host IP address

Provide user interface to input IP address of the host machine where CIMA is running

(3) Fix bugs

Bug 1 : Assume Ant was installed in the user machine and call Ant targets to install Ant for CIMA

Solution: Install Ant as the 1^st step for the whole installation

Bug 2 : Errors on setting environment variables

Solution : Modify env.sh and installcima.sh

2. Discuss issues related to multiple lab installation

Discussion

1. Issues related to multiple lab installation

(1) Content of user interface after the user logins to CIMA as root and a specific user account

(2) Provide user interface to install specific lab as user wants

(3) How we can tell who logins to the CIMA and just show the contents related to their own lab (most important)

2. CIMA CVS/SVN

3. Group for iframe portlet for each lab

Future work

1. Get feedback from Kia to improve the installation process

2. Fix bugs in the installation process

3. Some modifications on content of user interface after user logs in to the system as root or a specific user

4. Try to solve the problems described above in Discussion section

5. Submit an updated user guide and a package of the one-step process to Kia

Status Report for 11/08/2006 -- 11/15/2006

2006-11-15T15:02:00.000-05:00

Progress

1. Finish the 1.0 version of CIMA one-step installation with multiple labs. It expands the installation from one lab (IUMSC) to multiple labs (Purdue, ChemMATCars, Minnesota, Myers, JCU, CSAF)

(1) Design

A general process for installing cimadminprj, XRaySampleData, and showmysamples portlets for a new lab
A general process for adding a new tab for the new lab in GuestUserLayout.xml

(2) Implementation

Install 3 portlets for each new lab

- Run Ant by passing in parameters (here is just the labName, e.g. JCU) to an Ant target: Ant –DlabName=JCU addLabs

- In build.xml, load the parameter with

- Install cimadminprj portlet for the new lab

- Install XRaySampleData for the new lab

- Install showmysamples portlet for the new lab

Add new tab for each new lab in GuestUserLayout.xml

- Method 1 : Writing JAVA programs the read and write DOM XML document

- Method 2 : Using the standard Ant tasks ‘replace’ or ‘style/xslt’ and defining an XML style sheet.

- Method 3 : Using xmltask

Comparison results:

- Method 1 is complicated

(a) Use an XML parser to read the xml document

(b) Need a DOM document to put the XML into

- Method 2 works well for simple search/replace operations, but lacks the ability to perform powerful searches and complex replacement operations

- Method 3 provides the facility for automatically editing XML files as part of an Ant build. (Best choice)

2. Improves the installation process according to Kia’s feedback

Problem : Cannot run properly on Kia’s machine

Solution: Add a host IP address environment variable, and user can input the IP address to run the installation process

Problem : There are a few misunderstandings in the user guide

Solution : Clarify it

3. Submit an installation package of the one-step process including all CIMA sites to Kia

User Guide
Package

All installation files
CIMA portlet
Third party software

Discussion

1. Issues related to lab.properties

2. Build group for iframe portlet for each lab using ANT script

3. Checks CIMA code out from CVS/SVN

4. Create a test user account in Ant script

Future work

1. Get feedback from Kia to improve the installation process

2. Fix bugs in the installation process

3. Try to solve the problems described above in Discussion section

Status Report for 11/01/2006 -- 11/08/2006

2006-11-08T11:45:00.000-05:00

Progress

1. Finish the one-step process which includes two CIMA sites (IUMSC and JCU) and fix bugs in it

(1) Design

Add a process for installing cimadminprj, XRaySampleData, and showmysamples portlets for JCU
Add a process for adding JCU tab for JCU in GuestUserLayout.xml

(2) Implementation

Install 3 portlets for JCU

- Modify build.properties for cimadminprj, XRaySampleData, and showmysamples

- Run ant install

Add new tab for JCU in GuestUserLayout.xml

(3) Bugs

Bug 1 : Cannot show detailed information of each sample in sample data

Solution: Copy sample information data to related directory

Bug 2 : A wrong information related to labname in showmysamples portlet

Solution : Modify cima.properties

Bug 3 : Some small bugs related to user interface settings

Solution : Modify the files related

2. Reorganize the project directories and submit an updated user guide and a package of the one-step process to Kia

User Guide
Package

All installation files
CIMA portlets
Third party software

Discussion

1. Build group for iframe portlet for each lab using ANT script

2. Check CIMA code out from CVS/SVN

3. Create a test user account in Ant script

Future work

1. Get feedback from Kia to improve the installation process

2. Fix bugs in the installation process

3. Write general processes to install multiple labs

4. Try to solve the problems described above in Discussion section

Status Report for 10/25/2006 -- 11/01/2006

2006-11-03T10:11:00.000-05:00

Summary of previous work before this week

1. Install CIMA portal (with IUMSC) manually

Install all third part software(Java, Ant, Tomcat, Gridsphere)
Install CIMA portal

Deploy and configure “iframe portlet”
Get the 3 CIMA portlets(cimadminprj, showmysamples, XRaySampleData) installed in my GS portal and configure them
Make a tab(IUMSC) that has the following 3 sub-menus: Current Status, Public Sample portlet, and About

2. Install the "internal" build system that in a single step does all of the following:

Provide an organizational directory structure for the project, including third party tar files for Gridsphere, Tomcat, Ant
Install (unpacks, builds) all third party software(Ant, Tomcat, Gridsphere) from tars.
Build and deploy all of the CIMA portlets.
Automatically customize Gridsphere to have the look and feel, layout, color scheme, and logo of the CIMA portal.
Build anonymous user views and IUMSC tab.

This is based on Apache Ant.

Progress of this week

1. Fix many small bugs in earlier deliverable

2. Extend earlier deliverable of "internal" build system to include all CIMA sites (Purdue, ChemMATCars, Minnesota, Myers, JCU,CSAF)

3. Write a simple user’s guide to Kia and let her check the CIMA portal built by this one-step process

User Guide
Package

All installation files
CIMA portlets
Third party software

Discussion

1. Checks CIMA code out from CVS/SVN

2. Create a test user account in Ant script

Future work

1. Get feedback from Kia to improve the installation process

2. Fix bugs in the installation process

3. Try to solve the problems described above in Discussion section