Tuesday, July 31, 2007

Status Report for 07/18/2007 -- 08/01/2007

Progress

Summary of problems and solutions in Multithreaded Programming for IUMSC 12 sensors Atom Feed

1. Exception in thread occurred for the first time

  • Background & Problem

CIMA Client can receive parcels from all 12 sensors. When a parcel arrives, the in-memory cache which stores parcels for each sensor will be updated. Then the content of cache will be written to a Feed.

After CIMA Client ran for about 15 to 20 minutes, there was an exception in Thread and no parcel came afterward.

The error message is shown below:

Exception in thread "Thread-3" java.lang.IllegalStateException:

IllegalStateException:

at org.apache.axis.components.threadpool.ThreadPool.addWorker(ThreadPool.java:103)

at org.apache.axis.transport.http.SimpleAxisServer.run(SimpleAxisServer.java:243)

at java.lang.Thread.run(Thread.java:595)

  • Solution

(1) It seems that the program ran out of threads. It was addWorker that throws this when the thread pool runs out of threads.

(2) Check that the default maximum thread count (org.apache.axis.components.threadpool.ThreadPool, public static final int DEFAULT_MAX_THREADS) is set to 100 when the instance of ThreadPool is created.

(3) Change SimpleAxisServer() to SimpleAxisServer(int maxPoolSize, int

maxSessions). Both of the constructor parameters default to 100. Now we make them 1000 instead.

2. Random occurred exception in thread

  • Problem

No thread exceptions occurred for 1 hour after changes described above, but the problems were found by observing the output window in eclipse. That is, if only a few parcels came at the same time, the program had enough time to write them to the Feed and receive new incoming parcels. But sometimes many parcels came at the same time, and parcels should be written to the Feed one by one, thus during this period the program could not receive any parcel and it seemed the program had been dead since then.

Restart the CIMA Client, exception in thread arose randomly.

  • Solution – Step 1

Variable “maxPoolSize” and “maxSessions” were kept to 1000 and 1000, and set a "threshold" in myHandler() to 990, which meant that once there were 990 or more parcels were waiting for being processed the program would skip the latest one. Doing so was aiming to relieve the possible threads burst. As a result, there was no exception in thread for a longer time, but unfortunately the threadpool was full eventurally.

  • Solution – Step 2

Removed the writing to Feed part from myHandler(), and added/activated a thread which was just responsible for writing parcels to Feed. The result was that the speed of getting parcels and writing them to a remote disk did not match forever no matter how large the buffer we set and no matter whether the thread was added or not. So the burst of threadpool occurred at last.

  • Solution – Step 3

In myHandler(), the "writing to feed" part was commented and set variable maxPoolSize and maxSessions to only 5 and 5, and set a "threshold" in myHanlder() to 0. The result was that and it worked very well -- it could receive all the parcels and had never been dead. Then I wrote the Feed to /tmp/foo.xml(local disk, local file), and set variable maxPoolSize and maxSessions still to 5 and 5, and it still worked very well!

So I drew a conclusion that the bottleneck might be the "writing to feed" part which was not local operation and the speed of writing to feed did not match the speed of receiving the parcels, so resulted in the burst of the threadpool.

  • Solution – Step 4

Wrote the Feed to local disk and set a symbolic link to the remote one. – It worked very well and never had been dead for 3 days( Since I need to add one more functionality to CIMA Client, so I stopped it).

Discussion

1. Functionalities for users to select different sensors and/or just show the sensors with abnormal data.

Future work

1. Get sensor values based on the user’s requirement;

2. Show only values not in normal range;

3. The issues described in Discussion section

Friday, July 20, 2007

Status Report for 07/11/2007 -- 07/18/2007

Progress

Some problems and solutions in detail design for IUMSC 12 sensors Atom Feed

1. Synchronization in writing data to Atom Feed

  • Problem

It is possible that data from 12 sensors could come simultaneously, but each sensor data should be written to the Feed sequentially. In that case, if the data is being written to Feed, the other data should not be written to the Feed. On the other hand, the incoming data is updated in memory first, then written to the Feed. So the updating should be synchronized too.

  • Solution

Put the Feed updating statements in a synchronized block and this synchronized block should be as small as possible.

2. Exception in thread

  • Problem

After CIMA Client ran for about 15 to 20 minutes, there was an exception in Thread and no parcel comes afterward.

Error message is shown below:

Exception in thread "Thread-3" java.lang.IllegalStateException:

IllegalStateException:

at org.apache.axis.components.threadpool.ThreadPool.addWorker(ThreadPool.java:103)

at org.apache.axis.transport.http.SimpleAxisServer.run(SimpleAxisServer.java:243)

at java.lang.Thread.run(Thread.java:595)

  • Solution

(1) It seems that the program ran out of threads. It was addWorker that throws this when the thread pool runs out of threads.

(2) Check that the default maximum thread count (org.apache.axis.components.threadpool.ThreadPool, public static final int DEFAULT_MAX_THREADS) is set to 100 when the instance of ThreadPool is created.

(3) Change SimpleAxisServer() to SimpleAxisServer(int maxPoolSize, int

maxSessions). Both of the constructor parameters default to 100. Now we make them 1000 instead.

3. A problem caused by Synchronization

  • Problem

It has been no thread exceptions for 1 hour, but I found another problem by observing the output window in eclipse. That is, if only a few sensor data come at the same time, the program has enough time to write the data to the Feed and receive new incoming data after that. If sometimes many sensor data come at the same time, the program needs to write these data one by one, during this period the program cannot receive any parcel and it seems the program has been dead for twenty or more minutes.

  • Possible solution

Instead of the critical sections using "synchronized" around the whole feed writing process we need some other way to serialize the writing of the feed. Maybe we should add a queue data structure into which each change is added, then the writer reads everything available in the queue, writes the feed, and reads the next batch of changes. This would limit the critical section to updating and reading an in-memory data structure rather than reading and writing a file.

Discussion

1. Functionalities for users to select different sensors and/or just show the sensors with abnormal data.

Future work

1. Get sensor values based on the user’s requirement;

2. Show only values not in normal range;

3. The issues described in Discussion section