So that's what happened to Wave

I tried Wave, was unimpressed, but felt like it would follow the classic hype curve. At any rate, I'm working on my Masters of Science in Information Science at the School of Information and Library Science. It's a wonderful program, though challenging with family + full time job + music gig. I'm holding it together (but procrastinating on some homework right now).

I am interested in collaborative search and sense-making from back in the day at RENCI when I was looking at InfoMesa, and recent research in my summer class has me thinking about this again. Many aspects of collaborative search and sense-making were captured in the original intent of Wave. I never bought that it was a new form of communication, or that it was revolutionary in that respect. What I still think is that it has some aspects of a platform for collaborative search and sense making. I especially like how it can combine time-shifted asynchronous activities with synchronous activities. What's most compelling is to look at what Wave was doing as an application platform. Collaborative search and knowledge discovery could be a great one for Wave.

I guess I've been busy working, because I'd hardly noticed that Wave has gone over to Apache as 'Wave in a Box'. While it appears to be in early incubator stage (they probably don't know what to do with it either), it bears watching.

Working on releasing Jargon-core

I'm working on a beta release of Jargon core, and all the fun involved in setting up the maven release plug-in for git and Nexus. Needless to say, I've made a few U-turns.

Here's my tip (I'm saving this for myself) on deleting the git tag to tackle some config errors:

http://nathanhoad.net/how-to-delete-a-remote-git-tag

Thanks Nathan!

Recent Jargon Updates

There's a lot of activity on various types of interfaces, but I wanted to update folks on things at the API level, especially in the jargon-core API.

https://code.renci.org/gf/project/jargon/

This is late beta, we are working on moving to a schedule of releases, frankly, we need to work out maven and git and the maven release plug-in, there is a lot going on and we're trying to get that done.

Some highlights:

• The big push has been to put a new public API out that is easier to use and maintain. I think this is shaping up (you can tell me whether that's so). There are plenty of capabilities that have never been exposed outside of the C API that are either in there, or planned.
• There are lots of implementing and testing going on in several places, providing a nice amount of friction to help the API come along. As things settle in, we're starting to be able to shift perspective more to optimization. There are lots more places where buffering is implemented, and baby steps to looking at NIO. With the presence of the RENCI team, we're starting to stand up resources where we could start doing benchmarks to help guide optimization.
• We will be working to pull the jargon-trunk and php code out of the main iRODS trunk into separate areas for the next release. The schedules are pretty tight, so that's still a tentative plan.
• Several key community wish list items are under development, some things that may be of interest are below:
  

Configuration, setting options for operations, defaulting

With the 'clean code' practice of separating code from metadata in mind, there is a jargon.properties file now. This is where config info is being consolidated over time. The IRODSSession object is the place where expensive stuff like loading configuration properties, extensible metadata mappings, and such occurs. You can access the default jargon properties there, or override them. (So in Spring, you can wire in configured properties at startup for your web app, etc).

When you do transfers, you can call methods like below:

/**
* Put a file or a collection (recursively) to iRODS. This method allows
* registration of a TransferStatusCallbackListener that will
* provide callbacks about the status of the transfer suitable for progress
* monitoring
*
* @param sourceFile
*            File with the source directory or file.
* @param targetIrodsFile
*            {@link org.irods.jargon.core.pub.io.IRODSFile} with the target
*            iRODS file or collection.
* @param transferControlBlock
*            an optional
*            {@link org.irods.jargon.core.transfer.TransferControlBlock}
*            that provides a common object to communicate between the
*            object requesting the transfer, and the method performing the
*            transfer. This control block may contain a filter that can be
*            used to control restarts, and provides a way for the
*            requesting process to send a cancellation. This may be set to
*            null if not required.
* @throws JargonException
*/
void putOperation(
final File sourceFile,
final IRODSFile targetIrodsFile,
final TransferStatusCallbackListener transferStatusCallbackListener,
final TransferControlBlock transferControlBlock)
throws JargonException;

I wanted to point out the 'transferStatusCallbackListener'. You can implement this interface, pass it to the method, and get callbacks like initiation of operation, file transfer update, completion of operation. We are working on also providing a callback for 'messages', like 'starting parallel transfer', or 'computing checksum' so those can start surfacing in UI like iDrop. There are plans soon to provide optional 'intra-file' callbacks, so you could throw up a progress bar for what's going on inside of a file transfer. Look for that soon. Of course, you can leave that null if you don't care.

The 'transferControlBlock' is important. It is the communication pipeline between the thing calling the transfer, and the transfer itself. You can peek at aggregate numbers, set a cancel flag, etc. The point is, there is now also a transferOptions that can be set. The transfer options can include things like 'no parallel transfers', or 'max transfer threads', or buffer sizes, or things like the recently added 'compute and verify checksum'. You can either set a transfer options in that transfer control block, or leave it alone and defaults will be computed based on the settings in the jargon properties. The checksum validation is in there now (a community request), and the rerouting of gets/puts to the owning resource server should be there soon. The checksum and checksum validation functions were just added

Parallel Transfer Pools

Another community request in the optimization department was to set up a thread pool for parallel transfer threads. This has been done in a first implementation, but turned off by default. This should be handy in mid-tier apps, where you might want to cap the number of threads. More testing needs to be done, is it fixed? What are the time-outs, etc.

GSS/Kerberos, etc

There are threads here about security, GSS, etc. This is a high-priority item from the user group meeting, and this is next on the list of 'core' development. Folks who are really facile with GSS, Kerberos and the like who have comments and insight are invited to give input. I think once this part is in jargon-core, we can go to a full-on 'release'. The code is pretty solid, much more so than the older API, we're seeing, and will continue to see, performance gains, and it's much easier to use in mid-tier situations, or if you are 'Spring'-happy. Please let me know your experiences, comments, criticisms, and keep an eye out!

Another undocumented annoyance with testing Grails controllers

I switched from rendering JSON in my previous post to using a gsp to format an HTML ul tag based on a model containing my list of results. This is because I find it easier when using JQuery to do my AJAX using HTML content. JQuery seems to natively prefer HTML to JSON, whereas Dojo seemd to run on JSON.

At any rate, I changed my controller to something like this:

def loadTree = {  
  def parent = params['dir']
  log.info "loading tree for parent path: ${parent}"
  def collectionAndDataObjectListAndSearchAO = irodsAccessObjectFactory.getCollectionAndDataObjectListAndSearchAO(irodsAccount)
  def collectionAndDataObjectList = collectionAndDataObjectListAndSearchAO.listDataObjectsAndCollectionsUnderPath(parent)
  log.debug("retrieved collectionAndDataObjectList: ${collectionAndDataObjectList}")
  render(view:"loadTree",model:[collectionAndDataObjectList:collectionAndDataObjectList, parent:parent])
 }

I altered my unit test, per the docs, to inspect the model and view. I attempted to get at the model entry for 'parent', which contained the parent dir, like this:

def parent = mav.model.parent

It turns out that that does not work anymore, instead, you need to interpose a reference to linkedHasMap like so:

void testBrowse() {
        controller.params.dir = "/"
        controller.irodsAccessObjectFactory = irodsAccessObjectFactory
        controller.irodsAccount = irodsAccount
        controller.loadTree()
        def mav = controller.modelAndView
        def name = mav.viewName
        assertNotNull("null mav", mav)
        assertEquals("view name should be loadTree", "loadTree", name)
        def parent = mav.model.linkedHashMap.parent
        assertEquals("parent dir not found", "/", parent)
 
    }

Now my Grails controller and test are working again. My next step is to create a decent JavaScript object that can bridge between methods in this BrowseController and a lazy-loadable JQuery JSTree. Maybe the next post can share some developments there.

Testing a Grails Controller - getting the $*&@ thing to work

I wanted to post on two things. First, of general interest, I wanted to show how I got a working test on a Grails controller. Second, I wanted to highlight some of the new Jargon API and how it can fit into apps written using Groovy and Grails (this should also translate to Jython and JRuby, as well as other dynamic JVM languages).

There is lots of contradictory information on the web, even in the Grails docs, on testing Grails controllers. The biggest issues are understanding the ControllerUnitTestCase, and what it does and does not do.

First, here's a controller I'm developing that will back a JQuery JTree component. The first method will take a parent path name, and return a JSON representation of the iRODS file system for the parent path using some new Jargon methods that are meant to assist in Swing and JQuery tree development.

package org.irods.mydrop.controller

import org.springframework.security.core.context.SecurityContextHolder;
import org.irods.jargon.core.pub.*;
import org.irods.jargon.core.connection.*;
import org.irods.jargon.core.exception.*;
import grails.converters.*

/**
* Controller for browser functionality
* @author Mike Conway - DICE (www.irods.org)
*/

class BrowseController {

IRODSAccessObjectFactory irodsAccessObjectFactory
IRODSAccount irodsAccount

/**
* Interceptor grabs IRODSAccount from the SecurityContextHolder
*/
def beforeInterceptor = {
def irodsAuthentication = SecurityContextHolder.getContext().authentication

if (irodsAuthentication == null) {
 throw new JargonRuntimeException("no irodsAuthentication in security context!")
}

irodsAccount = irodsAuthentication.irodsAccount
log.debug("retrieved account for request: ${irodsAccount}")
}


/**
* Display initial browser
*/
def index = { }


/**
* Render the tree node data for the given parent.
* 

*  Requires param 'dir' from request to derive parent
*
*/
def loadTree = {
def parent = params['dir']
log.info "loading tree for parent path: ${parent}"
def collectionAndDataObjectListAndSearchAO = irodsAccessObjectFactory.getCollectionAndDataObjectListAndSearchAO(irodsAccount)
def collectionAndDataObjectList = collectionAndDataObjectListAndSearchAO.listDataObjectsAndCollectionsUnderPath(parent)
log.debug("retrieved collectionAndDataObjectList: ${collectionAndDataObjectList}")
render collectionAndDataObjectList as JSON

}

}

A few things to note. The new Jargon libraries are meant to provide simple POJO's that represent various domain objects within iRODS. There also is an idea of Access Objects, that roughly equate with the DAO pattern. iRODS is not a database, so the mapping is not the same, but to me, the idea of Access Objects for iRODS at least gives some comfort in familiarity. One access object I'm developing is meant to assist in tree depictions and searching on file and collection paths to support such elements in typical web and Swing GUI applications. We'll use that Access Object.


I've wired in my Spring security objects as described in an earlier post. My interceptor in my controller grabs the saved authentication token from the Spring Security layer and extracts the IRODSAccount object. In the loadTree method, this IRODSAccount object is used with a factory injected into my controller to get a CollectionAndDataObjectListAndSearchAO. My controller uses the method that takes a parent path, and returns JSON objects that represent iRODS files and collections under that parent path.

The approach taken in Jargon seems to map fairly easily into Grails apps, that's a good sign. This shows that quick development of arbitrary interfaces on top of iRODS will become easier with the newer Jargon libraries. Yay!

So, the problems really began with trying to test. First, let me share a working test, then I'll talk about some of the problems I ran into:

package org.irods.mydrop.controller


import grails.test.*
import java.util.Properties
import org.irods.jargon.core.connection.IRODSAccount
import org.irods.jargon.core.pub.IRODSAccessObjectFactory
import org.irods.jargon.core.pub.IRODSFileSystem;
import org.irods.jargon.core.pub.io.IRODSFile
import org.irods.jargon.core.query.CollectionAndDataObjectListingEntry
import org.irods.jargon.testutils.TestingPropertiesHelper
import org.irods.jargon.testutils.filemanip.FileGenerator
import org.irods.jargon.testutils.TestingPropertiesHelper
import org.irods.jargon.spring.security.IRODSAuthenticationToken;
import org.springframework.security.core.context.SecurityContextHolder;
import grails.converters.*



class BrowseControllerTests extends ControllerUnitTestCase {

IRODSAccessObjectFactory irodsAccessObjectFactory
IRODSAccount irodsAccount
Properties testingProperties
TestingPropertiesHelper testingPropertiesHelper
IRODSFileSystem irodsFileSystem


protected void setUp() {
super.setUp()
testingPropertiesHelper = new TestingPropertiesHelper()
testingProperties = testingPropertiesHelper.getTestProperties()
irodsAccount = testingPropertiesHelper.buildIRODSAccountFromTestProperties(testingProperties)
irodsFileSystem = IRODSFileSystem.instance()
irodsAccessObjectFactory = irodsFileSystem.getIRODSAccessObjectFactory()
def irodsAuthentication = new IRODSAuthenticationToken(irodsAccount)
SecurityContextHolder.getContext().authentication = irodsAuthentication
}

protected void tearDown() {
    super.tearDown()
}

void testBrowseNoLogin() {
controller.params.dir = "/"
controller.irodsAccessObjectFactory = irodsAccessObjectFactory
controller.irodsAccount = irodsAccount
controller.loadTree()
def controllerResponse = controller.response.contentAsString
def jsonResult = JSON.parse(controllerResponse)
assertNotNull("missing json result", jsonResult)

}
}

One of the first sources of confusion to me were documents that seemed to imply that running the grails command create-controller would put the relevant test in the integration tests directory. It didn't, it placed the test in the test/unit directory. Go figure. Maybe I'm missing something, but I'm taking what Grails does at it's word, so to speak. I tried moving the test to the integration test directory, but found that this resulted in the setup methods not running. Fine...I'll just forget this rabbit-hole.

Second, I had the hardest time setting up the params. My controller expects a request param of 'dir', with my test specifying the root '/' directory. I kept getting a "No such property: params" error when I tried doing the

controller.params.dir = "/"

test setup. It turns out that the ControllerUnitTestCase is wanting to wire in and extend your controller for you, creating a 'controller' variable automatically, with the test target controller defined by convention using the test name. So instead of creating my controller in a def using the 'new' keyword, I had to let the ControllerUnitTestCase do it for me.

I also ran into the issue of irodsAccount being null. The ControllerUnitTestCase does not run my interceptor for me. I was getting a null irodsAccount, and realized that I had to manually inject that variable in my test code.

Well, I was almost home, I kept getting a 'null' for my controller variable in my test case. Doh...I had plugged a bunch of code into setUp() in my test case, and I had to go back and add the super() call in my test case so that ControllerUnitTestCase could do it's magic on the controller variable. So now it works! There's still lots of refactoring I want to do in the controller, and I need to do things like parse and test the actual JSON response, but these are the details to work out now that I can get the stupid test cases to run.

Now that's progress...hopefully that will help you avoid some of my own frustrations. Here's to a New Year and new adventures in coding. I think Grails/Groovy feels as immature as most new dynamic scripting languages, but at the same time, I really feel like, combined with the new Jargon libraries, I'm well on my way to a rapid web development stack on top of iRODS. That's a good thing!