Instanceof Idea
Mike Conway's adventures in IT, and other misc ramblings.
Wednesday, July 20, 2011
So that's what happened to Wave
Tuesday, July 19, 2011
Working on releasing Jargon-core
Saturday, July 16, 2011
Recent Jargon Updates
https://code.renci.org/gf/project/jargon/
This is late beta, we are working on moving to a schedule of releases, frankly, we need to work out maven and git and the maven release plug-in, there is a lot going on and we're trying to get that done.
Some highlights:
- The big push has been to put a new public API out that is easier to use and maintain. I think this is shaping up (you can tell me whether that's so). There are plenty of capabilities that have never been exposed outside of the C API that are either in there, or planned.
- There are lots of implementing and testing going on in several places, providing a nice amount of friction to help the API come along. As things settle in, we're starting to be able to shift perspective more to optimization. There are lots more places where buffering is implemented, and baby steps to looking at NIO. With the presence of the RENCI team, we're starting to stand up resources where we could start doing benchmarks to help guide optimization.
- We will be working to pull the jargon-trunk and php code out of the main iRODS trunk into separate areas for the next release. The schedules are pretty tight, so that's still a tentative plan.
- Several key community wish list items are under development, some things that may be of interest are below:
Configuration, setting options for operations, defaulting
With the 'clean code' practice of separating code from metadata in mind, there is a jargon.properties file now. This is where config info is being consolidated over time. The IRODSSession object is the place where expensive stuff like loading configuration properties, extensible metadata mappings, and such occurs. You can access the default jargon properties there, or override them. (So in Spring, you can wire in configured properties at startup for your web app, etc).
When you do transfers, you can call methods like below:
/**
* Put a file or a collection (recursively) to iRODS. This method allows
* registration of aTransferStatusCallbackListener
that will
* provide callbacks about the status of the transfer suitable for progress
* monitoring
*
* @param sourceFile
*File
with the source directory or file.
* @param targetIrodsFile
* {@link org.irods.jargon.core.pub.io.IRODSFile} with the target
* iRODS file or collection.
* @param transferControlBlock
* an optional
* {@link org.irods.jargon.core.transfer.TransferControlBlock}
* that provides a common object to communicate between the
* object requesting the transfer, and the method performing the
* transfer. This control block may contain a filter that can be
* used to control restarts, and provides a way for the
* requesting process to send a cancellation. This may be set to
* null if not required.
* @throws JargonException
*/
void putOperation(
final File sourceFile,
final IRODSFile targetIrodsFile,
final TransferStatusCallbackListener transferStatusCallbackListener,
final TransferControlBlock transferControlBlock)
throws JargonException;
I wanted to point out the 'transferStatusCallbackListener'. You can implement this interface, pass it to the method, and get callbacks like initiation of operation, file transfer update, completion of operation. We are working on also providing a callback for 'messages', like 'starting parallel transfer', or 'computing checksum' so those can start surfacing in UI like iDrop. There are plans soon to provide optional 'intra-file' callbacks, so you could throw up a progress bar for what's going on inside of a file transfer. Look for that soon. Of course, you can leave that null if you don't care.
The 'transferControlBlock' is important. It is the communication pipeline between the thing calling the transfer, and the transfer itself. You can peek at aggregate numbers, set a cancel flag, etc. The point is, there is now also a transferOptions that can be set. The transfer options can include things like 'no parallel transfers', or 'max transfer threads', or buffer sizes, or things like the recently added 'compute and verify checksum'. You can either set a transfer options in that transfer control block, or leave it alone and defaults will be computed based on the settings in the jargon properties. The checksum validation is in there now (a community request), and the rerouting of gets/puts to the owning resource server should be there soon. The checksum and checksum validation functions were just added
Parallel Transfer Pools
Another community request in the optimization department was to set up a thread pool for parallel transfer threads. This has been done in a first implementation, but turned off by default. This should be handy in mid-tier apps, where you might want to cap the number of threads. More testing needs to be done, is it fixed? What are the time-outs, etc.
GSS/Kerberos, etc
Monday, January 3, 2011
Another undocumented annoyance with testing Grails controllers
At any rate, I changed my controller to something like this:
def loadTree = {
def parent = params['dir']
log.info "loading tree for parent path: ${parent}"
def collectionAndDataObjectListAndSearchAO = irodsAccessObjectFactory.getCollectionAndDataObjectListAndSearchAO(irodsAccount)
def collectionAndDataObjectList = collectionAndDataObjectListAndSearchAO.listDataObjectsAndCollectionsUnderPath(parent)
log.debug("retrieved collectionAndDataObjectList: ${collectionAndDataObjectList}")
render(view:"loadTree",model:[collectionAndDataObjectList:collectionAndDataObjectList, parent:parent])
}
I altered my unit test, per the docs, to inspect the model and view. I attempted to get at the model entry for 'parent', which contained the parent dir, like this:
def parent = mav.model.parent
It turns out that that does not work anymore, instead, you need to interpose a reference to linkedHasMap like so:
void testBrowse() {
controller.params.dir = "/"
controller.irodsAccessObjectFactory = irodsAccessObjectFactory
controller.irodsAccount = irodsAccount
controller.loadTree()
def mav = controller.modelAndView
def name = mav.viewName
assertNotNull("null mav", mav)
assertEquals("view name should be loadTree", "loadTree", name)
def parent = mav.model.linkedHashMap.parent
assertEquals("parent dir not found", "/", parent)
}
Now my Grails controller and test are working again. My next step is to create a decent JavaScript object that can bridge between methods in this BrowseController and a lazy-loadable JQuery JSTree. Maybe the next post can share some developments there.
Saturday, January 1, 2011
Testing a Grails Controller - getting the $*&@ thing to work
There is lots of contradictory information on the web, even in the Grails docs, on testing Grails controllers. The biggest issues are understanding the ControllerUnitTestCase, and what it does and does not do.
First, here's a controller I'm developing that will back a JQuery JTree component. The first method will take a parent path name, and return a JSON representation of the iRODS file system for the parent path using some new Jargon methods that are meant to assist in Swing and JQuery tree development.
A few things to note. The new Jargon libraries are meant to provide simple POJO's that represent various domain objects within iRODS. There also is an idea of Access Objects, that roughly equate with the DAO pattern. iRODS is not a database, so the mapping is not the same, but to me, the idea of Access Objects for iRODS at least gives some comfort in familiarity. One access object I'm developing is meant to assist in tree depictions and searching on file and collection paths to support such elements in typical web and Swing GUI applications. We'll use that Access Object.
package org.irods.mydrop.controller
import org.springframework.security.core.context.SecurityContextHolder;
import org.irods.jargon.core.pub.*;
import org.irods.jargon.core.connection.*;
import org.irods.jargon.core.exception.*;
import grails.converters.*
/**
* Controller for browser functionality
* @author Mike Conway - DICE (www.irods.org)
*/
class BrowseController {
IRODSAccessObjectFactory irodsAccessObjectFactory
IRODSAccount irodsAccount
/**
* Interceptor grabs IRODSAccount from the SecurityContextHolder
*/
def beforeInterceptor = {
def irodsAuthentication = SecurityContextHolder.getContext().authentication
if (irodsAuthentication == null) {
throw new JargonRuntimeException("no irodsAuthentication in security context!")
}
irodsAccount = irodsAuthentication.irodsAccount
log.debug("retrieved account for request: ${irodsAccount}")
}
/**
* Display initial browser
*/
def index = { }
/**
* Render the tree node data for the given parent.
*
* Requires param 'dir' from request to derive parent
*
*/
def loadTree = {
def parent = params['dir']
log.info "loading tree for parent path: ${parent}"
def collectionAndDataObjectListAndSearchAO = irodsAccessObjectFactory.getCollectionAndDataObjectListAndSearchAO(irodsAccount)
def collectionAndDataObjectList = collectionAndDataObjectListAndSearchAO.listDataObjectsAndCollectionsUnderPath(parent)
log.debug("retrieved collectionAndDataObjectList: ${collectionAndDataObjectList}")
render collectionAndDataObjectList as JSON
}
}
I've wired in my Spring security objects as described in an earlier post. My interceptor in my controller grabs the saved authentication token from the Spring Security layer and extracts the IRODSAccount object. In the loadTree method, this IRODSAccount object is used with a factory injected into my controller to get a CollectionAndDataObjectListAndSearchAO. My controller uses the method that takes a parent path, and returns JSON objects that represent iRODS files and collections under that parent path.
The approach taken in Jargon seems to map fairly easily into Grails apps, that's a good sign. This shows that quick development of arbitrary interfaces on top of iRODS will become easier with the newer Jargon libraries. Yay!
So, the problems really began with trying to test. First, let me share a working test, then I'll talk about some of the problems I ran into:
package org.irods.mydrop.controller
import grails.test.*
import java.util.Properties
import org.irods.jargon.core.connection.IRODSAccount
import org.irods.jargon.core.pub.IRODSAccessObjectFactory
import org.irods.jargon.core.pub.IRODSFileSystem;
import org.irods.jargon.core.pub.io.IRODSFile
import org.irods.jargon.core.query.CollectionAndDataObjectListingEntry
import org.irods.jargon.testutils.TestingPropertiesHelper
import org.irods.jargon.testutils.filemanip.FileGenerator
import org.irods.jargon.testutils.TestingPropertiesHelper
import org.irods.jargon.spring.security.IRODSAuthenticationToken;
import org.springframework.security.core.context.SecurityContextHolder;
import grails.converters.*
class BrowseControllerTests extends ControllerUnitTestCase {
IRODSAccessObjectFactory irodsAccessObjectFactory
IRODSAccount irodsAccount
Properties testingProperties
TestingPropertiesHelper testingPropertiesHelper
IRODSFileSystem irodsFileSystem
protected void setUp() {
super.setUp()
testingPropertiesHelper = new TestingPropertiesHelper()
testingProperties = testingPropertiesHelper.getTestProperties()
irodsAccount = testingPropertiesHelper.buildIRODSAccountFromTestProperties(testingProperties)
irodsFileSystem = IRODSFileSystem.instance()
irodsAccessObjectFactory = irodsFileSystem.getIRODSAccessObjectFactory()
def irodsAuthentication = new IRODSAuthenticationToken(irodsAccount)
SecurityContextHolder.getContext().authentication = irodsAuthentication
}
protected void tearDown() {
super.tearDown()
}
void testBrowseNoLogin() {
controller.params.dir = "/"
controller.irodsAccessObjectFactory = irodsAccessObjectFactory
controller.irodsAccount = irodsAccount
controller.loadTree()
def controllerResponse = controller.response.contentAsString
def jsonResult = JSON.parse(controllerResponse)
assertNotNull("missing json result", jsonResult)
}
}
One of the first sources of confusion to me were documents that seemed to imply that running the grails command create-controller would put the relevant test in the integration tests directory. It didn't, it placed the test in the test/unit directory. Go figure. Maybe I'm missing something, but I'm taking what Grails does at it's word, so to speak. I tried moving the test to the integration test directory, but found that this resulted in the setup methods not running. Fine...I'll just forget this rabbit-hole.
Second, I had the hardest time setting up the params. My controller expects a request param of 'dir', with my test specifying the root '/' directory. I kept getting a "No such property: params" error when I tried doing the
controller.params.dir = "/"
test setup. It turns out that the ControllerUnitTestCase is wanting to wire in and extend your controller for you, creating a 'controller' variable automatically, with the test target controller defined by convention using the test name. So instead of creating my controller in a def using the 'new' keyword, I had to let the ControllerUnitTestCase do it for me.
I also ran into the issue of irodsAccount being null. The ControllerUnitTestCase does not run my interceptor for me. I was getting a null irodsAccount, and realized that I had to manually inject that variable in my test code.
Well, I was almost home, I kept getting a 'null' for my controller variable in my test case. Doh...I had plugged a bunch of code into setUp() in my test case, and I had to go back and add the super() call in my test case so that ControllerUnitTestCase could do it's magic on the controller variable. So now it works! There's still lots of refactoring I want to do in the controller, and I need to do things like parse and test the actual JSON response, but these are the details to work out now that I can get the stupid test cases to run.
Now that's progress...hopefully that will help you avoid some of my own frustrations. Here's to a New Year and new adventures in coding. I think Grails/Groovy feels as immature as most new dynamic scripting languages, but at the same time, I really feel like, combined with the new Jargon libraries, I'm well on my way to a rapid web development stack on top of iRODS. That's a good thing!
Wednesday, December 29, 2010
Grails and Spring Security
My Spring Security library has a custom AuthenticationManager as well as a custom AuthenticationFilter and AuthenticationToken, as iRODS security uses not only user and password, but also zone, resource, host, and port. Because of the differences, I did not find the Spring Security Grails plug-in to be suitable. Besides, since I am already testing a set of custom libraries, I wanted to take advantage of the library and the XML wiring I had already created.
First, here's what my XML wiring looks like:
I had started to re-create this using Spring DSL in my resources.groovy file, but I was finding it difficult to translate (I'm still very new at Spring DSL, and it had more to do with just getting this done).
I added my mappings to the resources.xml file in my Grails app, and started things up just to see if Spring could wire this together. Knock me over with a feather but that worked! The one issue I had was a complaint about the Spring Security namespace, which was corrected by adding these items to my BuildConfig.groovy:
dependencies {The key was actually the spring-security-config dependency.
// specify dependencies here under either 'build', 'compile', 'runtime', 'test' or 'provided' scopes eg.
compile 'org.irods:jargon-core:0.0.3-SNAPSHOT'
compile 'org.irods:jargon-security:0.0.1-SNAPSHOT'
compile 'org.springframework.security:spring-security-core:3.0.5.RELEASE'
compile 'org.springframework.security:spring-security-web:3.0.5.RELEASE'
compile 'org.springframework.security:spring-security-config:3.0.5.RELEASE'
//test 'org.irods:jargon-test:0.0.1-SNAPSHOT'
// runtime 'mysql:mysql-connector-java:5.1.5'
}
Next, I started testing with a browser, only to find that the security was not being enforced. The missing step was to add the delegating filter proxy to my web.xml file. Well...what web.xml file? Like I said, I'm very new to Grails. It turns out that I needed to fire up the grails console and do a
grails install-templates
command. This adds templates for various artifacts to my project. Then it's a simple matter of editing the template web.xml file to add my delegating filter proxy mapping like so:
Now, I have not completed all the testing, but it does seem like I'm on the right path. I'll let you all know how it's going, but I think it'll work out.
Tuesday, December 28, 2010
Fedora 14 running full-screen on Virtual Box for Mac
http://www.sysprobs.com/install-fedora-14-virtualbox-working-guest-additions
As I was having endless headaches mining the Google for tips on getting Fedora Linux 14 to run in full-screen mode on my Mac using VirtualBox. The missing pieces were some yum updates that had to be run, it's now running beautifully.
As is typical, I had to wade through several pages of hits that didn't solve the problem, hopefully this will boost the real solution.