Vid from Supercomputing 08

Here's some Renci Vid from SC08, showing some of the technology in the Renci booth. As stated before, I'd love to put a touch table in as a centerpiece to the Social Computing Room, and have multi-touch applications that interact with the 360-degree viz display. This idea meshes well with the ideas demonstrated by InfoMesa.

Also cool to note is the HoloPanel that can be seen in the corner of the booth. A pair of these was used in the Social Computing Room as a part of the Spectacular Justice installation, and they were stunning!

All that said, really this is just to share some cool technology. I'm lucky to work with such smart and resourceful people at Renci.

Renci Multi-Touch blog

Here's the multi-touch blog from Renci. There are two form factors that Renci is working with, a large multi-touch wall at Duke, and a horizontal touch-table at Europa.




My hope is that a touch-table will grace the Social Computer Room. A long-term vision would be to extend our Collage/InfoMesa ideas in the SCR, using the 360-degree display to provide visual real estate. Imagine a group working around a touch table, shooting images out to the 360-degree wall with gestures on the touch table.

Putting Google Earth into a WPF window

This may end up being fruitless, but I was inspired by this multi-channel version of Microsoft Virtual Earth. It looks like they linked multiple version of Virtual Earth with different camera settings, and I wanted to try something similar with Google Earth for our Global Immersion dome. This four-projector rig needs a viewport and four camera views to work right. Can I create a full-screen app that has these four viewports, and have four synchronized version of Google Earth running? I don't know, but the first step was to see if I could create a WPF app that had Google Earth embedded in the WPF window. You can at least do that, and that's interesting in itself, because I can add a Google Earth widget to my InfoMesa/Collage experiments described here...

Anyhow, here's the window in all its (yawn) glory:



It was a bit of a slog to get it right, and I'll share the code that worked. First, I had to get Google Earth, which gives you this COM SDK. I did this using C#, Vis Studio 2008. I added a project ref to the COM Google Earth library, and created a class that extended HwndHost.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Windows;
using System.Windows.Controls;
using System.Windows.Data;
using System.Windows.Documents;
using System.Windows.Input;
using System.Windows.Media;
using System.Windows.Media.Imaging;
using System.Windows.Navigation;
using System.Windows.Shapes;
using System.Windows.Interop;
using System.Runtime.InteropServices;
using EARTHLib;

namespace GeTest
{
class MyHwndHost : HwndHost
{


[DllImport("user32.dll")]
static extern int SetParent(int hWndChild, int hWndParent);
IApplicationGE iGeApp;


[DllImport("user32.dll", EntryPoint = "GetDC")]
public static extern IntPtr GetDC(IntPtr ptr);



[DllImport("user32.dll", EntryPoint = "GetWindowDC")]
public static extern IntPtr GetWindowDC(Int32 ptr);

[DllImport("user32.dll", EntryPoint = "IsChild")]
public static extern bool IsChild(int hWndParent, int hwnd);


[DllImport("user32.dll", EntryPoint = "ReleaseDC")]
public static extern IntPtr ReleaseDC(IntPtr hWnd, IntPtr hDc);

[DllImport("user32.dll", CharSet = CharSet.Auto)]
public extern static bool SetWindowPos(int hWnd, IntPtr hWndInsertAfter, int X, int Y, int cx, int cy, uint uFlags);

[DllImport("user32.dll", CharSet = CharSet.Auto)]
public static extern IntPtr PostMessage(int hWnd, int msg, int wParam, int lParam);

//PInvoke declarations
[DllImport("user32.dll", EntryPoint = "CreateWindowEx", CharSet = CharSet.Auto)]
internal static extern IntPtr CreateWindowEx(int dwExStyle,
string lpszClassName,
string lpszWindowName,
int style,
int x, int y,
int width, int height,
IntPtr hwndParent,
IntPtr hMenu,
IntPtr hInst,
[MarshalAs(UnmanagedType.AsAny)] object pvParam);


readonly IntPtr HWND_BOTTOM = new IntPtr(1);
readonly IntPtr HWND_NOTOPMOST = new IntPtr(-2);
readonly IntPtr HWND_TOP = new IntPtr(0);
readonly IntPtr HWND_TOPMOST = new IntPtr(-1);
static readonly UInt32 SWP_NOSIZE = 1;
static readonly UInt32 SWP_NOMOVE = 2;
static readonly UInt32 SWP_NOZORDER = 4;
static readonly UInt32 SWP_NOREDRAW = 8;
static readonly UInt32 SWP_NOACTIVATE = 16;
static readonly UInt32 SWP_FRAMECHANGED = 32;
static readonly UInt32 SWP_SHOWWINDOW = 64;
static readonly UInt32 SWP_HIDEWINDOW = 128;
static readonly UInt32 SWP_NOCOPYBITS = 256;
static readonly UInt32 SWP_NOOWNERZORDER = 512;
static readonly UInt32 SWP_NOSENDCHANGING = 1024;
static readonly Int32 WM_CLOSE = 0xF060;
static readonly Int32 WM_QUIT = 0x0012;

private IntPtr GEHrender = (IntPtr)0;
private IntPtr GEParentHrender = (IntPtr)0;

internal const int
WS_CHILD = 0x40000000,
WS_VISIBLE = 0x10000000,
LBS_NOTIFY = 0x00000001,
HOST_ID = 0x00000002,
LISTBOX_ID = 0x00000001,
WS_VSCROLL = 0x00200000,
WS_BORDER = 0x00800000;



public ApplicationGEClass googleEarth;

protected override HandleRef BuildWindowCore(HandleRef hwndParent)
{

// start google earth
googleEarth = new ApplicationGEClass();

int ge = googleEarth.GetRenderHwnd();

IntPtr hwndControl = IntPtr.Zero;
IntPtr hwndHost = IntPtr.Zero;
int hostHeight = 200;
int hostWidth = 300;

// create a host window that is a child of this HwndHost. I'll plug this HwndHost class as a child of
// a border element in my WPF app,

hwndHost = CreateWindowEx(0, "static", "",
WS_CHILD | WS_VISIBLE,
0, 0,
hostHeight, hostWidth,
hwndParent.Handle,
(IntPtr)HOST_ID,
IntPtr.Zero,
0);


// set the parent of the Google Earth window to be the host I created here
int oldPar = SetParent(ge, (int) hwndHost);
// check to see if I'm now a child, for my own amusement
if (IsChild(hwndHost.ToInt32(), ge)) {
System.Console.WriteLine("now a child");
}

// return a ref to the hwndHost, which should now be the parent of the google earth window
return new HandleRef(this, hwndHost);
}

protected override void DestroyWindowCore(HandleRef hwnd)
{
throw new NotImplementedException();
}
}
}



The main window of my WPF app, in its constructor for the Window, just plugs this HwndHost as a child of a Border control:

public Window1()
{

InitializeComponent();
MyHwndHost hwndHost = new MyHwndHost();
border1.Child = hwndHost;


}


And you are off to the races! I found a lot of different approaches to this all over the web, but none of them seemed to work, as is often the case. Maybe this will work for you, or maybe it adds to the confusion.

UPDATE: can't run more than one Google Earth, so it's a bust, but still has use in my InfoMesa/Collage project. I wonder about Virtual Earth?

Video for Ubisense/MIDI demo

Here's the video that was described in this post. The point of this experiment was to see if we could reasonably map carpet squares in the Social Computing Room to MIDI notes, and output those notes to the on-board MIDI implementation.




Oh well...off to a Games4Learning event...see ya there!

A bit about the Social Computing Room

In the last blog entry, I had put down a spot to link to a description of the 'Social Computing Room', and realized that I didn't have one. So I wanted to fill in a few details and fix that link.

The Social Computing Room, (or SCR for short) is a visualization space at the Renci Engagement Center at UNC Chapel Hill. We're over near the hospital in the ITS-Manning building in Chapel Hill. It's one of three spaces, the other being the Showcase Dome (a 5-meter tilted Global Immersion dome), and Teleimmersion, which is a 4K (4 x HD resolution) stereo environment. We're working on some virtual tours for a new web site, so there should be some more info soon on those other spaces.

One of the primary features of the SCR is its 360-degree display. The room is essentially a 12,288x768 Windows desktop. (I've also tested a Mac in this environment, and it works as well). Here's a pic of the SCR...



The room has multiple cameras, wireless mics, multi-channel sound, 3D location tracking for people and objects, and is ultra-configurable (plenty of cat-6 for connecting things, Unistrut ceiling for adding new hardware). The room has so many possibilities that it gets difficult to keep up with all of the ideas. I think of it as a place where you can paint the walls with software, and make it into anything you want. There are currently a few emerging themes:

• The SCR is a collaborative visualization space. The room seems especially suited for groups considering a lot of information, doing comparison, interperetation, and grouping. There is a lot of visual real estate, and the four-wall arrangement seems to lend itself to spatial organization of data. As groups use the space for this purpose, I'm trying to capture how they work, and what they need. The goal is to create a seemless experience for collaboration. This is the reason I've been interested in WPF, and the InfoMesa technology demonstrator, as covered in this previous post.
• The SCR is a new media space. Its been used for art installations, and it has interesting possibilities for all sorts of interactive experiences, as illustrated by this recent experiment.
  
• The SCR is a place for interacting with the virtual world. We're working on a Second Life client that would have a 360-degree perspective, so that we can embed the SCR inside of a larger virtual enviroment, enabling all sorts of new possibilities.

These are just a few of the areas I'm interested in. Each of the areas can be enhanced by the use of different types of sensors and robotics, and I've been started with Wiimotes, SunSpots, and the Ubisense location tracking hardware.

That's a bit about the SCR, it's a really fascinating environment, and if you are on the UNC campus, give me a shout out and I'll show you around!

Music and Media in the SCR

UPDATE: delay on getting the video done, should be here by early this week...MC

Here's an interesting prototype that combines the Social Computing Room with Max/MSP/Jitter and UbiSense. Video will follow soon.

The Social Computing Room (described here) has many potential applications, and one intriguing use is as an 'interactive media space'. The idea is that 360-degree visuals, combined with various types of sensors, software, and robotics, can create new kinds of experiences. Two examples that I can point to include the 'Spectacular Justice' exhibit that occurred last winter, as well as student work with Max/MSP/Jitter.

In this case, a prototype was written that uses UbiSense, which provides location tracking in 3D through an active tag. A Max object was written in Java to take the UbiSense location data off of a multicast stream, and push it out into Max-land. A second Max object was created to take the x,y,z data from UbiSense, in meters, and convert it into numbers that match up to the carpet squares on the floor of the Social Computing Room. Given those two new objects, the 'pad' number of the carpet square can be mapped to a MIDI note, and sent out through the Max noteout object.

Here's a picture of a simple patch:



What I'm really trying to show are techniqes for interacting with music and video. I could see using objects in the room that can be arranged to create musical patterns using an arpegiator or a loop player, and this can be combined with video on all four walls. MIDI or other methods can simultaneously control lights and other electronics. You could create a human theramin by having two people move around the room in relation to each other.

It's also interesting to let several people move around in the SCR holding multiple tags, you can create semi-musical patterns by working as a group. It's a fun thing, but points to some interesting possibilities. I've also adapted a Wiimote in the same manner.

Adapting InfoMesa to the Social Computing Room

InfoMesa is an interesting initiative by Sam Batterman at Microsoft Life Sciences...quoting from Sam's blog:

InfoMesa is a project to allow scientists to do more science and more discovery in a collaborative and data-rich environment. The metaphor that we have elected to use as the underlying fabric of the InfoMesa is a Whiteboard.

InfoMesa allows any kind of data or visualization to be added to the Whiteboard. Far from static, these tools are interactive, allowing data to be absorbed from data sources like Oracle, SQL Server, Excel Spreadsheets, XML or even Cloud-based web services. InfoMesa, when complete will support imagery, video, 2D connected models, 3D models (lit in a photo realistically manner), web searches, results from web service calls, Image Tile Maps, ScatterPlots, Sticky Notes, Ink Notes, Rich Annotations and Associations.

Check the blog link for screen shots, it's a really interesting application, as well as a nice example of the capabilities of WPF (Windows Presentation Foundation). As a died-in-the-wool Java and Ruby programmer, I don't necessarily fit the typical Microsoft bandwagon profile, but I am having quite good success leveraging the WPF framework for the challenging environment in our Social Computing Room, which is essentially a 12,288 x 768 desktop.

A bit of a rewind, the Social Computing Room is part of the Renci Engagement Center at UNC Chapel Hill. The SCR has a 360-degree visual display running on all four walls, with 12 projectors per wall. The relevant model we were already working on was a supporting environment for researchers working in collaborative groups, and considering LOTS of data at one time. The SCR is a great venue for approaching problems that fit the 'small multiples' mold.

One of my colleagues, Dave Borland, had created a prototype called 'Collage'. This prototype used OpenGL and C++ (including some nifty wxWidgets sleight of hand to allow OpenGL to render such a large visual application). Collage handled cool things like letting the mouse, and any images, wrap all the way around the room. Collage could also play videos, and we were looking at adding other capabilities. Another cool part of Collage was the ability to intelligently 'lay out' images. For example, it was a common activity to expand each image to size to one 'projector', avoiding any stitch lines between displays. We were also working on displaying metadata about the images on command, sorting data various ways, and generally assisting 'small multiple' visualization tasks.

The downside of Collage is that it was a bit hard to extend, requiring a lot of OpenGL and wxWidgets prestidigitation to add new features. There were further plans to add Wiimote integration for multi-user input, and the ability to assign functionality to each wall. E.g., have a magnification wall, where thumbnails that were dragged to the magnification wall would automatically size for comparison.

After seeing InfoMesa in prototype form, I realized that many of the ideas we had in Collage mapped nicely onto the InfoMesa concept, and InfoMesa really moved the ball down the field. The first question I had was whether WPF would support, in a performant manner, a 12,288x768 desktop, and I was pleasantly suprised! The thing I've been working on for the past couple of weeks is taking the InfoMesa code, and adapting it to cover the functionality we already had in the OpenGL Collage prototype. I've been concentrating on the visual interface so far, leaving persistance, annotation, and metadata for later work. Here's a short vid of the CollageWPF adaptation of InfoMesa:

I wanted to hit on a few of the 'features' we've added, some requested by researchers who are using the prototype:

• InfoMesa is turned 'inside out', maximizing real estate. Wrapping controls and toolbars around the whole display doesn't work on a big wall, so I went with a right-click context menu. It would be cool for InfoMesa to expand full desktop or display the interface! Otherwise, it might be cool to concentrate on partitioning the application such that the host 'window' can be easily customized for various display types.
  
• Ability to automatically lay out and size imagery, which I implemented by creating a SceneManager to describe the environment, and a LayoutManager that can be subclassed for various layouts. The first LayoutManager does a scale and position to get one image per projector. The idea is that SceneManagers could be created for other viz environments, such as a 3x3 viz wall, or a 4K high-def display.
• I Started thinking about a base 'widget' that can be subclassed to create other tools. Here I've still got things to learn about InfoMesa! I also started thinking about how these subclassed widgets would keep and share metadata, and allow the host 'cavas', or 'Universe', to know its widgets, and be able to manipulate them for things like layouts.
• I added a widget to display 'time series' images in a player. It will eventually work by synchronizing multiple 'time series' viewers so researchers can consider different model runs simultaneously. I also added a widget to digest a power point, break into images, and then lay out those images.
  
• The original InfoMesa zooms the entire desktop. Researchers were really looking to scale individual images.
• I added mouse-over tool tips to display image metadata.

There's a good rundown of the use cases for InfoMesa here, and I think they are spot on. I think there's some adaptation that will be needed to get InfoMesa to fit the SCR form factor, and there are several other viz environments that also can serve as test cases, so it should be interesting. I'd like for a researcher to be able to log in to an InfoMesa room, create a white board, provision it, then go into the SCR, log in there, and see the white board depicted and ready to roll. Another angle would be to have some sort of event model that would publish changes on one viz wall to render in another...wall-to-wall collaboration would be way cool!

More later....be sure to vote tomorrow. As a political junkie, I'm sure I'll be pretty tired looking on Wednesday morning.