Adobe Flash Catalyst Alpha Released

Adobe Labs released the Alpha build of Flash Catalyst, previously codenamed “Thermo,” and distributed it at Adobe MAX last week. Flash Catalyst is a new design and development tool that lives halfway in between Photoshop/Illustrator/Fireworks and Flex Builder.

These days most applications are created by two separate teams: design and development. As applications increase in complexity, the number of individuals capable of effectively creating both sides of an application by oneself is rapidly diminishing. Therefore, at some point the designer needs to hand off the comps to a developer to “make work.” This step is a critical, complex and typically expensive part of the software development lifecycle. In fact, some companies devote entire departments to the task to improve the efficiency and accuracy of the translation from design into code.

Flash Catalyst is not intended for designers to create designs directly inside it. Photoshop, Illustrator and Fireworks have much richer design toolsets. Likewise, Flash Catalyst is not intended for developers to create primary application logic inside it. That’s what Flex Builder is for. Rather, Flash Catalyst is a bridge to allow designers to take what are essentially comps and begin to basic interactivity to them, allowing Catalyst to create actual usable MXML behind the scenes. This pushes the handoff point further down the development side and eases the burden on developers to interpret the designers intent.

For simpler projects like basic websites, electronic press kits, rich content ads, online portfolios, the final output of a Flash Catalyst file may have enough functionality and interactivity for the designer to not even need a developer to complete the project. For more complex functionality like applications (whether internet, SaaS, or enterprise) or projects that need back-end server connections, a developer can import the Catalyst project file directly into Flex Builder to finish the job.

To see an early version of Flash Catalyst, when it was still codenamed “Thermo,” check out this three-part video Aral Balkan recorded last year at Adobe MAX in Chicago.

Thermo preview, part 1

Thermo preview, part 2

Thermo preview, part 3

The functionality Flash Catalyst offers may be likened to Microsoft’s Expression Blend, which lives halfway in between Microsoft Expression Design and Visual Studio. While it is still too early to meaningfully compare and contrast Catalyst (still Alpha at this stage) and Blend (second major revision already released), I can still offer my expectation of the differences between the two products’ design-time experiences.

I should note there are a number of comparisons and contrasts that I deliberately avoid in this post.

• This is not about Flash vs. Silverlight. I’ve already covered that topic here.
• This is not about Actionscript vs. C#. Since they’re both ECMAScript languages that comparison is mostly just about the different implementations of the ECMAScript standard. Not very interesting.
• This is not about MXML vs. XAML. Since they’re both XML-based declarative markup structures that comparison is effectively meaningless.
• This is not about Flex Builder vs. Visual Studio. I’m sure you can find plenty of blogs that can pound the whole Eclipse vs Visual Studio argument into the ground for you.
• This is not about Photoshop/Illustrator/Fireworks vs. Expression Design. That comparison is so rediculous it doesn’t even merit discussion.
• This IS about Adobe Flash Catalyst vs. Microsoft Expression Blend. That’s it.

So the bottom line is: which product is better? Well, it’s hard to say definitively. Since the tools are both “integration applications” that exist to more seamlessly connect other major software suites, its almost impossible to evaluate the Catalyst vs. Blend argument without comparing the toolsets they live in between. I guess the only real litmus test is this: put Photoshop, Illustrator, Fireworks, Expression Design, Flex Builder and Visual Studio out of your head and think hypothetically. If both Catalyst and Blend could be used on either platform, which would people choose? Would you rather use Catalyst or Blend to turn a comp into an application? I think we will have to wait until Catalyst is released (or at least Beta) to answer that question fairly.

Adobe has made some big promises here. If Flash Catalyst lives up to them it will be a hit. Guaranteed. I’d even go so far as to suggest that it may be another “game-changer.” If you weren’t one of the lucky ones to get a copy of the Alpha build at Adobe MAX, you can sign up to be notified of when the public Beta is available early next year at http://labs.adobe.com/technologies/flashcatalyst/.

Adobe Creative Suite 4 Changes The Game

Earlier this year I wrote a post speculating that Adobe intended to add content aware resizing A.K.A. “seam carving” to Photoshop CS4. Well, now that CS4 has been released I can proudly announce that I was right and it is indeed a “game-changing” feature. However, now that I’ve had the opportunity to pore over all the new features in the suite I can post my favorite new and improved features. Note, I’m not going to review every single product in the Creative Suite family. I don’t use them all. I’ll just cover the products User Interface experts use most frequently: Photoshop, Illustrator, After Effects and Flash.

If you just want the bottom line instead of reading another long UIHero post here it is: Buy Adobe Creative Suite 4. Now. CS4 is, without question, the most compelling major revision Adobe has released since they added Layers to Photoshop in version 3.0 back in 1994. Yeah. It’s that good.

Photoshop CS4 Extended

Photoshop, along with Illustrator, is the backbone of the Creative Suite platform. Justifiably, Photoshop Extended has received a healthy dose of upgrade goodness. My favorite features:

3D Painting and Compositing
I can’t tell you how many times I’ve had to create a 3D model in a 3D modeling tool, import an alpha-mapped still into Photoshop, and paint all over it only to have the client make some change that requires me to go back and adjust the model and start all over. Well, those days are over. Now you can work with 3D models in Photoshop natively! Import a standard-format 3D model and paint directly on the 3D surfaces. Apply a 2D images to a 3D shape and it will wrap around the contours of the solid. Create true 3D objects from 2D objects like gradient maps and text.

Content-Aware Resizing
This feature is so awesome I covered it in its own post before I was even certain it would be a feature is CS4. Read about it here: http://uihero.wordpress.com/2008/09/16/seam-carving-in-adobe-photoshop-cs4/

Masks Panel
Masking in Photoshop is an art in and of itself. This panel adds new power to those that take their masks seriously. Now you can edit and refine your pixel- and vector-based masks in ways that were previously possible but required lots and lots of steps. Feather a mask, adjust its density, color sampling range, all with nondestructive adjustments.

Other top new features in Photoshop Extended CS4.

Illustrator CS4

The upgrades to Illustrator aren’t the same kind of radical technology improvements that 3D Painting or Content-Aware Resizing in Photoshop are but they’re long overdue. I’m happy to see Illustrator finally get the vector tools that other programs have had for years.

Transparency in Gradients
Wow. Illustrator’s greatest flaw has FINALLY been removed.

Blob Brush
You can now “paint” vector shapes the way you can in Flash. This is huge for people that think in brushes instead of pens. Sometimes it’s just easier to paint your shape the way you want it instead of dealing with Bezier curves.

Stackable Styles
You can now apply multiple Graphic Styles to an object and get cumulative effects. This feature alone is incredibly powerful. Create a library of shadow, texture and color styles and apply them in a modular way to very, very quickly create uniform interfaces.

Multiple Artboards
Now you can comp all your layouts for each format in one file, just switch back and forth between different artboards at will. You can also create multi-page PDFs from artboards in a file. Nice one Adobe.

In-Panel Appearance Editing
The Appearance panel now has convenient popout buttons to edit Fill, Stroke, and Effects so you don’t need to open those palettes so often.

Other top new features in Illustrator CS4.

After Effects CS4

Don’t sleep on the new After Effects! Even if all you do is motionography and don’t really want the rest of CS4, there are enough awesome new features in CS4 to make you rush out and After Effects CS4 as a stand-alone product.

Direct Export to Flash Professional CS4
Now you can export a compositions as layered, XFL format projects that can be opened and edited in Adobe Flash CS4 Professional. Text and vector artwork are preserved, not rasterized, and can be further edited and animated in Flash.

Import 3D Models from Photoshop
Really, the most interesting upgrade to After Effects is the integration with the new 3D features in Photoshop. However, don’t underestimate the power of that fact! If you can retain editable 3D models from end to end you can save enough money to pay for your CS4 license in one project!

Imagineer Mocha in After Effects
Mocha is a 2.5 D planar tracking system that was previously an expensive, but critical, tool for 2.5 D animation. Without planar tracking, compositors had to do deal with a lot of of time consuming hand tracking shots and deal with the inaccuracies and guesswork as a result. Then there was Mocha, which, for a price, would do that for you. Well, now Mocha is a native part of After Effects. This feature alone is worth the price of the software

Cartoon Effect
The cell-shaded effect seen in the films Waking Life and A Scanner Darkly, and in Earthlink commercials, is now a native part of After Effects. While it’s not a revolution, it’s still a cool tool to have in your bag of tricks. LiveTrace added the effect to stills. Now you have it available in motion too.

Other top new features in After Effects CS4.

Flash Professional CS4

Over the last year, the most dramatic improvements to Flash were to improvements in the Flash platform, player and virtual machine itself, not the Flash Professional IDE. That said, there are still a number of hot new features that will have you reaching into the bottom of your pockets to pony up the cash for it.

Object-Based Animation
Say you animate an object on the timeline. Later you need to move the entire object and all its animation to the left 30 pixels. Previously, you needed to find each keyframe and move them all individually. Well, no more. Timeline animations now follow the object itself! Tweens are now attached to objects instead of to keyframes. Also, Bezier handles on tweens are a great feature.

3D Animation
There is now a Z axis so you can translate and rotate 2D objects in 3D space. Awesome.

Motion Editor
Yay! The ultra-accurate keyframe property editor in After Effects is now in Flash Professional! Adjust rotation, size, scale, position, filters, and easing control with accurate, detailed graphical displays.

Inverse Kinematics
The new Bones tool (what a cute name) allows you to quickly create inverse kinematics, ragdoll objects and similar objects that used to take a pretty sophisticated knowledge of Actionscript and lots of time.

Other top new features in Flash Professional CS4.

These features are just my favorites. There are hundreds of of other improvements and new features in CS4. go out and buy it, play with it, and post a comment back telling me what you love most. It’s a great era to be a designer.

Where are all the cool WPF applications we were promised?

Ina Fried, a reporter for CNET that provided great coverage of IdentityMine’s EventPhotos Windows 7 / Surface application at PDC, asked a question in her live blog:

“why aren’t we seeing more of these Windows Presentation Foundation applications? They look super cool, but I only ever see them at Microsoft developer conferences.”

Great question, indeed, but there actually are lots and lots of WPF applications in the marketplace right now. What I believe Ina really asks is “why aren’t there more Windows applications with impressive user interfaces?” The answer lies in looking at the history of the tools that have been available to create UI for Windows applications. As a WPF developer and designer, as well as a Flash/Flex developer and designer, I believe I can provide some insight into this question.

Until the release of WPF, the tools used to create Windows applications, like WinForms, made it very difficult to create a Windows application that looked like anything other than, well, a Windows application. As a result, the vast majority of pre-WPF applications have boring user interfaces – the same square grey layouts, the same buttons, the same pull-down menus, etc. Except for a select few WinForms UI Ninjas, and believe me they are exceptionally rare, it was just too difficult to create dazzling designs with the tools Windows applications were written with a generation ago so most software companies just didn’t bother.

On the other side of the fence, Adobe products like Flash, Flex and AIR, have enabled designers and developers to easily create stunning user interfaces for years now. There are thousands of examples of flashy-looking applications, both well-executed and otherwise, that take advantage of the excellent design-time experience Flash and Flex offer. Consequently, when one thinks of a user interface with a sexy, eye-catching look, one usually thinks of Flash, not Windows.

Today, Windows Presentation Foundation gives Windows developers and designers the ability to create impressive-looking applications that rival anything created with Flash. However, just because the tools are there doesn’t mean people are actually using them to their full potential. Cool looking WPF applications don’t “just come out that way.” They have to be designed. Unfortunately, at most Windows software shops the user interface is still an afterthought, not an essential part of the application design process. So, even if these shops are creating their application with WPF, if they don’t use any custom graphics or animations, the application will look the same as an application written with WinForms or other older tools.

Another reason for the slow adoption of designed, custom user interfaces in Windows applications is simple: Windows user interface conventions have been around for over two decades (Windows will celebrate its 23rd birthday on November 20th). When a user sees an unstyled WinForms control they already know how to use it. There must be a compelling reason to create a button that doesn’t look like every other button. Likewise, it takes a great deal of expertise to create a custom user interface that communicates its purpose effectively. A custom user interface created by an inexperienced or ineffective designer will actually make an application less usable. Many software companies fear departing from tradition because of this danger.

Some companies are brave enough to undertake the risk and cost of redesigning their applications in order to gain a competitive advantage. When they succeed, their competitors take notice and the consumer begins to expect well-designed user interfaces. Slowly and steadily, as more companies realize the importance of the user interface in their products, they start to become interested in leveraging WPF. They start to want to create user interfaces and animations that feel like they could have been lifted from a high-end Adobe Flash, Flex or AIR application. However, this process takes time: top notch design professionals need to be hired (often, as in my case, taken from top-tier Flash shops and converted into WPF experts), new user interface design workflows need to be integrated with existing software design workflows, and old Windows development habits (like forgetting to consider the UI) need to be unlearned.

Because of the weight of this paradigm change, only a handful of software shops are ahead of the game at this point in time. At IdentityMine, the user interface is an essential part of any application. IdentityMine has always had a best-in-class team of designers and developers on staff to envision, design and create compelling UI in Windows applications. Just look at our reel below and you can see what can be done with WPF in capable hands. Eventually, more software shops will get up to speed and more and more Windows applications will have user interfaces as beautiful as Flash applications. Until then, just keep your eyes on the few shops, like IdentityMine, that are already ahead of the game.

World’s First Multi-Touch Windows 7 Application

This week, we here at IdentityMine are proud to unveil the world’s first application to demonstrate multi-touch in Windows 7 at the Microsoft Professional Developers Conference in Los Angeles. Additionally, this application, called EventPhotos, is the first application to use Windows 7 and Microsoft Surface together in concert. IdenitiyMine is the first company in the world other than Microsoft that can create a multi-touch-enabled application for Windows 7.

There certainly is a great deal of buzz around this application. Elinor Mills and Ina Fried gave us some great press coverage on CNET.

Here’s the video of EventPhotos: IdentityMine’s application that combines Microsoft Surface, Windows 7, Windows Vista and Wi-Fi memory cards.

As an added bonus, not only did we create the application in the video above, but we also ported our popular Surface Hockey game to Windows 7 as well. It has proven to be really challenging and fun playing Windows 7 Air Hockey it on a 42″ multi-touch plasma screen!

WPF Assembly Source Code Now Available

As promised so long ago, Microsoft has publicly exposed the source code for the WPF assemblies in .NET 3.5 SP1! Not only is the source available, but now developers can step into the inner workings of WPF with the Microsoft Reference Source Server when debugging!

This is FAR Microsoft exposing the entirety of .NET 3.5 SP1. Microsoft is not becoming Mozilla, here. Nonetheless, I whole-heartedly applaud Microsoft making this important step toward embracing open standards and principles, selfless information sharing and transparent development.

The source for all these assemblies is available:

• NPWPF
• ReachFramework
• PenImc
• System.Printing
• PresentationBuildTasks
• System.Windows.Presentation
• PresentationCFFRasterizer
• UIAutomationClient
• PresentationCFFRasterizerNative_v0300
• UIAutomationClientsideProviders
• PresentationCore
• UIAutomationProvider
• PresentationFramework.Aero
• UIAutomationTypes
• PresentationFramework.Classic
• WindowsBase
• PresentationFramework.Luna
• WindowsFormsIntegration
• PresentationFramework
• wpfgfx_v0300
• PresentationFramework.Royale
• PresentationFontCache
• PresentationHostDll
• PresentationHost
• PresentationHostProxy
• XamlViewer_v0300
• PresentationNative_v0300
• XPSViewer
• PresentationUI

See the Reference Source Code Center blog to get started. Happy coding!

Seam Carving in Adobe Photoshop CS4

[UPDATE] It has been confirmed that Seam carving is indeed part of Photoshop Extended CS4. Read my review of Adobe Creative Suite 4 here.

On Monday, 8 September, 2008, Adobe announced in the Adobe Photoshop World Keynote that they hired Shai Avidan, co-creator of Seam Carving (also called Content-Aware Image Resizing). Presumably, Adobe intends to integrate Seam Carving as an image transformation option in Adobe Photoshop CS4. This feature will be the single most impressive technological upgrade to Photoshop’s feature set since the introduction of Layers in Photoshop 3.0 (wow, I’ve been using Photoshop since 1994. Time flies…).

What is Seam Carving?

The simplest way to non-uniformly resize an image is to either crop it or to cut rows/columns out of it. To make the edit as seamless as possible one needs to remove the least important, lowest-transient columns or rows. Unfortunately, no matter how hard you try, just cutting straight rows/columns out of an image almost invariably result in obvious distortion artifacts.

Instead of cutting straight rows or columns out of an image, Seam Carving uses dynamic programming algorithms to determine the least noticeable paths from one side of an image to another, called seams, and cuts the image along those seams. These paths are not straight, like a row or column, but rather, they naturally follow the contours of the content in the image. This is why the technique is sometimes called Content Aware Image Resizing.

Seam Carving is to bitmap images what Granular Synthesis in Ableton Live is to audio: a way to make your images “elastic,” easily allowing the designer to adjust your proportion and composition in ways that previously required tremendous expertise with the Clone tool. Getting your composition to balance perfectly within the Golden Ratio will become easier than ever!

Explaining it in words doesn’t do it justice. Watch this video to really understand what the technology is.

Excited yet? If you’re a designer you should be. The world’s best bitmap editor is about to get a whole lot better.

PhysicsPanel: Physics-Enabled WPF and Surface Applications

One of the consistently frustrating things about working on the cutting edge of user interface design and technology is that most of what I work on is so top-secret that I usually can’t blog about it for fear of invoking the wrath of the Non-Disclosure-Agreement-Gods. Well, IdentityMine has finally given me permission to expose one of our hottest new technologies: PhysicsPanel.

A crack team here at IdentityMine created a WPF Panel called PhysicsPanel that adds physics interaction, like mass, inertia, gravity and friction, to ordinary WPF controls. The first proof-of-concept we made for this library was an air hockey game for Microsoft Surface. These videos demonstrate the game itself as well as how to use PhysicsPanel. As far as I know this is the first public demonstration of how to write code for Surface applications!

At the moment the PhysicsPanel libraries are not publicly available, though if you’d like to hire IdentityMine to create an application for you than requires it, you can reap the fruits of our labor that way. At some future date IdentityMine Enterprise Libraries may become available for licensing. I’ll be sure to blog it when that day comes.

By the way, the crack team is in the video. Andrew Whiddett and Victor Gaudioso are the two players and I wrote and narrated the vid.

PhysicsPanel and Surface Air Hockey walkthrough, part 1

PhysicsPanel and Surface Air Hockey walkthrough, part 2

Okay. Seriously. Is there anything IdentityMine can’t do?

IdentityMine Surface Videos

The promo video for the last few showcase applications IdentityMine created for Microsoft Surface is now available in an attractive, digestible video. Bonus points: Thievery Corporation on the soundtrack.

IdentityMine’s other videos, including the general reel, are also available on the IdentityMine YouTube and Vimeo sites, too.

N-Trig DuoSense – Wacom Killer?

Wacom – The 800 Pound Gorilla of Stylus Input

Every designer worth his or her salt is familiar with Wacom, the dominant player in the graphics tablet market. In my career. the vast majority of designers that use a tablet use a Wacom from the Intuos or (now discontinued) Graphire lines. I myself have had my trusty Wacom Intuos2 6×8 for years. In fact, I’m such a devoted Wacom user that I use it as my primary means of interaction with my computer. Forget Photoshop and Illustrator, I even use my Intuos2 in Firefox and Finder. I have to make a conscious effort to put the Wacom pen down and switch to my mouse (a Logitech VX Nano, best general purpose mouse ever) for applications like Ableton Live that aren’t tablet-friendly. I’m not alone, either: Wacom launched PenCollective, a promotional site dedicated to the rabid fandom Wacom users tend to develop.

However, as much as I love my good old Intuos2, if I had my druthers I’d use a Cintiq 21UX which is an integrated pressure-sensitive tablet and 21 inch LCD monitor. Doing fine-detail retouching or painterly artmaking directly on the image instead of a separate tablet is a tremendous pleasure. However, at $1999 (compared to $329 for an Intuos3 6×8), it certainly is an expensive luxury.

N-Trig – Poised to Usurp the Throne?

Enter a young company called N-Trig, who created an ultra-thin pen and touch overlay for existing monitors and laptops. While stylus overlays and touch overlays have existed for years, they have never achieved mainstream popularity with design professionals because of Wacom’s strong presence in the market. However, N-Trig may be poised to change all that. Their technology, called DuoSense, supports not only pen and touch interaction but multi-touch interaction as well. N-Trig DuoSense is now being incorporated into Tablet PCs available today including the Dell Latitude XT.

So, we’ve got integrated pen and multi-touch in the hardware now, no overlays required! Great! Now how about the software? Well, it’s hardly news now that Windows XP and Vista natively support pen interaction. By now it’s hardly news that the next version of Windows, currently called simply ‘Windows 7′ will feature native multi-touch support as well. In fact, the video shown below demonstrates multi-touch in Windows 7 on a DuoSense-powered Dell Latitude XT. It seems like everything is coming together for N-Trig.

Any good news for OS X afficionados? Well, there are more than a few rumors floating around that Apple is negotiating with N-Trig to incorporate DuoSense into future Apple machines (MacBook Touch, anyone?).

Wacom Will Retain Their Core Market

So how is Wacom going to survive this attack on their near-monopoly? Well, there are a few advantages Wacom retains over N-Trig.

First and foremost, DuoSense is not nearly as pressure-sensitive as a Wacom tablet. Professional Wacom tablets (meaning everything except the Graphire and Bamboo lines) feature 1024 levels of pressure sensitivity compared with 256 levels of pressure sensitivity in DuoSense.

The second tremendous market advantage is that professional Wacom tablets are sensitive to the tilt of the pen and even its barrel rotation (with the 6D Art Pen), not just its coordinates and pressure. Ask any illustrator who uses Adobe Photoshop, Adobe Illustrator or Corel Painter X in a painterly manner about how they get naturalistic stroke variation in their work and they’ll tell you it’s in the tilt or rotation.

One last advantage for Wacom is that their devices recognize distinct inputs – Wacom makes an airbrush that adds fingerwheel pressure and position to the types of inputs you can use in your work. Additionally, their tablets recognize different pens independently, even of the same model, so you have have two different pens set to different brushes or colors, allowing you to switch between them in a naturalistic manner

So, unless N-Trig starts exploring ways to incorporate Wacom’s trade advantages into DuoSense, I really don’t see Wacom in any danger of falling off the map. The entry-level Bamboo line of tablets may become obsolete, though. Oh well. Real artists don’t use Graphire or Bamboo tablets, anyway.

IdentityMine and REZN8 host IxDA Surface Event

Last night IdentityMine (my employer) and REZN8, IdentityMine’s strategic design partner, had the distinct privilege of hosting an event at our Los Angeles office to showcase Microsoft Surface for IxDA, a community of Interaction Design experts. We discussed the opportunities and challenges for designers and developers that Surface presents. Andrew Whiddett [IdentityMine] led the half of the presentation dedicated to developer concerns while John O’Keefe [REZN8] lead the designer discussion. Both presenters were eloquent and informed since IdentityMine and REZN8 are among the top three Surface shops in the world. The event went tremendously well (and not just because of all the wine and hors d’oeuvres) so I decided to post a summary here.

A Little History

What is now called Microsoft Surface started in the early 2000s as a Microsoft research & development project called ‘Play Table’ and was originally envisioned as a new platform for gaming. Microsoft quickly realized that there were opportunities for other types of uses than just games so full-scale development went into creating Surface as a brand-new application platform. If you’ve seen ‘Minority Report’ you’ve seen some of where Surface computing is heading.

IdentityMine and REZN8 first got the Surface alpha in 2006 and began working with Microsoft and Harrah’s on Food and Drink applications. The result of these efforts have been recently released at the Rio Hotel and Casino in Las Vegas.

Bill Gates publicly announced demonstrated [edit, thanks Josh Santangelo] Surface at CES 2008. The applications he demonstrated there (Wine Bar and Snowboard, shown in the videos below) are IdentityMine and REZN8 applications.

Recently, Surface applications have also appeared in Starwood hotels and AT&T stores.

Fingers, Not Mice

The primary means of interaction with Surface is fingers, not a mouse. Unlike most other touchscreen computers, Surface recognizes multiple points of contact at the same time. This means applications can support multi-finger gestures. Gestures like pinch and spin, that have become familiar for iPhone users, are possible when the machine recognizes more than one point of contact at a time. In the future, touchscreen gestures will become as much a part of the collective consciousness as right-clicking with a mouse. This will lead to more sophisticated, yet natural interactions with multi-touch enabled applications.

While the iPhone was the first truly mass-market multi-touch device, it was in development at the same time as Surface. The similarity of the gestures between the two devices further underscores the validity of the interaction. Indeed, natural gesturing as a means of human-computer interaction has been a field of extensive study and research since the 1970s. One of the earliest pioneers in the field, Bill Buxton, is a Principal Researcher at Microsoft.

Gestures themselves are a fascinating topic. Imagine you have a user interface construct – an orb that can be dragged or flicked with one finger and resized with two fingers. Inside that orb you have a carousel that can be rotated horizontally with one finger. The items in that carousel can be ripped out of the carousel vertically with one finger. Now, with all those nested controls, when a user puts a finger on the control, how do you determine what he or she means? Drag it? Rotate the carousel? Rip an item out of the carousel? All good questions.

Finger interaction presents unique challenges as well. For example, Surface applications avoid the need to use a mouse or keyboard. Therefore new and novel methods of interacting with applications must arise. Instead of typing an address into a field to steer a web browser, we must invent a new method of browsing, perhaps one more like the internet William Gibson envisioned in Neuromancer.

Going Beyond Touch

What distinguishes Surface in the world of multi-touch computers is its vision system. Capacitor-based multi-touch devices like Apple iPhone, Jeff Han’s Perceptive Pixel device and JazzMutant Lemur use electrostatic resistance to “feel” touch. Surface uses a camera system to “see” touch. This means that Surface can “see” other objects too. In fact, Surface can see the shadow cast from a finger, not just the fingertip, and determine the user’s orientation to the table from his or her finger’s shadow!

Surface can recognize objects like encoded tags, credit cards, mobile phones, pretty much anything, and then interact with that object in a way that no other platform can. Surface is really like a “mixed-media” environment where applications can use a combination of real objects and virtual objects together. This opens Surface up to a whole range of possibilities that are simply impossible on other multi-touch computers and utterly unimaginable on ordinary comptuers.

Multi-User Shared-Space Applications

Today’s applications are principally designed around a single individual sitting at a single workstation. There are very few single-workstation-multi-user applications in the world. The most notable are single-console multiplayer games, and in most cases each user owns a portion of the screen real estate, like split-screen racing games, for example. Surface is radically different in that it’s designed explicitly for multiple users to interact at the same time.

There are three main types of multi-user interactions in Surface: a ‘host-guest’ scenario where a host drives the application and offers features to a single client, like a concierge application; a ‘collaborative’ scenario where multiple users are working together toward a common goal, like a restaurant ordering application; and a ‘discrete’ scenario where multiple users are working independently and each wants a piece of the screen real estate for themselves. Each scenario is different and each requires a different approach to user interaction design.

One of the constant challenges in multi-user applications, especially in the ‘discrete’ scenario, is real estate. There are a limited number of pixels available on the screen. When four people using an application at the same time each want a piece of the table, you can quickly run into situations where there is a shortage of workspace. Traditional applications are like cubicles – each user gets their own little space to do with it what he or she pleases without affecting the other users. Surface applications are more like a large worktable with people gathered around it, sharing the space. If one user has a messy work style, scattering his or her objects around the space willy-nilly, its likely to get in the way of the other users, much like real-world space. Managing each users objects in the environment is a fun challenge to address. How do you know if a particular object belongs to you or another person? Color-coding? Tagging? A way to collect all your stuff in neat piles in front of you?

Omnidirectional Applications

In traditional applications there is a clear sense of top and bottom, clear borders, most everything lines itself up along right angles. Now think of Surface, an environment where a user can approach it from any direction. How do you know which way to orient the text when there is no real “top?” When a user approaches the device they expect the application to be facing them, regardless of where they are. How do you handle that?

Envision an application where two users are sitting across from each other, working together. If one user flicks an object, say a photo or a document, over to the other user’s side of the table, should it orient itself to face that user? At what point should it orient itself? These types of questions are the challenges omnidirectionality present but at the same time they offer tremendous flexibility. Surface applications are fluid, organic and amorphous. Rectangular windows and grid systems just feel out of place in Surface computing.

Physics and Virtual Objects

When an object is represented on a regular monitor, and manipulated with a mouse and keyboard, no matter how realistically the object is rendered there user never feels like he or she is interacting with a virtual object. On a Surface computer an object feels more visceral. Manipulating objects in Surface with the pointing device we were all born with, users feel a deeper physical connection with the object, bringing their real-world physical experience to the application. Consequently, they expect the object to behave in a manner consistent with real-world physics. Suddenly, gravity, friction, inertia, mass and collision become critical features of your application where previously these features were only found in gaming.

Here is a scenario that can help illustrate the importance of physics in a Surface application: imagine an application where a user has a number of photos scattered across the table. They also have a virtual toolbox in front of them that they can use for photo editing and manipulation. The photos and the toolbox all have physical properties – they can be dragged, flicked and spun around the application. Now, a user flicks a photo toward the toolbox. What happens? Does the photo bounce off the toolbox? Does it bump the toolbox, moving in response to the impact? Does it slide under the toolbox as if it were on a lower layer? These are the types of physics questions a multi-touch physics-enabled application engender.

The Future of Surface

One of the comments I hear frequently about Surface is its form factor. There was is popular spoof video on YouTube talking about the future of computing being a “big-ass table.” Yes, it is a big-ass table. We all know that. Microsoft knows that. Think back to the mainframes of the 1950s. The computing power that used to occupy an entire room is now eclipsed by the computer in your pocket. It is inevitable that the form factor will get smaller, lighter and more flexible. Due to a non-disclosure agreement I can’t say too much about this topic but I will say this: the coffee table is just the beginning…

In Closing

Thinking about designing and developing for Surface raises a tremendous number of new questions about application design. The answers? That will be up to the community of User Interface designers and developers, as well as the users themselves, to decide.