Tuesday, August 24, 2010

Green-eyed Monster

If you have been using the Emerald viewer, for now we would encourage you to consider either one of the Linden Lab viewers, or an alternative third-party viewer.

Philip Rosedale
Linden Lab (interim) CEO

Emerald is by far the most popular third party viewer in use in Second Life. Always the focus of some controversy, Emerald includes many power user features much-adored by its users, including a built-in radar, client-based animation overrider (which lets people set up their avatar's "body language" without using awkward in-world tools), performance improvements, and a large number of geeky and semi-geeky features that go well beyond the official SL viewer applications, or make things the standard SL viewers can do significantly easier. Oh, yes, and a preference to make female avatars' boobs bounce.

However, Emerald now finds itself at the sharp end of Linden Lab's pointy stick: earlier this month, Emerald was used to execute a denial of service attack against a rival's Web site, and it's not the first time the Emerald team—or a subset of the Emerald team—has shown a disregard for Emerald users and their privacy.

Linden Lab's introduction of Viewer 2.0—which has been poorly received by the Second Life user community—drove many of SL's power users to Emerald as an alternative. Many Emerald users are loyal and enthusiastic about the application, lauding its features and approving of the fact it's primarily developed by actual Second Life users rather than Linden Lab who, if Viewer 2.0 is any indiction, are significantly out of touch with how people use Second Life.

As an estimate of Emerald's popularity in the Second Life user community, Hamlet Au of New World Notes says "reliable sources" claim almost half of all users hours currently logged in Second Life are from users of the Emerald viewer. That's not the same as half of all SL users, but certainly, the more hours someone spends using SL, the more likely they are to appreciate and covet Emerald's feature set.

A disclaimer: I have never used Emerald. Like many Second Life users, I was interested in Emerald when I first heard about it, but decided to ask around before trying it out. One fine day in a sandbox, I happened to overhear one of Emerald's developers talking in open chat for a while…and decided I probably wasn't interested in anything with which that person was associated. Later I attempted to attend one of the Emerald teams open office hour events and was immediately barred from the Emerald Point sim. Watching Emerald's story unfold over the last few months has substantially reinforced my misgivings.

It is important to note that the Emerald viewer is not an enterprise that directly earns money for any members of the Emerald team. Although some Emerald developers and team members have in-world businesses and earn money in SL-related endeavors, they do not work on Emerald as employees of any company—Emerald is in essence a volunteer effort. Most of members of the Emerald development team are known only by their Second Life avatar names; it would seem they value their privacy.


Two weeks ago, users of the third-party Second Life viewer Emerald were unwittingly made accomplices in a distributed denial-of-service attack against a third party Web site. The apparent goal of the attack was to deluge the third party site with traffic, in hopes of taking the site offline or, potentially, racking up significant bandwidth charges for the site if the amount of material it served exceeded its hosting allotments. The attack was carried out using the Emerald Viewer's login screen—which every Emerald user loads by default when they start the application—and, of course, those user's Internet connection, whose bandwidth and Internet access was commandeered to perform the attack. Since thousands of people log into Second Life using Emerald every day—and account have each of those logins requesting over 4MB of data from the third party site—the amount of bandwidth involved was significant.

The attack was apparently engineered by Fractured Crystal, aka JCool, the "project leader" of the Emerald development team. The attack targeted the iheartanime.com site of Hazim Gazov, a admitted developer of copybot and ban-evading Second Life clients and reputed Second Life griefer.

The attack operated for about three days, and has since been removed from the Emerald login screen. Fractured Crystal has apparently fallen on his sword, issuing a contrite mea culpa and says he has turned over the Emerald project to other, unnamed, members of the Emerald team.

The Emerald viewer has not "yet" been barred from connecting to Second Life; however, the Emerald development team's use of the viewer (and its users) to launch a denial of service attack is in violation of Linden Lab's recently-implemented third-party viewer policy. Rosedale says Linden Lab will prevent Emerald from logging in to Second Life if the Emerald team can't live up to Linden Lab's standards. "We will not tolerate [..] development teams with a history of violating users' trust or disrupting their lives."

Emerald has precisely that kind of history.


In May of this year, the Emerald development team endured its first significant privacy scare when someone—probably the same Hazim Gazov—managed to break into the ModularSystems Web site (until this week, ModularSystems was the entity responsible for the Emerald viewer) using a very poorly secured password. He obtained—and forwarded to Linden Lab—a database of information gathered from Emerald viewers, ModularSystems site visitors, folks who created Second Life accounts using Emerald, and in some cases visitors to Emerald's in-world base on the Second Life grid (Emerald Point) and a few other locations. This information was apparently collected for several months, and included users avatar names and keys (unique numbers) and IP addresses. Portions of that information have been made public.

For some Internet users, IP addresses aren't particularly sensitive: perhaps they get a different one every time they sign on, or they access the Internet from many locations. Folks concerned about their privacy go to some lengths to obscure their IP addresses. However, associating an IP address with a physical region (say, a metropolitan area) is very simple, and in some cases an IP address can be used to identify a user's physical location with a great deal of precision.

IP addresses are the cornerstone of Internet communications: if remote computers didn't have your IP address, they wouldn't be able to send you any information at all—email, Web pages, IMs, video. However, being able to associate IP addresses with Second Life avatar names and (potentially) in-world and real-life locations creates possibilities for all sorts of cross-checking, potentially being able to determine what avatars are likely to be "alts" of a single individual and perhaps determine where a particular Second Life user lives or works. This kind of correlative analysis has significant privacy implications.

The Emerald team defended collecting this information as a "prototype" of a system intended to identify alts of griefers on the Emerald Point sim. The Emerald team said the system was created and maintained by a single member of the team; opinion seems to be that person was Fractured Crystal/JCool.

The in-world techniques used to collect users' information are remarkably similar to some employed by Skills Hak's controversial Gemini Cybernetic CDS service, which purports to be able to block Second Life users employing copybot-enabled viewers. Skills Hak was—and continues to be—a member of the Emerald development team. Hak sells CDS independently of work with the Emerald project.

The emkdu Library

On July 14, Hazim Gazov—admitted developer of clients that violate Second Life terms of service—struck again, publishing substantial proof that the Emerald client was encoding details about user's computers into baked textures that comprise an avatar's visible base-layer clothing. The library behind emkdu is Kakadu, and it's commercial software: the idea behind the tagging was, apparently, to be able to identify "legitimate" copies of the Emerald viewer from third-party clients merely posing as Emerald, enabling Emerald to issue piracy complaints against any viewers that lifted the library for their own use. In some cases, the details used to make those tags included the viewer application's working directory on the user's computer. For some users, this information is innocuous, but for others it might include personal information about the user—for instance, their username or computer name. Gazov, of course, wound up building a tool to systematically scan for this information in-world.

Accounts vary regarding who on the Emerald team knew about this information being encoded in textures. Gazov claims the feature was implemented by Emerald developer Zwagoth Klaar, but other Emerald devs he contacted were unaware of the data being stored in baked textures. In his mea culpa, Fractured Crystal/JCool claimed the idea was suggested by "others" but had his support, and he still feels it is "harmless."

Fractured Crystal/JCool says the enkdu metadata has since been removed from Emerald, and will "never occur again."


The emkdu metadata apparently generated a significant disruption in the Emerald development team. Two of Emerald's developers left the project during this time period, with LordGregGreg Back—now characterized as a "minor ex-dev"—specifically departing over the issue.

Some portion of the Emerald development team was distinctively annoyed at Hazim Gazov. Apparently during the second week of August—several weeks after Gazov's publishing information about the emkdu metadata—the Emerald dev team decided, "amid an atmosphere of pride and boasting," to target Gazov's Web site with a flood of traffic generated by Emerald users—and this effort to "show off" the size of Emerald's user base led to Fractured Crystal/JCool embedding iframes (in this case, kind of an non-displaying sub-window) within the Emerald login page that pointed to material in Gazov's blog. The embedded links were, in theory, loaded by any Emerald user to logged into Second Life; the items were intentionally selected to be the largest images in Gazov's blog, thereby maximizing the bandwidth that would be consumed every time an Emerald user logged in to Second Life.

The Emerald team described the actions as mere "shenanigans," and specifically denied it constituted a distributed denial of services (DDOS) attack. This description, apparently authored by Fractured Crystal/JCool himself, failed to hold water with the broader user community and, perhaps more importantly, with the Emerald development team.

Significantly, during this time the Emerald team announced picking up two former Linden Lab employees as members: the former Qarl Linden, who was one the Linden Lab render team and apparently responsible for implementing much of Second Life's beloved "sculpties," and the former Data Linden, who will apparently be helping out with Emerald support. It's not clear at this point if either of these former Linden Lab employees are still associated with Emerald in the wake of Fractured Crystal/JCool's "shenanigans."

Where Things Stand…Today

As of this moment, Linden Lab continues to permit the Emerald viewer to log into the Second Life grid.

The Emerald development team has announced they are reorganizing as a transparent, democratic operation that will have no single project leader. "All decisions, changes, and alterations to any code or anything at all, will be done transparently and democratically."

The Emerald team is in the process of setting up a new domain—emeraldviewer.net—to separate themselves from Fractured Crystal/JCool's ModularSystems entity. This probably means that most of the links I have made to the Emerald team's statements will break.

The Emerald team intends to re-apply to be included in Linden Lab's third-party viewer directory.

What To Make Of This?

Although I'm a mere human, I've tried to present the events and information above plainly. I can't claim to have an eagle's eye view, and, to my knowledge, I don't know any of the people involved personally. However, the Internet being the way it is, it's possible that scruffy-looking guy with the big-ass Dell notebook over at the other end of the coffee shop is one of the Emerald developers. He sure is scowling a lot.

But here's my take: Linden Lab is between a rock and a hard place. If a significant portion (half?) of Second Life's logged hours are from the Emerald viewer, banning Emerald from the Second Life grid will alienate a substantial number of Second Life's most ardent users—and, undoubtedly, that includes many content creators, power users, builders, and folks who run in-world businesses, successful or not.

On the other hand, if Linden Lab lets Emerald back into its directory of viewers that self-certify they conform to the Lab's third party viewer policy, then, clearly, the third party viewer policy—the subject of much drama and gnashing of teeth—is utterly meaningless. Emerald claimed it conformed to the terms of Linden Lab's third party viewer policy, and has now repeatedly and willfully violated that policy.

My guess is that Linden Lab and the re-constituting-itself Emerald development team will try to strike some sort of compromise, perhaps a "probationary" period wherein Emerald will still be permitted to connect to the main grid but will not be listed as conforming to the third party viewer policy until the viewer has a clean record for, say, a year, and the development team proves it can keep its new glass house in order. If I were Linden Lab—and I didn't want the entire third party viewer program thrown out the window—I would set conditions to any such probation. One of those conditions would be that Emerald must inform every user on every login that the viewer has violated Linden Lab's third party viewer policy, with a link with complete disclosure of the violations and what the Emerald team is doing to rectify the problems.

No matter what, I'm not going to be touching Emerald anytime soon.


  • The Emerald development team says Linden Lab has issued a set of undisclosed requirements Emerald has to fulfill before it may re-apply to the third-party viewer directory.
  • I've made some minor tweaks to my text above to clean up some sloppy grammar, add a few links, and correct some production issues. I was working quickly, and Blogger isn't my idea of a proper editing environment.

Tuesday, August 17, 2010

Lines in the Sand

Philip Rosedale has been back in the Linden Lab CEO seat for a few weeks now, after saying it was going to take an (understandable) few weeks to get his head back into the day-to-day of running Second Life. This past week, at the 6th annual Second Life Community Conference, Rosedale actually outlined a set of concrete goals for Second Life. You can watch a video of it; so far, Linden Lab has yet to publish much of this material on their site, although they seem to be getting around to it. Bottom line, Rosedale's remarks are the first time he's gotten specific about how Linden Lab plans to get real about its new "Fast, Easy, Fun!" mantra…you know, now that they have one-third less people.

How SL has been feeling lately
(Image from the maze under Silent Sparrow)

Some of Rosedale's goals are a bit technical and dweeby; others seem likely to have a significant impact on Second Life. Here are the ones I believe will have major, visible impact:

Mesh imports will enter public beta by the end of 2010
Like Second Life's existing sculpts, meshes are a way to model 3D objects in a more-organic way than using SL's geometric prims; it's the technology used in high-end 3D rendering programs and modeling systems. Like sculpts, however, creating meshes won't be possible in Second Life: folks will have to use external tools like Maya and Blender.
Users will be able to edit and make their own "display" names for their avatars, although underlying usernames will still be unalterable.
Right now, avatar names are fixed at sign up and unchangeable; users can only select from a small set of available surnames. Linden Lab apparently plans to get rid of fixed surnames and let users create a small number of personalized display names for their avatars.
Group chat failures and the huge performance hits when crossing between regions will be fixed by the end of 2010
So far as I can determine, group chat has never worked reliably in Second Life: attempting to talk privately to a group is as likely to pop up an error dialog that locks you out of the viewer application as it is to actually work. Lag crossing sims is a huge problem for many users, particularly, if they use vehicles like boats, planes, trains, cars, UFOs, or merely walk.
Linden Lab will be shutting down the Teen Grid
Right now, Second Life is ostensibly limited to people age 18 of over (although, of course, there's no real way to enforce that). Linden Lab now plans to let 16- and 17-year-olds onto the Main Grid.

Here are some geekier but still-important promises:

Linden Lab will be adopting a Scrum development process for its official viewer application.
Scrum is an iterative code development practice that's all kinds of hip these days. The upside of scrum processes is that they can adapt and implement changes fast, and pump out lots of incremental updates rather than big monolithic releases. The downside is that it's very hard for Scrum projects to see beyond the end of their own noses (e.g., the current "sprint") and they often suffer from poor quality testing and documentation.
Viewer updates updates will be background downloads
Right now every time Linden Lab wants to issue a patch, they force users to download a completely new (ginormous) viewer application. This ties in with Linden Lab's idea to release updates to the viewer more often.
Textures and other assets will be sent to clients directly from database servers, using TCP and HTTP
A long time coming, these changes should both reduce loads on individual sims and improve rezzing. It's already rolling out.
Linden Lab will eliminate new user orientation
After a substantial effort last year to revamp the "first hour" experience of new residents, Linden Lab apparently plans to drop new users directly onto the Main Grid at events and places they indicate they might be interested in.
Linden Lab plans to make an iPad client for Second Life.
Of course, once you make an iOS viewer for an iPad, the iPhone and iPod touch won't be far behind. And there's no reason something similar couldn't be done for Android and similar smartphone/mobile operating systems (webOS? MeeGo?) No timetable has been announced.

As with so many things in Second Life, Rosedale's remarks weren't without a misfire. Rosedale was initially going to phone in a keynote address from some summer vacation location, but that went over like a lead balloon and at the last minute Rosedale decided to address the convention in person.

So, why does any of this matter?


Meshes are a more-sophisticated way to represent 3D objects than Second Life's built-in geometric primitives. Instead of geometrically "pure" cubes and spheres and tori, you can think of meshes as a bit like chickenwire: you can bend and twist it and flop it around, pinch it narrow or stretch it out, and paint textures (e.g., graphics) on it in fairly complicated ways. Where cubes and spheres are useful for some sorts of things, meshes are useful for more-organic shapes and forms—and, depending on the implementation, may finally enable virtual clothing that can move semi-realistically with an avatar—a mesh could stretch and bend with an avatar's motion, similar to the way SL's painted-on system cloths do now. Second Life's existing "sculpt" primitives are kind of simple meshes, with a maximum of 64 vertices. True meshes promise to offer a lot more capability.

Mesh technology is what gets used in high-end 3D rendering applications, and Second Life's professional content creators—mainly virtual fashion people, but plenty of others—have been demanding that Second Life support mesh for years. It looks like they're finally going to get their wish.

Meshes are a much-demanded feature from SL's content creation community, and promise to turn the content market on its ear, much the same way sculpts and flexies did before it—suddenly anything that doesn't use mesh will be old and tired. Meshes also promise to make content creation for Second Life even less accessible to everyday users, and an even more expensive and exclusionary proposition. This is nothing new: while Second Life does let people create basic primitives, nothing else can be made in-world. Want to make textures? You've got to use something like Gimp or PhotoShop and import them. Want to make animations? You need something like Poser. Want to make sculpts? You need something like Blender. Want to make mesh? Blender, Maya, Strata 3D, Lightwave 3D, 3ds Max. These applications are just examples, but they all have a few things in common, including requiring substantial investments of time, effort, and (often) money to use effectively. Once you've imported a mesh you'll be able to scale and position it in-world, but—just like sculpts—you won't be able to edit or change it.

Second Life's viewer software will have to be updated to support mesh; it's highly unlikely Linden Lab will support meshes in anything but Viewer 2 and its revisions.

Display Names

Although it wasn't mentioned in Rosedale's speech, Linden Lab is going to let avatars set and change their "display names," which they're heralding as a great new thing to improve "self-expression" in Second Life.

At this point, all avatar names are fixed and unchangeable: once you create your account, you can't change your avatar's name. (Although there are persistent reports if you pay Linden Lab an ungodly sum, they will change it for you.) Users also don't get to choose their last names: they're presented a short list of available names when they sign up, and that's all they get. (I wound up with "Netizen" because it was merely the least hideous of the bunch I was offered.)

Avatars will still have unique usernames that will be "easily discoverable" within SL, so ownership of land and items will still use the same mechanisms that are in place today, and anyone who, say, sets their display name to "Lou Netizen" can easily be ratted out as not really being me. If anyone bothers to look.

There are some huge plusses to this idea. First, it gives people full control over their names, which, speaking as someone with a lame last name, huzzah! The new system will also enable users to use Unicode characters in their names, so European and Asian scripts will all work—dunno about things like Hebrew and Arabic, but that would be neat. Folks who "partner"—the SL equivalent of marriage—can change their last names to match if they like, or hyphenate. And roleplayers will go nuts: those macho types in their wild west sim can all be "Quickdraw Aimes" and "Ace McGee" and other genre-appropriate things.

On the other hand, the potential for troublemaking is huge. It ought to be trivial to make chatlogs completely unreliable by changing your display name to someone you disagree with—or, even better, showing up somewhere with a "Linden" last name and wreaking a little havoc. The demonstration video Linden Lab has posted also shows other problems: when someone changes their display name, it changes system-wide, including other people's Friends Lists, radars, groups, you name it. That'll make finding people almost impossible: how are you to know that Lou Netizen is now Bumpersticker Foozlebrain?

Shutting Down the Teen Grid

Since 2005 Linden Lab has operated a separate Teen Grid, a smaller, separate Second Life for users aged 13 to 17. When a Teen Grid resident turns 18, their avatar (and their stuff, including any regions) gets transferred to the Main Grid. Under the new policy, 16- and 17-year-olds will be allowed to set up accounts on the Main Grid directly; existing Teen Grid account holders in that age range will be transferred to the Main Grid. Folks under 16 who want to use Second Life will be left out in the cold.

Rosedale expressed the opinion that the maturity rating system Second Life has rolled out in the last year or so—including the adult ghetto of Zindra (aka the Continent of Dildos and Bazooms)—is enough to protect minors from…inappropriate content and experiences in Second Life. Basically, there are three "ratings" for content in SL, as determined on a parcel-by-parcel basis. "General" is basically content appropriate for anyone, "Mature" is the vast majority of SL (most bars, malls, music venues, and other builds), while "Adult" areas of explicit adult and/or intensely violent content.

I'm of two minds about this. On one hand, I've kind of appreciated that a lot of the "adult" content has been sequestered out into the hinterland of Zindra: it's not my thing. It's not all gone by any means: there's plenty of explicitly adult content and intense violence still floating around on "Mature" and "General" portions of Second Life. It's not everywhere, but it's common enough that I'd be hesitant to turn a 16 year-old loose even in just "General" areas.

On the other hand, Second Life has tremendous potential for education. If someone had showed me SL when I was 13—you can make stuff! and script it! and run around like a freakazoid!—I'd have been all over it. The demise of the Teen Grid would seem to mean that educators can't bring students into Second Life as part of their classwork until they're 16. And then they'll be on the nowhere-near-as-sanitary-as-Rosedale-thinks Main Grid.

HTTP Assets

One thing I'm kind of psyched about it the HTTP Assets project, which is probably the geekiest thing here. When Second Life was being developed, a technical decision was made to use UDP to transfer things like animations and textures from sim servers to users' client software. UDP is a low-level Internet protocol that has several cool features: you don't need to do handshaking to setup communications with UDP: you can just start sending stuff. UDP has less communications overhead than the more-common TCP, but it also fails silently: UDP offers no guarantee that your data is going to get to its destination, get there in order, or get there intact. That's why some objects and avatars in SL always stay grey, some sculpts just appear as lumps, and some things never rez. The SL client literally doesn't know what it's missing.

Now, in some ways, using UDP was the right call in Second Life's early days: bandwidth was scarcer, and all the overhead of trying to confirm, resend, and reorder packets would just use a bigger chunk of what bandwidth was available for housekeeping. However, as the Internet has evolved damn near everything is using TCP, rather than UDP. TCP has all those housekeeping features UDP omits—data integrity, packet ordering, re-requesting bits that got lost, etc.—so switching to TCP means a bit more overhead but also fewer lump sculpts, grey textures, and, um, unfortunate ruthing.

Using HTTP on top of TCP to transfer texture and other assets is also a benefit. HTTP is the primarily high-level protocol used for the Web. ("High-level" here means that HTTP is actually built on top of TCP, which is a "low-level" protocol.) The Internet has kind of taken a mild interest in Web servers, which means there's a metric f*ckton of software out there designed to handle HTTP as efficiently as possible. Relying on HTTP means Linden Lab gets to benefit from all that optimization instead of having to write and implement (and optimize) their own system from scratch.

And it's not just servers: routers, firewalls, and traffic management systems—the kinds of things that actually make up the Second Life grid—have put a ton of effort into optimizing their performance for TCP and HTTP. So Linden Lab will benefit from all that work too.

Furthermore, letting viewers bypass the current sim and make requests directly to Linden Lab database servers for textures and other assets not only removes the local sim as a "middle man" for these transactions (hundreds of which are typically involved just TPing into somewhere), but also reduced overall load on the sim, making more cycles available to handle things like, or, say, CHAT! Bottom line: the HTTP Asset project promises to strike at the heart of lag, and I can get behind that.

And Viewer Too

A lot of these changes are going to require support in the Second Life Viewer…and, unfortunately, Linden Lab kinda laid a turd on the user community with Viewer 2.0. Sure, Viewer 2.0 has a few things going for it (I like the address bar) and a lot of under-the-hood get-ready-for-the-future things. The interface of the previous 1.x viewers was—well, I'll be polite, it was and remains deeply problematic. Viewer 2 improved on a few things but adopted a new interface model that is simultaneously more intrusive and less flexible than the previous model, with the result that many long-time residents refuse to use it. (I can think of many reasons not to use Viewer 2, but the chat interface—both new school and "plain text"—are dealbreakers for me, since I live in text.)

Linden Lab seems to have realized that response to Viewer 2 has been less than enthusiastic in many circles. "We've got a lot of improvements we want to make to the Viewer 2 user experience," Esbee Linden wrote in the company blog. "Some of the Viewer's workflows are cumbersome for some Residents and this has hurt Viewer adoption. We really can improve the Second Life user experience by rethinking the way our Viewer works and making it (and its features and functionality) faster, easier, and more fun for everyone."

So Linden Lab has announced Project Snowstorm, which will attempt to roll out updates to the official viewer on a regular basis—they're aiming for every other week—using a development methodology called Scrum. I make no secret that I am not a fan of Scrum, but I'll try to describe it: basically, developers work against a "backlog" of tasks in phases called "sprints." Items get cherrypicked out of the backlog and assigned to a particular sprint, then everyone goes whiteknuckle to get their sprint tasks compiling and checked in before deadline. Then the whole thing starts again.

The problem with Scrum as a management system is that it's very short-sighted: it has trouble dealing with issues that can't be neatly broken down into sprintable tasks. It also leads to what I call "trainwreck" development where everyone is pushing hard to get their work in under deadline—and that often means there's little or no testing of individual features (or interoperability), and nobody has time to document their work when the sprint wraps.

In the brave Internet world of perpetual betas and "we'll push an update," that's probably OK in Linden Lab's eyes: the bi-weekly sprints will be for early adopters and enthusiasts, not everyday users. But the large number of significant changes Linden Lab plans to roll out in the next 180 days, a larger number of people are going to have to be looking at those builds…and we might see why every Scrum project I've seen has been one of those aforementioned "trainwrecks."