Tuesday, August 24, 2010

Green-eyed Monster

If you have been using the Emerald viewer, for now we would encourage you to consider either one of the Linden Lab viewers, or an alternative third-party viewer.

Philip Rosedale
Linden Lab (interim) CEO

Emerald is by far the most popular third party viewer in use in Second Life. Always the focus of some controversy, Emerald includes many power user features much-adored by its users, including a built-in radar, client-based animation overrider (which lets people set up their avatar's "body language" without using awkward in-world tools), performance improvements, and a large number of geeky and semi-geeky features that go well beyond the official SL viewer applications, or make things the standard SL viewers can do significantly easier. Oh, yes, and a preference to make female avatars' boobs bounce.

However, Emerald now finds itself at the sharp end of Linden Lab's pointy stick: earlier this month, Emerald was used to execute a denial of service attack against a rival's Web site, and it's not the first time the Emerald team—or a subset of the Emerald team—has shown a disregard for Emerald users and their privacy.

Linden Lab's introduction of Viewer 2.0—which has been poorly received by the Second Life user community—drove many of SL's power users to Emerald as an alternative. Many Emerald users are loyal and enthusiastic about the application, lauding its features and approving of the fact it's primarily developed by actual Second Life users rather than Linden Lab who, if Viewer 2.0 is any indiction, are significantly out of touch with how people use Second Life.

As an estimate of Emerald's popularity in the Second Life user community, Hamlet Au of New World Notes says "reliable sources" claim almost half of all users hours currently logged in Second Life are from users of the Emerald viewer. That's not the same as half of all SL users, but certainly, the more hours someone spends using SL, the more likely they are to appreciate and covet Emerald's feature set.

A disclaimer: I have never used Emerald. Like many Second Life users, I was interested in Emerald when I first heard about it, but decided to ask around before trying it out. One fine day in a sandbox, I happened to overhear one of Emerald's developers talking in open chat for a while…and decided I probably wasn't interested in anything with which that person was associated. Later I attempted to attend one of the Emerald teams open office hour events and was immediately barred from the Emerald Point sim. Watching Emerald's story unfold over the last few months has substantially reinforced my misgivings.

It is important to note that the Emerald viewer is not an enterprise that directly earns money for any members of the Emerald team. Although some Emerald developers and team members have in-world businesses and earn money in SL-related endeavors, they do not work on Emerald as employees of any company—Emerald is in essence a volunteer effort. Most of members of the Emerald development team are known only by their Second Life avatar names; it would seem they value their privacy.

EmeraldGate

Two weeks ago, users of the third-party Second Life viewer Emerald were unwittingly made accomplices in a distributed denial-of-service attack against a third party Web site. The apparent goal of the attack was to deluge the third party site with traffic, in hopes of taking the site offline or, potentially, racking up significant bandwidth charges for the site if the amount of material it served exceeded its hosting allotments. The attack was carried out using the Emerald Viewer's login screen—which every Emerald user loads by default when they start the application—and, of course, those user's Internet connection, whose bandwidth and Internet access was commandeered to perform the attack. Since thousands of people log into Second Life using Emerald every day—and account have each of those logins requesting over 4MB of data from the third party site—the amount of bandwidth involved was significant.

The attack was apparently engineered by Fractured Crystal, aka JCool, the "project leader" of the Emerald development team. The attack targeted the iheartanime.com site of Hazim Gazov, a admitted developer of copybot and ban-evading Second Life clients and reputed Second Life griefer.

The attack operated for about three days, and has since been removed from the Emerald login screen. Fractured Crystal has apparently fallen on his sword, issuing a contrite mea culpa and says he has turned over the Emerald project to other, unnamed, members of the Emerald team.

The Emerald viewer has not "yet" been barred from connecting to Second Life; however, the Emerald development team's use of the viewer (and its users) to launch a denial of service attack is in violation of Linden Lab's recently-implemented third-party viewer policy. Rosedale says Linden Lab will prevent Emerald from logging in to Second Life if the Emerald team can't live up to Linden Lab's standards. "We will not tolerate [..] development teams with a history of violating users' trust or disrupting their lives."

Emerald has precisely that kind of history.

Datamine

In May of this year, the Emerald development team endured its first significant privacy scare when someone—probably the same Hazim Gazov—managed to break into the ModularSystems Web site (until this week, ModularSystems was the entity responsible for the Emerald viewer) using a very poorly secured password. He obtained—and forwarded to Linden Lab—a database of information gathered from Emerald viewers, ModularSystems site visitors, folks who created Second Life accounts using Emerald, and in some cases visitors to Emerald's in-world base on the Second Life grid (Emerald Point) and a few other locations. This information was apparently collected for several months, and included users avatar names and keys (unique numbers) and IP addresses. Portions of that information have been made public.

For some Internet users, IP addresses aren't particularly sensitive: perhaps they get a different one every time they sign on, or they access the Internet from many locations. Folks concerned about their privacy go to some lengths to obscure their IP addresses. However, associating an IP address with a physical region (say, a metropolitan area) is very simple, and in some cases an IP address can be used to identify a user's physical location with a great deal of precision.

IP addresses are the cornerstone of Internet communications: if remote computers didn't have your IP address, they wouldn't be able to send you any information at all—email, Web pages, IMs, video. However, being able to associate IP addresses with Second Life avatar names and (potentially) in-world and real-life locations creates possibilities for all sorts of cross-checking, potentially being able to determine what avatars are likely to be "alts" of a single individual and perhaps determine where a particular Second Life user lives or works. This kind of correlative analysis has significant privacy implications.

The Emerald team defended collecting this information as a "prototype" of a system intended to identify alts of griefers on the Emerald Point sim. The Emerald team said the system was created and maintained by a single member of the team; opinion seems to be that person was Fractured Crystal/JCool.

The in-world techniques used to collect users' information are remarkably similar to some employed by Skills Hak's controversial Gemini Cybernetic CDS service, which purports to be able to block Second Life users employing copybot-enabled viewers. Skills Hak was—and continues to be—a member of the Emerald development team. Hak sells CDS independently of work with the Emerald project.

The emkdu Library

On July 14, Hazim Gazov—admitted developer of clients that violate Second Life terms of service—struck again, publishing substantial proof that the Emerald client was encoding details about user's computers into baked textures that comprise an avatar's visible base-layer clothing. The library behind emkdu is Kakadu, and it's commercial software: the idea behind the tagging was, apparently, to be able to identify "legitimate" copies of the Emerald viewer from third-party clients merely posing as Emerald, enabling Emerald to issue piracy complaints against any viewers that lifted the library for their own use. In some cases, the details used to make those tags included the viewer application's working directory on the user's computer. For some users, this information is innocuous, but for others it might include personal information about the user—for instance, their username or computer name. Gazov, of course, wound up building a tool to systematically scan for this information in-world.

Accounts vary regarding who on the Emerald team knew about this information being encoded in textures. Gazov claims the feature was implemented by Emerald developer Zwagoth Klaar, but other Emerald devs he contacted were unaware of the data being stored in baked textures. In his mea culpa, Fractured Crystal/JCool claimed the idea was suggested by "others" but had his support, and he still feels it is "harmless."

Fractured Crystal/JCool says the enkdu metadata has since been removed from Emerald, and will "never occur again."

Shenanigans

The emkdu metadata apparently generated a significant disruption in the Emerald development team. Two of Emerald's developers left the project during this time period, with LordGregGreg Back—now characterized as a "minor ex-dev"—specifically departing over the issue.

Some portion of the Emerald development team was distinctively annoyed at Hazim Gazov. Apparently during the second week of August—several weeks after Gazov's publishing information about the emkdu metadata—the Emerald dev team decided, "amid an atmosphere of pride and boasting," to target Gazov's Web site with a flood of traffic generated by Emerald users—and this effort to "show off" the size of Emerald's user base led to Fractured Crystal/JCool embedding iframes (in this case, kind of an non-displaying sub-window) within the Emerald login page that pointed to material in Gazov's blog. The embedded links were, in theory, loaded by any Emerald user to logged into Second Life; the items were intentionally selected to be the largest images in Gazov's blog, thereby maximizing the bandwidth that would be consumed every time an Emerald user logged in to Second Life.

The Emerald team described the actions as mere "shenanigans," and specifically denied it constituted a distributed denial of services (DDOS) attack. This description, apparently authored by Fractured Crystal/JCool himself, failed to hold water with the broader user community and, perhaps more importantly, with the Emerald development team.

Significantly, during this time the Emerald team announced picking up two former Linden Lab employees as members: the former Qarl Linden, who was one the Linden Lab render team and apparently responsible for implementing much of Second Life's beloved "sculpties," and the former Data Linden, who will apparently be helping out with Emerald support. It's not clear at this point if either of these former Linden Lab employees are still associated with Emerald in the wake of Fractured Crystal/JCool's "shenanigans."

Where Things Stand…Today

As of this moment, Linden Lab continues to permit the Emerald viewer to log into the Second Life grid.

The Emerald development team has announced they are reorganizing as a transparent, democratic operation that will have no single project leader. "All decisions, changes, and alterations to any code or anything at all, will be done transparently and democratically."

The Emerald team is in the process of setting up a new domain—emeraldviewer.net—to separate themselves from Fractured Crystal/JCool's ModularSystems entity. This probably means that most of the links I have made to the Emerald team's statements will break.

The Emerald team intends to re-apply to be included in Linden Lab's third-party viewer directory.

What To Make Of This?

Although I'm a mere human, I've tried to present the events and information above plainly. I can't claim to have an eagle's eye view, and, to my knowledge, I don't know any of the people involved personally. However, the Internet being the way it is, it's possible that scruffy-looking guy with the big-ass Dell notebook over at the other end of the coffee shop is one of the Emerald developers. He sure is scowling a lot.

But here's my take: Linden Lab is between a rock and a hard place. If a significant portion (half?) of Second Life's logged hours are from the Emerald viewer, banning Emerald from the Second Life grid will alienate a substantial number of Second Life's most ardent users—and, undoubtedly, that includes many content creators, power users, builders, and folks who run in-world businesses, successful or not.

On the other hand, if Linden Lab lets Emerald back into its directory of viewers that self-certify they conform to the Lab's third party viewer policy, then, clearly, the third party viewer policy—the subject of much drama and gnashing of teeth—is utterly meaningless. Emerald claimed it conformed to the terms of Linden Lab's third party viewer policy, and has now repeatedly and willfully violated that policy.

My guess is that Linden Lab and the re-constituting-itself Emerald development team will try to strike some sort of compromise, perhaps a "probationary" period wherein Emerald will still be permitted to connect to the main grid but will not be listed as conforming to the third party viewer policy until the viewer has a clean record for, say, a year, and the development team proves it can keep its new glass house in order. If I were Linden Lab—and I didn't want the entire third party viewer program thrown out the window—I would set conditions to any such probation. One of those conditions would be that Emerald must inform every user on every login that the viewer has violated Linden Lab's third party viewer policy, with a link with complete disclosure of the violations and what the Emerald team is doing to rectify the problems.

No matter what, I'm not going to be touching Emerald anytime soon.

25-Aug-2010

  • The Emerald development team says Linden Lab has issued a set of undisclosed requirements Emerald has to fulfill before it may re-apply to the third-party viewer directory.
  • I've made some minor tweaks to my text above to clean up some sloppy grammar, add a few links, and correct some production issues. I was working quickly, and Blogger isn't my idea of a proper editing environment.

3 comments:

  1. I like Emerald. It's the viewer that has consistently given me better performance, more flexibility and features, and greater ease of use. My dislike for the standard LL viewers comes from actual use, not hearsay or third-party opinion.

    However, I grow increasingly weary of being a pawn in the political/financial/ethical/technical/ideological (pick one) infighting between LL and Emerald and whoever else decides to wade into the battle for viewer domination. I just want a quality in-world experience, without being taken down some Rosedale-covered path or being unwittingly used in someone else's technovellian revenge maneuvers.

    Increasingly, I feel my days in SL becoming numbered, and the numbers in question are growing smaller at a rapidly increasing rate.

    ReplyDelete
  2. Lou, I have used Emerald for some time now...as far as security, the info I got was that nothing private is on x86 files where all viewer data is. Despite that...I won't use Emerald until a new cleaned version is made available if then... while I do miss a few of the radar options the Emerald viewer offered...The Imprudence viewer I'm currently using seems to load textures a lot faster while overall seems a tad slower.

    I watched the new Emerald team members talking about what had happened on Treet TV...maybe I'm just foolish, but I believe them.

    I don't see LL stopping the Emerald viewer any time soon. The Lab knows 2.ought oh doesn't make the customers happy. I do know that 2.ought oh is a disaster that the Lab has spent too much money on to give up at this time, but pxxxxxg off the number of us that use a TPV isn't good business.
    As for the numbers of users? Download Imprudence and go around in SL and look at all the green tags....there's a lot of green. And the ages of those using that green tag means that's a lot of "green" that older accounts spend.

    ReplyDelete
  3. Becks: I'm with you on not wanting to be a pawn, and as much as I think I would like some of Emerald's features, I have several hesitations unrelated to my level of trust with the Emerald development team...and that was very low to start with and even lower now. But you've hit the nail on the head. Second Life ought to be getting better; instead, it's getting bitter.

    Brinda: I watched as much of the Paisley Beebe/treet.tv thing as I could stomach. I give Paisley and the two Emerald reps points for trying but nothing I heard reassured me in the slightest - in fact, the piece raised a number of additional questions in my mind that, I suppose, will only be borne out as Emerald attempts to form its no-pun-intended crystal-clear organization and process.

    I haven't tried Imprudence either, and I will not until I look into it and the development team more deeply.

    ReplyDelete

Comments are moderated. You can use some HTML tags, such as <b>, <i>, <a>. If you'd like to contact me privately, use a blog comment and say you don't want it published.