Behavioral Classification on the Click Graph

A bipartite query-URL graph, where an edge indicates that a document was clicked for a query, is a useful construct for finding groups of related queries and URLs. Here we use this behavior graph for classification. We choose a click graph sampled from two weeks of image search activity, and the task of ``adult'' filtering: identifying content in the graph that is inappropriate for minors. We show how to perform classification using random walks on this graph, and two methods for estimating clas...

Comic for 28 Mar 2008

“Machine Listening” is the idea that computers can be programmed to interpret audio signals the same way humans do. This means that they can tell when a song belongs to the blues genre rather than techno. And they can detect musical characteristics like tempos, transition types, and harmonies.

The technology has some obvious practical uses. It could be used to compile collections of music with the same sound or with similarities to the music someone already knows they like. Applications could also be designed to create the perfect mixtapes, with songs picked and ordered in just the right ways.

The Echo Nest is a company that’s bringing machine listening to Web 2.0. It was founded by two MIT PhD students and is supported by a government grant. Today, the company releases the first of several “Musical Brain” APIs intended to improve three main aspects of music-related web services: search, recommendations, and interactivity.

The first API, which focuses on signature analysis and is being released through Mashery, can be used to retrieve an XML file with information about a particular song. A proof of concept website called This is my jam has been set up to demonstrate its capabilities. Load up a few of your favorite artists and it will automatically arrange songs from them in an order deemed most suitable given their audio characteristics.

The Echo Nest will lend all of its APIs to non-commercial projects for free, but it will charge commercial sites with a usage fee. The company plans on showcasing a website for each of its APIs, but it doesn’t currently have any plans to create a consumer destination of its own with the tech.

CrunchBase Information

The Echo Nest

Information provided by CrunchBase

Crunch Network: MobileCrunch Mobile Gadgets and Applications, Delivered Daily.

Paper: On Designing and Deploying Internet-Scale Services

Greg Linden links to a heavily lesson ladened LISA 2007 paper titled On Designing and Deploying Internet-Scale Services by James Hamilton of the Windows Live Services Platform group. I know people crave nitty-gritty details, but this isn't a how to configure a web server article. It hitches you to a rocket and zooms you up to 50,000 feet so you can take a look at best web operations practices from a broad, yet practical perspective. The author and his team of contributors obviously have a lot of in the trenches experience. Many non-obvious topics are covered. And there's a lot to learn from.

The paper has too many details to cover here, but the big sections are:

Recommendations

Automatic Management and Provisioning

Dependency Management

Release Cycle and Testing

Operations and Capacity Planning

Graceful Degradation and Admission Control

Customer Self-Provisioning and Self-Help

Customer and Press Communication Plan

In the recommendations we see some of our old favorites:

read more

03/21/08 PHD comic: 'Ph.D. Training'

Piled Higher & Deeper by Jorge Cham		www.phdcomics.com
title: "Ph.D. Training" - originally published 3/21/2008 For the latest news in PHD Comics, CLICK HERE!

Semantic startups and projects are hot right now. (See Radar Networks, Freebase, Blue Organizer, Hakia, even Yahoo). But what do you do if you are a little-known technology company in Rochester, New York with a powerful semantic-analysis engine on your hands that you want to turn into new businesses?

You offer a $1 million prize to anyone who can come up with the most profitable application for your technology and call it the Semantic Hacker challenge. The challenge starts today, and is being sponsored by TextWise, a private company backed by pension-fund adviser Bill Manning that has been around since 1994. TextWise uses natural-language processing and semantic analysis to automatically categorize Web pages and create contextual ads for them. But it wants to see what the crowd can do with its technology. It is opening up its APIs, much like semantic search engine Hakia did yesterday. But instead of merely licensing the technology, which it is willing to do, it hopes to generate actual business ideas that it can run with. CEO Connie Kenneally explains how the challenge will work:

The winners of the challenges would turn over rights to their idea. We would award them $100,000 immediately, we would likely make them or their team job offers, and we would build out whatever is required. Then they would receive 50% of the first year’s revenues, up to $1 million.

Not a bad deal for simply coming up with a killer semantic application. Before somebody else takes it, my idea is a semantic search engine that actually works at Web scale (please send the check to my home address). Don’t worry. There can be more than one winner.

Kenneally is hoping for specific suggestions to apply semantic analysis to different industries. Any idea is fair game, except for four works-in-progress TextWise is already developing: a browser plug-in that replaces ads with content related to the page you are on (foof), semantic bookmarks that bring up related content from the top 6,000 blogs on Technorati (Gyzork), a shopping discovery tool, and a Facebook app that automatically provides shareable links relevant to a given conversation (Festoon). That Gyzork idea is my favorite. Instead of saving bookmarks, you save concepts and the relevant links are added to the bookmark over time.

The way the technology works is that it creates “Semantic Signatures” from any text that is fed into the system. You put text in and it spits out categories it thinks the text fits under, as well as related Wikipedia articles. On the Semantic Hacker site you can try it out by cutting and pasting some text and seeing what it comes up with. For instance, I put in the text from a post I wrote about China blocking YouTube, and it generated a Semantic Signature with these categories:

Society/Issues/Territorial_Disputes/Tibet 68

…/Religion_and_Spirituality/Buddhism/Lineages/Tibetan/Dalai_Lama 48

Society/Religion_and_Spirituality/Falun_Dafa 22

Computers/Internet/Searching/Search_Engines/Google 17

Computers/Internet/Searching/Directories/Volunteer-Edited 17

That is pretty accurate. (The numbers weight the relevance of each category on a scale of 1 to 100). And it produced ten related Wikipedia articles about Tibet as well. “It is like decoding the DNA of the text—creating a semantic map of the text,” explains Kenneally. She says it can scale to hundreds of millions of Web pages, but for the challenge she is limiting applicants to 20,000 separate requests a day, and up to 100,000 characters per request. Unlike, say, the Netflix challenge to come up with a better recommendation engine, TextWise isn’t looking for someone to improve its algorithm, which it is keeping secret. It just wants to know what applications and markets to go after.

If you were going to build a semantic startup, what would it be?

CrunchBase Information

TextWise

Hakia

Radar Networks

Information provided by CrunchBase

Crunch Network: MobileCrunch Mobile Gadgets and Applications, Delivered Daily.

Yahoo Mail Gives Users Trojan Horses

I got this picture from a reader of the site. Apparently the reader was simply viewing Yahoo mail and poof, RogueIframe trojan. We are starting to see a lot more of this kind of stuff, but it’s really disappointing that third party ads are being displayed on otherwise sensitive apps (or at least I think most people feel they are sensitive). Here’s the picture:

Click to enlarge

We’ve seen this exact hack hit before, against Facebook. But I think this kind of thing may be the beginning of a epidemic. As long as you can end up with your advertisements on any site that is even vaguely sensitive, you can start either taking over the site, or delivering malware. Whatever best suits the attacker’s needs. I think this all goes back Tom Stripling’s speech at OWASP where he in painstaking detail explained why you cannot trust third party JavaScript on your site, and yes, that definitely includes advertisements. Anyway, I hope this gets cleaned up quickly.

Paper: Consistent Hashing and Random Trees: Distributed Caching Protocols for Relieving Hot Spots on the World Wide Web

Consistent hashing is one of those ideas that really puts the science in computer science and reminds us why all those really smart people spend years slaving over algorithms. Consistent hashing is "a scheme that provides hash table functionality in a way that the addition or removal of one slot does not significantly change the mapping of keys to slots" and was originally a way of distributing requests among a changing population of web servers. My first reaction to the idea was "wow, that's really smart" and I sadly realized I would never come up with something so elegant. I then immediately saw applications for it everywhere. And consistent hashing is used everywhere: distributed hash tables, overlay networks, P2P, IM, caching, and CDNs. Here's the abstract from the original paper and after the abstract are some links to a few very good articles with accessible explanations of consistent hashing and its applications in the real world.

read more

Closures: Control Abstraction, Method References, Puzzler Solution

The Java Closures prototype now supports control abstraction and
implements restricted closures and function types. The syntax has
changed slightly. Also, as hinted in the
draft JSR
proposal, there is now support for eta abstraction, which
is called method reference in
Stephen
Colebourne's FCM proposal. We haven't updated the
specification, so this will serve
as a brief tutorial on the changes until we do. I don't know if this
will be the syntax we will end up with, but it will do for now. Finally,
we look at solutions to the closure puzzler in my previous post.

Control Abstraction

The first thing you'll notice when using the new prototype is that
the compiler gives a warning when a closure uses a local variable from an enclosing scope:

Example.java:4: warning: [shared] captured variable i not annotated @Shared
        Runnable r = { => System.out.println(i); };
                                             ^

There are a few ways to make this warning go away:

• declare the variable final; or


• annotate the variable @Shared; or


• make sure the variable is not the target of any assignment expression; or


• put @SuppressWarnings("shared") on an enclosing method or class; or


• use an unrestricted closure, by using the ==> token
  instead of the => token (when possible).

The => token builds a restricted closure that
triggers this warning. Restricted closures also do not allow a
break or continue statement to a target outside
the closure, nor a return statement from the enclosing method.
You will rarely want to write an unrestricted closure; many (but not all) of
the things you need to do with an unrestricted closure can be expressed more
clearly with a control invocation statement instead.

You're not allowed to assign an unrestricted closure to a restricted
interface. A number of existing JDK interfaces, such as
java.lang.Runnable, have been modified to be restricted.

Error: cannot assign an unrestricted closure to a restricted interface type
        Runnable r = { ==> System.out.println(i); };
                       ^

In the less common case that you're writing a method intended to be used as a control
API, you can write a function type with the (new) ==> token to designate
an unrestricted function (interface) type. Let's do that to write a method,
with, that will automatically close a stream for us. The idea is to be able
to replace this code

FileInputStream input = new FileInputStream(fileName);
try {
    // use input
} finally {
    try {
        input.close();
    } catch (IOException ex) {
        logger.log(Level.SEVERE, ex.getMessage(), ex);
    }
}

with this

with (FileInputStream input : new FileInputStream(fileName)) {
    // use input
}

which is an invocation of the following method

public static void with(FileInputStream t, {FileInputStream==>void} block) {
    try {
        block.invoke(t);
    } finally {
        try {
            t.close();
        } catch (IOException ex) {
            logger.log(Level.SEVERE, ex.getMessage(), ex);
        }
    }
}

This is among the simplest control APIs, but it has some limitations:

• It works with the type FileInputStream, but not any other Closeable types


• It does not support exception transparency


• It does not support completion transparency

Completing the API by repairing these defects is left as an exercise to the reader.
A solution will be discussed in
my
JavaOne talk Closures Cookbook.

Method References

A natural companion to closures is a way to refer to an existing method instead
of writing a closure that accepts the same arguments and just invokes the method. This is
sometimes known as eta
abstraction or method
references. We expect closures in their final form to include support for this
convenient feature, which is why it is called out in the
draft JSR proposal. The
latest version of the prototype supports this, with a syntax based on javadoc conventions.
Here are a few examples:

{ int => Integer } integerValue = Integer#valueOf(int);
{ Integer => String } integerString = Integer#toString();
{ int, int => int } min = Math#min(int, int);
{ String => void } println = System.out#println(String);
{ => String } three = new Integer(3)#toString();
{ Collection<String> => String } max = Collections#max(Collection<String>);
{ => Collection<String> } makeEmpty = Collections#<String>emptySet();
Runnable printEmptyLine = System.out#println();

Writing code as a method is sometimes more convenient than writing it as a closure:

void doTask() {
    // a complex task to be done in the background
}


Executor ex = ...;
ex.execute(this#doTask());

Puzzler Solution

A couple of weeks ago we looked at
a
Java puzzler involving closures, and a number of people discussed the underlying issue. My favorite is David's post "Color-flavor locking breaks chiral symmetry". Lessons include not exposing public fields (accessors are better) and being careful to avoid cyclic initialization dependencies.

The enum language feature provides support for one solution to the puzzle: specialize each instance of the enums.

import java.util.*;

enum Color {
    BROWN {
        public Flavor flavor() {
            return Flavor.CHOCOLATE;
        }
    },
    RED {
        public Flavor flavor() {
            return Flavor.STRAWBERRY;
        }
    },
    WHITE {
        public Flavor flavor() {
            return Flavor.VANILLA;
        }
    };
    abstract Flavor flavor();
}

enum Flavor {
    CHOCOLATE {
        public Color color() {
            return Color.BROWN;
        }
    },
    STRAWBERRY {
        public Color color() {
            return Color.RED;
        }
    },
    VANILLA {
        public Color color() {
            return Color.WHITE;
        }
    };
    abstract Color color();

}

class Neapolitan {

    static <T,U> List<U> map(List<T> list, {T=>U} transform) {
        List<U> result = new ArrayList<U>(list.size());
        for (T t : list) {
            result.add(transform.invoke(t));
        }
        return result;
    }

    public static void main(String[] args) {
        List<Color> colors = map(Arrays.asList(Flavor.values()), { Flavor f => f.color() });
        System.out.println(colors.equals(Arrays.asList(Color.values())));

        List<Flavor> flavors = map(Arrays.asList(Color.values()), { Color c => c.flavor() });
        System.out.println(flavors.equals(Arrays.asList(Flavor.values())));
    }
}

Another elegant solution, due to 5er_levart, uses closures:

enum Color {
    BROWN({=>Flavor.CHOCOLATE}),
    RED({=>Flavor.STRAWBERRY}),
    WHITE({=>Flavor.VANILLA});

    private final {=>Flavor} flavor;

    public Flavor flavor() { return flavor.invoke(); }

    Color({=>Flavor} flavor) {
        this.flavor = flavor;
    }
}

enum Flavor {
    CHOCOLATE({=>Color.BROWN}),
    STRAWBERRY({=>Color.RED}),
    VANILLA({=>Color.WHITE});

    private final {=>Color} color;

    public Color color() { return color.invoke(); }

    Flavor({=>Color} color) {
        this.color = color;
    }
}

In both solutions the idea is to compute the value lazily, a key technique to break dependency cycles.

Security In IE7 & IE8.

I had some fun today on Internet Explorer and tested the new IE8 also. Along, I also compared the new security features to IE7. And there are many things to say about both versions. Don't get me started on the usability. To be honest it looks horrible. Don't know if you ever noticed it, but selecting text with your mouse has become a real challenge. It has some strange selection boundary which results in selecting text you don't want to select. Typically Microsoft, onto the security features. You might have heard or read about a new feature called XDR that allows cross-domain requests. If not, I'll explain it here in short detail as well as the other new features before I'm going into reverse engineering IE7 & IE8.



The XDR object.

xdr = new XDomainRequest();  

xdr.open('POST', 'http://www.mr.bigglesworth.com'); 

xdr.send(data);

Now, Mr.Bigglesworth needs to approve the send XDomainRequest header, but we can approve the call by returning this header to the server that requested legitimacy:

Response.AppendHeader("XDomainRequestAllowed","1");

Great, XSS made easy. No need for hijacked iframes, css or images. Nope pure Javascript does the trick for us. This obviously can bypass many XSS filters in use today, so if you run one be sure to check this beast out. In my opinion this will broaden the attack landscape since there are more ways of launching XSS or spreading worms. The XDR object also returns the responseText that gives access to:

xdr.onerror

xdr.ontimeout

xdr.onprogress

xdr.onload 

xdr.timeout

Useful, if you're into worms and all.



next, I saw that they implemented cross-document messaging in the form of the object postMessage. Opera already has it, and from a security standpoint I don't trust it. It basically means that a webpage can write into another page that is running in the same session and on the same host by attaching an event listener. Spoofing comes to mind, and maybe other attacks as well. The real question is of course: what is it for? I don't know.



Implementing it is a breeze:



page 1:

var doc = document.getElementsByTagName('iframe')[0];

doc.contentWindow.postMessage('Hello Mr. Bigglesworth!');

page 2:

document.attachEvent('onmessage',function(e) {

    if (e.domain == 'example.com') {

        if (e.data == 'Hello Mr. Bigglesworth!') {

            e.source.postMessage('Meow! Meow! Dr. Evil!');

        } else {

            alert(e.data);

        }

    }

});

Hash write access.



Another thing that caught my eye was write access to the hash of an url. Doesn't sound smart because I don't want Javascript to manipulate the hash. Not only can it be annoying, it can lead to security issues depending on the setting of your website.



Webslices.



If I understand it correctly this feature allows users to favorite the slice or put it in their feed reader. Better expect some buffer overflows here since IE8 now listens for a tag called 'hslice' on any page it opens, would be nice to fuzz this feature.

<div class="hslice" id="main">

<h2 class="entry-title">All I want are friggin' sharks with friggin' lazer beams attached to their heads! </h2>

</div> 

GlobalStorage & SessionStorage.



IE8 jumped on the Mozilla bandwagon and implemented the Session object. I can't say I'm that impressed because I as I said before; allowing 10MB of data to be stored in such object (XML file in IE8) isn't smart. Let alone the permanent storage of user tracking details, XSS worms and other spy-ware.



IE8 GlobalStorage

<script>



var storage = globalStorage[location.hostname];



storage.some_string = '

Ladies and Gentlemen welcome to my underground lair. 

I have gathered here before me the worlds deadliest assassins. 

And yet each of you has failed to kill Austin powers. 

That makes me angry. And when Dr. Evil  get angry, Mr. Bigglesworth gets upset. 

And when Mr. Bigglesworth gets upset...people DIE!!! 

Why must I be surrounded by freakin idiots. Mustafa, Frau Farbissina...

';



</script> 

Reverse Engineering IE7 & IE8.



Okay, this is fun. I'm going to show you a couple things I found out about Internet Explorer. First off IE8 prevents header forwards on files, pity this was pretty 'evil' in MSIE 7 where it is still possible to change the location of a file to a local file stored on your computer. It's very simple:

<?

header("location: localfile ");

?>

And IE7 follows it, whereas IE8 refuses to follow.



The reason why this is dangerous is because of this XML file that contains system information which we could parse. Useful for reconnaissance and possibly other attack schemes.

<?

header("location: res://ieframe.dll/24/123");

?>

Results in IE7:

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?> 

- <!--  Copyright (c) Microsoft Corporation 

  --> 

- <assembly xmlns="urn:schemas-microsoft-com:asm.v1" xmlns:asmv3="urn:schemas-microsoft-com:asm.v3" manifestVersion="1.0">

  <assemblyIdentity name="Microsoft.Windows.InetCore.ieframe"processorArchitecture="x86" version="5.1.0.0" 

type="win32" /> 

  <description>Windows IE</description> 

- <dependency>

- <dependentAssembly>

  <assemblyIdentity type="win32"name="Microsoft.Windows.Common-Controls"version="6.0.0.0" processorArchitecture="*" 

publicKeyToken="6595b64144ccf1df" language="*" 

/> 

  </dependentAssembly>

  </dependency>

- <trustInfo xmlns="urn:schemas-microsoft-com:asm.v3">

- <security>

- <requestedPrivileges>

  <requestedExecutionLevel level="asInvoker" uiAccess="false" /> 

  </requestedPrivileges>

  </security>

  </trustInfo>

- <asmv3:application>

- <asmv3:windowsSettings xmlns="http://schemas.microsoft.com/SMI/2005/WindowsSettings">

  <dpiAware>true</dpiAware> 

  </asmv3:windowsSettings>

  </asmv3:application>

  </assembly>

If you notice correctly I read res://ieframe.dll/24/123 located on ieframe.dll which is the IEDataObjectWrapper (InProcServer32) I don't know why they still allow this to be browsable, because you can resource it on iframes, XML and as a Javascript source. So I went further to find all data object in IE8 and a few in IE7.



IE7/8 data sources are:

res://ieframe.dll/MUI/1

res://ieframe.dll/TYPELIB/1

res://ieframe.dll/UIFILE/{20481,20482,20483,20484,20484,20485,20486,20487,39216,41555}

res://ieframe.dll/WEVT_TEMPLATE/1

res://ieframe.dll/Version Info/1

res://ieframe.dll/23/ABOUT.js

res://ieframe.dll/23/ANALYZE.js

res://ieframe.dll/23/ANCHBRWS.js

res://ieframe.dll/23/DOCBROWS.js

res://ieframe.dll/23/ERROR.js

res://ieframe.dll/23/HTTPERRORPAGESSCRIPTS.js

res://ieframe.dll/23/IEERROR.js

res://ieframe.dll/23/IMGBROWS.js

res://ieframe.dll/23/INVALIDCERT.js

res://ieframe.dll/23/ORGFAV.js

res://ieframe.dll/23/PHISHSITE.js

res://ieframe.dll/23/POLICY.js

res://ieframe.dll/23/PREVIEW.js

res://ieframe.dll/preview.dlg (dialog)

res://ieframe.dll/23/PSTEMPLATES.js

res://ieframe.dll/24/123 (XML file)

IE6 has a few too:

res://mshtml.dll/REGINST/REGINST

res://mshtml.dll/23/ABOUT.MOZ

res://mshtml.dll/23/BLANK.HTM

res://mshtml.dll/23/REPOST.HTM

As well as others.



These are nice to play with some more, I haven't digged any deeper yet but this is quite nice to have a look at. So, enough building blocks to pentest IE a little further. If you find anything notable, do let me know.



Have fun.