There really isn’t a great reason why the compiler can’t infer the type of the variable on the right hand [...]]]> I got into a mild argument about static vs. dynamic typing. I recognize that static typing can be verbose to the point of being repetitious. Take Java generics for example:

List<String> astr = new ArrayList<String>();

There really isn’t a great reason why the compiler can’t infer the type of the variable on the right hand side of the assignment. C# already implements type inference for this case, and C++ is adding it. ML and Haskell are strongly typed and have practiced type-inference since their inception. So we should actually dismiss the verbosity objection to static typing right now, because it’s an artifact of implementation that the more popular languages, C++ and Java, represent really poor examples of what could otherwise be a really good thing.

In my opinion, a static typing system is actually a proof system over your code. We shouldn’t complain about having compiler errors, rather we should rejoice that the compiler is able to automatically detect cases where we were ambiguous or tried to do something ill-defined. We should try to write our code so that the compiler can tell us when we make a mistake. Really, we want to express as many constraints as possible, so that the machine can do more checking and we end up with less buggy code. Statically typing all our variables and expressing systematic constraints is an effort that pays off in spades for large code bases.

But couldn’t we all just use the more flexible dynamic typing languages, and catch the bugs with testing? In my opinion, no. Testing should be done anyway, but it isn’t enough to prove the absence of a bug. Only a proof checker, such as a static typing system, can come close to doing that. I think I can really drive this point home by examining web applications.

The Problem

Web applications are really glorified string processors. HTML requests come in as strings, and web pages are emitted as strings. JavaScript processes more strings in the page layout, potentially requesting even more information from the server in response to user-generated events. Forums, Social Networking, and other participatory applications allow for user generated content. This widespread and popular practice actually leaves our glorified string parser (web app) at risk: for, if we are not careful, a malicious user can supply a string which, if it appears in the ‘wrong’ context, might be interpreted as legitimate JavaScript code by the application. That is, malicious users can execute arbitrary code, with the full rights and privileges as the application itself. This vulnerability is known as Cross-Site Scripting (XSS).

So, we find ourselves writing a string processor which must deal with strings of various encodings, special characters, and escape conventions. Namely, HTML, JavaScript, XML, CSS, URL. If one of these strings (even from our own database) manages to arrive in a context without first going through a filter to sanitize it, then our application has a security vulnerability. Do you think that it’s possible to write test cases (or even auto-generate them) given all the code paths, all the different sources (user, cookie, url, database, etc) and all the contexts in which a string might appear. In my opinion, the exponential complexity makes testing an infeasible approach. What we really need, then, is a proof system to verify that no strings end up in the wrong context.

Static Typing to the Rescue

If we are willing to go back to our application and examine it in detail, we find that we should really be treating each of the above strings as different types. HtmlString should be a different type from JSString, which are again both different from UrlString. Simply expressing each context as a different type enables our static typing system to verify that we never use the wrong kind of string in the wrong context. We can also provide explicit conversion functions, which provide the proper escaping and sanitization when moving from one context to another.

void addToDocument(HtmlString hStr);
HtmlString fromURL(UrlString uStr);
UnsafeString HttpRequest(UrlString uStr);

Language Support

What’s most unfortunate about this approach is that neither C++ nor Java provide us with an easy way to distinguish two strings. We certainly don’t want to use C’s typedef, because that enables automatic coercion between the different kinds of string, which defeats the point. So, we’re forced into creating a separate class for each of these strings, including implementing all the operators that make for convenient string manipulation. I’d really love a language that would allow me to extend my existing string type without fully re-implementing everything, yet still be able to treat the extension as a completely different type.

Conclusion

Essentially we’re using the static typing as a proof system to constrain our programming practices. The static type verification provides a proof that we never use a string in the wrong context. In my opinion, this coding technique is of enormous benefit, and represents a use-case that dynamic typing + unit testing simply cannot approach.

The real trick is recognizing that two strings aren’t necessarily the same type.

Just for reference, I did not come up with this example myself.
Joel Spolsky advocates using Hungarian notation, which I think is too weak for solving security vulnerabilities.
Tom Moertel provides an inplementation of this approach in Haskell.

There’s really a ton of stuff to think about. Mostly the field of computer science concerns itself with taming complexity. All too often [...]]]> Reddit modded up a nice review of Considerations When Designing your Own Programming/Scripting Language (it’s worth following the links provided there to Clementson’s Blog, to get a larger picture of the issue).

There’s really a ton of stuff to think about. Mostly the field of computer science concerns itself with taming complexity. All too often software projects buckle under the weight of feature creep or code bloat. Eventually, in the career of any decent programmer, the ability to clearly see and identify the common organizational patterns becomes a core focus. Anyone with exposure to several different languages, notices that these organizational patterns are expressed differently among the babble, that even the same old problems (i.e. parsing, printf, or search/logic) can be cast in completely different light by a shift in linguistic perspective.

We see that language design suffers exactly the same problems as other domains. But, because it is a language, botching your solution to these problems, will affect the way that programmers solve their problems. Once such non-trivial task that involves many sub-problems which can clearly bring to light these issues is that of writing a compiler. Compilers involve a great deal of data structures and algorithms: text-manipulation, parsing, trees, graphs, fixed-point algorithms (dataflow analysis), NP-complete problems (register allocation, instruction scheduling). From this simple observation, we see that we should follow Wirth’s language-complexity metric: that languages should be compared based on the relative sizes of their self-compilers. (It really is an elegant fixed-point metric to the circularity involved in the complexity-of-expression/complexity-of-implementation trade-off).

So, to highlight these ideas, let’s take a particularly poor specimen: C++. This linguistic monstrosity is both verbose and inconvenient. Yes, it let’s you get close to the ‘bare-metal’, and it requires you to really think about what you are doing. (to a fault actually: I’ve noticed that the more C++ I learn the more tricky and horrendous my bugs become). But, even though I use it most often, I can’t help but feel that it score’s incredibly poor in the design space. The features don’t interact well together: inheritance vs templates, memory management vs exceptions, etc. The lower-level exposure consistently prevents higher-level conveniences. For example, pointer arithmetic prevents garbage collection (although garbage collection was proposed for C++0x it got thrown out at the last minute, hopefully it will be introduced eventually. I’m not sure I want to know what horrible contortions of logic must be followed to actually implement it.) Finally, it should be noted, that the complexity of implementing C++ in C++ is incredibly daunting.

So, designing a language is incredibly difficult. The implementation details of certain features can easily wreak havoc on the design. What’s it take to implement closures? Do you want a fast linear stack, or would you like to have continuations? Do you want type-safety or does that drive you to drop first-class functions out of implementor’s laziness? Nor do the features always work well together, as demonstrated by the C++ trade-offs above. Yes, before designing your own language, take great heed of the accumulated wisdom within that introductory post.

Supposing you were asked to perform the well known Dijkstra’s Algorithm. Someone hands you a graph of generic items, let’s call [...]]]> I’ve come across another programming language feature that I would like to have. The last one was a bit outlandish, and I’d really like to refine it a bit. Dress it up a little.

Supposing you were asked to perform the well known Dijkstra’s Algorithm. Someone hands you a graph of generic items, let’s call them Nodes, and you know that each Node has a List<Node*> of edges. So far, so good.

Now, Dijkstra’s Algorithm works pretty well if you are allowed to tag or mark the Nodes in some way. If you have control over the Node class, then you might be tempted to ‘gift’ the class with a member field to hold this tag. But, you should resist this temptation, because it’s not really appropriate.

It will pollute your class. You are increasing the size of each and every Node, regardless of wether it participates in Dijkstra’s Algorithm. The additional field, if not commented, might confuse you later. If you’re paranoid you might never touch it again; even after removing Dijkstra’s Algorithm from your codebase. Even if commented, you might later hijack the field for use in other algorithms for which marking or tagging is convenient. Cleverly using the same field for dual missions, and later confusing yourself or getting into an inconsistent state.

So, we don’t want to make an additional field. Fine, what other options are there?

We could subclass Node to make a MarkableNode. But, in the implementation given above, we see that Nodes point only to other Nodes. To really give a seamless touch to the algorithm, we’d have to make a complete copy of the graph we were given, out of the MarkableNodes.

Alternatively, We could write a wrapper class, MarkableNode that wraps a Node*. This will work generically, yet you have to write a ton of other features for equality tests, worry about creating two different MarkableNodes around the same Node*, etc. What a crapload of tedious busywork.

class MarkedNode
{
public:
    MarkedNode(Node *node)
        :m_node = node;
        ,m_mark = false; {
    }
 
    void mark() {
        m_mark = true;
    }
 
private:
    Node *m_node;
    bool m_mark;
}

How ’bout this last option. We could temporarily attach to Node* a method (and storage) for marking the nodes. This method would only exist in the scope that it’s needed, so it won’t pollute the Node class. It would behave like a subclass or a wrapper class, but will need to be syntactically lighter weight, and semantically the same as having a Node* except for the specific extensions for marking.

Class MarkedNode : extends Node*
{
    m_marked = false;
 
public:
    void mark() {
        m_marked = true;
    }
}

So, essentially we can call this method on any Node*, and we allow the compiler to figure out how it will map (1-to-1) each m_marked field to each Node*. We also let the compiler figure out how to mangle the mark() method. Doing the mapping of Node* to extended data will be tricky, especially if we want to clear out the data when a Node* is deleted.

If you drew the benchmark results on an XY chart you could name the four corners. The fast but verbose languages would cluster at the top left. Let’s call them system languages. The elegantly [...]]]> Guillaume Marceau has used data from the Computer Language Benchmark Game to provide a graphical comparison of many different languages.

If you drew the benchmark results on an XY chart you could name the four corners. The fast but verbose languages would cluster at the top left. Let’s call them system languages. The elegantly concise but sluggish languages would cluster at the bottom right. Let’s call them script languages. On the top right you would find the obsolete languages. That is, languages which have since been outclassed by newer languages, unless they offer some quirky attraction that is not captured by the data here. And finally, in the bottom left corner you would find probably nothing, since this is the space of the ideal language, the one which is at the same time fast and short and a joy to use.

Of course the C compilers do a very good job on performance, but seem to do average on verbosity (better than I expected). Haskell (ghc) does a surprisingly nice job. I wish that I’d thought to do this kind of visualization, it’s really pretty neat. The only improvement that I could think of, would be to do the performance axis on a logarithmic scale.

Given a very large corpus of sentences, such as a digitized version of the Library of Congress, or a less noisy version of the Internet, how can you automatically generate a Thesaurus?

At first I thought the problem should be fairly easy, but [...]]]> Last week, I landed on another PhD worthy research project.

Given a very large corpus of sentences, such as a digitized version of the Library of Congress, or a less noisy version of the Internet, how can you automatically generate a Thesaurus?

At first I thought the problem should be fairly easy, but the more I thought about it the more difficult and daunting the task became. For example, as a first approach, we might assume that textual substitution would be a good proxy for identifying synonymous terms. That is if a couple terms really are synonymous, then they ought to be substitutable for each other in a sentence. With a large enough set of sentences, we should be able to identify such situations, and thereby bootstrap the building of the thesaurus. But there’s a small problem, provided by my good friend EvB:

The sky is blue.
The ocean is blue.

But sky is not the same as ocean. Sure they are similar. A poet could compose a nice metaphor of fish swimming through their sky above the bottom feeders. But this metaphorical relationship isn’t one that would necessarily make it into a human compile thesaurus. So, textual substitution can easily lead us astray.

Continuing with EvB’s particularly good example we can also identify another problem. Suppose that we incorporate a bit of natural language understanding, enough to pull out parts of speech. Then, the system would easily identify the equation of sky with blue, or ocean with blue. But neither of these statements is true either. Usually people take the example to mean not that sky and blue are the same thing, but that the sky belongs to the set of objects that have a property called color, the value of which is blue. So this understanding depends on what the definition of ‘is’ is (obviously not a simple affair). We also would like to avoid drawing a relationship between any of the pronouns and the rest of the language.

Next lets look at how people tend to write. Any good library is gonna be full of metaphor, simile, pun, allusion, word play, sound play, and other such highly nuanced expression. All of these things will trump any reasonably simple attempt at drawing a link between synonymous words. Political propaganda and polemic, will probably be particularly bad at equating terms that should probably be kept logically distinct. Furthermore, at least when I write, I’m reminded of other things during the process, things that are associated, but not necessarily synonymous. These remindings are an important part of the essay writing process, but will certainly throw noise into the digital library.

But if it’s so hard to make a mechanical system for identifying synonyms, then how do humans do it? Here I have a hypothesis: that similar words stimulate similar patterns in the brain. Thus when a human tries to think up synonyms it’s really the same as playing word association with a filter. First, the word stimulates the brain, bring up certain associations. These associations will be based on ‘brain distance’, a measure of the similarity of brain activity for certain words and thoughts. But some associations will be radically different from the synonyms that we’re looking for. For example, antonyms and non-sequiturs often come up in word association games. So a filter is applied to weed these out, and what’s left is passed through a dictionary/meaning check. Anything passing this process will be reported as a synonymous term.

So, in order to really generate a thesaurus, we do need AI (or at least an underlying cognitive model). When I first thought of the thesaurus problem, I was hoping that it was paired down enough, small and simple enough that it would be doable without all this complexity. We might have to reduce the problem further, make it looser. Say, build an association dictionary, rather than a thesaurus. An association dictionary might be possible, because it forgoes the understanding of meaning and similarity, it doesn’t have to question or measure why two words should be associated, only record that they are used similarly.

So, if you can automate the building of a thesaurus, you should get a PhD in Linguistics.