Infrequently updated blog!

Silly placeholder post

2009-04-04T04:10:00.000-07:00

I should update this blog more. Maybe just a little thing right now: I really like the option parsing library Trollop for Ruby. It's a great little system, and its values (or the ones important to me, at least) are that it is as easy as possible for the library consumer to use. I've been experimenting with making one with similar values for Scala, adding in the type safety and the like that you get with that language, while aiming to still keep a fluent interface.

Using it looks like this:

object Main extends OptionApplication {
 val options = new Options {
   banner = """
This is my test application.
There are many like it, but this one is mine.

Usage: Main [options]
"""
  
   val name = string( "Name to greet", default( "Nobody" ) )
   val alternativeGreeting = flag( "Use an alternative greeting" )
 }

 import options._

 def main() {
   val greeting = if( alternativeGreeting.value ) {
     "Bonjour, %s!"
   } else {
     "Hello, %s!"
   }
  
   println( greeting format name.value )
 }
}

Calling it from the "command line" (okay, I did it through Eclipse):

> Main --help
This is my test application.
There are many like it, but this one is mine.

Usage: Main [options]

Options:
-a, --alternative-greeting - Use an alternative greeting
               -h, --help - This help page
               -n, --name - Name to greet (default: Nobody)

> Main -n Calum --alternative-greeting
Bonjour, Calum!

> Main --pleasehelpme
Problem with options: unrecognised argument 'pleasehelpme'
Try --help for help.

Anyway, beyond just a random tour of that, why do I think this sort of thing is interesting? I like the idea that Scala works as a statically-type "scripting" language, as well as in the domains where Java works. I like the idea that command line tools and so on could be built with this easily. Java tends to only be used for huge big systems run inside containers and the like, because it's geared very much towards building the sort of huge system where this makes sense. Scala is fluid enough to give a lot of the advantages in fluidity you see from languages like Python and Ruby, and while it's a little more verbose, I think it's a useful alternative depending on your values.

What Scala is really missing, to me, is little "fun" libraries like you find in the form of Python Eggs and Ruby Gems. There's a lot of nice, impressive, projects but I'd like to see more in the realm of small projects that aren't necessarily a work of architectural genius, but are useful (or quirky) and encourage people to dabble and play. Java lacks a sense of fun and there's no reason that Scala should end up down that route. I might go into more detail with that feeling later; this is pretty poorly-drafted. Oh well!

Simple JVM sandboxing

2008-06-22T10:01:00.001-07:00

I have written an dumb IRC bot which (among other things) can run scripts in arbitrary languages, provided they have Java implementations wrapped by the JSR-223 scripting API (a.k.a. javax.script in Java 6). At the moment I've got it doing Ruby, Groovy, Javascript and Scheme.

As part of this I decided that I needed to security restrict the script runner, since by design people will essentially be able to run arbitrary code in it. Java is so kind as to provide this sort of sandboxing functionality in its core framework (this is the same sort of security protection offered when you ran Java as an applet on a webpage years ago when applets were still cool), and although it's very well documented on a high-level online, I found it difficult to find a low-level explanation on exactly how to use the code to build a sandbox like I wanted, so here's how I ended up doing it.

It's worth noting that my IRC bot is actually written in Scala, so the actual Java code here (translated) is probably not well-tested. I've put the Scala wrapper I made at the foot of the entry in case it's useful to anyone.

First step – enable a SecurityManager

The first thing to realise (I certainly didn't) is that none of the security infrastructure in Java does anything until you enable it. By default the security policy checks are disabled (presumably for performance reasons); to enable them you need to attach an instance of SecurityManager to the system. You only need to do this once, and I'd recommend doing it as part of your app setup. It's really easy to do:

System.setSecurityManager(new SecurityManager());

The important thing to notice is that the current security policy — as defined by your .java.policy file in your home directory (I believe this is the same on Windows, incidentally), which will fall back to a very restrictive policy (the one used for applets) if the file doesn't exist. Now really I'd like any code that resides on my machine to run without sandboxing even when a security manager is in place (I like to live dangerously), so I can enter the following into my /home/calum/.java.policy file:

grant codeBase "file:/home/calum/-" {
  permission java.security.AllPermission;
};

The /- at the end matches all files below that directory recursively. One can also use /* to match all JAR and class files in a directory, / to match only class files, and specify individual files. For more information on policy files, check out this tasty document.

Important: Once you create this file, it overrides the default Java policy for anything you have a rule for. Do not just blanket grant permissions without a codeBase, you will be opening yourself to a world of hurt, in security terms.

Once you've restricted your JVM's permissions by adding a SecurityManager, then un-restricted them by granting the permissions back, you're ready to actually manipulate security on parts of code in your app.

Constructing an simple sandbox

Code can be invoked at modified privilege levels by the static AccessController.doPrivileged(PrivilegedAction,AccessControlContext) method. For this, we need to construct a PrivilegedAction<T> (essentially a Callable<T>) which encapsulates the logic to be run (if the logic throws an exception, you can use PrivilegedExceptionAction<T> instead). For our simple case we can fill this context with a single ProtectionDomain with a CodeSource (codesource can never be null) which is constructed will null values to indicate unsigned code from an unknown location, and a PermissionCollection consisting only of the permissions we'd like to grant our code (represented below as perms:

// Cast to Certificate[] required because of ambiguity:
ProtectionDomain domain = new ProtectionDomain( new CodeSource( null, (Certificate[]) null ), perms ); 
AccessControlContext context = new AccessControlContext( new ProtectionDomain[] { domain } );
T result = AccessController.doPrivileged(action, context);

Here's a full Java program which demonstrates the use of these methods (huge, I'm afraid — I've omitted the imports for brevity):

public class SandboxTest {
 private static final File file = new File( "/etc/motd" );
 
 public static void main(String[] args) throws Exception
 {
  // Do not forget this line!
  System.setSecurityManager( new SecurityManager() );
  
  System.out.println( "Calling method directly:" );
  
  System.out.println( readFirstFileLine(file).run() );
  
  System.out.println( "Calling method in restrictive sandbox:" );
  
  System.out.println( callInSandbox( new Permissions() ) );
  
  System.out.println( "Calling method with file read permission:" );
  
  Permissions perms = new Permissions();
  perms.add( new FilePermission( "/etc/motd", "read" ) );
  
  System.out.println( callInSandbox(perms) );
 }
 
 private static String callInSandbox( PermissionCollection perms )
 {
  ProtectionDomain domain = new ProtectionDomain( new CodeSource( null, (Certificate[]) null ), perms );
  AccessControlContext context = new AccessControlContext( new ProtectionDomain[] { domain } );
  try
  {
   return AccessController.doPrivileged( readFirstFileLine(file), context );
  }
  catch( Exception e )
  {
   return e.toString();
  }
 }
 
 private static PrivilegedExceptionAction<String> readFirstFileLine( final File file )
 {
  return new PrivilegedExceptionAction<String>() 
  {
   @Override
   public String run() throws Exception {
    BufferedReader reader = new BufferedReader( new FileReader( file ) );
    
    try
    {
     return reader.readLine();
    }
    finally
    {
     reader.close();
    }
   }
  };
 }
}

On my machine, this yields:

Calling method directly:
Linux thoth 2.6.24-19-generic #1 SMP Wed Jun 4 16:35:01 UTC 2008 i686
Calling method in restrictive sandbox:
java.security.AccessControlException: access denied (java.io.FilePermission /etc/motd read)
Calling method with file read permission:
Linux thoth 2.6.24-19-generic #1 SMP Wed Jun 4 16:35:01 UTC 2008 i686

Touch of Scala

Since I developed my app in Scala, here's a little object I built for simplifying this process. It works similarly to the callInSandbox method above, but also wraps Scala functions so you can use closures pretty simply:

import java.security._
import java.net.URL

object Security {
  private val codeSource = new CodeSource( null: URL, null: Array[cert.Certificate] )

  def sandboxed[T]( perms: PermissionCollection )( f: => T): T = {
    val domain = new ProtectionDomain( codeSource, perms )
    val context = new AccessControlContext( Array( domain ) )

    AccessController.doPrivileged( new Action( f ), context )
  }

  private class Action[T]( f: => T ) extends PrivilegedAction[T] {
    override def run() = f
  }
}

In order to use this, construct your Permissions that you want to grant to the item, import Security.sandboxed, and just wrap like this:

sandboxed( perms ) {
  println( "This code is running within the sandbox!" )
}

I hope this helps someone out!

Better classing through Scala

2008-03-16T06:25:00.000-07:00

Rather than the promised follow-up to my last post (it's coming, honest), I'm just going to update the way that I did the first section of that post. I realised earlier that rather than making my hastily-constructed RichIterable a class, I should have made it a trait. In this way, other classes can use its functionality explicitly if they know about it and, critically, can override the functionality in a way that makes more sense to them.

To do this and still be able to do the implicit cast thing I showed, we also need a concrete class to mix the trait in with. Since you can't (as far as I know) mix in traits after you've created an object, if we just create a wrapper class for Iterable objects:

class IterableWrapper[A]( underlying: Iterable[A] ) extends Iterable[A]
{
  override def elements = underlying.elements
}

Which means we can now express the implicit function from before as so, creating a new wrapper with our trait mixed in:

implicit def iterable2RichIterable[A]( iterable: Iterable[A]) = {
  new IterableWrapper( iterable ) with RichIterable[A]
}

This means that any other collection can just extend the RichIterable trait if they don't want to require the type conversion to use the new methods, or if they want to replace the functions with better implementations. Our actual RichIterable trait can be a subtrait of Iterable, too, meaning less indirection. It's wonderful I tells ya.

The extra curly brackets here do absolutely nothing; I just feel it's clearer to add them in if an expression like this one is too long to fit on a line.

Here's a feature I made earlier

2008-03-15T11:45:00.000-07:00

After being suitably wowed by Max Bolingbroke's implementing the Disposable pattern from C# in my current language-of-obsession Scala, I was looking at some of the other features of C#/.NET which are quite nice, and wondering how they could be implemented.

One set of stuff that is available in C# is the much-talked-about LINQ system, which lets you combine and query lists in a way fairly similar to SQL. They also do a bunch of clever things (like actually translating these queries into SQL, from what I can tell) but let's gloss over that for now.

In any case, there was a few of these operators don't exist in the Scala library, and since it's fairly easy to push new things onto objects in Scala, I decided to see how easily I could model them. For kicks.

Before I start proper I should mention two things:

I'm assuming programming knowledge, and probably some Scala experience here. That said, if you're used to this type of language you shouldn't have too much trouble reading the code examples, and hopefully you'll like the way the language works. If there's any questions, though, just comment. I don't imagine I'm a great writer.
There is probably better ways to do this stuff; I'm not hugely experienced with the language (I have at least thought of alternate ways to do some of the things). I'd be interested to hear any other approaches people have on this.

This is the first of two posts. In this one I'm going to set the stage, and do something simple. In the next one I'm going to continue down the same road, and so something a little more complicated.

"Adding" our own methods onto an existing class

UPDATE: This follow-up article shows a way of doing this that should make the code a bit more re-usable. The principle is pretty much the same, though.

This is just a bit of preparation for other things. Trust me when I say that we're going to want to put (or appear to put) extra methods onto objects with the Iterable trait. It's what all the cool kids are doing.

Following the naming scheme put forth by the Scala standard library, we can create a RichIterable class which will house the extra stuff we're going to be putting on the normal Iterable class, and make it a class which "wraps" an existing iterable object. With an implicit type conversion function which will wrap it up transparently, we can treat any Iterable as a RichIterable whereever the function is imported. This is done like so:

object RichIterable
{
  implicit def iterable2RichIterable[A]( iterable: Iterable[A]) = new RichIterable( iterable )
}

class RichIterable[A]( inner: Iterable[A] )
{ 
  // Extra functions go here...
}

Now all one needs to do is add the line import whatever.package.RichIterable._ and the power of these new methods will be theirs.

Pushing things into a map – `toMap`

This first one is the simplest of the methods I decided to implement; it is based on the ToDictionary method available in .NET. Basically the motivation is this:

I have a set of items with some distinguising property. I would like to put them into a Map object keyed on this property so I can look them, or maybe some property derived from them, up quickly.

Here's a more concrete example; we have our typical terrible-example-of-object-orientation class representing a car by its registration plate number and the name of its driver, which looks like so:

class Car( registration: String, driver: String )
{
  val getRegistration = registration
  val getDriver = driver
}

We have a big list of cars (a List[Car], if you will), but what we really want is a map from the registration plate number to the driver name, so we can do something like this:

val guiltyDriver = registrationMap( registrationOfCarSeenLeavingInAHurry )

We'd like to generate this map quickly and easily, and this is where the toMap function comes in; we want to be able to do something like:

val registrationMap = carList.toMap( _.getRegistration, _.getDriver )

This is fairly easy to achieve via a number of means, but I chose to build the inputs to the immutable Map's factory method. It takes any number of two-item tuples (doubles) and gives you a Map object back. So to construct those tuples from values in the list in this case we could do something like this:

a => ( a.getRegistration, a.getDriver )

Or more generally for a given function for the key, and one for the value:

a => ( keyFunc(a), valFunc(a) )

So to map an entire Iterable object (like our List) using these two functions, we could do this:

val tuples = inner.map( a => ( keyFunc(a), valFunc(a) ) ).toStream

The toStream at the end is just there because the fussy Map constructor wants a Seq. Presumably because it wants some guarantee that there's a finite number of elements. Wuss.

So to put this into a function we just need to take in the two functions, and push it out to the Map factory method:

def toMap[K,V]( keyFunc: (A) => (K), valFunc: (A) => (V) ): Map[K,V]  = {
  val tuples = inner.map( a => ( keyFunc(a), valFunc(a) ) ).toStream
    
  Map( tuples: _* )
}

This uses K for the type of the key, and V for the type of the value. The _* type parameter makes the Seq of tuples appear like a bunch of arguments, since this isn't automatic (this is similar to params in C# or the ... thing in Java).

This now works fine; so we can do something like this:

import RichIterable._
  
val cars = List(
    new Car( "SANTA1", "Santa Claus" ),
    new Car( "PANDA1", "P.C. Plod" ),
    new Car( "12345", "Count Count" ) )
      
val registrationMap = cars.toMap( _.getRegistration, _.getDriver )

println( registrationMap )
  
println( registrationMap("SANTA1") )

And we get the output:

Map(SANTA1 -> Santa Claus, PANDA1 -> P.C. Plod, 12345 -> Count Count)
Santa Claus

...which is of course ridiculous, since Santa rides in a sleigh rather than driving a car.

For convenience, we can also fulfil the situation where we just want to push our objects into a map keyed by the result of a function; that is, the same as the above, but the value should just be the input object. This is likely to be a fairly common use-case, so let's not make people type a lot to do it:

def toMap[K]( keyFunc: (A) => (K) ): Map[K, A] = toMap( keyFunc, a => a )

This is just calling the existing toMap function with a => a (the identity function, a fancy name for a function which does nothing at all) as valFunc.

In the next entry, I'm going to do something more substantial; an analogue to GROUP BY in SQL, which was the feature of LINQ I wanted the most. If you're too anxious to wait, this code is available here. It's a little untidy and uncommented at present, though (and doesn't currently contain a bunch of improvements I've since added).

Java gotcha o' the day

2008-02-18T12:17:00.000-08:00

Here's something in Java that I just couldn't predict what it was going to do:

try
{
    try
    {
        throw new RuntimeException("Lost?");
    }
    finally
    {
        throw new RuntimeException("Interloper");
    }   
}
catch (Exception e)
{
    System.out.println(e);
}

In hindsight I guess it's obvious that the exception thrown in the finally block would take precedence, but losing exceptions without having to explicitly deal with them seems quite the dangerous thing. It's just strange to say. That said, anyone throwing exceptions from a finally block should probably be strung up anyway.

Oh good lord

2007-12-17T14:20:00.002-08:00

April? That's when I last touched this? And to update on why I hadn't been updating? This is a disgrace. I'll have to start posting something soon. Honest, I'll make a better effort. I do post reasonably frequently to my Tumblog, which is the sort of bloggin' that suits me best - low content. I've been busy at work (since I had to get another job, which will hopefully stick a little longer), that's my excuse.

Update!

2007-04-22T10:04:00.000-07:00

Whoops, I kinda let this stagnate. In any case, since I last posted I was made redundant from my old job; nothing I did wrong, just not enough work coming into the company. Unfortunately this means I've had to abandon the projects mentioned in the last post, as they were all PDA-based, and I no longer have access to a PDA. Furthermore I've switch to using the recently-released (and excellent) Feisty Fawn (7.04) release of Ubuntu, which has the knock-on effect of meaning I no longer have access to the development tools for that environment. What I've been doing in place of that is doing some work porting Shaim so that it works with the open-source implementation of .NET, Mono. Although I've had success getting the back-end to work, I need to write an entirely new front end, as the current one is written in Microsoft's super-flashy next generation UI library WPF, which is unlikely to work with Mono any time soon. Most of my personal projects have been put on the back-burner while looking for a job, though, and although I've found one now, I'm away on holiday for a week before starting. Hopefully I'll be back into the swing of these systems before long, though.

Obligatory "first post" post. (first, obligatory)

2007-01-25T06:47:00.000-08:00

Well, this is theoretically going to replace the previous blog I had on zootm.co.uk, because really most of the time there's no real reason to host your own blog, and I realised that I could use my GMail/Google account to write on this one. So this is going to chronicle things I make and do, since I've decided that I need to spend my copious (exaggeration) free time doing something other than yelling at people online about things that don't matter. At the moment the only thing of note I've got going is that I'm attempting to shrink down open source .NET instant messenger Shaim so that it (well, not the fancy frontend, but the core) works on the .NET Compact Framework, and Mono. The project is hosted on the fancy-pants code hosting thing Google Code project hosting, and at the moment it's mostly just an experiment, we'll see how it goes. The project is hosted under the name shaim-cf if anyone wants to look at it, but all I've done is branch the original Shaim code for now. I've also written a little sketchpad for my PDA, and I might well write something else which takes the images off of it automagically and chucks them up here at some point. We'll see what happens on this blog anyway. It might interest people. Maybe.