Massive parallelism – F# in the cloud?

I’m still trying to learn a bit of F# and I thought of a quite nice experiment.
Since F# supports quotations (for you C# devs, think Linq Expressions on roids) wouldn’t it be possible to serialize such quotation and pass it to a webservice and execute that code there?

Imagine the following code for a fractal calculation:

for y = 0 to 100000 do
            let Fractal xx yy =
                .....fractal calculation....

            for x = 0 to 100000 do
                Fractal x y

If “CloudExec” passes the code quote for individual scanlines of the fractal to the cloud, we could get some massive parallelism.
It would be just like PLinq but instead of executing a delegate in multiple threads, we would execute blocks of code on multiple threads on multiple machines (please ignore how naïve the code sample above is).

The biggest problem as far as I can tell would be to pass a result set back to the client in some way (that is missing in the sample code).
Input data doesn’t seem to be a problem since values defined outside the quotes are represented as constants in the quote.

It would ofcourse madness to expose such services to the public since you could pass in any code you want, but maybe it would work in an isolated environment.

Have anyone done such thing already?


7 thoughts on “Massive parallelism – F# in the cloud?”

  1. Yes indeed, now you say new Thread { IsBackground = true }

    But soon you will tell new CloudThread { … :)

    The code wil automatically be proxied to multiple servers!

  2. We have started a project like that a few years back. It consists of a generic client that (after authentication) executes the .NET code that is sent over the wire and passes the result back to the caller. This can be done pretty safely because the code can be executed in a sandbox.

    The part of executing stuff remotely and bringing the result back to the server is solved. However we ran into massive problems with asynchronous communication which I think is needed for good performance, so we have just started our third rewrite of the damn thing!

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s