Whatever occurs to me at the time to discuss, on the general subjects of IT and programming.

Showing posts with label erlang. Show all posts
Showing posts with label erlang. Show all posts

Tuesday, March 25, 2008

Erlang is a Specialist Language

We've all heard a lot about the "coming multicore revolution" and about how all us simpleminded programmers better get our stuff together to "program for multicores." This in part is what has given Erlang a certain amount of buzz, and the appearance of the very nice book by Joe Armstrong, Programming Erlang, on the Practical Programmer imprint (the fine fellows who piloted the Ruby boat that O'Reilly missed, thus destroying O'Reilly's cool factor).

(Side note: if you want an excellent book on Erlang, you should pick that up. If you want a terrible book, you should also pick it up. In fact it's pretty much the only book in town, so whether you love it or hate it (I quite like it) it's your only choice. The second edition of Concurrent Programming in Erlang is long out of print and hard to get. The other "recent" book is in French.)

As is well-known, this has led to a fair number of people being attracted to some of Erlang's technology (strict message-passing concurrency, transparent distributed programming, very lightweight processes) and turned off by a "bad" first impression of the language. I'm not going to get into all the things that turn people off (except to say that I don't really mind if they are; as far as I'm concerned, the question isn't whether Erlang is right for you, but whether you're right for Erlang). But I want to address this point:

Erlang is a specialist language, and your task X falls outside of it, so it's not really a fair criticism.

This surprised me--the idea that somehow Erlang fit only a small set of problem domains and was a "domain-specific language" (for, say, telecom applications, I guess) for these areas. And it wasn't a a good "general purpose" language.

This view is sometimes supported by Erlang's biggest boosters:

9 ++++ interface to flash using flex 2. Solve the GUI problem once and
for all as follows

repeat after me: client = flash in the browser, server = erlang.
Intermediate protocol = flash AMF

The "GUI problem" Armstrong refers to here is the fact that Erlang's built-in graphic facility is based on the famously ugly Tk toolkit, and shares some of its limitations. To me this feeds into the idea that Erlang is at most a "server side" language and not the sort of thing you'd want to write a desktop application in.

To all this pigeon-holing, I say hogwash. Erlang is not "domain-specific" and it's a good choice of programming language for all kinds of applications. It's just that Erlang is so good at writing servers that it seems like its role in life.

We accept that Erlang is great for writing systems with the following characteristics:
  • "Soft" real-time performance
  • High concurrency
  • High availability
  • Network-oriented
To me, it's the rare application that doesn't have at least one of these as a desirable characteristic. (The meaning of "soft" real-time has been debated before: to me, it means the absence of performance "surprises", not necessarily the idea that everything happens immediately. I've had a lot of these with other languages: Ruby has seemed full of them; Java has its share--every request gets served in 100 ms except for sometimes, when the garbage collector hits and it's 2000).

Who writes any kind of application, even a script, and wants to make sure it performs inconsistently, does operations serially, crashes often and doesn't use any network protocols?

These are really all well-known virtues of the Erlang technology: the Erlang OS, the Erlang/OTP platform, and so forth. In theory they could be had using a entirely different language: look at Lisp-Flavoured Erlang or Termite Scheme.

But I want to apply a little love to the language itself. I don't accept that having a Prolog-derived syntax is a necessary negative. In fact, it makes a great deal of sense it control a program as a series of clause selections by pattern. It's quite "natural"--it's a lot more natural than nested conditional constructs and the kinds of control structures offered by other languages. Arithmetic and function application follows the straightforward algebraic convention. Except for Haskell, Erlang is probably the language that looks the most straightforwardly like math to me.

I don't really know of a common programming task for which I wouldn't prefer to write an Erlang program--in the areas where it shines it's better than just about anything else (except for exotica like array programming languages) and, otherwise, is just as good.

Back to Armstrong's GUI problem. It unfortunately misses the bigger point, and rather badly. The "coming multicore revolution" isn't about programming for multiprocessing on servers. Servers have had multiprocessing for gobbles of time. Every server application is concurrent. Servers have plenty to do even without doing funny things with lightweight processes. Heck, Erlang itself didn't have an SMP-enabled scheduler until just "recently" (in the last couple of years)--and this didn't really limit its ability to take advantage of multiprocessing servers.

The point of "the multicore revolution" is that no processor can just be made faster now--every processing application will require multiprocessing in order for it to scale. The generation of "newly concurrent" applications will be on things like desktop machines, entertainment consoles and small devices; not server applications, where concurrency is old hat (though it's true that essentially all server applications could benefit from Erlang's concurrency model).

In short, saying that Erlang's GUI should live in a web browser is busted: Erlang is (or should be) a great language to write a web browser in. Or a MMORPG client. Or a photo processing program. Erlang models tasks as a set of concurrent processes receiving messages and replying to them--what about that isn't an excellent mapping to the event-based programming of GUIs?

Friday, March 7, 2008

Erlang shell as Unix shell

I'm not the first to suggest it, though I hadn't read anything about the idea, but it seemed to me that in some circumstances, the Erlang shell could be a really useful general-purpose shell. At the expense of, well, not being designed as an interactive Unix shell, you gain some significant benefits:
  • You can ssh to your Erlang nodes or start remote shells on them via Erlang distribution. From a practical perspective this access method may be a lot more flexible and enable-able than actual ssh.
  • Importantly, you can do that in the same way for servers running Windows or Unix. If most of your "shell commands" are implemented in Erlang they can work the same way on both platforms.
  • Being a "real" programming language, Erlang is a lot better at handling structured data--proper quoting alone causes huge headaches in shell languages, and is very rarely got right.
In order to realize this last advantage, "shell utilities" need to be implemented in an Erlang-conscious way rather than as an attempt to make the closest possible syntactic mapping between Erlang and shell utilities.

That means that they should generally operate on and return Erlang terms. Consider the stock c:ls/0 and c:ls/1 that come with Erlang. They print output, they don't return values. If the Erlang equivalent of a pipeline is nested function invocation, this just isn't useful. So the first thing we need to do is replace c:ls/0,1 (this is part of the code at the erlang-command-shell google code project):

ls() ->
ls(".", []).

ls(Opt = [O|_]) when is_tuple(O); is_atom(O) ->
ls(".", Opt);
ls(Paths) ->
ls(Paths, []).

ls(Path, Opt) ->
case opt(long, Opt) of
true ->
gather_info(shortls(glob(Path), [], Opt), [], Opt);
_ ->
shortls(glob(Path), [], Opt)
end.

shortls([], Result, _) ->
lists:reverse(Result);
shortls(["."|Paths], Acc, Opt) ->
{ok, Files} = file:list_dir("."),
shortls(Paths, lists:foldl(fun(F, A) -> [F|A] end, Acc, Files), Opt);
shortls([P|Paths], Acc, Opt) ->
case filelib:is_dir(P) of
{long, true} ->
{ok, Files} = file:list_dir(P),
shortls(Paths, [{P, Files}|Acc], Opt);
_ ->
case filelib:is_file(P) of
true -> shortls(Paths, [P|Acc], Opt);
_ -> shortls(Paths, [{no_such_file_or_directory, P}|Acc], Opt)
end
end.

gather_info([], Acc, Opt) ->
lists:reverse(Acc);
gather_info([N = {no_such_file_or_directory, P}|Paths], Acc, Opt) ->
gather_info(Paths, [N|Acc], Opt);
gather_info([{Dir, Files}|Paths], Acc, Opt) ->
gather_info(Paths,
[{Dir, lists:map(fun(F) ->
case file:read_file_info(F) of
{ok, FI} -> {F, FI};
{error, Reason} ->
{cannot_stat, F, Reason}
end
end, Files)}|Acc], Opt);
gather_info([F|Paths], Acc, Opt) ->
gather_info(Paths,
[case file:read_file_info(F) of
{ok, FI} -> {F, FI};
{error, Reason} -> {cannot_stat, F, Reason}
end|Acc], Opt).

glob(Files) ->
glob(Files, []).

glob([], Acc) ->
Acc;
glob(File, Acc) when is_atom(File) ->
glob([atom_to_list(File)], Acc);
glob(File = [X|_], Acc) when not is_atom(X), not is_list(X) ->
glob([File], Acc);
glob([File|R], Acc) ->
List = case filelib:wildcard(File) of
[] -> [File];
X -> X
end,
glob(R, Acc ++ List).

This is just a first draft--as you can see, it only supports one option, long, the Erlang proplist equivalent of -l or --long. And instead of printing results, it returns them. Files are returned as strings. If you list a directory, you get a tuple with the name of the directory and all the files in it. You can pass shell-like wildcards. For ease of typing, you can pass atoms instead of lists: ls(user_default.erl) works just fine. If you use the long option, you'll get a tuple back with the filename and a file_info record.

Without this new ls/0,1,2, the proposed example in the orbitz-erlang blog post won't even work:

lists:map(fun(X) -> print([X]) end, ls())

The author suggests this as an equivalent to the bash pipeline:

for i in *; do echo $i; done

And says he just doesn't see Erlang working as a command shell. Well: the problem with fake examples is that it can be hard to see any value. I mean, in neither of these situations would you actually use either command, since they are both the equivalent of perfectly available single commands (ls and ls/0). Even still, it seems to me that there's potential in the Erlang expression. Let's try something not-unusual but a little more complicated--finding out how much space is being taken up by all the Erlang source code files.

In bash, we might do something like this (following the template of the above):
tot=0; for f in *.erl; do size=`ls -l $f| cut -f5 -d\ `; let tot=tot+size; done; echo $tot
Looks kind of ugly to me (it's also pretty slow). Now, there's a lot of ways to do that. This is maybe one of the better ones:
ls -l *.erl | awk '{ tot += $5; } END { print tot }'
That at least doesn't fork for every file. But in any case, it's actually pretty hard to come up with these bash solutions interactively--both of the above required a fair amount of trial and error (at the very least, you tend to have to look at the output a few times to find out which column you're interested in, and whether there's a header row or not).

How would we do this in our "Erlang Unix shell"? How about this:
lists:foldl(fun({F, S}, Tot) -> Tot + S#file_info.size end, 0, lsl("*.erl")).
Here we're using list functions to iterate over our list rather than a weird embedded programming language (awk) or weirder strings-as-data expressions. We can refer to the file size symbolically and our "pipeline" is around the same length as the bash ones. Oh, I should also mention that I have an "alias" of lsl that invokes ls with the long option.

There are two main disadvantages that I see. One is that the "pipeline" is backwards, or at least, inside-out. In the Unix shells, production ties into consumption in a left to right direction, whereas in Erlang it happens in an inside-out direction. It means you have to think about what you're doing at the "outer" or "final" layer before you think about what data it will operate on (you don't really have to--the Erlang shell has command-line editing, so you can build your command from the inside-out, too).

The second is that there is a bit more typing in the very common cases--due to statement termination, string quoting (though, as I said, you should usually be able to use atoms for filenames) and so forth. And the benefit won't be realized for everyone. But if you're an experienced Erlang programmer and you're interacting with the Erlang shell this way all the time, it feels pretty natural to start doing shell tasks with it.

One thing I haven't quite settled is the bread and butter of Unix shells--command invocation. A lot of commands are mere shell utilities, and should just be implemented in our Erlang command shell (wc, cut, cat, grep, etc.). But what about invoking an editor or the make utility and that sort of thing? How about job control?

I don't know final answers for this, though I have created a system/1,2 function for invoking commands and a bg/1 function for running jobs in the background (as you might expect, jobs/0 shows you your running jobs and you can kill them or get the output from them when they're finished.

One thing that does seem clear is that some structure should be imposed when operating with command output or file content. I think the standard should be that when getting output from a command or catting a file, the contents are returned as a list of lines (strings) with no line terminator. It's straightforward to see how these could be operated on by a grep/2 or list function. In general such functions should also accept options to not chomp/1 the line endings or to read in a binary.

The current source code for all this wackiness is at the erlang-command-shell project at Google Code.

Perl-like split for Erlang

I just know someone has already done this, and probably numerous times. But I was working on a user_default module to customize my Erlang shell, with the intention of making it a good interactive shell for file and system management (more on this in the future). And I came across a couple places where I really wanted the perl utility (split()) instead of the typical Unix shell one (cut).

In particular I think Perl's split is nice, and I wanted an equivalent in Erlang. Here it is (this is a snippet rather than a module because right now it's just in my user_default:

split(Pred, List) ->
split(Pred, List, -1).

%% our fun split is the inverse of lists:splitwith/2 because I feel it's
%% more obvious to write: split(fun(C) -> C =:= $. end, List), given the
%% other options. If you provide a character "predicate" or a list "predicate"
%% then you are *splitting on* (not retaining) the character/sublist
%% that satisfies the condition. -pd
split2(Pred, List) when is_function(Pred) ->
{First, Rest} = lists:splitwith(fun(C) -> not Pred(C) end, List),
case Rest of
[] -> [First, []];
[_|Last] -> [First, Last]
end;
%% Certain common cases given as special atoms
split2(Space, List) when Space == ' '; Space == space ->
split2({rx, "\\s+"}, List);
split2(Comma, List) when Comma == ','; Comma == comma ->
split2({rx, "\\s*,\\s*"}, List);
split2({RxSpec, RX}, List) when RxSpec == rx;
RxSpec == regex;
RxSpec == regexp ->
tuple_to_list(splitrx(RX, List));
split2(Pred, List) when is_list(Pred) ->
tuple_to_list(splitsub(Pred, List, []));
split2(Pred, List) ->
split2(fun(X) -> X =:= Pred end, List).

%% We don't use regexp:split because we only want two limbs at a time
%% for counting up
splitrx(RX, List) ->
case regexp:first_match(List, RX) of
{match, 1, 0} ->
throw({regex_error, zero_width_match_at_start_of_list});
{match, Start, Len} ->
{lists:sublist(List, Start - 1),
lists:nthtail(Start + Len - 1, List)};
nomatch ->
{List, []}
end.

%% Note--does not include substring in split, unlike lists:splitwith/2
splitsub(_, [], Acc) ->
{lists:reverse(Acc), []};
splitsub(Str, List = [H|T], Acc) ->
case lists:prefix(Str, List) of
true -> {lists:reverse(Acc), lists:nthtail(length(Str), List)};
_ -> splitsub(Str, T, [H|Acc])
end.

split(Pred, List, N) ->
split(Pred, List, N, []).

split(_, [], _, Chunks) ->
lists:reverse(Chunks);
split(_, Rest, 1, Chunks) ->
lists:reverse([Rest|Chunks]);
split(Pred, List, N, Chunks) ->
[First, Rest] = split2(Pred, List),
case First of
[] ->
split(Pred, Rest, N, Chunks);
_ ->
split(Pred, Rest, N - 1, [First|Chunks])
end.
The split/2 and split/3 functions work a lot like Perl split() (except there is no split/1 and no default variable of course).

These functions split a list (usually but not necessarily a string--they're more akin to lists:splitwith/2 than string:tokens/2) on the given separator (which in the code above, I've confusingly called Pred). The three-argument version also has a limit--the maximum number of sublists to generate. The functions return a list of lists.

The separator can be given in one of the following ways:

  • An integer, such as $,
  • A string (list)--the list will be split at the sublist.
  • A fun, the list will be split on a single element for which the fun returns true. Note: This is the opposite sense from lists:splitwith/2.
  • A tuple of the form {rx, RegularExpression}--the string will be split on the substrings which match the regular expression. Care must be taken to avoid using a regular expression like " *" which makes a zero-width match at the beginning of the string; the split functions will throw an exception.
  • Certain "magical" atoms:
    • ' ' or space - Equivalent to {rx, " +"}
    • ',' or comma - Equivalent to {rx, "\\s*,\\s*"}

One of the useful features of these functions is that, unlike lists:splitwith/2, they do not include the separator sublist in the resulting lists.

I also think you could come up with a really nice set of commonly-used split patterns that you can refer to by atom.

Another thing that would be nice (as usual) is to also be operating on binary strings, but that runs into encoding issues, of course.