Whatever occurs to me at the time to discuss, on the general subjects of IT and programming.

Friday, March 7, 2008

Erlang shell as Unix shell

I'm not the first to suggest it, though I hadn't read anything about the idea, but it seemed to me that in some circumstances, the Erlang shell could be a really useful general-purpose shell. At the expense of, well, not being designed as an interactive Unix shell, you gain some significant benefits:
  • You can ssh to your Erlang nodes or start remote shells on them via Erlang distribution. From a practical perspective this access method may be a lot more flexible and enable-able than actual ssh.
  • Importantly, you can do that in the same way for servers running Windows or Unix. If most of your "shell commands" are implemented in Erlang they can work the same way on both platforms.
  • Being a "real" programming language, Erlang is a lot better at handling structured data--proper quoting alone causes huge headaches in shell languages, and is very rarely got right.
In order to realize this last advantage, "shell utilities" need to be implemented in an Erlang-conscious way rather than as an attempt to make the closest possible syntactic mapping between Erlang and shell utilities.

That means that they should generally operate on and return Erlang terms. Consider the stock c:ls/0 and c:ls/1 that come with Erlang. They print output, they don't return values. If the Erlang equivalent of a pipeline is nested function invocation, this just isn't useful. So the first thing we need to do is replace c:ls/0,1 (this is part of the code at the erlang-command-shell google code project):

ls() ->
ls(".", []).

ls(Opt = [O|_]) when is_tuple(O); is_atom(O) ->
ls(".", Opt);
ls(Paths) ->
ls(Paths, []).

ls(Path, Opt) ->
case opt(long, Opt) of
true ->
gather_info(shortls(glob(Path), [], Opt), [], Opt);
_ ->
shortls(glob(Path), [], Opt)
end.

shortls([], Result, _) ->
lists:reverse(Result);
shortls(["."|Paths], Acc, Opt) ->
{ok, Files} = file:list_dir("."),
shortls(Paths, lists:foldl(fun(F, A) -> [F|A] end, Acc, Files), Opt);
shortls([P|Paths], Acc, Opt) ->
case filelib:is_dir(P) of
{long, true} ->
{ok, Files} = file:list_dir(P),
shortls(Paths, [{P, Files}|Acc], Opt);
_ ->
case filelib:is_file(P) of
true -> shortls(Paths, [P|Acc], Opt);
_ -> shortls(Paths, [{no_such_file_or_directory, P}|Acc], Opt)
end
end.

gather_info([], Acc, Opt) ->
lists:reverse(Acc);
gather_info([N = {no_such_file_or_directory, P}|Paths], Acc, Opt) ->
gather_info(Paths, [N|Acc], Opt);
gather_info([{Dir, Files}|Paths], Acc, Opt) ->
gather_info(Paths,
[{Dir, lists:map(fun(F) ->
case file:read_file_info(F) of
{ok, FI} -> {F, FI};
{error, Reason} ->
{cannot_stat, F, Reason}
end
end, Files)}|Acc], Opt);
gather_info([F|Paths], Acc, Opt) ->
gather_info(Paths,
[case file:read_file_info(F) of
{ok, FI} -> {F, FI};
{error, Reason} -> {cannot_stat, F, Reason}
end|Acc], Opt).

glob(Files) ->
glob(Files, []).

glob([], Acc) ->
Acc;
glob(File, Acc) when is_atom(File) ->
glob([atom_to_list(File)], Acc);
glob(File = [X|_], Acc) when not is_atom(X), not is_list(X) ->
glob([File], Acc);
glob([File|R], Acc) ->
List = case filelib:wildcard(File) of
[] -> [File];
X -> X
end,
glob(R, Acc ++ List).

This is just a first draft--as you can see, it only supports one option, long, the Erlang proplist equivalent of -l or --long. And instead of printing results, it returns them. Files are returned as strings. If you list a directory, you get a tuple with the name of the directory and all the files in it. You can pass shell-like wildcards. For ease of typing, you can pass atoms instead of lists: ls(user_default.erl) works just fine. If you use the long option, you'll get a tuple back with the filename and a file_info record.

Without this new ls/0,1,2, the proposed example in the orbitz-erlang blog post won't even work:

lists:map(fun(X) -> print([X]) end, ls())

The author suggests this as an equivalent to the bash pipeline:

for i in *; do echo $i; done

And says he just doesn't see Erlang working as a command shell. Well: the problem with fake examples is that it can be hard to see any value. I mean, in neither of these situations would you actually use either command, since they are both the equivalent of perfectly available single commands (ls and ls/0). Even still, it seems to me that there's potential in the Erlang expression. Let's try something not-unusual but a little more complicated--finding out how much space is being taken up by all the Erlang source code files.

In bash, we might do something like this (following the template of the above):
tot=0; for f in *.erl; do size=`ls -l $f| cut -f5 -d\ `; let tot=tot+size; done; echo $tot
Looks kind of ugly to me (it's also pretty slow). Now, there's a lot of ways to do that. This is maybe one of the better ones:
ls -l *.erl | awk '{ tot += $5; } END { print tot }'
That at least doesn't fork for every file. But in any case, it's actually pretty hard to come up with these bash solutions interactively--both of the above required a fair amount of trial and error (at the very least, you tend to have to look at the output a few times to find out which column you're interested in, and whether there's a header row or not).

How would we do this in our "Erlang Unix shell"? How about this:
lists:foldl(fun({F, S}, Tot) -> Tot + S#file_info.size end, 0, lsl("*.erl")).
Here we're using list functions to iterate over our list rather than a weird embedded programming language (awk) or weirder strings-as-data expressions. We can refer to the file size symbolically and our "pipeline" is around the same length as the bash ones. Oh, I should also mention that I have an "alias" of lsl that invokes ls with the long option.

There are two main disadvantages that I see. One is that the "pipeline" is backwards, or at least, inside-out. In the Unix shells, production ties into consumption in a left to right direction, whereas in Erlang it happens in an inside-out direction. It means you have to think about what you're doing at the "outer" or "final" layer before you think about what data it will operate on (you don't really have to--the Erlang shell has command-line editing, so you can build your command from the inside-out, too).

The second is that there is a bit more typing in the very common cases--due to statement termination, string quoting (though, as I said, you should usually be able to use atoms for filenames) and so forth. And the benefit won't be realized for everyone. But if you're an experienced Erlang programmer and you're interacting with the Erlang shell this way all the time, it feels pretty natural to start doing shell tasks with it.

One thing I haven't quite settled is the bread and butter of Unix shells--command invocation. A lot of commands are mere shell utilities, and should just be implemented in our Erlang command shell (wc, cut, cat, grep, etc.). But what about invoking an editor or the make utility and that sort of thing? How about job control?

I don't know final answers for this, though I have created a system/1,2 function for invoking commands and a bg/1 function for running jobs in the background (as you might expect, jobs/0 shows you your running jobs and you can kill them or get the output from them when they're finished.

One thing that does seem clear is that some structure should be imposed when operating with command output or file content. I think the standard should be that when getting output from a command or catting a file, the contents are returned as a list of lines (strings) with no line terminator. It's straightforward to see how these could be operated on by a grep/2 or list function. In general such functions should also accept options to not chomp/1 the line endings or to read in a binary.

The current source code for all this wackiness is at the erlang-command-shell project at Google Code.

No comments: