Programming and IT Nuggets

Whatever occurs to me at the time to discuss, on the general subjects of IT and programming.

Tuesday, November 24, 2009

WYSIWYG is undesirable for collaborative editing

WYSIWYG editors are great, aren't they? They can be, especially when dealing with non-strictly-text applications, but they are the wrong tool for collaborative editing, like that done with most wikis. By encouraging editors to focus on making things look right, they are actually less likely to be making content that is right.


It Looks Right


If What You See Is What You Get, then users naturally focus on making what they see correct. The problem is that what you see isn't what's actually there. There is a structure to documents1 that must be correct if you are to have a hope of maintaining them.


This isn't so bad if only one person is going to be editing their documents. But it's a nightmare in a collaborative environment, as anyone who has had to edit a lot of other people's Microsoft Word documents will attest.


A Small Example


Let's say you like your second-level headings (a subsection heading, like the one at the beginning of this essay) to be large and blue. If you are using MediaWiki, you would write that like this:



==And Now For Something Completely Different==

Anything else results in the wrong result: something that looks wrong. The easiest way to get something that looks right in this instance is to make something that is right.


On the other hand, in the WYSIWYG world, the easiest way to make your heading look like the other headings is to make the text large and blue, usually by pressing a text size enlarger and text color picker buttons. Which, behind the scenes, results in something like this:2



<span style="color: blue; font-size: large">And Now For Something Completely Different</span>

In this world, the easiest way to do something is actually the wrong way to do it. To do it right, your knowledge actually has to be more subtle and arcane than in the plain text world: you have to appreciate a hidden document structure (that is, you have to be conscious of the semantic structure of the document behind the rendered version you're actually viewing--yet you can't see that structure), know the arcane name for a second-level heading (Maybe it's H2--intuitive, but only for HTML users. Maybe it's "subhead" or "subsubhead" if "head" is reserved for a document title), and select it from a dropdown box of styles. You have to know that what you're applying is a style, and pick the right one. This is surely no easier, and may be harder, than the equivalent wikimarkup; it's certainly harder than altering the text appearance ad-hoc.


One might reasonably ask, "What's wrong with that?" And the answer is that it's harder to maintain in various ways, which is fine if you're the only one editing and you want to deal with it. But it's not fine in a collaborative environment.


Specifically:



  • The document structure can no longer be interpreted by the software, so the section, for example, will be missing from a generated table of contents.

  • A similar objection obtains that you can't link to the heading directly, because there is no anchor.

  • If you ever decide a different style of heading is desirable (for example, you're now generating a PDF of your document or just changing the look) you'll have most of your headings in your new look (maroon small caps, for example) but not the several "wrong ones." You've added a great deal of overhead to any such change.


What You Don't See Can Be Important


There's another problem with WYSIWYG editors in this context. Collaborative editing is much more demanding from the perspective of change-tracking and dynamic content. It's important to be able to factor sections of a document into separate chunks, for example, or be able to include or refer to parts of a document dynamically elsewhere.


WYSIWYG-edited documents are full of spurious reformatting and extra whitespace--this happens because users do not see these things. Yet they interfere with tracking changes in the line-oriented text file that underlies the document. An opaque proprietary document format is even worse; the contortions that Microsoft Word, for example, goes to to display changes make things pretty ugly and hard to understand, and don't have nearly the same quality of information as a list of versions and standard diffs (it's more like "cvs annotate"; imagine if that were your only view of document changes?).


Worse still, WYSIWYG editing completely breaks down conceptually given the dynamic nature of most wiki documents. What is the WYSIWYG representation of a TWiki %SEARCH% result or a MediaWiki parserfunctions expression? Or even a template inclusion? Even something as simple as symbolically referring to another section is much harder in a WYSIWYG context. Again, it's much easier to do the wrong thing (link directly to a hardcoded URL, for example), than the right thing.


Adequate Is the Enemy of Good


For a variety of reasons (some of which I've mentioned) well-maintained documents factor out common markup and replace it with symbolic intent. The use of WYSIWYG discourages this. This is related to both phenomena above--making things look right instead of things that are right; and hiding important information. When I create a tag called %BUTTON%, and use it to indicate the names of buttons in my program documentation, I'm capturing an important datum of intent. If on the other hand, I just note that buttons look like [this], then it's up to the good behavior of individual editors to get it right; I've lost the actual intent around the button decoration, and I've lost an opportunity to factor out common formatting into something consistent.


WYSIWYG Editors Make Easy Things Easy


In any written introduction to a plain text markup system, there are a few very easy things covered first: how to make bold text, italic text, change text color and so forth. These also correspond to the buttons that appear on almost every WYSIWYG editor.


I do believe that it's (marginally) easier to "learn" how to make bold, italic or underlined text using a WYSIWYG editor. But it's also pretty easy to learn how to do these few tasks using plain text markup. And certainly no harder to use once learned (it's certainly easier for me to type *bold* than take my hands off the keyboard to click a button before and after typing a short word).


The question is, how valuable is it to make these low-hanging fruit hang incrementally lower on the tree? At the expense of making the slightly more complicated tasks (links, document structure, bulleted lists, tables) actually harder to do correctly? And difficult tasks (dynamic content) essentially impossible? I see little to no value, except perhaps in socialization (a WYSIWYG screen looks friendly and appears easier to the newbie than a blank edit box).


I do think some things could be friendlier and easier than a naked <textarea/> on a web form. Forum-style "helper" editors, that insert plain text markup in response to button pressing would be a good enhancement.


These Issues Are Not Individually Very Important


Any given one of these maintainability and correctness issues is not terribly important. In Perl Best Practices,3 Damian Conway warns against discounting maintainability issues because they're marginal. Collectively, all these tiny issues contribute to sabotage the correctness and maintainability of documents, which is poison to effective document management in a collaborative setting.


On the other hand--do not confuse what I'm saying. I'm not saying that usability is less important than correctness. I'm saying that a WYSIWYG editor makes it harder and is not more usable than being exposed to wikitext markup. It is a humane consideration for editors, assuming they want to produce maintainable documents that other people can edit.



1 This structure is not a technical detail: I don't mean whether the document is HTML, or wiki markup, or anything like that. Conceptually and philosophically, the document has a structure. Some text has membership in sections, which in turn are part of higher sections. Some runs of text are the names of ships, and appear differently. And so forth.


2 And I'm being pretty generous in assuming what it actually generates here.


3 Conway, Damian. Perl Best Practices. 2005. O'Reilly and Associates, Sebastopol, California.


Tuesday, March 25, 2008

Erlang is a Specialist Language

We've all heard a lot about the "coming multicore revolution" and about how all us simpleminded programmers better get our stuff together to "program for multicores." This in part is what has given Erlang a certain amount of buzz, and the appearance of the very nice book by Joe Armstrong, Programming Erlang, on the Practical Programmer imprint (the fine fellows who piloted the Ruby boat that O'Reilly missed, thus destroying O'Reilly's cool factor).

(Side note: if you want an excellent book on Erlang, you should pick that up. If you want a terrible book, you should also pick it up. In fact it's pretty much the only book in town, so whether you love it or hate it (I quite like it) it's your only choice. The second edition of Concurrent Programming in Erlang is long out of print and hard to get. The other "recent" book is in French.)

As is well-known, this has led to a fair number of people being attracted to some of Erlang's technology (strict message-passing concurrency, transparent distributed programming, very lightweight processes) and turned off by a "bad" first impression of the language. I'm not going to get into all the things that turn people off (except to say that I don't really mind if they are; as far as I'm concerned, the question isn't whether Erlang is right for you, but whether you're right for Erlang). But I want to address this point:

Erlang is a specialist language, and your task X falls outside of it, so it's not really a fair criticism.

This surprised me--the idea that somehow Erlang fit only a small set of problem domains and was a "domain-specific language" (for, say, telecom applications, I guess) for these areas. And it wasn't a a good "general purpose" language.

This view is sometimes supported by Erlang's biggest boosters:

9 ++++ interface to flash using flex 2. Solve the GUI problem once and
for all as follows

repeat after me: client = flash in the browser, server = erlang.
Intermediate protocol = flash AMF

The "GUI problem" Armstrong refers to here is the fact that Erlang's built-in graphic facility is based on the famously ugly Tk toolkit, and shares some of its limitations. To me this feeds into the idea that Erlang is at most a "server side" language and not the sort of thing you'd want to write a desktop application in.

To all this pigeon-holing, I say hogwash. Erlang is not "domain-specific" and it's a good choice of programming language for all kinds of applications. It's just that Erlang is so good at writing servers that it seems like its role in life.

We accept that Erlang is great for writing systems with the following characteristics:
  • "Soft" real-time performance
  • High concurrency
  • High availability
  • Network-oriented
To me, it's the rare application that doesn't have at least one of these as a desirable characteristic. (The meaning of "soft" real-time has been debated before: to me, it means the absence of performance "surprises", not necessarily the idea that everything happens immediately. I've had a lot of these with other languages: Ruby has seemed full of them; Java has its share--every request gets served in 100 ms except for sometimes, when the garbage collector hits and it's 2000).

Who writes any kind of application, even a script, and wants to make sure it performs inconsistently, does operations serially, crashes often and doesn't use any network protocols?

These are really all well-known virtues of the Erlang technology: the Erlang OS, the Erlang/OTP platform, and so forth. In theory they could be had using a entirely different language: look at Lisp-Flavoured Erlang or Termite Scheme.

But I want to apply a little love to the language itself. I don't accept that having a Prolog-derived syntax is a necessary negative. In fact, it makes a great deal of sense it control a program as a series of clause selections by pattern. It's quite "natural"--it's a lot more natural than nested conditional constructs and the kinds of control structures offered by other languages. Arithmetic and function application follows the straightforward algebraic convention. Except for Haskell, Erlang is probably the language that looks the most straightforwardly like math to me.

I don't really know of a common programming task for which I wouldn't prefer to write an Erlang program--in the areas where it shines it's better than just about anything else (except for exotica like array programming languages) and, otherwise, is just as good.

Back to Armstrong's GUI problem. It unfortunately misses the bigger point, and rather badly. The "coming multicore revolution" isn't about programming for multiprocessing on servers. Servers have had multiprocessing for gobbles of time. Every server application is concurrent. Servers have plenty to do even without doing funny things with lightweight processes. Heck, Erlang itself didn't have an SMP-enabled scheduler until just "recently" (in the last couple of years)--and this didn't really limit its ability to take advantage of multiprocessing servers.

The point of "the multicore revolution" is that no processor can just be made faster now--every processing application will require multiprocessing in order for it to scale. The generation of "newly concurrent" applications will be on things like desktop machines, entertainment consoles and small devices; not server applications, where concurrency is old hat (though it's true that essentially all server applications could benefit from Erlang's concurrency model).

In short, saying that Erlang's GUI should live in a web browser is busted: Erlang is (or should be) a great language to write a web browser in. Or a MMORPG client. Or a photo processing program. Erlang models tasks as a set of concurrent processes receiving messages and replying to them--what about that isn't an excellent mapping to the event-based programming of GUIs?

Terseness in Erlang

Paul Graham's release of Arc got me thinking about terseness as an end in itself. To me, terseness as a desirable quality is a measure of the signal to noise ratio of source code. How much of the source code represents the programmer's actual intent in addressing a problem; and how much is structure, scaffolding or other artifacts imposed by the language that do not express that intent?

I'll probably have more to say about this general subject. Now I'm going to change the subject to Damien Katz's "famous" What Sucks About Erlang blog post. Or at least, to his particular point about how much single assignment sucks because instead of doing this:
   f(X) ->
X = foo(X),
X = bar(X),
baz(X).
He has to do this:
   f(X) ->
X1 = foo(X),
X2 = bar(X1),
baz(X2).
Which makes it "hard" to add a function in the middle, because you have to renumber all those Xn variables to new variable names.

Well, as I've said before, the problem with toy examples is that it's easy to miss the point. Katz was so close to finding a real criticism here, but missed it: because X1, X2, X3... in series like this is icky programming, it's true. It's also the sort of thing people do when they're new to Erlang and they're trying to write C or Ruby code in it.

Learning Erlang consists of two epiphanies and an increasing zen. The first epiphany (if it's your first FP language) is about functional programming. At some point, you "get" functional programming and things seem cool, exciting and straightforward that used to seem abstruse and weird.

The second epiphany is OTP. It's hard to explain why the combination of modules and design principles in OTP is so good without going down the path of writing a big application yourself. But once you've gone down those ratholes and started implementing your own ad-hoc server state management, request protocols and restart schemes, then you "get" OTP and you have that "aha!" moment about why supervisors work the way they do and why there are trees of them and what transient children are for and that kind of thing.

The increasing zen of Erlang, though, is the kind of insight that gradually makes things like serially numbered variables all but disappear from your code; that makes you intuitively put cases at just the right spot so you avoid having some branches with recursive calls and some without, or makes you realize why sometimes the best way to return a certain value is to call yourself again.

There is one pernicious pattern, though, that does require ugly serial binding, and it's the one that goes like this:
   f(X) ->
{value, {X, X1}} = lists:keysearch(X, 1, ?valuetab),
{atomic, X2} = bar(X1),
{ok, X3} = baz(X2),
X3.
This is not just some dumb coding, and it can't be as easily fixed as Katz's toy example. Katz can just do this:
   f(X) ->
baz(
bar(
foo(X))).
But the pattern-matched examples can't be so easily fixed.

I don't think this is a "bad" characteristic of Erlang. What it reflects is that two powerful features--function composition, and pattern matching--are in tension. I don't think there's a way to "fix" that. But I think there might be a way to make it look better, at least.

One way of doing this is to write wrappers over the functions. How many times have you written some thin wrapper over one of the key-lookup style functions? I've done it a lot (and I might have a bit more to say about that in a future post, as well).

Since Erlang has this nifty way of doing pattern-matching, functions will tend to return structures like that all the time--structures that are not the most convenient form for passing to the next function.

What we want is a way to specify the pattern inline, have it extract a value, and use that as the value of the whole pattern match. Well, maybe we don't want that. But I think I do:
   f(X) ->
{ok, ?_} = baz(
{atomic, ?_} = bar(
{value, {X, ?_}} = foo(X))).
I'm using ?_ in my example because it turns out that's one way of making it work, and my other ideas do not, without modifying the Erlang scanner. So what we want is to magically transform expressions like this:
   {ok, [?_|_]} = blah(X)
into this:
   begin {ok, [Something|_]} = blah(1), Something end
Luckily, Erlang has such magic, a parse_transform.

Here's a wrinkle. You've probably noticed that "Something" is not a very good variable name for us to be inserting wherever we want to use this feature. We could to this "gensym()" in a number of ways--if we do it at parse_transform time, we could insert it where we find our marker. We can even scan the parse tree to look for conflicts.

Unfortunately, we still need a marker of some kind--something that's not going to conflict with variables in the source. If we use a special atom or tuple, we run into the possibility of such a legitimate conflict. And honestly, if our "magic" item is anything more than one character (or, okay, two characters I guess), it defeats the point of the exercise. Nicely, however, we can use our ?_ macro to expand to whatever we want. Now, unfortunately, _ is the only punctuation character that can be used as a macro, and I could see all kinds of client code conflicting with it. But as long as what it represents can't conflict, that client code could redefine the marker to something else (?r for return?).

Luckily, we can name a variable something that we can (just about) guarantee will never conflict with client code--at least, to the extent that random nonces and GUIDs "never" conflict, which is frankly good enough. We can just use a variable with 128 bits of random goodness and define our _ macro to that. Then the parse transform can look for a match operation in the code tree that has that magic variable on the left-hand side. And when it does, it encloses it in the block and adds the variable expression to the end to make the block evaluate it.

This also makes our code terser. So yes, it brings us back to that subject. I went looking for some other examples of repetition, and found a few. One consists of the clauses in function definitions:
   mydo({foo, Foo}) -> Foo;
mydo({bar, Bar}) -> Bar;
mydo({baz, Baz}) -> Baz.
Couldn't this be written:

   mydo({foo, Foo}) -> Foo;
({bar, Bar}) -> Bar;
({baz, Baz}) -> Baz.
I would like to be able to do without them: funs do. But since the result does not produce an Erlang form, I can't effect that with a parse transform.

Another common source of repetition is in self-recursion. This is especially obvious when the leaves of a case expression involve self-recursive calls:
   find_by_traversing_lookups(K, Keylist, Levels) ->
case {value, {K, ?_}} = lists:keysearch(K, 1, Keylist) of
1 -> Levels + 1;
Next -> find_by_traversing_lookups(Next, Keylist, Levels + 1)
end.
If you do that a lot (and you do), that's a lot of unnecessary text to scan. Since the only thing we're interested in in a self-recursive call is the argument list, you can get rid of the visual clutter by replacing self-recursive calls with _:
   find_by_traversing_lookups(K, Keylist, Levels) ->
case {value, {K, ?_}} = lists:keysearch(K, 1, Keylist) of
1 -> Levels + 1;
Next -> _(Next, Keylist, Levels + 1)
end.
This actually can't conflict with other function calls because _ is never bound to a variable and therefore can't be used to dispatch a call. Of course, another parse transform could use this--it's handy "syntax" because the form scans fine, unlike other characters (using <- came to mind but again, I would have had to modify the scanner).

The code for these experiments is at google code. They're not much now, but I am already finding the pattern-matched returns handy.

Friday, March 7, 2008

Erlang shell as Unix shell

I'm not the first to suggest it, though I hadn't read anything about the idea, but it seemed to me that in some circumstances, the Erlang shell could be a really useful general-purpose shell. At the expense of, well, not being designed as an interactive Unix shell, you gain some significant benefits:
  • You can ssh to your Erlang nodes or start remote shells on them via Erlang distribution. From a practical perspective this access method may be a lot more flexible and enable-able than actual ssh.
  • Importantly, you can do that in the same way for servers running Windows or Unix. If most of your "shell commands" are implemented in Erlang they can work the same way on both platforms.
  • Being a "real" programming language, Erlang is a lot better at handling structured data--proper quoting alone causes huge headaches in shell languages, and is very rarely got right.
In order to realize this last advantage, "shell utilities" need to be implemented in an Erlang-conscious way rather than as an attempt to make the closest possible syntactic mapping between Erlang and shell utilities.

That means that they should generally operate on and return Erlang terms. Consider the stock c:ls/0 and c:ls/1 that come with Erlang. They print output, they don't return values. If the Erlang equivalent of a pipeline is nested function invocation, this just isn't useful. So the first thing we need to do is replace c:ls/0,1 (this is part of the code at the erlang-command-shell google code project):

ls() ->
ls(".", []).

ls(Opt = [O|_]) when is_tuple(O); is_atom(O) ->
ls(".", Opt);
ls(Paths) ->
ls(Paths, []).

ls(Path, Opt) ->
case opt(long, Opt) of
true ->
gather_info(shortls(glob(Path), [], Opt), [], Opt);
_ ->
shortls(glob(Path), [], Opt)
end.

shortls([], Result, _) ->
lists:reverse(Result);
shortls(["."|Paths], Acc, Opt) ->
{ok, Files} = file:list_dir("."),
shortls(Paths, lists:foldl(fun(F, A) -> [F|A] end, Acc, Files), Opt);
shortls([P|Paths], Acc, Opt) ->
case filelib:is_dir(P) of
{long, true} ->
{ok, Files} = file:list_dir(P),
shortls(Paths, [{P, Files}|Acc], Opt);
_ ->
case filelib:is_file(P) of
true -> shortls(Paths, [P|Acc], Opt);
_ -> shortls(Paths, [{no_such_file_or_directory, P}|Acc], Opt)
end
end.

gather_info([], Acc, Opt) ->
lists:reverse(Acc);
gather_info([N = {no_such_file_or_directory, P}|Paths], Acc, Opt) ->
gather_info(Paths, [N|Acc], Opt);
gather_info([{Dir, Files}|Paths], Acc, Opt) ->
gather_info(Paths,
[{Dir, lists:map(fun(F) ->
case file:read_file_info(F) of
{ok, FI} -> {F, FI};
{error, Reason} ->
{cannot_stat, F, Reason}
end
end, Files)}|Acc], Opt);
gather_info([F|Paths], Acc, Opt) ->
gather_info(Paths,
[case file:read_file_info(F) of
{ok, FI} -> {F, FI};
{error, Reason} -> {cannot_stat, F, Reason}
end|Acc], Opt).

glob(Files) ->
glob(Files, []).

glob([], Acc) ->
Acc;
glob(File, Acc) when is_atom(File) ->
glob([atom_to_list(File)], Acc);
glob(File = [X|_], Acc) when not is_atom(X), not is_list(X) ->
glob([File], Acc);
glob([File|R], Acc) ->
List = case filelib:wildcard(File) of
[] -> [File];
X -> X
end,
glob(R, Acc ++ List).

This is just a first draft--as you can see, it only supports one option, long, the Erlang proplist equivalent of -l or --long. And instead of printing results, it returns them. Files are returned as strings. If you list a directory, you get a tuple with the name of the directory and all the files in it. You can pass shell-like wildcards. For ease of typing, you can pass atoms instead of lists: ls(user_default.erl) works just fine. If you use the long option, you'll get a tuple back with the filename and a file_info record.

Without this new ls/0,1,2, the proposed example in the orbitz-erlang blog post won't even work:

lists:map(fun(X) -> print([X]) end, ls())

The author suggests this as an equivalent to the bash pipeline:

for i in *; do echo $i; done

And says he just doesn't see Erlang working as a command shell. Well: the problem with fake examples is that it can be hard to see any value. I mean, in neither of these situations would you actually use either command, since they are both the equivalent of perfectly available single commands (ls and ls/0). Even still, it seems to me that there's potential in the Erlang expression. Let's try something not-unusual but a little more complicated--finding out how much space is being taken up by all the Erlang source code files.

In bash, we might do something like this (following the template of the above):
tot=0; for f in *.erl; do size=`ls -l $f| cut -f5 -d\ `; let tot=tot+size; done; echo $tot
Looks kind of ugly to me (it's also pretty slow). Now, there's a lot of ways to do that. This is maybe one of the better ones:
ls -l *.erl | awk '{ tot += $5; } END { print tot }'
That at least doesn't fork for every file. But in any case, it's actually pretty hard to come up with these bash solutions interactively--both of the above required a fair amount of trial and error (at the very least, you tend to have to look at the output a few times to find out which column you're interested in, and whether there's a header row or not).

How would we do this in our "Erlang Unix shell"? How about this:
lists:foldl(fun({F, S}, Tot) -> Tot + S#file_info.size end, 0, lsl("*.erl")).
Here we're using list functions to iterate over our list rather than a weird embedded programming language (awk) or weirder strings-as-data expressions. We can refer to the file size symbolically and our "pipeline" is around the same length as the bash ones. Oh, I should also mention that I have an "alias" of lsl that invokes ls with the long option.

There are two main disadvantages that I see. One is that the "pipeline" is backwards, or at least, inside-out. In the Unix shells, production ties into consumption in a left to right direction, whereas in Erlang it happens in an inside-out direction. It means you have to think about what you're doing at the "outer" or "final" layer before you think about what data it will operate on (you don't really have to--the Erlang shell has command-line editing, so you can build your command from the inside-out, too).

The second is that there is a bit more typing in the very common cases--due to statement termination, string quoting (though, as I said, you should usually be able to use atoms for filenames) and so forth. And the benefit won't be realized for everyone. But if you're an experienced Erlang programmer and you're interacting with the Erlang shell this way all the time, it feels pretty natural to start doing shell tasks with it.

One thing I haven't quite settled is the bread and butter of Unix shells--command invocation. A lot of commands are mere shell utilities, and should just be implemented in our Erlang command shell (wc, cut, cat, grep, etc.). But what about invoking an editor or the make utility and that sort of thing? How about job control?

I don't know final answers for this, though I have created a system/1,2 function for invoking commands and a bg/1 function for running jobs in the background (as you might expect, jobs/0 shows you your running jobs and you can kill them or get the output from them when they're finished.

One thing that does seem clear is that some structure should be imposed when operating with command output or file content. I think the standard should be that when getting output from a command or catting a file, the contents are returned as a list of lines (strings) with no line terminator. It's straightforward to see how these could be operated on by a grep/2 or list function. In general such functions should also accept options to not chomp/1 the line endings or to read in a binary.

The current source code for all this wackiness is at the erlang-command-shell project at Google Code.

Perl-like split for Erlang

I just know someone has already done this, and probably numerous times. But I was working on a user_default module to customize my Erlang shell, with the intention of making it a good interactive shell for file and system management (more on this in the future). And I came across a couple places where I really wanted the perl utility (split()) instead of the typical Unix shell one (cut).

In particular I think Perl's split is nice, and I wanted an equivalent in Erlang. Here it is (this is a snippet rather than a module because right now it's just in my user_default:

split(Pred, List) ->
split(Pred, List, -1).

%% our fun split is the inverse of lists:splitwith/2 because I feel it's
%% more obvious to write: split(fun(C) -> C =:= $. end, List), given the
%% other options. If you provide a character "predicate" or a list "predicate"
%% then you are *splitting on* (not retaining) the character/sublist
%% that satisfies the condition. -pd
split2(Pred, List) when is_function(Pred) ->
{First, Rest} = lists:splitwith(fun(C) -> not Pred(C) end, List),
case Rest of
[] -> [First, []];
[_|Last] -> [First, Last]
end;
%% Certain common cases given as special atoms
split2(Space, List) when Space == ' '; Space == space ->
split2({rx, "\\s+"}, List);
split2(Comma, List) when Comma == ','; Comma == comma ->
split2({rx, "\\s*,\\s*"}, List);
split2({RxSpec, RX}, List) when RxSpec == rx;
RxSpec == regex;
RxSpec == regexp ->
tuple_to_list(splitrx(RX, List));
split2(Pred, List) when is_list(Pred) ->
tuple_to_list(splitsub(Pred, List, []));
split2(Pred, List) ->
split2(fun(X) -> X =:= Pred end, List).

%% We don't use regexp:split because we only want two limbs at a time
%% for counting up
splitrx(RX, List) ->
case regexp:first_match(List, RX) of
{match, 1, 0} ->
throw({regex_error, zero_width_match_at_start_of_list});
{match, Start, Len} ->
{lists:sublist(List, Start - 1),
lists:nthtail(Start + Len - 1, List)};
nomatch ->
{List, []}
end.

%% Note--does not include substring in split, unlike lists:splitwith/2
splitsub(_, [], Acc) ->
{lists:reverse(Acc), []};
splitsub(Str, List = [H|T], Acc) ->
case lists:prefix(Str, List) of
true -> {lists:reverse(Acc), lists:nthtail(length(Str), List)};
_ -> splitsub(Str, T, [H|Acc])
end.

split(Pred, List, N) ->
split(Pred, List, N, []).

split(_, [], _, Chunks) ->
lists:reverse(Chunks);
split(_, Rest, 1, Chunks) ->
lists:reverse([Rest|Chunks]);
split(Pred, List, N, Chunks) ->
[First, Rest] = split2(Pred, List),
case First of
[] ->
split(Pred, Rest, N, Chunks);
_ ->
split(Pred, Rest, N - 1, [First|Chunks])
end.
The split/2 and split/3 functions work a lot like Perl split() (except there is no split/1 and no default variable of course).

These functions split a list (usually but not necessarily a string--they're more akin to lists:splitwith/2 than string:tokens/2) on the given separator (which in the code above, I've confusingly called Pred). The three-argument version also has a limit--the maximum number of sublists to generate. The functions return a list of lists.

The separator can be given in one of the following ways:

  • An integer, such as $,
  • A string (list)--the list will be split at the sublist.
  • A fun, the list will be split on a single element for which the fun returns true. Note: This is the opposite sense from lists:splitwith/2.
  • A tuple of the form {rx, RegularExpression}--the string will be split on the substrings which match the regular expression. Care must be taken to avoid using a regular expression like " *" which makes a zero-width match at the beginning of the string; the split functions will throw an exception.
  • Certain "magical" atoms:
    • ' ' or space - Equivalent to {rx, " +"}
    • ',' or comma - Equivalent to {rx, "\\s*,\\s*"}

One of the useful features of these functions is that, unlike lists:splitwith/2, they do not include the separator sublist in the resulting lists.

I also think you could come up with a really nice set of commonly-used split patterns that you can refer to by atom.

Another thing that would be nice (as usual) is to also be operating on binary strings, but that runs into encoding issues, of course.