Better is better than worse

(Disclaimer: this .txt file is a work of fiction. No attempt has been made to run code examples, and the startup thing is a nightmare I had. Apologies to the notorious R.P.G for appropriating the snowclone pattern of his profound essays for what amounts to a reaction GIF in .txt format.)

1. INTROITUS

I'm one of two "technical cofounders" in a tiny "bootstrapped" startup. Come what may, we're launching whatever's finished in two months.

It's a stressful time. A third technical cofounder who had most of the ML technology had some kind of mental breakdown and just left, even giving back his share. Being the team's resident mathy person I inherited his responsibilites and messy Python code while also leaning in hurricane fashion a range of skills, up to setting up properly my own production servers. Like Rodney Dangerfield, I get no respect.

So what's 98 out of 100 dentists's recommendation for dealing with the scope creep and looming deadline? Why, I started to learn a new programming language -- and even began writing bits and pieces of important TODO functionality in it. That's what everyone does, right?

2. KYRIE

You've probably seen the xkcd comic where Lisp is compared to lightsabers and otherwise seen the general Shinto-like awe (and derision of this awe and derision of deriders etc) at Lisp like the thousand-year language that will see us through trouble and peace.

Yet no one uses Lisp, and whenever someone makes a stand about it, it quickly gets written when enetering production. Of course, network economies and switching costs being what they are, most of the time it's very complicated to stop using PHP and start writing node or whatever. Whomever is switching to lesser known languges (Ocaml, Erlang etc) is doing it for exclusive superpowers that are not to be found in Lisps.

But this dynamic is different for Hy. (It might be similar in Clojure with Java; I wouldn't know)

3. REX TREMENDAE

In essence, Hy is a language that compiles to the Python AST. Most of the time it actually has a readable idiomatic translation to Python. In this way it's more of an alternate syntax for Python than a completely different way of life with different everything. In the most obvious ways one can write modules in Hy and `import` them in Python code -- which means you can plug it in, in bite-sized increments, and integrate it with existing, functioning, real world-sized Python codebases.

Inversely, one can import any given Python module in Hy code. Since Python is a top 10 language and especially popular in scientific computing, it means you can start doing useful things right away. This *dissolves* the problem where Haskellers or whomever tell you "yes, you can do ML/DS work, there's a library". The tools you know and hate but still use are all there. Pandas and networkx, in Lisp, today!

(Is Hy a Lisp? If Python was metaprogrammable, Hy might have been written in Python as something close to core Python. But then, if Python was metaprogrammable it would almost have certainly been Hy. I'm not a Shinto priest, but it does feel that Hy is the highest possible form of Python, completely transcending petty ideas about what "Pythonic" means and going for the core jewels. Maybe the gods meant Python as a transitional syntax for Hy.)

4. TUBA MIRUM

That's a lot of philosophy and opinions, and I promised insights from a week going "live" with Hy -- switching from water hoses to something wireless invented by Tesla in the middle of putting out a fire. These are not necessarily important points; these are just what I've seen in my first week.

4.1. The ineffable lightness of inlining code

I was just saying Python was not metaprogrammable, but we create small functions all the time that are effectively extensions to the syntax, usually short-hand for commonly-repeated patterns that others may just copy-paste around. This is a good thing; I've seen it called "once and once only", and has the important effect that the repeated bit of syntax can be changed in one place swiftly.

In data science work, some ML thingajimajigs expect target features to be 2D arrays with one column rather than 1D vectors. You know this because they tell you to say `y.reshape(-1,1)`. This is for practical purposes part of the syntax in "data python".

But often in data science work, your values are kept in "data frames" and need to be extricated first using the method "values". However, arrays themselves don't have a "values" method so you can't say `y.values.reshape(-1,1)` with confidence. A "convenience function" comes in handy:

def prepare_target(y):
    if hasattr(y, "values")
        return target.values.reshape(-1,1)
    else:
        return target.reshape(-1,1)

The straightforward translation of this "convenience function" to Hy is:

(defn prepare-target [y]
    (if (hasattr y "values")
        (. target values reshape (, (-1 1)))
        (. target reshape (,(-1 1 )))))

(I haven't run any of these examples). But wait, do we need to call a function everytime we want to just prepare a target? That kind of thing has a performance penalty, and besides, before the dataframe detection came up, this was just a snippet of code we had memorized -- the usual developer doesn't write functions like

def straighten_target(y):
    return y.reshape(-1,1)

Lisps, including Hy, have the aditional concept of a macro. Macros are metaprogramming tools: one writes program-construction programs; when these are evaluated at compile-time, they explicitly change other code we had written. For example, one might write

(deftag | [y] `(. ~y reshape (, (-1 1))))

and then rewrite `prepare-target` as follows:

(defn prepare-target [y]
    (if (hasattr y "values")
        #|(. y target values )
        #|(. y target )))

We can also make the whole prepare-target "utility function" a macro so it gets automatically inlined -- the way we did by hand when all we meant was `#|y`:

(defmacro prepare-target [y]
    `(if (hasattr ~y "values" )
        #|(. target values )
        #|(. target reshape)))

Now we can continue to use "prepare-target" (or "prepare_target" if importing back on Python).

4.2. Naming and necessity

Often Python makes one write lots of intermediary variables if we want to have readable code at all

def diff_model_predictions(model1, model2, X,y):
    y1 = model1.fit_transform(X,y)
    y2 = model2.fit_transform(X,y)
return abs(y1-y2)

This is the direct equivalent in Hy:

(defn diff-model-predictions [model1 model2 X y]
    (setv y1 (. model1 (fit_transform X y)))
   ( setv y2 (. model2 (fit_transform X y)))
   (abs (- y1 y2)))

but why not say this instead?

(defn diff-model-predictions [model1 model2 X y]
( abs (-
    (. model1 (fit_transform X y))
    (. model2 (fit_transform X y)))))

This is obviously possible in Python, but to my ear less readable:

return abs(model1.fit_transform(X,y) - model2.fit_transform(X,y))

(and with less brackets, someone might note). But see how the parallelism between the expression is lost unless one goes out of one's way to say

return abs(model1.fit_transform(X,y)\ - model2.fit_transform(X,y))

Also check out this version

(deftag #f [model] `(. ~model fit_transform (X,y)))
(defn diff-model-predictions [model1 model2 X y]
    (abs (- #f model1
          #f model2)))

4.3. Braces Hy

So far it seems that I'm talking about saving characters -- at the price of an explosion of (round and [square brackets]).

But you can not only manage all of those braces but also make them work for you with one weird trick: a "rainbow brackets" extension for your go-to code editor. This will give you colorful visual indication of which are matching parentheses. I'd say this is useful in any language, but in Lisplikes such as Hy, rainbow bracketing effectively shows you the complete structure of your code.

Yeah, man, the brackets are not a drag, they actually diminish the quantity of things you have to juggle on your head while trying to solve problems. There was an article on HN today with a title like "Think in Math, Implement in Code". The comments were mostly to the effect that making up a math formalism rarely helps ordinary everyday problems. But also some pointed out, conversely, that "thinking in code", hands-on, is typically associated with highly-nested loops, non-MECE if-else clauses and worse.

My transitional experience between Python and Hy reinforces an idea that I already had from watching people transition from PHP to node.js: because one tends to sit at the keyboard to figure out problems rather than find an available whiteboard and go into deep thinking, one tends to conceptualize problems in the ways that the language makes most easy. I've known people in finance who were still wed to Matlab and used 2D matrices to represent binary trees.

A concrete example from just yesterday is as follows: I was extracting network information from some shapefiles, which is easy thanks to the vast universe of interoperable Python libraries. The problem was that names were lost in such a way that I was obtaining

{1: [1,2,4], 2: [3,4]} # etc.

when I wanted

{"bart": ["lisa", "homer", "ned"], "lisa": ["beethoven", "ned"]} # etc.

What I did have was the mapping {1: "bart"} etc in dict form. After poking the libraries some more, I came up with this in not more than ten minutes (this in the unbearable open plan office of the day job I haven't quit yet)

(defn cross-dict [coll cross]
    (setv keyed (dfor key coll
                    [(get cross key) (get coll key)] ))
    (dfor item keyed
        [ item (lfor val (get keyed item) (get cross val)) ]))

You kind of need to understand the Hy synax for list and dict comprehensions, but with your rainbow bracket coloring on, code like this grows by thinking in code like this. What, if I was on a dumber language, I might have been neck-deep in double loops already; indeed, much of the conceptual footwork to give us this functionality has been done by Python who came with (the Pythonic version of) list comprehensions (which yes, were in Haskell before). But look how messed up `cross_dict` gets in Python:

def cross_dict(coll, cross):
        let keyed = {cross[key]: coll[key] for key in coll}
        return {item:[cross[val] for val in keyed[item]] for item in keyed}

I have written nested list comprehensions in Python before -- with much thought and care and painfully aware that comprehensions are pretty much an embedded DSL that doesn't talk much to how Python is otherwise structured. But this is a dict comprehension of a list comprehension of a dict comprehension. I can't reason like this "by the seat of my pants". S-expressions on the other hand let me do it. And Hy already has everything Python in it to begin with, while giving me a non-messed up syntax.

I do admire the work done by the creators and developers of languages like Python and Ruby and so on, who try to make complex things easy by means of cute notation. But how often there's an impedance mismatch between problems at hand, and how often do we avoid thinking in ways that annoy Guido van Rossum? Hy is your ticket to freedom, baby.

5. AGNUS DEI

I've seen Hy dismissed as "yet another Lisp", but this is the exact adjoint of how it should be dismissed -- as an extension or plugin to Python, something like decorators but with good pedigree. But I should reiterate the fundamentals: Hy might be the ultimate form and the generalization of "Pythonicity" beyond petty details (that do represent the more abstract and fundamental meaning of "Pythonic" but get stuck at the level of using syntax well).

Yet as it stands now, Hy has evolved (and even significantly changed syntax-wise) a lot over a few years and seems to be converging on a final form. Regardless of my evaluation that the combination of Lispishness (macros and s-expressions) and Pythonicity is gunpowder, revolution-in-a-box, the language hasn't "taken off" yet. Moreover the Hy core team appears to be severely understaffed, documentation is sometimes patchy and a non-insane person might want to hold back on Hy because their code will need to be still running in two years and worse, other people will have to read it.

This is a travesty. A travesty. Python is already an extraordinary language enjoying extraordinary success (it's even replacing Scheme for SICP-type hardcore introductions to programming) and learned men like Norvig even claim it's a workable pseudo-Lisp. But a Lisp with all the virtues of Python exists and it's literally at arms reach.

RPG's narrative is that C took over Lisp because worse is better. Dynamic languages like Python in turn are taking over because hardware makes performance irrelevant for most use cases and because somewhat-better or less-worse is better than worse. But this is bull. Better is better. Hy underpromises and overdelivers: it's not just a workable alternative to Lisp, it *is* a Lisp and enjoys a huge initial leg up. It stands on the shoulders of a giant (Python) and balances another giant (Lisp) on top of this.

Better is better, guys. pip install hy.

---

This text file is licensed more or less like those CCC and L0pht l77t h4ckr txts of old. Also this pastebin setup is quite rinky; copy away and maybe please please host it somewhere better.