Why metaprogramming instead of functions?

I'm a JavaScript programmer who is very intrigued by Lisp after hearing people like Eric Raymond claim that is often enlightening. Disclaimer: I'm sure there are various ways I'm being a noob and various things I'm just not understanding.

So far it seems to me like the big benefit to Lisp really comes down to the fact that it makes metaprogramming easier. So then, I'm trying to understand why metaprogramming is so useful. But as Matthew Butterick talks about, people don't really give concrete examples of why it is useful. It's more "you're going to have to learn some Lisp and see for yourself".

I'm a little skeptical of that though. I don't see why examples can't be provided. And so, I've been searching for them. I haven't come across much, unfortunately. For the examples I have come across, I always find myself saying "Can't this just be a function?"

This lif example from Reddit is probably the best and easiest to talk about. In short, instead of doing this:

(let ([x (get-x)])
  (if (valid? x) 
      (use-x x) 
      (error "no x available" x)))

You can write a macro that allows you to do this:

(lif [x (get-x)]
     (valid? x)
     (use-x x) 
     (error "no x available" x))

Here's my question: why a macro instead of a function? To me, lif looks like a function that takes get-x, valid?, use-x, error and "no x available" as inputs and gives you whatever as the output.

Meta programming comes in many different forms.
Meta programming via macros is just one Flavor.

Meta programming with macros means extending the programming language via source code transformations. A macro takes source code and outputs new source code - either at compile time or during runtime. Macros in Lisp are designed such that they can be executed during compilation.

Let's look at the source

If we have an expression like this in Lisp, then what are the parts?

(let ((x (get-x)))
  (if (valid-p x) 
      (use-x x) 
    (error "no x available" x)))

LET is a built-in special operator with its own evaluation rules. It is not a function.

IF is a built-in special operator with its own evaluation rules. It is not a function.

GET-X, VALID-P, USE-X and ERROR might be functions.

X is a variable, introduced with a LET binding.

Thus neither (LET ...) nor (IF ... ) are function calls.

Remember the rule to evaluate a function call in Lisp:

• evaluate all the argument expressions to their values

• find the function

• call the function with the values

• execute the function code

• return a result from the function call

The LET does not evaluate ((x (get-x))) . It treats it as a binding list. X is a variable. (GET-X) is a form. (x (get-x)) is a binding list. Evaluate the form (GET-X) and bind the new lexical variable X to the result.
The evaluate all the body expressions of the LET form in this lexical scope.

a new control structure

Now your LIF is also not a function, because it also uses different evaluation:

(lif (x (get-x))
       (valid? x)
     (use-x x) 
  (error "no x available" x))

(x (get x)) does not get evaluated.

(use-x x) and (error ...) are getting evaluated based on the evaluation result of the second sub-expression (valid-p ...).

Thus this can't be a function call.

Now, LIF is not built-in, but with a macro we can transform the LIF form into a valid Lisp form, the form we saw about with the IF and LET.
A form is an expression meant to be evaluated.

That lets you expand the syntax and the semantics of your programming language on a user/programmer level. The LIF macro takes code which might not be valid Lisp code and transforms it into something else, which eventually should be valid Lisp code.

Now you only need to ask yourself, what kind of transformations one can do with this. The answer actually is. that this is left to the imagination of the programmer. At first one wants to write macros which easily and simply fit into the language. But then we can also bend the rules&conventions.

An example in Common Lisp is the LOOP macro:

(LOOP FOR i FROM 0 BY 3
      REPEAT 10

      WHEN (evenp i)
       COLLECT i)

-> (0 6 12 18 24)

Its purpose is to a) be a sublanguage that is focused on iteration patterns and b) it should expand into efficient Lisp code (which would not be easy to be written manually and which would not be very readable). But what is strange (and controversial) about it, is that it introduces a lot of syntax into Lisp and the code does not look like the normal Lisp with a lot of parentheses. The LOOP form above expand into a page of lower-level code. The implementation of LOOP is complex (-> SBCL LOOP implementation, see how it includes even more macros implementations to implement the LOOP macro -> we can use macros in macros, too) and in effect it is a compiler written in Lisp, for an embedded language, with Lisp itself being the target language.

Thus ONE purpose of a macro is to implement embedded domain specific sublanguages, here a new language for iteration.

Here is why metaprogramming is useful. It is 1958 in a variant of our universe (so, in particular, none of this happened quite like this) and I am inventing a programming language based on λ-calculus. Functions in the paper version of this language look like

> λ x y z: ... something ...

But it's 1958 and parsers haven't really been invented, let alone unicode (I will pretend that lower-case has been invented). So I invent a simple syntax, together with an in-memory data representation and a reader and printer, which I call s-expressions and use this to represent my language. The above function is then represented as

(lambda (x y z) ... something ...)

Well, I can now read this into memory and evaluate it using a program I've written in machine code which walks this data representation. I have, in fact, invented Lisp.

Well, in the textual form of my original language function application might look like

> (λ x y z: ... something ...)(thing 1 thing 2 thing 3)

say, and I turn this into my s-expression representation as

((lambda (x y z) ...) 1 2 3)

say.

So now I start to write the first programs in this language, and one thing I discover is that local variables are useful. Well, I can do local variables:

((lambda (x)
   ... x is a local variable here ...)
 1)

But this is hard to read, especially if there is lots of code:

((lambda (x)
   ... 300 lines ...
   ...)
 1)

is just painful, because to see what the initial value of x is I have to look through 5 pages of printout. I want to be able to write something that makes it easier to read my program. I think, well, I will have to modify my evaluator function (which is in machine code) and this will be very painful, especially as I am not yet sure what the new syntax should be.

Then the light goes on: the source code of my program is a datastructure – an s-expression – and I can write a program to take an s-expression in the nice syntax and turn it into one in the hard-to-read one. In particular I come up with this function:

(lambda (form)
  (append
   (list
    (cons (quote lambda)
          (cons
           (mapcar (lambda (binding)
                     (cond
                      ((symbolp binding)
                       binding)
                      ((listp binding)
                       (car binding))))
                   (cadr form))
           (cddr form))))
   (mapcar (lambda (binding)
             (cond
              ((symbolp binding) (quote nil))
              ((listp binding) (cadr binding))))
           (cadr form))))

And if I name this function let-expander I can try it on an s-expression using a syntax which I've decided I want to use:

> (let-expander (quote (let ((x 1) (y 2))
                         (+ x y))))
((lambda (x y) (+ x y)) 1 2)

So now, I can write programs which have local variables written as

(let ((x 1) ...)
  ...)

feed them through my let-expander function and then just use my old evaluator to evaluate the resulting s-expression.

I have just invented a new language: a language which is like my old language but has this new let construct in it. It is much easier to read programs in this new language! (And let can't be a function: its first argument is a list of variables and values, not something to be evaluated: let is an extension to the language, not a function defined in it.)

But wait: I don't have to stop there, do I? Every time I decide I want a new feature in my language I can write one of these 'expander' functions and now I have that feature. I can also use the language extended by all the previous expanders to write new ones if I'm careful.

Pretty soon I write a sort of meta-expander which knows about a table of these expanders and walks over an s-expression dutifully calling the expanders until there is nothing left to expand. I add this as a front-end to my evaluator.

The end result of this is something completely extraordinary: I started with a primitive language which had lambda, cond, mapcar (which I can write in the language itself), and some primitives for dealing with and building these s-expression objects, and suddenly I have an infinite family of languages. Some of these have extra general-purpose constructs, like if and case and when and unless and so on, all of which I can just implement using expanders and which become more-or-less standard. Others have special extensions to deal with domain-specific things: the language used for the first Jupiter missions had a lot of code that looked like:

(with-ensured-spacecraft-state (:pyros 'armed :ullage-motors 'ready)
  ...
  (fire-pyros 1 2 4)
  ...
  (ullage-motors 2.8)
  ...)

Constructs like this were obviously critical to the safety of the system: the code in the body of this form could not run unless the preconditions were met. Sadly we did not anticipate the alien incursion however.

And all this came from one thing (apart from the aliens): I represented the source code of my programs in an extremely simple data structure (in particular that data structure was not full of assumptions about what things meant) which was available to the programs themselves.

This is why metaprogramming is useful: metaprogramming is building programming languages.

Notes:

• 'expanders' are macros, and 'expander functions' are macro functions in CL;
• in Common Lisp let is a 'special operator' – something built into the evaluator – but it does not have to be and is not in my variant world:
• same for if.

If you've seen ReactJS, you've already seen an example of what you can do with metaprogramming that you can't do with regular functions. Metaprogramming is very common in the JS community. There is a plethora of "compile-to-JavaScript" languages, and they're all examples of metaprogramming.

Metaprogramming allows you to do things that would otherwise require a new compiler.

The advantage you get from having macros built into the language is that you don't have to write an external tool to support alternative syntax like that of ReactJS. If someone was to implement ReactLisp, it would only be necessary to write some reader macros that would parse the component syntax into ordinary Lisp code.

Reader macros are functions that return Lisp code, so there's no need to write temporary files to disk.

It's so well integrated that it wouldn't even be necessary to separate ReactLisp source files from ordinary Lisp files (which you do in a React.js project by using the .jsx extension instead of .js).

In addition to reader macros, Lisp also has AST macros, which can be used to implement less radical syntax extensions. For example, years ago, someone in the JavaScript community came up with these things called "promises", and you had to call the then method on them to carry out the next step.

Then, because using promises was so cumbersome, they decided to add the async and await keywords to JavaScript. They're just syntactic sugar for the idioms around Promises.

You could implement promises in Lisp (but be advised that there's something called a "promise" in Scheme, and it has nothing to do with the construct that JavaScript calls a promise), and then extend the language with async and await macros which would work just as well as the async and await keywords that are built into JavaScript. Since macros are built into Lisp, there's no need for there to be a change to the language. The extension could be a library.