Titel

The exciting news is that the user is able to extend the language; without macros, only the language implementors can add new language constructs. This hinders experimentation with new language features.

This tutorial provides a bottom-up introduction to macros.

The idea behind macros is deceptively simple. A macro (syntax transformer) simply rewrites one piece of syntax into another.

Even though I promised a bottom-up introduction, we will begin with an example chosen to give an overview of the process of writing a macro from scratch to finish. In the next chapter we go into great detail with respect to the topic of syntax objects.

Let’s say I discover this pattern in my code:

(define x 42)
(define y x)
(define a 43)
(define b a)

That is, for some reason, I often have the need to declare and set two variables to the same value. Following the DRY principle DRY means “Don’t repeat yourself”. Wikipedia I would like to write this instead:

(define2 x y 42)
(define2 a b 43)

To do this I need to bind

define2

to a syntax transformer (macro), a normal function from syntax objects to syntax objects. The input to the syntax transformer is a piece of syntax

stx

representing a use of the code, such as

(define2 x y 42)

. The output is a newly constructed piece of syntax representing the rewritten code.

(define-syntax define2  ; bind define2 to
  (lambda (stx)         ; this syntax transformer
    <analyze stx and return a new syntax object>))

Given, say,

(define2 x y 42)

as input, the syntax transformer needs to analyze the input to find out which variables were used. This is often done via pattern matching with the help of

syntax-parse

. In its simplest form

syntax-parse

looks like this:

(syntax-parse stx
  [pattern1 expr1]
  [pattern2 expr2]
  [pattern3 expr3])

It will match the syntax object

stx

against the patterns

pattern1

,

pattern2

,

pattern3

. When it finds a pattern that matches, the corresponding expression is evaluated. In our example we only need one pattern:

(_define2 var1 var2 expr)

When this pattern is matched against

(define2 x y 42)

, the pattern variables

var1

,

var2

and

expr

will be bound to pieces of syntax representing

x

,

y

and

42

respectively. This is what we have so far:

(define-syntax define2
  (lambda (stx)
    (syntax-parse stx
      [(_define2 var1 var2 expr)         ; pattern
       <construct the output syntax>]))) ; expression

In order to construct the output we will use the form

syntax

.

(syntax template)

The form

syntax

constructs a new syntax object from a given template. It takes the template and replaces identifiers that are bound as pattern variables with their corresponding values.

Since we need to produce two definitions with only one template, we wrap our definitions in a

begin

: In a module or in the body of a function

(begin form ...)

will splice the forms into the enclosing context.

(syntax
  (begin
    (define var1 expr)
    (define var2 var1)))

The resulting macro becomes:

(define-syntax define2
  (lambda (stx)
    (syntax-parse stx
      [(_define2 var1 var2 expr)
       (syntax
        (begin
          (define var1 expr)
          (define var2 var1)))])))

A complete, runnable program with a few tests:

define2.rkt

#lang racket
(require (for-syntax syntax/parse))
(define-syntax define2           ; bind define2 to a
  (lambda (stx)                  ; syntax transformer that
    (syntax-parse stx            ; matches stx against
      [(_define2 var1 var2 expr) ; this pattern, and produces
       (syntax                   ; a new syntax object
        (begin                   ; following this template
          (define var1 expr)
          (define var2 var1)))])))

(define2 x y 42)
(define2 a b 43)
(list x y a b) ; (42 42 43 43)

Tinker with the example.

Try defining a

define3

.

Could we have used this template

(syntax
  (begin
    (define var1 expr)
    (define var2 expr)))

instead of the one used above?

Hint: What happens when

expr

has a side effect?

What happens when you run a Racket program?

The language model divides the process into passes. The model tells us “how to think” about the evaluation process. How the details are actually implemented are irrelevant here.

\[\begin{align} \textrm{Source} & \xrightarrow{\texttt{read}} \textrm{Syntax Object} \\\\ & \xrightarrow{\texttt{expand}} \textrm{Syntax Object} \\\\ & \xrightarrow{\texttt{compile}} \textrm{Compiled Expression}\\\\ & \xrightarrow{\texttt{eval}} \end{align} \]

-: A read pass turns a character stream into a syntax object.
-: An expand pass transforms the syntax object into a fully parsed one.
-: Finally, compile and/or eval to get the job done.

Usually the last step, evaluation, involves compiling into machine code.

Macros affect the expand pass. During program expansion, if the expander finds a macro application, it will call the associated syntax transformer. The expansion continues with the value returned from the syntax transformer.

After expansion the result is a fully parsed program in which there are no macro applications left.

The fact that macros are called only during expansion allows us to conclude:

-: Use a
reader
if you need to change the lexical structure. Macros can’t be used to change the lexical structure
(the lexical structure specifies how a stream of characters is divided into tokens (numbers, symbols, names etc) by the reader).
-: Macros are a compile time concept.
More on runtime vs compile time later. Macro expansion happens once - not each time the program is run.

The following example shows the results of the various passes during the evaluation of a small program fragment. This section is about the big picture, so there are details of the example program, that we haven’t yet dicussed (such as the use of the namespace). However, the program gives you something to try out in the repl.

fact-example.rkt

#lang racket
(require racket/format)

(define source
  "(begin
     (define (fact n)
        (cond
          [(= n 0) 1]
          [else    (* n (fact (- n 1)))]))
     (fact 5))")

(define ns (make-base-namespace))         ; expand and eval
(parameterize ([current-namespace ns]     ; need a namespace
               [read-accept-compiled #t])
  (define input    (open-input-string source))
  (define before   (read    input))
  (define after    (expand  before))
  (define compiled (compile after))
  (define result   (eval    compiled))
  (define out      (list "-- before expansion --"
                         before
                         "-- after expansion --"
                         after
                         "-- compiled expression --"
                         (substring (~a compiled) 0 20)
                         "-- result --"
                         result))
  (for-each displayln out))

The output shown below has been slightly altered in order to make it easier to read.

output of fact-example.rkt

-- before expansion --
(begin (define (fact n)
         (cond
           ((= n 0) 1)
           (else (* n (fact (- n 1))))))
       (fact 5))
-- after expansion --
#<syntax
   (begin
     (define-values (fact)
       (lambda (n)
         (if (#%app = n (quote 0))
             (let-values ()
               (quote 1))
             (let-values ()
               (#%app *
                      n (#%app (#%top . fact)
                               (#%app - n (quote 1))))))))
     (#%app (#%top . fact)
            (quote 5)))>
-- compiled expression --
#~ 7.5racket...bytes omitted...
-- result --
120

In historical Lisp-systems with only S- expressions and no syntax objects,

read

handled the read pass. To our disappointment the example shows us that

read

produces an \(S\)-expression - not a syntax object. It turns out that the read pass is actually done by

read-syntax

.

Think of a syntax object as an \(S\)-expression with extra information, which keeps track of where identifiers and numbers come from (line numbers, start and end positions in the character stream is recorded). The function

read

consumes a character stream and produces an \(S\)-expression. It is often better to use

read-syntax

which produces a syntax object instead.

Change the example to use

read-syntax

instead. Remember to pass a source-name.

Check out the description of

read-syntax

in the Reference first. You can use, say,

"fact.rkt"

as a dummy source name.

For the second reading only: Change the example to use

expand-syntax

.

Read the description of

expand

and

expand-syntax

carefully first. Can you spot the difference?

Hint: Use

(namespace-syntax-introduce before ns)

to “enrichen” the syntax object before expansion.

During the expansion pass, macros rewrite program fragments. Thus, program fragments need to be stored as values in memory (as opposed to a stream of characters in a file).

How program fragments are represented (stored in memory) differs from language to language. In macro assemblers and \(C\) programs, fragments are represented as plain strings. In some Lisp systems, program fragments are represented as \(S\)-expressions. Wikipedia: Abstract Syntax Trees A common representation in non-Lisp languages are abstract syntax trees (ASTs).

Using plain strings or \(S\)-expressions looks alluring at first sight. They are easy to understand and manipulate. However, they both have one important downside. There is nowhere to attach extra information.

Say, during macro expansion, the expander finds an

"x"

or

'x

that has no binding. It needs to generate an “x is unbound” error, but since there is no information attached to the

x

, the expander can’t show the user which of the many

x

‘s in the program that are causing the problem.

Racket represents program fragments as

syntax objects

. A syntax object consists of a normal Racket value combined with extra information. The most important pieces of extra information are source-location information and lexical information. For completeness, here is a list of information contained in syntax objects:

-: a simpler Racket value (non-syntax object)
-: lexical information
-: source-location information
-: syntax properties

In the next section, we will look at how to explore the contents of a syntax object using a syntax browser.

During the process of writing a macro, it is very helpful to inspect the information buried in a syntax object.

Let’s produce a syntax object to experiment with:

> (define (to-syntax s [source-name #f])
(read-syntax source-name (open-input-string s)))
> (to-syntax "(foo bar baz)")
#<syntax:string::1 (foo bar baz)>

The output #<syntax:string::1 (foo bar baz) doesn’t reveal much about the extra information stored in the syntax object. It looks almost like a plain \(S\)-expression.

I recommend using DrRacket while reading this section. It is however also possible to open a syntax browser from Emacs. See Appendix A. Luckily DrRacket has a builtin syntax browser. In DrRacket, a click on the blue triangle in front of a syntax object will reveal additional information.

We didn’t provide a source name to

read-syntax

, so the source simply says

'string

. We also see that

(foo bar baz)

is read from position 1 (remember that file positions count from \(1)\). The span is 13, which means that the number of characters in

(foo bar baz)

was 13. Since the position of the open parenthesis ‘(‘ is 1, the position of the close parenthesis ‘)‘ is 13.

Note that

read-syntax

marks the output as “original”. This can be used to distinguish between original syntax read from a file and syntax introduced by a macro.

Figure 1: Focus on the syntax info. alternative text

Figure 2: Focus on the line and column numbers.

In Emacs line numbers are enabled with
M-x line-number-mode Note that the the line and column numbers are missing from in figure 1. The port returned by

open-input-string

doesn’t track line and column numbers by default, but it’s easy to enable them with

port-count-lines!

.

> (define (to-syntax s [source-name #f])
    (define input (open-input-string s))
    (port-count-lines! input)
    (read-syntax source-name input))
> (to-syntax "(foo bar baz)")
#<syntax:string:1:0 (foo bar baz)>

It is now possible to see both column and line numbers in the second figure above (figure 2).

Figure 3: Focus on bar. Finally, let’s click on

bar

to check that source-location information is tracked not only for the whole list, but for all elements. Notice that now only

bar

is green.

Since

bar

is an identifier, the syntax browser also displays any known bindings. The syntax object we are examining has neither been expanded nor been enriched with extra lexical information, so there is no binding for

bar

stored in the syntax object.

So far, we have only explored unexpanded syntax objects. We have seen that the syntax browser can show us source-location information. However, before expansion there is no lexical information to see.

During the expansion pass, the macro expander adds lexical information to a syntax object. The syntax browser can show the binding information attached to identifiers.

Evaluate the expression below in the DrRacket REPL.

Click on the various identifiers.

What is the difference between, say,

=

and

fact

?

(expand
(syntax
  (define (fact n)
    (if (= n 0)
        1
        (* n (fact (- n 1)))))))

In this section we will study how to construct syntax objects based on templates.

In previous sections, we have seen how to turn a file into a syntax object with the help of

read-syntax

. In this section, we will look at template-based construction instead.

The form

syntax

constructs a syntax object based on a template. The grammar below describes how templates are written. I have simplified the grammar of

syntax

slightly. The full grammar also allows boxes and prefab structures, ellipsis quoting and the operators

~?

,

~@

. Later, we will return to the operators.

syntax
(syntax template)

template = const
| id
| (head-template ...)
| (head-template ...+ . template)
| #(head-template ...)

head-template = template
| head-template ellipsis ...+

ellipsis = ...

When you read the grammar, pronounce = as “is a”, and | as “or”. The first two lines read: A template is either a constant or an identifier or an ...

; a string constant
> (syntax "Hello World")
#<syntax:eval:1:0 "Hello World">
; a number constant
> (syntax 42)
#<syntax:eval:2:0 42>
; a character constant
> (syntax #\a)
#<syntax:eval:3:0 #\a>
; an identifier
> (syntax fish)
#<syntax:eval:4:0 fish>

The third, fourth and fifth lines describe templates that construct syntax objects representing lists, improper lists and vectors respectively.

In general

...

indicates “0 or more”, and

...+

indicates “1 or more”. The third line says that a template can have the form

(head-template ...)

. This will construct a syntax object representing a list. The elements of the list have the same shape as head-template. The ellipsis,

...

, after head-template means that a list template consist of 0 or more subtemplates (all head templates).

See Reading Pairs and Lists for for more on reading improper lists (and dotted pairs). The fourth line says that a template can have the form

(head-template ...+ . template)

.

This constructs a syntax object representing an improper list (unless the last template produces a list). The ...+ means that the template needs at least 1 subtemplate before the dot.

Note that #(...) is normally used to create vector literals. The fifth line says that a template can have the form

#(head-template ...)

. This constructs a syntax object representing a vector.

; a list
> (syntax (foo "bar" 42))
#<syntax:eval:5:0 (foo "bar" 42)>
; an improper list
> (syntax (foo "bar" . 42))
#<syntax:eval:6:0 (foo "bar" . 42)>
; a vector
> (syntax #(foo "bar" 42))
#<syntax:eval:7:0 #(foo "bar" 42)>

The idea behind the template syntax is that it corresponds to the syntax used to write literals.

> '(foo "bar" 42)
'(foo "bar" 42)
> '(foo "bar" . 42)
'(foo "bar" . 42)
> '#(foo "bar" 42)
'#(foo "bar" 42)

In fact it is standard practice to write #’ instead of

syntax

. It is at read time that #’ is turned into

syntax

. See

parse-quote

.

> #'(foo "bar" 42)
#<syntax:eval:11:0 (foo "bar" 42)>
> #'(foo "bar" . 42)
#<syntax:eval:12:0 (foo "bar" . 42)>
> #'#(foo "bar" 42)
#<syntax:eval:13:0 #(foo "bar" 42)>

What makes

syntax

the ideal tool to construct syntax objects, is that it is possible to substitute parts of a template with syntax objects.

A quick example wherein an example of Pythagoras is constructed.

> (with-syntax ([a (syntax 3)]
                [b (syntax 4)]
                [c (syntax 5)])
    (syntax
     (= (sqr c) (+ (sqr a) (sqr b)))))
#<syntax:eval:14:0 (= (sqr 5) (+ (sqr 3) (sqr 4)))>

The forms

with-syntax

or

syntax-parse

are used to make an identifier into a pattern variable. If a pattern variable (identifier) appears in a template,

syntax

won’t insert the identifier, but rather insert a syntax object associated with the pattern variable.

In other words, we can use an identifier in a template as a place holder. Use

with-syntax

to bind the identifier (pattern variable) to a piece of syntax. Then

syntax

will insert the piece in place of the identifier.

Here the identifiers

a

,

b

and

c

were substituted for syntax objects representing the numbers 3, 4 and 5.

As a convenience, if the expression on the right hand side of a

with-syntax

clause does not evaluate to a syntax object, it is automatically converted to one using

datum->syntax

. That is, we can also write:

> (with-syntax ([a 3]
                [b 4]
                [c 5])
    (syntax
     (= (sqr c) (+ (sqr a) (sqr b)))))
#<syntax:eval:15:0 (= (sqr 5) (+ (sqr 3) (sqr 4)))>

Now,

syntax

will substitute identifiers (pattern variables) that are bound to syntax objects. Before examining how

syntax

handles pattern variables, let’s take a closer look at how

with-syntax

binds pattern variables to syntax objects.

syntax
(with-syntax ([pattern stx-expr]
...)
body ...+)

The form

with-syntax

wraps a body in a series of clauses consisting of patterns and expressions.

The expressions

stx-expr

... are evaluated in order. If an

stx-expr

doesn’t produce a syntax object it is converted to one using

datum->syntax

. Each syntax object is then matched with the corresponding pattern. An identifier in the pattern will be bound as a pattern variable to the part of the syntax object it matched.

Finally, the bodies are evaluated in order. Within the bodies,

syntax

will substitute any pattern variable (identifier) to the syntax object it is bound to.

If some pattern doesn’t match, an exception will be raised. Only if all patterns match successfully will the body of

with-syntax

will be evaluated.

The most important rules followed by

syntax

are:

-

If an identifier is not bound as a pattern variable, we get the actual identifier.

-

If an identifier

id

is bound as a pattern variable, then

id

as a template inserts the value to which it is bound.

-

If an identifier is bound as a pattern variable in a pattern of the type

id

...

, then the template id ... as a template will insert (splice) the elements of the list.

We have seen examples of the first two rules already. Let’s examine the third rule.

> (with-syntax ([x '(1 2 3 4 5)])
(syntax
(list "first" x "last")))
#<syntax:eval:16:0 (list "first" (1 2 3 4 5) "last")>

> (with-syntax ([(y ...) '(a b c d e)])
(syntax
(list "first" y ... "last")))
#<syntax:eval:17:0 (list "first" a b c d e "last")>

In the first example, the list is inserted as a list. In the second example, the elements of

y

are spliced into the syntax object.

If we need to splice the elements of

x

, we can’t just add an ellipsis after

x

in the template. This leads to an error:

> (with-syntax ([x (list 1 2 3 4 5)])
    (syntax
     (list "first" x ...  "last")))
eval:18:0: syntax: too many ellipses in template
  at: ...
  in: (syntax (list "first" x ... "last"))

Note that the error says “syntax: too many ellipses in template”. It is implicit that the “too many” compares the number of ellipses in the pattern versus the template. In other words, the problem can also be too few ellipses in the pattern.

We need to change the pattern:

> (with-syntax ([(x ...) (list 1 2 3 4 5)])
(syntax
(list "first" x ... "last")))
#<syntax:eval:19:0 (list "first" 1 2 3 4 5 "last")>

Now that the pattern and the template have the same number of ellipses following the

x

, everything works as expected.

Before we end this section, we have one loose end left to tie. In the grammar for templates, we have both template and head-template. Why do we have both?

The reason is that the position after the dot in the template for an improper list is special. Only one literal/identifier can occur after the dot, which means that we are forced to have two productions (rules) in the grammar: one that allows ellipsis and one that doesn’t. A head template in turn is either a template or a head template followed by one more ellipses (i.e. one more triple dot). Since all templates are head templates, templates of the form

If you are familiar with runtime vs compile time, we will work at runtime only in this section. We will look at runtime vs compile time later. This makes experimenting easier – and can be a useful debugging technique later on. The job of a macro is to transform some input form into an output form. In this section, we will practise the use of

with-syntax

and

syntax

on some common use cases. In this section we will focus on the transformation.

Our first example will be a verbose version of

let

that will print the variable and value pairs before evaluating the body.

(let/verbose ([a 11] [b 12])
(list a b))

should be transformed into

(let ([a 11] [b 12])
(displayln (list "let: " (list 'a a) (list 'b b)))
(list a b))

When run, we will see:

(let: (a 11) (b 12))
'(11 12)

The first job is to write a pattern that matches the input. As an input example, think of:

(let/verbose ([a 11] [b 12]) (list a b))

The example has two binding clauses, but the actual number varies. We need to use an ellipsis in the pattern:

(define (let/verbose-transformer stx)
  (with-syntax
    ([(_let/verbose ([x expr] ...) body) stx])
    <produce-the-output>))

In order to test that our pattern works as expected, we can temporarily insert displays to show what was matched:

> (define (let/verbose-transformer stx)
    (with-syntax
      ([(_let/verbose ([x expr] ...) body) stx])
      (displayln #'(x ...))
      (displayln #'(expr ...))
      (displayln #'body)
      #'we-still-need-to-produce-some-output))
> (let/verbose-transformer
   #'(let/verbose ([a 11] [b 12]) (list a b)))
#<syntax:eval:1:0 (a b)>
#<syntax:eval:1:0 (11 12)>
#<syntax:eval:2:0 (list a b)>
#<syntax:eval:1:0 we-still-need-to-produce-some-output>

Given these pieces we want to produce the following output:

(let ([a 11] [b 12])
(displayln (list "let: " (list 'a a) (list 'b b)))
(list a b))

To do that, we need to write a template:

> (define (let/verbose-transformer stx)
    (with-syntax
      ([(_let/verbose ([x expr] ...) body) stx])
      (syntax
       (let ([x expr] ...)
         (displayln (list "let: " (list 'x x) ...))
         body))))
> (let/verbose-transformer
   (syntax(let/verbose ([a 11] [b 12]) (list a b))))
#<syntax:eval:3:0 (let ((a 11) (b 12)) (displayln (list "let: " (list (quote a) a) (list (quote b) b))) (list a b))>

And finally we can try the output in repl:

> (let ((a 11) (b 12))
    (displayln (list "let: " (list 'a  a) (list 'b  b)))
    (list a b))
(let:  (a 11) (b 12))
'(11 12)

Let’s for a moment consider a variation. Instead of the output

(let: (a 11) (b 12))

we want

(let: a 11 b 12)

.

It is not enough to change the template to

(displayln (list "let: " 'x x ...))

.

Doing so leads to the error
"syntax: missing ellipsis with pattern variable in template at: x". The problem is that the ellipsis in our template only applies to the

x

before it, not to the

x

in

'x

. Simply adding an extra ellipsis, as in

(displayln (list "let: " 'x ... x ...))

does result in a program that compiles, but now the order is incorrect. The names are printed first, followed by the values.

The simplest solution is to change

(list 'x x) ...

to

(~@ 'x x) ...

.

The use of @ to mean “splicing” follows the quasiquote convention, where ,@ means

unquote-splicing

. Here

~@

can be thought of as “splicing-list”. The elements are inserted directly without an enclosing list.

> (define (let/verbose-transformer stx)
    (with-syntax
      ([(_let/verbose ([x expr] ...) body) stx])
      (syntax
       (let ([x expr] ...)
         (displayln  (list "let: " (~@ 'x x) ...))
         body))))
> (let/verbose-transformer
   (syntax(let/verbose ([a 11] [b 12]) (list a b))))
#<syntax:eval:6:0 (let ((a 11) (b 12)) (displayln (list "let: " (quote a) a (quote b) b)) (list a b))>

We will end this section with a look at simplifying a nested use of

with-syntax

.

Consider this example:

> (with-syntax ([((sym str) ...)
                 (syntax ((a "11") (b "22") (c "33")))])
    (with-syntax ([(num ...)
                   (map string->number
                        (syntax->datum
                         (syntax (str ...))))])
      (syntax
       (list (list sym num) ...))))
#<syntax:eval:8:0 (list (list a 11) (list b 22) (list c 33))>

The outer

with-syntax

is used to extract the strings. The inner

with-syntax

then converts the strings into numbers, which appears in the output.

Note, that we can’t write:

> (with-syntax ([((sym str) ...)
                 (syntax ((a "11") (b "22") (c "33")))]
                [(num ...)
                 (map string->number
                      (syntax->datum
                       (syntax (str ...))))])
      (syntax
       (list (list sym num) ...)))
eval:9:0: syntax: no pattern variables before ellipsis in
template
  at: str
  in: (syntax (str ...))

The problem is that the scope of the pattern variable

str

doesn’t include the following clauses – the pattern variable is only in scope in the body.

Analogy:

with-syntax*

is to

with-syntax

\[as \]

let*

is to

let

. Instead we, can use

with-syntax*

.

> (require racket/syntax)
> (with-syntax* ([((sym str) ...)
                  (syntax ((a "11") (b "22") (c "33")))]
                 [(num ...)
                  (map string->number
                       (syntax->datum
                        (syntax (str ...))))])
      (syntax
       (list (list sym num) ...)))
#<syntax:eval:11:0 (list (list a 11) (list b 22) (list c 33))>

The formal description of

with-syntax*

is as follows.

syntax
(with-syntax* ([pattern stx-expr]
...)
body ...+)

The module

racket/syntax

exports

with-syntax*

. Similar to

with-syntax

, but the pattern variables of each pattern are not only bound in the bodys, but also in the stx-exprs of subsequent clauses. Also, the patterns no longer need to bind distinct pattern variables; later bindings shadow earlier bindings.

Replace

???-pattern

and

???-template

in the following snippets in order to make all snippets evaluate to true.

(define result0
  (with-syntax ([(num ...) (syntax (1 2 3))])
    (syntax ???-template)))

(equal? (syntax->datum result0)
        (list 1 2 3))

(define result1
  (with-syntax ([(num ...) (syntax (1 2 3))])
    (syntax ???-template)))

(equal? (syntax->datum result1)
        (vector 1 2 3))

(define result2
  (with-syntax ([(str ...) (syntax ("a" "b" "c"))]
                [(num ...) (syntax ( 1   2   3))])
    (syntax ???-template)))

(equal? (syntax->datum result2)
        '(("a" 1) ("b" 2) ("c" 3)))

(define result3
  (with-syntax ([(str ...) (syntax ("a" "b" "c"))]
                [(num ...) (syntax ( 1   2   3))])
    (syntax ???-template)))

(equal? (syntax->datum result3)
        '("a" 1 "b" 2 "c" 3))

(define result4
  (with-syntax ([(str ...) (syntax ("a" "b" "c"))]
                [(num ...) (syntax ( 1   2   3))])
    (syntax ???-template)))

(equal? (syntax->datum result4)
        '("a" "b" "c" 1 2 3))

(define result5
  (with-syntax ([(num ...) (syntax (1 2 3))])
    (with-syntax ([(str ...) ???-expr])
      (syntax ???-template))))

(equal? (syntax->datum result5)
        '("1" "2" "3"))

(define result6
  (with-syntax
    ([???-pattern (syntax ((a (1 "alpha") (2 "beta"))
                           (b (3 "gamma") (4 "delta"))))])
    (syntax ???-template)))

(equal? (syntax->datum result6)
        '((a b) (1 2 3 4) ("alpha" "beta" "gamma" "delta")))

In this section we examine a simple macro.

“A Freudian slip is when you say one thing and mean your mother.”
– Bob Hope But first, a small reminder. The idea is to write one thing, \(A\), in our source code, but the compiler must pretend we wrote another, \(B\).

\[\begin{align} \textrm{Source} & \xrightarrow{\texttt{read}} \textrm{Syntax Object} \\\\ & \xrightarrow{\texttt{expand}} \textrm{Syntax Object} \\\\ & \xrightarrow{\texttt{compile}} \textrm{Compiled Expression}\\\\ & \xrightarrow{\texttt{eval}} \end{align} \] The reader turns \(A\) into a syntax object (buried in a very large syntax object representing the entire program). If the expander can tell that \(A\) is a macro call, it will call the associated syntax transformer. The return value \(B\) will then be used in place of \(A\).

As a concrete example, let’s make a macro

backwards

that allows us to write \(A\):

(backwards
   (display 3)
   (display 2)
   (display 1))

but will compile as if we had written \(B\):

(begin
   (display 1)
   (display 2)
   (display 3))

Let’s write our transformer first. The general form of the input \(A\) is:

(backwards form ...)

And we want to produce

(backwards rev-form ...)

where the

rev-form

forms appear in the reverse order of the original forms.

> (define (backwards-transformer stx)
    ; First use a pattern to get parts of the input
    (with-syntax ([(backwards form ...) stx])
      ; Compute auxiliary data needed in output
      (define rev-forms
        (reverse (syntax->list (syntax(form ...)))))
      ; Use a template to produce the output
      (with-syntax ([(rev-form ...) rev-forms])
        (syntax
         (begin rev-form ...)))))

We can test the transformer in the repl:

> (backwards-transformer
   (syntax
    (backwards
      (display 1)
      (display 2)
      (display 3))))
#<syntax:eval:1:0 (begin (display 3) (display 2) (display 1))>

We get the expected result, so we are ready to tell the expander that

backwards

must invoke the transformer

backwards-transformer

.

Our first attempt is:

> (define-syntax backwards backwards-transformer)
eval:3:0: backwards-transformer: unbound identifier;
also, no #%top syntax transformer is bound in the
transformer phase
in: backwards-transformer

What happened?

The issue is that syntax transformers run at compile time (aka in the transformer phase), but our transformer

backwards-transformer

is defined as a normal definition, which means it isn’t available until the final program runs.

We could define the transformer in a different module, and then use

(require (for-syntax "helper.rkt"))

. However, we will instead mark our definition of

backwards-transformer

, so the system knows it is supposed to run at compile time. The form

begin-for-syntax

is what we need.

> (begin-for-syntax
    (define (backwards-transformer stx)
      (with-syntax ([(backwards form ...) stx])
        (define rev-forms
          (reverse (syntax->list (syntax(form ...)))))
        (with-syntax ([(rev-form ...) rev-forms])
          (syntax
           (begin rev-form ...))))))
> (define-syntax backwards backwards-transformer)
> (backwards
   (display 3)
   (display 2)
   (display 1))
123

Since this is a tutorial, I wanted to make a point of defining the macro

backwards

and the transformer

backwards-transformer

separately to make the concepts clear. However, in practise the common case is to define both at the same time.

> (require (for-syntax racket))
> (define-syntax (backwards stx)
    (with-syntax ([(backwards form ...) stx])
      (define rev-forms
        (reverse (syntax->list (syntax(form ...)))))
      (with-syntax ([(rev-form ...) rev-forms])
        (syntax
         (begin rev-form ...)))))
> (backwards
   (display 3)
   (display 2)
   (display 1))
123

At this point we have a working macro. We are not done yet though.

We need to consider error situations. Can a user invoke our syntax transformer with bad input?

The void value is also called the invisible value, since the default repl doesn’t print a lone void value. The edge case

(backwards)

expands to

(begin)

, which produces a void value. No problem there.

In our code we have:

(with-syntax ([(backwards form ...) stx])
<more>)

If the input syntax in

stx

doesn’t match the the pattern

(backwards form ...)

, the user will get an error. Two examples illustrate the problem.

The second example shows

backwards

being used as an identifier macro.

> (backwards 1 2 . 3)
eval:2:0: with-syntax: binding match failed
in: (backwards form ...)
> (list 1 2 backwards 3 4)
eval:2:0: with-syntax: binding match failed
in: (backwards form ...)

From a user perspective “with-syntax: binding match failed” is a somewhat confusing error - nowhere in the user code is

with-syntax

used. This can be especially confusing, if the person who uses the macro and the author of the macro isn’t the same. Error messages should always be worded in terms of the macro itself – not to some arbitrary part implementing the macro.

To fix this, we will use

syntax-parse

instead of

with-syntax

to pick the input apart. Note that

raise-syntax-error

will highlight the offending user code correctly in DrRacket/racket-mode. Note that we have added

syntax/parse

to the

require

form to make

syntax/parse

available in the transformer phase.

> (require (for-syntax racket syntax/parse))
> (define-syntax (backwards stx)
    (syntax-parse stx
      [(backwards form ...)
       (define rev-forms
         (reverse (syntax->list (syntax (form ...)))))
       (with-syntax ([(rev-form ...) rev-forms])
         (syntax
          (begin rev-form ...)))]
      [_ (raise-syntax-error
          'backwards
          "expected (backwards form ...), got: "
          stx)]))
> (backwards
   (display 3)
   (display 2)
   (display 1))
123
> (backwards 1 2 . 3)
eval:4:0: backwards: expected (backwards form ...), got:
  in: (backwards 1 2 . 3)
> (list 1 2 backwards 3 4)
eval:5:0: backwards: expected (backwards form ...), got:
  in: backwards

Our macro now works both for well-formed and malformed input.

The steps we followed to in the preceding section can be thought of as a recipe for writing simple macros.

1.

Decide the input form of a macro

2.

Experiment with concrete examples to find a general transformation

3.

Turn the examples into tests.

4.

Write the macro by filling in:

(define-syntax (name stx)
  ; Use syntax-parse destructure the input syntax.
  (syntax-parse stx
    [(_name form ...)
     ; Compute auxillary information.

     ; Use with-syntax to bind the information
     ; to pattern variables.
     (with-syntax ([...])
       ; Construct output using a template.
       (syntax
         (begin rev-form ...)))]
    ; Repeat if several input patterns are needed.
    ; Finally use _ to catch all erroneous uses.
    [_ (raise-syntax-error
        'name
        "expected (name form ...), got: "
        stx)]))

5.

Run tests.

There is no builtin syntax browser in Emacs racket-mode, but fear not - there is a solution.

The library

mrlib/syntax-browser

contains a function

render-syntax/window

that can be used to explore a syntax object.

syntax-browser-in-racket-mode.rkt

#lang racket
(require mrlib/syntax-browser)
(render-syntax/window #'(foo bar baz))

The main resource when learning the art of macros is the chapter Macros in the Racket Reference. The second and third section of the chapter Macros are also relevant.

The reference is the most precise description, but can be a bit unwieldy in the beginning. A more practical introduction to macro writing is Greg Hendershott’s Fear of Macros.

The tutorial "Macros and Languages in Racket" by Ryan Culpepper and Claire Alvis explains basic macrology and has a lot of exercises. The blog post Writing syntax-case Macros by Eli Barzilay is an introduction to

syntax-case

written for people who knows Lisp-style macros (defmacro) The collection of examples "Syntax Parse Examples" curated by Ben Greenman. For old school

syntax-rules

macros, section section "4.3 Macros" in R5RS is a nice, concise introduction. The appendix contains examples of

syntax-rules

macros.

If you decide to dig into the papers on macros, take a look at the bibliography formerly on readscheme.org: Macros. Some highlights:

R. Kent Dybvig.	Syntactic abstraction: the syntax-case expander (pdf)
R. Kent Dybvig.	Writing Hygenic Macros in Scheme with Syntax-Case
Matthew Flatt.	Composable and Compilable Macros: You Want it When?
Matthew Flatt.	Binding as Sets of Scopes

Finally, if you are looking for some concrete mini projects to practise on, check out Racket Macro Exercises by Alexis King.