Language-oriented Programming in Racket

OK, what is lan­guage-ori­ent­ed pro­gram­ming, real­ly?

Rack­et’s more than just a bat­ter­ies-in­clud­ed Lisp. It’s about solv­ing prob­lems by mak­ing lan­guages. Lan­guage-ori­ent­ed pro­gram­ming is the name of the game.

The idea has an im­me­di­ate in­tu­itive ap­peal. Per­haps it’s what drove you to Rack­et in the first place. And if it wasn’t, chances are good, if you stick around the com­mu­ni­ty for a while, that you’ll find your­self think­ing about lan­guages.

But what in the world are we real­ly talk­ing about when we talk about lan­guage-ori­ent­ed pro­gram­ming?

The term has a va­ri­ety of mean­ings

Look­ing around the Rack­et com­mu­ni­ty & among its fel­low trav­el­ers, one finds that make your own lan­guage has a va­ri­ety of sens­es. Al­though some ideas are shared, they get ex­pressed in dif­fer­ent ways. At times, one won­ders: Are we all talk­ing about the same thing?

To help fos­ter the growth of lan­guage-ori­ent­ed pro­gram­ming in gen­er­al—and Rack­et in par­tic­u­lar as the best en­vi­ron­ment for solv­ing prob­lems in that way—it’s im­por­tant to un­der­stand the di­ver­si­ty of mean­ings of lan­guage-ori­ent­ed pro­gram­ming and kin­dred ex­pres­sions.

So who can say what it means?

A good place to help de­vel­op an un­der­stand­ing of lan­guage-ori­ent­ed pro­gram­ming is the Rack­et com­mu­ni­ty it­self. See be­low for a sam­pler of quotes.

Lan­guage-ori­ent­ed Pro­gram­ming with Rack­et: A Cul­tur­al An­thro­pol­o­gy is an ebook (482 pages) that con­tains the re­sult of a sur­vey I’ve tak­en of a wide spec­trum of Rack­et pro­gram­mers about what make your own lan­guage means to each of them.

Fore­word by Mar­tin Ward, who, in 1994, coined the term lan­guage-ori­ent­ed pro­gram­ming.

(Go here to get your copy of the ebook.)

Splash image for Language-oriented Programming in Racket—A Cultural Anthropology

Meet the lan­guage mak­ers

Here are snip­pets from each of the in­ter­views. (A PDF sam­pler of these quotes is avail­able here.)

That kind of grad­ual de­vel­op­ment from an al­most iden­ti­cal to Rack­et lan­guage to one that has all of the fea­tures that you need, and be­haves dif­fer­ent­ly (pos­si­bly very dif­fer­ent­ly) from Rack­et—that’s where most of the ben­e­fits of LOP come in, and that’s where Rack­et’s ap­proach shines.
I could write an ex­ter­nal pre­proces­sor, but this is a sep­a­rate lan­guage that gen­er­ates code. I could try to write a plu­g­in, but this re­quires step­ping out­side the lan­guage. If I were work­ing in Rack­et, I would write a few macros to ex­pand all of those things.
Rack­et pro­vides a pars­er gen­er­a­tor li­brary but what makes Rack­et dif­fer­ent as a lan­guage build­ing ma­te­r­i­al is two fea­tures: (i) its macro sys­tem, and (ii) the pos­si­bil­i­ty to over­ride what func­tion ap­pli­ca­tion means.
I real­ly had no ex­po­sure to the idea of DSLs. Rather, I want­ed a cer­tain re­sult, and crept up on DSLs ac­ci­den­tal­ly (it just seemed like the most di­rect way to do what I was af­ter).
Cer­tain vo­cab­u­lary emerges out of rudi­men­ta­ry Rack­et func­tions. It is like the Rack­et lan­guage is en­riched in the di­rec­tion of game play­ing, pop­u­la­tion match­ing, re­gen­er­at­ing, etc. They car­ry sig­nif­i­cant mean­ing out­side the de­fined range of orig­i­nal Rack­et.
I’ve al­ways been fas­ci­nat­ed with dif­fer­ent pro­gram­ming lan­guages and even with the idea of cre­at­ing my own, but I al­ways saw it, and much of the lit­er­a­ture and the pub­lic dis­course pre­sent­ed it, as some im­pos­si­bly com­pli­cat­ed task not meant for mere mor­tals.
If the pro­gram­ming mod­el af­fects the struc­ture of pro­gram frag­ments sig­nif­i­cant­ly then a new li­brary is a sub-op­ti­mal so­lu­tion as it in­tro­duces lay­ers of en­cod­ings. A new lan­guage can hide all this and re­sult in con­cise rep­re­sen­ta­tions.
I saw that I had an up­com­ing need to write lots of vari­a­tions of the same kind of pro­gram, and want­ed to make those pro­grams as hu­man-leg­i­ble as pos­si­ble, which means ab­stract­ing away over­head tasks and hav­ing log­i­cal names for things.
The in­struc­tor em­pha­sized that it was com­mon­place in the Lisp world to pro­gram in two steps: first cre­ate the lan­guage that you want, then im­ple­ment your so­lu­tion in that lan­guage. That idea stuck with me, and I lat­er ap­plied it even when I wasn't pro­gram­ming in Lisp.
For me the real aha! mo­ments didn’t oc­cur when I re­al­ized I could use lan­guage-ori­ent­ed pro­gram­ming to solve some prob­lem. The big aha! mo­ment was when I re­al­ized that LOP-heavy projects aren’t al­ways worth do­ing.
I want [my stu­dents] to see that there are lan­guages every­where. Sports teams have lan­guages for plan­ning plays; recipes are lan­guages; med­ical pro­to­cols are lan­guages.
Af­ter you use high­er-or­der func­tions for a while, you’re not will­ing to go back. Macros are like that, so I think it’s just a mat­ter of time for enough pro­gram­mers to catch on.
I find mak­ing lan­guages to be the same as nor­mal pro­gram­ming: some­times is easy, some­times its hard. And of­ten when its hard its be­cause you picked the wrong de­sign at the start!
I was kind of do­ing LOP any­way when I cre­ate new li­braries for peo­ple. When I dis­cov­ered Rack­et and be­gan re­search­ing it, I re­al­ized I could cre­ate more than just func­tions for peo­ple. I re­al­ized I could cre­ate new syn­tax and ba­si­cal­ly make the shape of the code be any­thing I want­ed it to be.
Lan­guage-ori­ent­ed pro­gram­ming, to me, means tweak­ing, aug­ment­ing, or chang­ing an un­der­ly­ing com­pu­ta­tion­al mod­el—or com­ing up with a com­plete­ly new com­pu­ta­tion­al mod­el.
I un­der­stand the term lan­guage-ori­ent­ed pro­gram­ming as an ap­proach to prob­lem solv­ing where, be­fore ac­tu­al­ly solv­ing the prob­lem, one fo­cus­es on ex­press­ing this prob­lem in an op­ti­mal way.
I’d al­ready gone from think­ing about prob­lems in terms of data struc­tures to think­ing about prob­lems in terms of types and so when I heard Rack­et was good for think­ing about prob­lems in terms of a tai­lor-made lan­guage that just sound­ed good.
Hav­ing a com­mon lan­guage meant the ops team was free to tweak the mod­el and draft pat­terns while I bolt­ed a CSV file pars­er onto a fi­nite-state ma­chine. With­out ex­plic­it for­mal­ism, we solved a com­pli­cat­ed prob­lem el­e­gant­ly by de­vel­op­ing a pat­tern lan­guage to­geth­er.
I think it’s the case that any­one en­gaged in bot­tom-up pro­gram­ming is ac­tive­ly do­ing lan­guage-ori­ent­ed pro­gram­ming. As a re­sult, you can look at vir­tu­al­ly any li­brary out there and see a lan­guage in it. The types, and func­tions ex­port­ed are the prim­i­tives used to build up lay­ers of ab­strac­tion to form more com­pli­cat­ed ideas.
I was al­ways in­ter­est­ed in pro­gram­ming lan­guages and liked to learn about new ones, but the idea of re­al­is­ti­cal­ly mak­ing and us­ing po­ten­tial­ly many lan­guages nev­er seemed prac­ti­cal un­til I learned about em­bed­ded DSLs made with macros.
[In Rack­et] it is rel­a­tive­ly seam­less to go be­tween the two: to take a li­brary and add some lan­guage be­hav­ior to it, or to take a lan­guage-en­riched li­brary and just drop the lan­guage parts.
My lan­guage de­sign is al­most en­tire­ly ori­ent­ed around some sort of op­ti­miza­tion that I want to do. I need to re­strict the lan­guage so the op­ti­miza­tion ap­plies then I need to build a lan­guage that en­forces that re­stric­tion.
A lot are put off by Lisp's lack of syn­tax. Macros may be viewed as a black-art. If peo­ple tried them out and be­gan to see that macros are an ab­strac­tion lay­er, they would dis­cov­er that they are em­i­nent­ly us­able with­out hav­ing to sac­ri­fice (many) goats.
Think about the count­less hours poured into writ­ing tran­spilers for Java­Script. All that ef­fort to side­step in­ter­preters when a DSL or macro sys­tem could have achieved the same. Give me a ful­ly-fea­tured and pro­duc­tion ready Rack­et­Script any day.
I think there are two ways of lan­guage-ori­ent­ed pro­gram­ming. There are the macros you write to make your own code eas­i­er, and there are the lan­guages you write for oth­ers to use. I’ve used Rack­et for both.
I un­der­stood from a long way back that rather than sim­ply cod­ing in a giv­en tar­get lan­guage there was virtue in bring­ing the de­scrip­tive lan­guage to the prob­lem. This can be­gin with func­tion­al ab­strac­tion and a ruth­less de­sire to elim­i­nate du­pli­ca­tion and boil­er­plate.
The cor­rect jus­ti­fi­ca­tion [of ho­moiconic­i­ty] is that paren­the­ses fa­cil­i­tate care­ful metapro­gram­ming, the com­pile-time cre­ation of code. And metapro­gram­ming is at the heart of what makes Rack­et great.
I don’t think mak­ing a lan­guage is a yes-or-no de­ci­sion: You cre­ate ab­strac­tions, at some point you ab­stract over syn­tax and maybe lat­er down the path you dis­cov­er that just do­ing re­quire is awk­ward or repet­i­tive, which is when you do a lan­guage.
The in­ter­face was orig­i­nal­ly pro­vid­ed strict­ly as a li­brary, but com­pos­ing us­ing it felt like shoe­horn­ing the pat­terns into a pro­gram: quot­ing, boil­er­plate, etc. So I built a #lang in­ter­face in­stead.
At some lev­el, noth­ing in Sex­p­tex couldn't have been done with TeX macros in­stead. But the sur­face syn­tax of Rack­et is so much eas­i­er on the eyes than back­slash­es and curly braces, which I think is an un­der­rat­ed rea­son to do metapro­gram­ming.
When I'm writ­ing in the Anato­my lan­guage, it al­most doesn't feel like pro­gram­ming, be­cause all I have to think about is the struc­ture of di­nosaur skele­tons, which is ul­ti­mate­ly the only thing that feels rel­e­vant to the prob­lem I'm try­ing to solve.
At the mo­ment the project I am work­ing on is a web­site where stu­dents can solve math prob­lems. Such a project con­sists of many small parts, but the lan­guage ori­ent­ed ap­proach pops up re­peat­ed­ly. In­spired by scrib­ble and pollen I made #lang math­scrible.
Orig­i­nal­ly, I thought lan­guage-ori­ent­ed pro­gram­ming was just about us­ing macros to build in­ter­est­ing di­alects of Rack­et, such as Typed Rack­etor Lazy (Rack­et). But it’s not real­ly about writ­ing di­alects nec­es­sar­i­ly, since the lan­guages peo­ple make are of­ten hid­den as a tool used to im­ple­ment some oth­er pro­gram.
I find Rack­et’s ecosys­tem a good proof-by-con­struc­tion of the fea­si­bil­i­ty of [lan­guage-ori­ent­ed pro­gram­ming], con­sid­er­ing the im­pres­sive tow­er of mini-lan­guages that are used in con­cert.
The most fre­quent stick­ing point is the ques­tion of when some­thing is a lan­guage ver­sus a li­brary. I adopt a lib­er­al de­f­i­n­i­tion of a lan­guage that blurs the dis­tinc­tion be­tween the two, as I find that this makes the con­cept of mak­ing one’s own lan­guage more ap­proach­able.
What’s nice for me about the Rack­et ap­proach is that I only have to deal with the de­tails of those as­pects of the lan­guage im­ple­men­ta­tion that I want. If I don’t want to deal with lex­ing and pars­ing, for ex­am­ple, I can use the read­er.
I start­ed out build­ing a li­brary of func­tions for build­ing com­pil­ers in Rack­et. It was a sim­ple API built us­ing Rack­et's amaz­ing C-FFI. While I was us­ing this API to build a com­pil­er I slow­ly built a set of macros which made work­ing with these crude func­tions much eas­i­er. These set of macros slow­ly evolved and be­came an em­bed­ded lan­guage.
The par­tic­u­lar do­main of lan­guage con­struc­tion might be es­pe­cial­ly amenable to do­main-spe­cif­ic lan­guages. (There’s a rea­son yacc stands for yet an­oth­er com­pil­er com­pil­er. And yacc is old.)

The #langs un­der dis­cus­sion

A lot of Rack­et #langs and non-Rack­et pro­gram­ming lan­guages are dis­cussed:

2d anato­my Arc Asi64 ASL at-exp BA­SIC Bigloo bit­syn­tax book­cov­er brag Brain­fuck BSL C C++ ca­noe CFDG Chez Scheme Clo­jure cmx Cof­fee­Script Com­mon Lisp Com­pjure-api CSS css-expr cu­ni­form cur dat­a­log DSSL DSSL2 Er­lang Es­ter­el event Fairy­log FIST Flask Forth For­tran frog/con­fig fr­time F# gtp-mea­sure/man­i­fest Guile GW-Ba­sic hack­ett Han­di-Wrap Haskell heresy Hob­bit HTML info ipoe ISL Java Java­Script joy JSX Ju­lia La­TeX lazy Less LIPIcs Logo Lua mar­grave Mark­down match Math­Jax math­scrib­ble Maya MetaO­Caml MetaWSL mic1 mini­Kan­ren ML nanopass non-det OCaml ott page-lang paren­log Pas­cal pl PLAI play pollen pop-pl PRIMP Puz­zle­Script py-fizz Pyret Python QBA­SIC R rack­et rack­et/base rack­et/gui ragg Rash rash-vsa Re­act Re­ac­tiveML re­ac­tor re­dex reg­exp remix remo­ra Rie­mann ro­man-nu­mer­als rosette Ruby Rust s-exp Sass Scala Scheme Scheme 48 pl schlac Scratch scrib­ble scrib­ble/ac­mart scrib­ble/base scrib­ble/man­u­al scrib­ble/text scsh Scur­ry SDL send-exp sex­p­tex Sham shill SIMP Sina­tra Slick slideshow slideshow/sim­ple Smalltalk some­thing Son­ic Pi Spoofax SQL Sty­lus SVG syn­di­cate syn­tax-parse synth teachlog Ter­ra TLA+ Tur­bo Pas­cal turn­stile Typed Rack­et Type­Script Un­seem­ly urlang Ver­ilog video VI­O­LET vr-lang web-serv­er web-serv­er/in­s­ta Web­As­sem­bly XML xsmith

Get a copy

Go here to get your copy of the ebook (PDF for­mat). It comes out to 482 pages; there’s a ton of in­sight to be had.