Ruby DSL

macro没有利用任何反射能力，和类型也没有关系

Lisp macro(我只知道Common Lisp)的macro主要是开放编译器内部的结构，把词法分析、语法分析和语义分析和执行中间割裂（不过S-Expression非常简单，基本上没有语法)，Lisp的S-expression只管形式正确，而Lisp Form才管语义，用户编写的macro扩展的过程是把s-expression翻译成Lisp Form,经过Lisp的编译，然后才被执行。也就是说同一个macro针对不同的数据可能产生不同的Lisp Form(或者说不同的Lisp代码），执行效率也非常高。这是我讲的对同一个类的同一个方法，第一次执行，我们用Ruby的instance_eval 动态根据不同对象生成它自己的singleton method的过程，第2次执行就类似于已经“编译”了。

虽然和你讲的东西都有关系，但我表达的主题不再这些东西上。这并不是动态语言的特性，也不是函数型语言的特性，也不是反射的问题。虽然都有关系。

我不太喜欢看到评论的时候把Ruby/PhP/Python这样列一遍，他们是不同的，只浮在这个层次，那讨论没有任何意义

另外：http://wiki.rubytao.com/RubyTAO/page/show/RubyBlock+vs+LispMacro

"DSL" mechanisms in Ruby

i'm trying to get the zen of building DSLs using Ruby. After reading a dozen or so pieces referenced by my favorite search engine, I have a feeling I'm still not quite getting it.

I grok the various forms of eval (eval, module_eval and instance_eval) as well as the extremely cool Binding feature.

I've also seen BlankSlate for eviscerating an object of the methods it picks up from Object.

At the end of the day, however, it looks like the whole Ruby/DSL thing is just inventive use of eval + some clever string munging, both of which are pretty doable in Javascript or Python (or even (gasp) C# using the CodeDOM compiler).

I'd love nothing more than for the Ruby folks to point me in the right direction.

I met someone at the MS Tech Summit last month who's writing a book on this - perhaps he reads my blog and will give me the clue(s) I need.

---- comments
One of the aspects that makes Ruby nice for DSLs (IMHO) is the flexibility on use of parentheses in invocations. It seems like a trivial thing, but parentheses are more for machines than for people.

----
还有很多回复。值得一看。

http://jimweirich.umlcoop.net/articles/buildingwithrake/index.html

http://www.martinfowler.com/articles/rake.html

Everything Old is New Again - The Ruby DSL hype
Update: I've received a few mails asking how to go about creating dsl s "the right way" in Ruby. If you are in Bangalore, I suggest you attend the Ruby DSL talk at the Agile India 2006 by my friends Badri and Sudarshan (both Thoughtworkers). Such things are best learned by demonstration than reading about it and the Badri and Sudarshan are probably the best people in Bangalore to learn from.

If you are not in Bangalore find someone who knows and ask him to teach you. :-)

[Rant] These things have been true from the dawn of programming languages

   1. A language with a built in eval function allows you to transform code into a parse tree at runtime.
   2. To the degree that the language has a uniform/flexible syntax, this transition between code and data is painless
   3. All language transforms consist of two steps . First you convert a string into a data structure called a "syntax tree" . This is called parsing. Then various transformations can be done on this data structure to produce the effect you want.

      Thus the string "2 + 3" can be converted into a datastructure like "{ operator: "+", operand1:"2", operand2:"3"}".

      Then you can define various transforms that work on (or "visit" in oo terms) the components of the data structure and do whatever you want.

      e.g : (pseudocode) def print(anAst) puts anAst.operand1,anAst.operator,anAst.operand2 end

      def printReversePolish(anAst) puts anAst.operand1,anAst.operand2,anAst.operator end

      def evaluate(anAst) return anAst.operand1 + anAst.operand2 end

      anAst = parse("2 + 3")

      print(anAst) => 2 + 3

      printReversePolish(anAst) => 2 3 +

      evaluate(anAST) => 5

      and that is all there is (conceptually) to any langauge transform.
   4. Ruby "DSL" s allow people who haven't studied the fundamentals of computer science to think they are discovering something new and amazing. Yes, Ruby allows a somewhat seamless mixing of "pure" ruby and "dsl" code, but this is just as true for any langauge with a built in eval and clean syntax. (lisp, forth, even smalltalk). The built in "eval" takes care of the parsing (as long as the syntax does not vary too widely from ruby syntax)

To understand ruby "dsl" s, ignore all the hype (and the forthcoming stream of books explaining how ruby's 'dsl ability' is the next silver bullet that will save enterprise programmers , give their arid lives meaning, and solve world hunger), understand how interpreters work (read "The Essentials Of programming Languages") and meditate on these concepts in ruby - class eval, object_eval, method_missing. if you understand the basics of how languages work, ruby's "dsl" is some (very) old wine in some sparkling new bottles.
# The next time someone tries to tell you how he created this uber cool "dsl" in ruby AND that this takes some fundamentally new technique, laugh in his face.

Ok , biting your tongue and walking away quietly is almost as good.

DSLs like any other programming technique, when used appropriately gives some beneficial effects. But it is no panacea and a casual perusal of the net will reveal a bunch of atrocious dsls . Beware the hype and snake oil. [end Rant]

----------

Comments:
I agree with your main point but you've missed what makes people excited about Ruby DSLs. It's not that you can parse some text into an AST and then interpret the AST, it's that you can define a DSL *without* having to parse text into an AST.

Any wierdy-beardy Smalltalk or Lisp programmer will tell you that this has been around for a long, long time. In fact, most experienced programmers will tell you that defining languages in which to express the problem and it's solution is the very essence of programming.

Yes laugh in their faces. But point them at Abelson and Sussman's Structure and Interpretation of Computer Programs, Paul Graham's On Lisp or any number of books about Smalltalk or functional programming.
# posted by Anonymous : 5:02 PM

you say ,

"But point them at Abelson and Sussman's Structure and Interpretation of Computer Programs, Paul Graham's On Lisp or any number of books about Smalltalk or functional programming."

which part of "read 'essentials of programming languages' don't you understand?


you say
"t's not that you can parse some text into an AST and then interpret the AST, it's that you can define a DSL *without* having to parse text into an AST."

again which part of

'The built in "eval" takes care of the parsing (as long as the syntax does not vary too widely ffrom ruby syntax)" don't you understand?'

And of course you twist my words.
I said

"The next time someone tries to tell you how he created this uber cool "dsl" in ruby AND that this takes some fundamentally new technique, laugh in his face"

note the three letters in bold? the "AND"? hopefully you learned what "and" meant in primary school?

It is not the designing or use of dsls which warrant derision , but the claim that is a fundamentally new technique, which exposes ignorance of basic CS.

This is exactly the kind of clueless reaction I was warning readers against. Must be nice that you are protected by your anonymity form your own stupidity.

Read the damn post before posting vapid rebuttals.

Regards,
# posted by Ravi : 5:27 PM

I see that the usual anonymous idiots have already surfaced :-)

You are right to excoriate anyone who claims that ruby's dsl is a fundamentally new programming tecnique. It is no such thing, at least not to people who have had a basic computer science education, let alone any experience with lisp or smalltalk.

However, it may be "new" for someone whose only experience in programing is confined to the j2ee or dotNet stacks. If prisoners who've been clapped in irons appear bewildered or ecstatic on being released into the prison courtyard for some exercis in the sunlight, perhaps that is all right :-)

I think your problem is that you consistently overestimate the intelligence and programming skill of the average enterprise programmer. Herein the UK, the very best programmers don't work on banking software, they get hired by google. Expecting the average database drone to understand compilers and language processing and compiler theory may be too optimistic :-).Let them part with their money and buy the "Enterprise DSL With Ruby" style books. The book authors have to eat :-)

EOPL is an awesome book btw. Your reccomendation is well thought out. If there is one book that cuts through "dsl" hype (or any other buzzwords about languages), EOPL is it.
# posted by Joe Williams : 8:25 PM

Who is this Joe Williams, who derides the average enterprise programmer ? Provide a link please.Let me read your posts ! Kindly mention your worthiness before ridiculing others.
And yeah, Do you work for Google ? I am happy to stay anonymous as i neither work for google nor ridicule the average !!
# posted by Anonymous : 1:08 AM

dear Anonymous Idiot,

You might want to at least make sure your post makes sense before submitting it.

"I am happy to stay anonymous as i neither work for google nor ridicule the average"

This kind of claptrap as a substitute for logic almost guarantees you are a bad programmer my anonymous friend. If you want to stay anonymous, do so. That has nothing to do with "work for google" or "ridicule the average".

No field of work which requires application of logic, would hire you if the above is sample of your thinking. Read up on the "Ad Hominem" fallacy. Criticize what people say, not who they are.

As for me, I don't have a blog (so you can't "read my posts") or a web page of my own. Last I checked, neither was mandatory to surf the web (and post blog comments). And I am a scientist, not a programmer.

Now to my comments. I said "Expecting the average database drone to understand compilers and language processing and compiler theory may be too optimistic".

Now which part of this statement do you disagre with. Why? If you have a logical counter argument, (vs "Joe Williams doesn't have a blog nyah nyah") let us hear it.
# posted by Joe Williams : 7:53 AM

whoops.

This is degenerating into commenters calling each other names again.

"Anonymous" did post a reply

here is about the only publishable sentence

"The gist of what i said is that posting a comment under the garb of anonymity is no different from posting one with a non blogger name."

The rest came perilously close to insulting whole nations. ("Englishmen... ").

None of that on this blog please.

Anyway I am not very sure I agree with the idea that what id people chose is particularly important *as long as* they make sensible arguments.

There are good reasons to avoid a blogger id (or prefer "anonymous"). Having said that someone who puts his name on a comment (even without a blogger id) is preferred to one who stays 'anonymous', especially if he/she also writes to me offline or is otherwise known to me.


I had hoped that 'anonymous' would take up joe's challenge to identify precisely that he objected to in his *main* argument(== "you can't expect the average enterprise programmer to understand compiler theory" ) and kick start a good discussion.

I don't care (much) what id people use. I do care what they post here using that id makes logical sense and targets a good discussion.

oh well,
# posted by Ravi : 9:53 AM

Anonymous,

You are so cool and absolutely stupid !!

Why do you get so personal, when someone ridicules the average idiot. You stay anonymous, yet give clues to your self..
:) Bwaaahaahaa -)


"Never argue with an idiot. A third person can't make out the difference"
# posted by Anonymous : 10:20 PM

Joe said: "You are right to excoriate anyone who claims that ruby's dsl is a fundamentally new programming tecnique."

I'd maintain that Ravi is right to contradict or correct them.

As to 'excoriate'? Well, that's a question of politeness and decency.

I believe the tone of Ravi's post borders on smug. But since I am commenting on the style of the post and am engaging in a variety of 'ad hominem', I guess I am just about as guilty as he isI suppose I mention all this, because there's no way that I'd treat someone in the way that's being advocated, because I was once that way -- the wide-eyed convert. Not all of us have come to programming via academia or with any strong CS background. Some of us learn it later in our careers. But, since he qualifed it with [rant] I'll assume Ravi just wants to get it off his chest.

Anyways, the _content_ of Ravi's post is dead-on, and lucid. It boils down the essence of the metasystems out there, and I'll probably use the same wording when explaining the topic to others.

I do have an itch, which cannot be scratched by the typical CS/math analysis of the situation. That itch is the question: why do certain languages permit embeddable DSLs better than others.

For a very long while, I had accepted the conventional wisdom that ONLY very-simple syntactical systems with consistent and thorough paradigms could produce these internal DSLs. Smalltalk and Lisp are the poster children of this philosophy: lisp for its near-zero syntax to AST impedance, and Smalltalk for its consistency and access to the Compiler & Stack as objects.....

But then Ruby proved that it could be done too. And by 'done' I mean 'done easily' and in a way that seems to encourage programmers to use it. There's no better indicator to success than results: there are clearly more embeddable DSLs in the Ruby world than in a lot of the big enterprise languages.

So what is it about Ruby that permits this? It seems to take the opposite tact than a lisp; indeed a lisp purist would certainly disparage its philosophy: tons and tons of syntactic sugar. And yet it has gotten a lot of the same power. (don't get me wrong, when push comes to shove, Lisp will always when out in expressive power)

Just something to think about...

Daniel Eklund
# posted by Daniel Eklund : 12:52 AM

"Not all of us have come to programming via academia or with any strong CS background. "

I haven't anyway :-).

I majored (if spending four years running after girls and rarely attending class can be called "majored") in Industrial engineering and have no formal training in CS and am very much a self taught programmer.

Having said that, I get irritated when *basic* computer science is "rediscovered" and worse, repackaged as the next silver bullet(as is beginning to happen with ruby dsls).

"But, since he qualifed it with [rant] I'll assume Ravi just wants to get it off his chest"

Oh Absolutely. I was doing some technical oversight for a venture capitalist here and someone essentially repackaged html as a ruby "dsl" and wanted funding because of his "cutting edge technology." [hit head on wall]

Ruby *is* a marvellous language. "DSL" is a valuable technique to have in your arsenal.


But DSLs are NOT a "fundamentally new" technique. It just appears that way to ignoramuses.


"why do certain languages permit embeddable DSLs better than others"

I would say presence of eval + (relative) simplicity of syntax + the language would enable this. To make it *easy* though some part of the language substarte would need to be exposed in the language itself.


Ruby is very much a "son of smalltalk" and though the syntax is not as regular as that of smalltalk or lisp , the uniform use of objects throughout and modelling the language substrate as objects (ruby's weird notion of "metaclass" notwithstanding) enables the meta hackery, imo.

Thanks Daniel for a lucid comment.

不太理解为什么一定要贴近自然语言。
自然语言的二义性有些不是在语法和语义的层面能够解决的。语言的表达能力（对一个东西能不能表达，花多大代价表达）太强未必是好事情。
对这类扩展方式实现dsl的另一个想法，我估计即使这个玩意儿会流行开来(我不是抱很大希望)，也会经历一个优胜劣汰的标准化过程。语言设计的进入成本太低会直接抬高沟通成本