When it comes to web apps, the Ruby community gets it. Rack is a great example of this. What is it? In a nutshell:

Any Ruby web framework now jives great with any Ruby web server. I can scale down from a fully load-balanced production system, to a local Apache, to a quick in-process web server without even thinking about it. I can also add components to the HTTP chain in a snap. Thanks to Rack’s amazing simplicity, it took less than one year to move from Rack 1.0 to widespread Rack compliance. As Sam Ruby put it: I love it when a plan comes together.

.http://www.flickr.com/photos/russmorris/202752043/

Kinetic’s web servers run Phusion Passenger, which is Rack-compliant. Since both Rails and Sinatra are Rack-compliant, I can hook either web framework up to Passenger. Today, my choice is Sinatra for its bare-bones simplicity…perfect for prototyping! But let’s say, six months from now, I also hop on the Rails bandwagon. Thanks to Rack, that hop will be painless.

Or maybe, just maybe, I decide to swap out Phusion Passenger for Unicorn. Or add a caching server to speed up the site. With Rack, I can freely mix-and-match. I love it, it’s plain brilliant.

As a consultant, I charged my clients $60-$100 an hour to build software on their terms. I was a mercenary. I traded my finite time for money. It was deeply unfulfilling. On June 1, 2005, I notified my clients that I would no longer take on new business.

As a software artisan, I can bring my vision to life, in congruence with my values. I can craft software with limitless leverage. The deep joy I experience when I bring beautiful code to life is profound and deeply satisfying.

Chris Dixon neatly captures the “builder” philosophy:

Tim O’Reilly poses a question every entrepreneur and investor should consider: are you creating more value for others than you capture for yourself? …I think of people who aim to create more value than they capture as “builders” and people who don’t as “extractors.” Most entrepreneurs are natural-born builders. They want to create something from nothing and are happy to see the benefits of their labor spill over to others.

http://www.flickr.com/photos/nodx2/287404267/

I try to inch forward a bit each day. Steps so far towards getting the basic prototype working:

• Setup of a Ruby webserver
• Installed Graphviz on my webserver
• Developed a basic set of HTML templates for the initial primitive prototype
• Created a rudimentary landing page
• Re-familiarized myself with the core Ruby language
• Learned the Sinatra web framework

Photo by flickr user used under a Creative Commons license.

Now that I’ve decided Kinetic should have a web interface, the next question is: What features should it have?

I’m learning towards just creating a Minimum Viable Product. I’ll be able to get early feedback on my ideas. Plus it’ll force me to resist my natural tendency to be a perfectionist. Ok, so what might a bare-bones, no-fluff app look like?

1. The user enter texts which describes a diagram.
2. Kinetic then generates the diagram (in PNG image format) from that textual description.
3. Kinetic then displays the generated PNG diagram to the user.

That’s as basic as it can get, and it all fits on a single web page:

So that’s my plan for the next several weeks–to get version 0.0001 up-and-running. It’ll be pathetic and primitive, but at least it’ll be something. What do you think?

Call me fickle if you want. I’ll shift Kinetic priorities from time to time, as I gain new insights.

A month ago, top items on my TODO were:

1. Define the language syntax
2. Complete the text file parser
3. Complete the diagram generator

See the problem? I’ll never have enough time to do it all myself. And I’m not willing to steal from time I’d rather spend with family and friends.

http://www.flickr.com/photos/hockadilly/446538765/sizes/l/

Since starting this blog two weeks ago, I’m realizing that I need to re-focus my efforts. I think my time will be better spent on setting up the infrastructure that will allow other folks to help. My top TODO items are now:

1. Refine the architecture diagram, to focus on interfaces and not components. I’d like to outsource some component development work to freelance developers. In order for that to succeed, I’ll need to be excrutiatingly clear when I communicate the architectural interfaces.
2. Launch a pre-ALPHA Kinetic site, and deploy the simplest primitive version that demonstrates end-to-end functionality. I’d like to get early feedback on the basic viability of the idea. Will other folks find it usable? Will other folks find it useful?
3. Keep posting to this blog every workday. Bounced ideas are the best ideas.

I hate shaving the yak.

Saturday, 9:30am: I decide to test-drive MarsEdit‘s offline blog editing features. Well the latest MarsEdit only works on Mac OSX Snow Leopard. I figure an OS upgrade is long overdue anyhow, so I reluctantly upgrade. Post-reboot, I wait for an additional 1.12GB in MacOS software updates to complete. I discover that the Ruby version on my machine is no longer 1.9.1; it has somehow reverted to 1.8.6. I use MacPorts to manage the development tool installations, but the Snow Leopard version of MacPorts needs Xcode 3.2.1, not the 3.1.2 installed with Snow Leopard. Ugh. It takes me awhile but I finally hunt down a copy of Xcode 3.2.1, which makes it possible to re-install MacPorts. I then re-install all the ports which were working before, including Ruby 1.91.

Saturday, 12:20pm: I re-learn an old lesson:

http://www.flickr.com/photos/revcyborg/22883042/

A kick of inspiration for the coming weekend:

Victory by Herbert Kauffman

You are the Man who used to boast
that you’d achieve the uttermost,
some day.

You merely wished to show,
to demonstrate how much you know
and prove the distance you can go.

Another year we’ve just passed through.
What new ideas came to you?
How many big things did you do?

Time left twelve fresh months in your care
how many of them did you share
with opportunity and dare
again where you so often missed?

We do not find you on the list of makers good.
explain the fact!
Ah No, ‘Twas not the chance you lacked!
As usual – you failed to act!

What’s the spark behind Kinetic? One day last June, while working on an architecture diagram using Microsoft Visio, several of my neurons misfired, and it’s an itch that I’ve been trying to scratch ever since. The idea is simple: How can we use words to describe complex diagrams? For example, is it possible to have a tool that will allow me to update an organizational chart or a network diagram just by changing a few lines of text?

Granted, this isn’t exactly a new idea. Google and you’ll find dozens of programs. But I’ve yet to find one that satisfies my twin requirements of: 1) usability and 2) flexibility.

1. The tool needs to be simple to use. The litmus test is always, “Yeah, but will my grandma be able to use it?” It should be dead simple to update the organization chart.  AT&T Research’s Graphviz rocks, but in my opinion, the syntax might be a bit too arcane for the average user.
2. The tool needs to be flexible. Can the software handle a variety of diagrams? Ideally, I’d like to be able to use the same software to generate both organizational charts as well as network diagrams. Tools such as Diagrammr are limited in the types of diagrams they can generate.

As it turns out, meeting both requirements is a hard problem. It’s a design challenge, a tug-of-war. Make the tool flexible, and usability suffers. And vice versa.

http://www.flickr.com/photos/kitone/2478029256/

I know it’s a challenging problem. The scary thing is–it might not even be solvable using current technologies. And that’s the very reason I’m drawn to it. Paul Graham calls this “running upstairs.”

Use difficulty as a guide not just in selecting the overall aim of your company, but also at decision points along the way. At Viaweb one of our rules of thumb was run upstairs. Suppose you are a little, nimble guy being chased by a big, fat, bully. You open a door and find yourself in a staircase. Do you go up or down? I say up. The bully can probably run downstairs as fast as you can. Going upstairs his bulk will be more of a disadvantage. Running upstairs is hard for you but even harder for him.

What this meant in practice was that we deliberately sought hard problems. If there were two features we could add to our software, both equally valuable in proportion to their difficulty, we’d always take the harder one. Not just because it was more valuable, but because it was harder. We delighted in forcing bigger, slower competitors to follow us over difficult ground. Like guerillas, startups prefer the difficult terrain of the mountains, where the troops of the central government can’t follow. I can remember times when we were just exhausted after wrestling all day with some horrible technical problem. And I’d be delighted, because something that was hard for us would be impossible for our competitors.

Thanks for joining me on this stair-run.

I met my real-life American Idol yesterday–and I even snagged his autograph!

Ok, so Edward Tufte autographs his books for everyone who attends his awesome “Presenting Data and Information” course, but that’s besides the point. And the $380 course fee may sound expensive for a 6-hour seminar, but I think it’s actually a tremendous bargain. In addition to the privilege of hearing Edward Tufte in person, everyone who attends the course also receives four books.

Each book normally costs $50 or so, although you can get the full set for $160 from edwardtufte.com.

• The Visual Display of Quantitative Information
• Envisioning Information
• Visual Explanations
• Beautiful Evidence

I first discovered these books in 2002. By chance, I came across Envisioning Information at the Montclair Barnes & Noble bookstore. And as soon as I started reading the first couple pages, I was immediately hooked. Smitten might be a better word. I remember that night, just deciding to take the plunge (damn the torpedoes!) and ordering the three books off of Amazon.com, even though it cost what-was-then-a-staggering $150. (The fourth book, Beautiful Evidence, wasn’t published until 2006). And when they finally arrived, I devoured them. Pored over every sidenote. Studied every single diagram. Pondered every single one of Tufte’s design principles. I know this may sound strange, but it wouldn’t be hyperbole to describe those next two weeks as bliss.

Over the years, I’ve fully re-read each book at least twice, and still enjoy scanning them for inspiration. I still regard them as one of my smartest purchases ever.

Tufte writes, designs, and publishes these books himself, and the production quality is top-notch. The 3-D popup pyramid on page 16 of Envisioning Information demonstrates the same visualization technique used in the 1570 edition of Euclid’s Elements. Guess how much it costs to include this pop-up pyramid in every book? Tufte told us, it’s an extra 32 cents per printing. Passion doesn’t always come cheap.

Now, if you:

1. are an information architect
2. want to be a graphic designer
3. study financial charts
4. deal with information presentation in any way (and in today’s world, who doesn’t?)

then do yourself a favor and buy the full set today. You don’t have to take my word for it. Just check out the reviews on Amazon (“Best 100 books of the 20th century.”) or Tufte’s site (“If this book were a house, it would have been designed by Frank Lloyd Wright”). The high praise is deserved.

These books form the foundation of my visual design thinking. I respect the Tufte’s depth of his wisdom and his clear way of thinking. I love that his integrated design principles are ethical and respectful of the viewers’ intelligence. In future blog posts, I’ll cover how I try to apply them as part of the Kinetic experiment.

As if I haven’t gushed enough, I’ll close out this post with yet another reason why I respect Edward Tufte so fully:

But he is no stranger to iffy new enterprises. Tufte, who taught at Princeton from 1967 to 1977 and Yale from 1977 to 1999, entered publishing in 1982, when an Ivy League university press agreed to publish his first book, then told him that someone else would design it and that the book would sell for more than $64. Tufte had other ideas. He wanted control of the design, and a lower price.

Tufte enlisted the help of a seasoned book designer and self-published “The Visual Display of Quantitative Information” — and priced it at $32. A second mortgage on his house paid for the venture, even though the best interest rate he could get was 18.7% — a particular horror for a trained statistician, but the strongest possible incentive to market fiercely.

Yesterday, I quoted Larry Ellison saying that software is the “only industry that’s more fashion-driven than women’s fashion.”

Peek at the current issue of any IT management magazine, and you’ll probably find an article or two covering “cloud computing” or “virtualization.” These disruptive technologies are deservedly topics du jour. I see Amazon’s EC2 as a bona-fide game-changer. I can’t live without VMware Fusion on my MacBook Pro.

However, look past the allure…these “new and hot” technologies will not fulfill their promise for every business. I think this is what Larry Ellison meant by the industry being “fashion-driven.” Many businesses will feel the push to adopt these “fashionable” technologies.

In the world of personal finance, they affectionately call it:

affluenza, n. 1. The bloated, sluggish and unfulfilled feeling that results from efforts to keep up with the Joneses. 2. An epidemic of stress, overwork, waste and indebtedness caused by the pursuit of the American Dream. 3. An unsustainable addiction to economic growth.

The problem with “keeping up with the Joneses” is that the Joneses are quickly going broke. Likewise, “technology affluenza” (my term for the uncritical adoption of “fashionable” technologies) is harmful when it doesn’t contribute to business value.

As I’m developing Kinetic, I’m often tempted to incorporate “fashionable” technologies. For me, these “Sirens of Technology Affluenza” come in varied guises: Ajax, Maven, Amazon EC2, Nutch and Hadoop, CouchDB, Google Wave, and on and on…

http://www.flickr.com/photos/gael_lin/2670110513/

For example: I’m excited about Google Wave. Not just because of the handsome instant-messaging/email/collaboration client; what really gets my creative juices going are the Google Wave APIs and the underlying Google Wave Federation Protocol.

Wow! If I implement the Google Wave APIs, multiple users can edit the same Kinetic diagram at the same time! Sally can format a box here, while Joseph can add a line there–and they’ll be able to see each other’s edits immediately! It’ll be so cool!

Last week, I officially purged this task from my TODO list.

I now realize: Core functionality must come first. I can’t (yet) justify the business value for using the Google Wave API. Respond to the other Sirens with the same reasoning:  “Wouldn’t it be cool to–”

• –use Ajax to make the website feel more responsive?” Only if the business value justifies it.
• –deploy Amazon EC2 instances, so that my website can scale to a million users?” Only if the business value justifies it.
• –use CouchDB to store my diagram data?” Only if the business value justifies it.

I’m naturally susceptible to “technology affluenza.” So, like Odysseus, I’ll need to plug my ears to the siren calls.

(Did you enjoy this blog post? I’d love to hear your feedback.)

Recently I found myself nodding to a Larry Ellison quote. Yes, that Larry Ellison. The Oracle CEO. The one in the Silicon Valley joke:

Q: What’s the difference between God and Larry Ellison?
A: God doesn’t think he’s Larry Ellison.

Here’s what he said:

I have to say that the only way I can understand the computer industry–the computer industry is the only industry that’s more fashion-driven than women’s fashion. I was reading W magazine and I found that orange is the new pink–and cloud is the new SaaS, or the new virtualization–I mean, it is the most nonsensical–I read these articles and I have NO idea what–and maybe I’m an idiot–but I have NO idea what anyone’s talking about! It’s really complete gibberish.

All of my co-workers know: Without a whiteboard marker, I’m like a one-legged man in an ass-kicking contest.

I have to scribble in order to get my ideas across.

On Monday, I had an aha! moment: Even moreso than readership or investment funding–this blog needs diagrams! It makes sense, right?

So, I picked up a Wacom Bamboo tablet. My sketches look clumsy and I still can’t get pressure-sensitivity to work with the MacOS version of GIMP (even after installing the latest version of the X11). Other than that, it seems to work fine.

So look for my clumsy sketches in future blog posts. In the meantime, I’ll start getting ready for the new readers and eager VC investors.

I consider Kinetic “an experiment” because:

One, I have *no clue* whether my half-baked ideas are even feasible (given my grasp of current technologies), and the only way to find out is to actually try them out.

http://www.flickr.com/photos/clivrow/2733115397/

And two: I continually tweak/mash/purge/brainstorm/test new ideas as I clumsily evolve the project towards something potentially useful.

Joel Spolsky calls this “Fire and Motion“:

When I was an Israeli paratrooper a general stopped by to give us a little speech about strategy. In infantry battles, he told us, there is only one strategy: Fire and Motion. You move towards the enemy while firing your weapon…It took me another fifteen years to realize that the principle of Fire and Motion is how you get things done in life. You have to move forward a little bit, every day. It doesn’t matter if your code is lame and buggy and nobody wants it. If you are moving forward, writing code and fixing bugs constantly, time is on your side.

Expect Kinetic to be under constant flux. I have a directory on my MacBook Pro desktop, named “Language syntax ideas.” I’ll try out the promising ideas, and discard those that make the product confusing.

Ready–fire–aim.

Yesterday, I touched on two strategies for developing domain-specific language (DSL) parsers: Hardcore DSLs may warrant using a full-fledged parser generator (such as JavaCC or ANTLR) with support for BNF grammar rules. For lighter-weight DSLs, regex hackery and substring functions often suffice to get the job done.

My challenge is–Kinetic’s language waffles somewhere in the middle of the “hardcore DSL” vs. “lightweight DSL” spectrum. Language design is largely about making hard choices about your language’s limitations. Right now, Kinetic is far from “hardcore.” It doesn’t support if/else/while control statements, exception handling, class definitions, and many amenities found in modern programming languages. And I’m ok with that: The target user for Kinetic isn’t the ninja coder; rather, Kinetic is intended to appeal to less-technical folks who want an easy way to describe complex diagrams.

But my ego won’t allow me to classify it as a “lightweight DSL” either. The current Kinetic language spec supports macro definitions, variable assignments, file references, embedded scripting, and a fairly expressive node-selection syntax. And new language features will be added as the project evolves.

Lucky for me–this middle-ground happens to be Ragel‘s sweet spot.

So what exactly does Ragel do? It’s a tool to describe finite state automata (FSA). Basically, an FSA defines the DSL parser’s behavior as it encounters each and every character in the input text. An FSA describes “states” and “transitions”:

http://www.complang.org/ragel/number_lg.png

For example, if a parser sees a “a” followed by a “d” followed by another “d”, it can conclude that it saw that word “add.” As the parser hops from “state” to “state” (from letter to letter), we can instruct it to perform special actions during those “state transitions”. For example, the parser can increment a counter variable every time it transitions from “a” to “d”.

The elegance of Ragel actually makes it fun to create DSL parsers. Remember how earlier I noted that “hardcore parser-generators” offer the benefit of BNF notation? Well, Ragel’s syntax isn’t quite as powerful as BNF, but it gets us mostly there. These Ragel rules are more maintainable than regexes. And, since it turns out that every regex is in fact an FSA (every regex engine starts by converting a regex to an FSA), if you’re already familiar with regexes, you’ll find it incredibly easy to pick up Ragel’s slick syntax.

Here’s some Kinetic language grammar defined using Ragel (compare to BNF):

unquoted_string        = ([^"]+);
quoted_string          = (["] unquoted_string ["] | (["] ["]));
node                   = (upper (('_' | alpha | digit)*?))
tagname                = (lower+ ('_' | lower)*)
variable_name          = (alpha (('_' | alpha | digit)*?))

Then, Ragel performs its magic –voila! — and generates code for your custom DSL parser. Peek at the Ragel-generated code and you’ll see thousands upon thousands of lines like this:

self._parser_actions = [
 0, 1, 0, 1, 1, 1, 2, 1,
 2, 6, 22, 2, 7, 4, 2, 7,
 23, 2, 16, 24, 2, 19, 23, 2,
 19, 24, 2, 20, 24, 2, 23, 0,
 3, 1, 6, 22, 3, 6, 1, 22,
 3, 8, 13, 23, 3, 10, 18, 23,
 3, 10, 18, 24, 3, 11, 15, 23]

And that, dear friend, is precisely why you never want to write a finite state automata by hand.

The generated Ruby/Java/C++ parser code is completely standalone. In other words, once you use Ragel to generate your parser, there’s no need to include Ragel libraries in your code distribution. This is simply brilliant.

If you’ve read this far, you may also want to check out Ragel’s other features. Frankly I think Ragel is underrated, and is worth adding to your quiver.

Thanks Adrian Thurston, for your gift to us. Ragel rocks!

Zed Shaw is spot-on: “The Ragel State Machine Compiler is one kick-ass piece of software.”

As part of Kinetic development, I need a way to parse several custom-designed languages, each with its custom grammar. The term popularized by Martin Fowler is DSLs, short for “domain-specific languages.”

http://www.flickr.com/photos/dilaudid/278649026/

In the past, when faced with this task of parsing DSLs, I took one of two approaches:

Approach #1: For parsing complex DSLs, the typical starting point, and the one taught in college compiler design courses, is to first define the language’s custom syntax via a comprehensive set of BNF grammar rules.

In junior high, I was one of those kids who actually enjoyed diagramming sentences. Well, defining BNF grammars for DSLs requires a similar masochistic mindset. Parser generators such as ANTLR allow these BNF grammars to be defined, and custom logic to be executed when the constituent words/tokens are “found” in the input text.

I’m over-simplifying the process of course, as any CS major who wasted away Memorial Day weekend staring at lex/yacc debugging messages can attest. And that is in fact the primary drawback of this approach of using parser generators: it’s usually too heavy-handed unless you really want to design a fairly rich custom DSL.

It’s no surprise that parsers and compilers are written by a relatively-tiny and arcane community of computer engineers. That business developer you’re paying $90/hour for? Chances are, he or she has never written a custom DSL parser.

Thankfully, I’ve only had to go down this road twice in my programming career (both for personal projects). I absolutely love the power of this approach–using full-fledged parser generator tools gives me the freedom to define complex DSLs–but the price in terms of software complexity and time is difficult to justify for most projects.

Approach #2: For that reason, I generally opt to keep my DSLs simple. Simpler DSLs means simpler parsing. This allows me to just cobble together straightforward text-parsing routines. These routines rely on basic multi-stage regular expression (regex) matchers and core string-manipulation functions. Nothing fancy, it’s anything any engineer can do.

Yes, it’s simplistic and dirty with the taint of hackiness. But–more importantly, it gets the job done every time.

The ugly catch? Debugging and maintaining these simplistic parsers is no fun. There’s a pearl of programming wisdom:

Some people, when confronted with a problem, think “I know, I’ll use regular expressions.” Now they have two problems.”

Step away from several complicated regexes, and try patching that code a few years later. I’ve had to do that on several occasions, and each time, I growled at my former self. I would’ve had more luck trying to decipher the green letters in the Matrix. Add a pair of parentheses to an existing regex, and you risk breaking your back-references (Ruby 1.9′s support of named groups only helps very slightly here). The pain of maintaining regexes and convoluted string-manipulation logic generally leads to parsing code collecting more than its fair share of dust.

Which is why I was thrilled to discover Ragel.

(to be continued tomorrow…)

Ralph Waldo Emerson quipped: “Who you are speaks so loudly I can’t hear what you’re saying.” Let it marinate for a bit. I think it’s quite profound.

All human creations are unwitting reflections of their creators. And so, like every other project I touch, Kinetic will necessarily reflect my values and ideals.

At first, I considered using today’s post to outline this project’s “core values.” But I then realized it wasn’t necessary: After all, the project’s values will eventually find a way to “speak loudly” on their own.

Last week, my friend Furqan and I had an insight: Many software developers build fragile systems, because they tend to focus on technical components rather than the functional interfaces between those technical components. For example, a developer may devote much of his/her time on the code for the billing adapater implementation, without spending adequate time to define/re-factor a robust billing interface.

Off the cuff, I sketched a simple whiteboard diagram: Two small boxes inside of a larger box.

My theory is that most people will gravitate towards seeing the two small boxes; in other words, they focus on the “figure” (in the artistic sense of “figure vs. ground”) However, the proper focus should be on the empty space (the “ground”) between the small boxes and the larger box. As humans, we’ve evolved to focus on concrete visual images which make up the “figure.” As long-term system-builders, we need to fight this “concrete thinking bias” and instead focus our minds-eye on the “ground” instead.

Agile software development requires continuous software refactoring. Functional interface design facilitates refactoring and help build robust systems. Unfortunately, clean functional interface design is often neglected in order to “just get the code working.” Treated as a luxury, it is eagerly sacrified at the altar of expediency.

Imagine my smug self-satisfaction — how fun to criticize others for the fundamental flaw in their conceptual thinking!

Funny how life finds a way to hold up a mirror to your face:

This morning, I reviewed the current high-level architectural diagram for Kinetic. I’m embarassed to admit that it also focuses too heavily on components, and not interfaces. The Parser/Interpreter reads data from an Intermediate Data Model. The Diagram Generator generates a PDF. Very concrete component definitions. Very fuzzy interface definitions. In a nutshell, poor design. It lacks the rigor of deliberate functional thinking.

Please excuse me while I wipe the egg off my face, thanks.

http://www.flickr.com/photos/melaniedefazio/3444863022/