I am outraged at the SF Muni and their adoption of the TransLink card.  Soon, there are certain bus lines that will only accept the TransLink card during rush hour... you won't be able to ride the bus except with one of these evil devices.

So, what's my outrage?

I spent most of today trying to obtain a TrankLink card for my kids.  I read on the TransLink site that the youth cards had to be obtained in person rather than at the normal retail outlets.  So, I took my kids downtown figuring I could buy a card at a Muni station, but no dice.  Most of the folks in the Muni stations had no clue.  Finally, somebody pointed me to Golden Gate Transit in the Ferry Building, so off we went to the Ferry Building.

We find the right booth to buy the pass.  Now, my kids are in kindergarten.  There's no question that they are youths and are elegible for youth discounts.  In fact, most Muni drivers say, "These kids look too young to pay."  My response is, "Muni rules say that 5 year olds pay and we follow rules in our family."

So, I ask for 2 youth passes.  The woman behind the counter says, "give me their birth certificates and we'll give you the youth passes."  I say, "that's a joke, right?  Look at them, do they look anywhere near 16 or 18 or whatever the cutoff age is?"  She responds, "Our policy is that we only issue youth cards with a birth certificate."

We've had to show the kids' birth certificates for their social security cards and for entrance to school.  We did not have to produce birth certificates for library cards, health insurance, so I'm wondering what is going on here.  I ask for a supervisor.

I talk to the supervisor and he tells me, "It is our policy that you need a birth certificate to obtain a youth TransLink card."  Me: "Why?  Do these kids look like they're too old for the card?"  Him: "We bond the birth certificate to the card to avoid fraud."  Me: "You what?"  Him: "We make sure that the discount card is associated with the person."  Me: "So, you're telling me that in order to get a discount, you guys are going to track the movements of my kids throughout the TransLink system?  How is this supposed to reduce fraud?"  Him: "That's our policy, sir."  Me: "Your policy is Orwellian and there's no way I'm letting you associate my kids' public transportation usage with their birth certificate.  Instead of trying to buy cards for the kids, I'm going to burn mine."

So, we walk away.  The supervisor comes running after me.  "Sir, your kids look young enough that they don't need to pay a fair anyway."  Me: "First, they have to pay on the SF Muni and we follow the rules so we pay.  Second, giving me a 'free-be' today don't mitigate your Big Brother policy of tracking my kids where-abouts."  Him: "I'm just trying to be helpful."  Me: "If you want to be helpful, go back to your boss and explain to him that this is the United States and we move freely in this country."

My kids asked me why I was cranky.  I said, "Because the bad people want to know where you go."  Kids: "We're not going anywhere bad, so why don't we want people to know where we've gone?"  Me: "It is our right as Americans to come and go as we please.  It's our right to be private.  But more importantly, historically, once the government starts tracking their citizens, especially their children, they will find a way to use the information against you.  It's not that we have anything to hide, it's that any time people want to always know, they are up to no good... they are bad people."

So, we walk through the Ferry Building and I see a guy with an EFF hat.  Now, this is San Francisco, so it's not unexpected to see some guy sporting an EFF hat.  I go up to him and say, "Are you a member of the EFF?  I was one of the first donors to the EFF so don't get freaked."  He says, "Yes."  I say, "The TransLink folks want my kids' birth certificates in order to get a youth TransLink card and they bond the birth certificate to the card so they track the kid's movement throughout their system."  His jaw dropped to the ground.  We shared a brief moment of liberal outrage and my kids and I went on our way.

When I got home, I checked the TransLink site.  There is no mention of requiring a birth certificate to obtain a youth pass.  I called the TransLink customer service line and they did verify that a birth certificate is required.

First, TransLink is hiding the fact that a birth certificate is required.  It's not on their public web site although it is mentioned in a press release.  Why are they hiding this information?  Do they think that if they don't publish it on their web site that somehow it'll make it okay?

Second, TransLink is managed by the Metropolitan Transportation Commission, but this is not mentioned anywhere on the TransLink web site and is only mentioned once in the TransLink cardholder "license agreement".  Why is the MTC hiding?

Third, if the MTC is a governmental body, where was the public hearing and the public comment request relating to requiring a birth certificate and bonding that information to youth passes?

Forth, what great problem are they looking to solve by associating a card with a person?  What kind of failures is the current youth (and elderly) pass mechanism (a different colored pass) justify bonding my child's birth certificate to their transit card and track my children's comings and goings?

So, I am outraged.  I will burn my TransLink card and until this Orwellian, KGB-esq policy of requiring a birth certificate to get a youth TransLink, I will not in any way participate in the TransLink system.  I urge you to do the same.

Back in June, I started chatting about Goat Rodeo: a highly scalable mechanism for building distributed applications.  My first set of concepts for Goat Rodeo were wrong, most notably trying to do distributed Software Transactional Memory.  I've spent the last bunch of months revising the concept and code for Goat Rodeo... and today, I'm excitied to announce the 0.1 alpha code for Goat Rodeo.

First, some background on the problem:

Far and away the most common way of persisting and calculating state is in a relational database (RDBMS).  RDBMSs are awesome creatures.  They sit on top of some excellent and well understood mathematics: set theory.  They have well known and well understood concurrency mechanisms: transactions.  They have been designed, built, tested, and optimized over the last generation. RDBMSs offer a simple set of commands (SELECT, DELETE, INSERT, UPDATE) as well as a generally human understandable set of semantics: people understand that RDBMSs are a sets of things and there are simple ways to ask about these sets.  RDBMSs have evolved along with ERP systems and have evolved to meet the needs of these systems.

However, there are well known things that RDBMSs don't do well that include tree structures (yeah, Oracle and others have extensions for tree walks, but nothing is part of the SQL spec and the performance of these extensions is not always the same as other models: a tree-walk in an RDBMS costs O(log n) for each node where a tree walk in an OO system costs O(1)).  Social networks/social graphs are another place where RDBMSs do not excel.

Let's dive down into this.

A naive implementation of a social messaging site runs something like these tables:

• Users(id, name, password)
• Friends(owner, friend)
• Messages(id, poster, content, date)

So, if we wanted to calculate the timeline for a given user at a given instant, the query would look like:

SELECT messages.* FROM messages, friends WHERE friends.owner = current_user AND messages.poster = friends.friend ORDER BY messages.date DESC LIMIT 20

Assuming we've got indexes on friends.owner, messages.poster and messages.date, the query still results in O(n log n) where n is the aggregate number of messages posted.  This is non-trivial and if you follow someone who has posted 20,000 messages, then log n cost becomes non-trivial.

Basically, each time a client asks for the latest timeline, you've got an O(n log n) operation to determine state.  This doesn't scale.

The first obvious response to the issue is caching (capturing the state beyond the duration of a short-lived session).  I'm going to skip caching for a moment and do a more sophisticated implementation of timelines so wecan get better performance.

Let's create a mailbox table.  Each time someone publishes a message, a reference to that message will be put in a Mailbox(owner, message, date) table and we'll create an index on the table: (owner, date DESC)

This changes the query to:

SELECT messages.* FROM messages, mailbox WHERE mailbox.owner = current_user AND messages.id = mailbox.message ORDER BY mailbox.date DESC LIMIT 20

Depending on your RDBMS, you will wind up with an O(log n) operation.  You find the newest mailbox entry by user (O(log n)) and do an index walk until you've found 20 entries (I'm putting aside the fact that looking up the 20 messages is an O(n log n) operation because 20 is a small number and the messages will likely be in the database's cache... this operation is going to be fast.)

I'm going to sidetrack for a moment.  I had the pleasure of talking over a few beers at a baseball game with one of the senior engineers at Facebook. We were talking about Facebook's scaling success.  His comment was that it was successful but very expensive.  If there were more than 3% cache misses from MySQL queries, the system would back up.  If they got more than 2% cache misses from the memcached stuff in front of their MySQL servers the system would back up.  So, basically Facebook has 195% of their data in RAM.

The net is that O(log n) is only going to work if you've got your entire index in the cache of your RDBMS.  Even a dozen disk reads is going to turn a 10ms query into a 250ms query and if you've got 1,000 users asking for a status update, you'll wind up with disk thrashing and ultimately you will not be able to satisfy all of those requests.

Let's make our discussion more concrete.  I'm assuming that an social networking instance will support 25,000 users.  On average, a user will follow 100 people (100x fan-out of messages).  Users will post one message every 30 minutes (48 messages a day).  The day lasts 10 hours (this is a reasonable approximation for peakiness... basically, you're compressing 48 message sends in to a 10 hour period).  There are 300 days in a year.  These numbers are averages and there will be some folks who are above average in terms of fan out (the CEO will have a 25,000x fan out) and some folks are above average in number of messages per day.

So, that means that each year, there will be 36,000M (36B) mailbox entries. If each entry costs us 16 bytes of RAM for index purposes, that means we're at 576B bytes of index.  There's no way that amount of index will fit in RAM.  So, what happens if the average messages/day drops to 1, you're still looking at 10GB of index.   Alternatively, you could purge messages after 3 weeks or limit timelines to a certain number of messages.  That's not unreasonable, but it's also adding a constraint to the system to deal with limitations of the RDBMS.  There are other alternatives.

Let me talk memcached for a minute.  In my opinion, memcached means that you have a failed design.  Memcached means abandoning all the awesome things that you get with an RDBMS: a mathematical model, a concurrency/transactional model, durability guarantees, etc.

But, we could move our state from the calculate-on-demand model of the RDBMS to the a calculate once and cache model using memcached.  This means that you only take the nasty hits if the cache is not valid.  Putting aside the cost of cache invalidation (I haven't covered the costs of updates in this discussion because there's no need to go there... the implementation failures can be demonstrated with just reads), if you have a simple cache invalidation scheme, most of the cache entries will not survive for 15 minutes (I can go through the math, but I'm going to leave this one to the reader).  You risk cache stampedes (more than 1 process rebuilding the cache entry).  Basically, the naive memcached implementation buys you a little bit of head room over the naive (non-mailbox) approach.  In order to get more than 5x or so improvement (something that will serve a few thousand rather than a few hundred users), you need to manipulate the cache entries inserting/deleting individual messages.

The above paragraph in fact leads us in the direction of a better answer.

But first, let me state that I have proven that an RDBMS cannot be the sole locus of state for a social messaging site that services more than a few hundred users.  Period.  We must move state somewhere else and manage the cached state manually rather than with queries and indexes. Let's walk through a design that gives us a concurrency model as well as the performance we want.

Imagine a model where you interact with a User with a limited set of (asynchronous) messages:

• add/remove friend
• add message to timeline
• post message (the user has created a message and it needs to be processed)
• get current timeline (with offsets and number of entries)

These are the basic messages needed to implement a social messaging site. If we guaranty that a User will only process 1 message at a time, we have a concurrency model.  It's simple and simple is good.  We have not defined how/where Users store there state (it could be on a filesystem, in an RDBMS, in a NoSQL store, who knows).  But we can say that adding a message is an O(1) operation (prepending to the head of a singly linked list).  Each User can have a caching policy (and that caching policy could be dynamic based on the access characteristics for the User).  The sender of the message doesn't block on the processing of the message (although the get current timeline message will have an asynchronous response that the sender will likely block on).

We have changed our abstraction from one where all data (tables and indexes) are created equal to one where certain data structures are more prominent (User and Message) than others (mailbox, friends).

We have lost something: transactions.  In this model, if I add Dick as a friend, I am not guaranteed that I will receive Dick's next update... it may take time for the messages to propagate to Dick's User and his Message may be sent before the "add friend" message gets to him.  In the case of a financial transaction, this would be fatal.  In the case of social networking, this is a perfectly reasonable trade-off.

One of the ways that RDBMSs get performance (and the way products like Oracle distinguish themselves from the likes of MySQL) is the ability to cache optimized query plans, cache the right data, and invalidate the right caches at the right time.  The same requirements are going to come up in a social networked application.

So far, I've described an Actor based system... and there are a plethora of Actor-based systems around.  Erlang is the the best system for distributed Actors.  Akka is the best Scala Actor implementation around.  But Actors provide a single service: ordered messaging.  The additional services that are needed to create scalable social systems are:

• A unified transaction model such that different components in the system have a single mechanism to engage in transactions;
• A simple interface to extract data from the system without being and Actor in the system;
• A scalable storage model that provides transactions (integrated with the system's transaction mechanism) and a richer data model than key-value pairs.

Additionally, the system should behave the same way when everything is running in a single JVM or distributed across the network.  The application-level code should not know or be different if the resources are local or remote.  Code should go from the developer's laptop to the production system unchanged.  Additionally, the business logic in the code should be testable without having to set up a large number of different software systems.

Goat Rodeo acheives all the above goals... plus one additional goal: the single machine implementation is open source under an Apache 2.0 license.  The scalable multi-machine version is closed source.  What this means if if you've got a site that's going to run on a single system, you can build it with Goat Rodeo.  If you become the next Twitter of FourSquare, you can rest assured that your system will scale just by adding money.  The benefit to the open source community is that development is based on money rather than the goodness of someone's heart.  Anyway, back to the bits and bytes.

Goat Rodeo insures, at the compiler level, that all messages can be JSON serialized and distributed across the network.  Anything in Goat Rodeo that could be serialized must be a subclass of the QBase trait.  Goat Rodeo includes a compiler plugin that insures that all classes that extend QBase can be serialized (they contain only immutable List, Map, String, primative types or things that subclass from QBase.)  This means that all messages are immutable.

The container of business logic in Goat Rodeo is a Worker.  A Worker is like an Actor, but there are enough syntactic and semantic differences, that I choice a new name.  Actors are typed based on the WorkerId:

trait WorkerId extends QBase {
  type MyType <: WorkerId
  type MsgType <: QBase
  type IdType

  def myType: Manifest[MyType]

  def id: IdType

  def uniqueNameForFile: String
}

Here's an example of a SimpleId:

final case class SimpleId(id: Long) extends WorkerId {
  type MyType = SimpleId
  type MsgType = SimpleMsg
  type IdType = Long
  def myType = manifest[SimpleId]

  def uniqueNameForFile: String = "simple_"+id
}

A Worker is defined:

/**
 * The interface to a Worker.  A Worker receives messages, performs computations and communicates with
 * other workers
 */
trait Worker[IdType <: WorkerId] extends SimpleActor[IdType#MsgType] {
  /**
   * The unique ID of the Work
   */
  def id: IdType

  /**
   * Send a message to the worker
   */
  def !(msg: MsgType): Unit
}

You can only send an asynchronous message to a Worker.  Here's an example of an asynchronous message:

case class Moo(num: Int) extends SimpleMsg

You can obtain a Worker[WorkerId] from the WorkerMgr:

WorkerMgr.find(SimpleId(4)): Box[Worker[SimpleId]]

And here's the code for a Worker[SimpleId]:

class SimpleWorker(id: SimpleId, calcFunc: SimpleId => ConnectionManager) extends WorkerImpl[SimpleId, SimpleMsg](id, calcFunc) {
  var mooRes = 0
  private var balance: Long = id.id

  def doMoo(in: Moo): Unit = {
    mooRes = in.num
  }
}

The code does not look like the typical Actor's react/receive loop.  While it's possible to write that kind of dispatch for Workers, I opted to use reflection to build the dispatch table.  Public methods that start with "do", "perform" or "handle" and take a subclass of IdType#MsgType are placed in the Worker's dispatch table.  So, you can send a message to a Worker (either from within another worker or from the outside world):

        for {
          worker <- WorkerMgr.find(SimpleId(17))
        } worker ! StartTrans(SimpleId(18))
   

So far, nothing revolutionary.  But, you may be asking, how does the outside world ask a question of a Worker?  All messages that lead to a response must mix in:

/**
 * A message that generates a response with a particular type
 */
trait MsgWithResponse[T <: QBase] extends QBase

So, all messages that lead to a response are strongly typed.  So, we might have the following message:

case class GetBalance() extends SimpleMsg with MsgWithResponse[QLong]

And from the outside world, we can send it to a worker:

 OutsideOfWorker on myWorker complete {resp: Box[QLong] => resp.foreach(q => println("The balance is "+q))} ask GetBalance()

The method on the Worker looks like:

  def doGetBalance(in: GetBalance): QLong = {
    balance
  }

Alternatively:

  def doGetBalance(in: GetBalance, answer: Box[QLong] => Unit): Unit = {
    answer(Full(QLong(balance)))
  }

So, we can send messages to Workers and get answers to questions from Workers from outside the Worker infrastructure.  We can also ask questions from within the Worker context:

on myWorker complete {resp: Box[QLong] => resp.foreach(q => println("The balance is "+q))} ask GetBalance()

The only difference between the two is that from within the context of a Worker, you ask questions via the Worker.  The answer function is executed within the context of the Worker's mailbox and within the context of the Worker.

Workers can also engage in transactions:

for {
      t <- msg.xacts
      fromWorker <- WorkerMgr.find(t.from)
    } transaction avec fromWorker send SetupTransfer(t)

To send a message transactionally to another Worker (of the same type or of a different type), you designate that the message is transactional, the transaction is with (avec) another worker and the message to send.  There are a number of transaction modifiers:

postCommit {
      f(Full(QLong(balance)))
    }

This says that after the commit operation, apply the function f... send the balance to the asker of the question.  We can also define the action to take in case of a rollback:

rollback {(a,b) =>
      TestRecord.findAll().foreach{r => balance = r.balance}
      f(ParamFailure("rollback", Empty, Empty, (a, b)))}

Read the balance from the database and reply to the balance inquiry with a Failure.

Particular messages can be marked as non-re-entrant... they can only be serviced once per transaction:

transaction notReentrant "doSetup called a second time with "+msg

Transactions can be rolled back explicitly:

if (balance < msg.transfer.amount) transaction rollbackTransaction("Insufficient balance: "+balance+" need "+msg.transfer.amount)

But if all messages that are part of a transaction are processed successfully and do not themselves send transactional messages, then transaction commits... basically, the leaf nodes of the transaction are the message handlers that do not send messages.  If all the leaf message handlers succeed, they notify their callers that they are available to commit.  This process rolls up to the original handler and once all the methods have completed, the worker that initiated the transaction sends out a Commit message which ripples through the Workers that participated in the transaction.

During a transaction, a worker can only process messages related to that transaction.  I am working on deadlock detection (right now, it's via timeout, but it could be more sophisticated) as well as supporting two-phase commits.

Goat Rodeo includes a powerful backing store: SQL.  Each Worker has its own private relational data store.  Each Worker can store complex data structures and perform powerful queries against a relational database.  Each Worker's RDBMS commits (or rolls back if the handler exits via an exception or if transaction is explicit rolled back) at the end of each message process for messages that are outside of a Goat Rodeo transaction.  For messages within a Goat Rodeo transaction, the RDBMS connection is committed or rolled back based on the success or failure of the transaction.

Goat Rodeo's semantics for message passing, ask/answer and transactions are identical for local systems or remote systems.  The Worker implementation will behave the same way in either case.  How is this acheived?  All messages are guaranteed to be serializable and not contain any mutable references.  There's no way to know where a message came from.  The Worker interface is simple (there are two additional private methods used for out-of-band communications but those methods take limitted and well-defined system control messages).  All communication between Workers is handled via messages.

If you are interested in a distributed version of Goat Rodeo, please contact me and we can discuss your project, your needs, etc.

So, the Goat Rodeo code is up at Assembla (http://liftweb.assembla.com/spaces/goat_rodeo/stream) [I chose Assembla over GitHub because Assembla has a much better ticketing system and it has more flexible administrative controls, but I might move it back to GitHub if there's a good reason].  There's a Google Group for discussing Goat Rodeo at http://groups.google.com/group/goat-rodeo.  I look forward to seeing what people think and seeing how Goat Rodeo can evolve. 

Oh... and a few side notes.  Goat Rodeo runs on Scala 2.8 Beta 1, so it's not ready for production.  Goat Rodeo was inspired by Philipp Haller's awesome work with Scala Actors and concurrency.  I think that the work Jonas and his team have done with Akka is great and I am interested in seeing how we can share ideas, code, inspiration, and exploration to make Goat Rodeo and Akka part of one larger, better thing.  I'll get the Maven bits and pieces up on scala-tools.org sometime the week of 2/15/2010.  Right now, Goat Rodeo uses Maven as its build system, that will change to SBT very soon.

Thanks!

Wow, it's been 3 years I've been in Scala-land

Three years ago, today November 20th, I discovered Scala... and how my life has changed for the better.

Mime-Version: 1.0 (Apple Message framework v752.3)
Content-Transfer-Encoding: 7bit
Content-Type: text/plain;
 charset=US-ASCII;
 format=flowed
To: scala-subscribe@listes.epfl.ch
From: David Pollak <dpp@xxxx.com>
Subject: Subscribe
Date: Mon, 20 Nov 2006 09:02:26 -0800
Subscribe

After RubyConf 2006, I realized that Ruby was not on the right track for me. I went searching for a new language.

I've been a JVM guy since '96, so finding a language that was as on the JVM was a plus for me. I was looking for a statically typed language with high performance, but with the syntactic economy of Ruby. I bounced around a couple of language listing sites and found Scala. Three years ago, I fell in total love with Scala. That love continues today.

What is Scala to me?

• A language that strikes the right balance between immutability and mutability.
• A language with economical syntax and type safety.
• A object oriented language with a tremendously powerful type system and compositional model.
• A high performance language.
• A language with a wealth of libraries: all the Java libraries.
• A language with a highly stable, broadly available runtime: the JVM.
• A language that convinced me that computer science is really a science.
• A language that excels at almost every task I've thrown at it.

I discovered Scala and the awesomely warm community. The likes of Martin Odersky, Lex Spoon, Burak Emir, Adriaan Moors, Jon Pretty and Jamie Webb hung out in a forum chatting about high level language topics and helped a Java-slinging, Ruby-toting, null-loving, stateful guy like me find a new way and a better way.

These guys and some others in the Scala community enlightened me as to a better way to describe to the computer how to interact with data and users.

Three years ago, Scala was the next Pizza... a little known language that was the brainchild of one of the most amazing practical/theoretical minds in programming language design: Martin Odersky. Scala was on version 2.2.0... just stretching its legs as a self-hosted language (rather than the 1.x branch that had a compiler written in Java.) Scala had hints of being a practical mainstream language, but it wasn't well know and the community felt fairly academic.

I first met Martin 6 months later at JavaOne in May 2007. He came over to San Francisco to do a Scala presentation at JavaOne. By that time, Bill Venners had become an instrumental part of the Scala community. Martin was truly awesome. He was quiet. He was focused. He loved Scala and loved the potential of what Scala could be. He spent a lot of time fostering the right community on the Scala mailing list... that excellent community continues through today. At the time, I had no clue that this was the first JavaOne mention of Scala. I had no clue as to the lack of mainstreamness of Scala. I had been out of JVM-land for long enough that I just thought that I had missed something.

But, by the time I met Martin, I had embarked on Lift as a project. We got some early recognition from Tim O'Reilly on his blog.

About a year after I found Scala, Lift had matured into something beyond a one man show. David Bernard, Jorge Ortiz, Steve Jenson and others had joined the Lift project. Scala-tools.org was online. Scala-blogs.org was attracting posters and readers. By December 2007, Scala was on more and more people's radar.

For me, 2008 was a bad dream. There was a ton of personal tragedy, but I found that the Scala and Lift communities continued to be a source of positive energy and a place where I could find outlets for my coding needs. During 2008, Scala the language matured. Programming in Scala was released and it was an excellent work. Scala became a language that IT organizations were looking at seriously. Companies including LinkedIn, Twitter and Office Depot were public about their Scala adoption. We had the first Scala Lift Off in San Francisco. The buzz about Scala was growing and the energy in the community continued to be great.

2009 seems to be the year that Scala reached official “alternative mainstream” status along with Groovy and JRuby. I wrote Beginning Scala and a number of other excellent Scala books appeared. At JavaOne this year, it was a Scala-fest with standing-room-only at most of the Scala presentations. We had two Scala Lift Offs: West coast and East coast. Tool chain support got better and better between IDE support, the increasingly excellent SBT, and other efforts. Scala and Lift were featured in BEA's Devoxx keynote highlighting alternative language support on WebLogic.

Is Scala at the self-sustaining level that Python and Ruby are? Not yet. But, for the first time, I'll predict that Scala has a > 50% chance of reaching Python-level mainstream status. Scala and Lift has been embraced this year by hot start-ups (FourSquare) as well as big companies (Novell). The “Scala Bet” is looking less and less risky and the rewards of using Scala are increasingly well documented and understood.

But... more about me. Three years ago today, I though I had found a place where I could learn... a place where I could find the right tools for expressing what was in my mind's eye... a place that I could contribute back to. Three years later, I am so happy, so excited (overjoyed really), that I found Scala. Scala rocks. Scala is fun. Scala is great.

So, a great big thanks to Martin and the current and former EPFL folks for making Scala something great. A big thanks to the early Scala community for planting the right seeds and shaping things so the Scala community is what it is today. A super big thanks to the Lift community and the Lift committers for giving me a place to learn, grow, and have fun.

Three years later, Scala is better than anything I could have imagined on November 20^th, 2006.

Lift recently made the shift away from Scala Actors to a more generic Actor solution that allows for the use of Lift's lightweight Actors as well as Akka Actors.

Here's a brief overview of the new Actor structure and how to change from Scala Actors to Lift Actors.

Jonas and I worked on the simplest and most generic client interface for Actors that we could.  This is the interface that a caller sees and says nothing about implementation.  This is how we can swap LiftActor and Akka Actor implementations.

The most basic Actor is defined as something that can accept a message (these definitions are found in net.liftweb.common.Actor.scala):


This allows you to create an Actor that accepts a limited type of messages.  If you want an Actor to accept any kind of message (like Scala Actors):


Many Actors have a send/receive mechanism (you send a message and hang out waiting for a response).  This is defined by TypedActor:


Note that the message type, T, and the response type, R, are type parameters.  If you want a generic send/receive kind of Actor, use:


So, this is the interface.  How do you use a concrete implementation?

LiftActor (found in the net.liftweb.actor) package is a trait that implements the SimplestGenericActor interface.  Lift Actors differ from Scala Actors in:

• Lift Actors share a single scheduler which, by default, is event base and uses the java.util.concurrent library for thread pooling.
• Lift Actors do not support any form of monitor hierarchy.  This means there's no link/unlink method. If an exception is thrown during the processing of a message, it will be processed by the PartialFunction defined by the exceptionHandler method.  This means your actor doesn't stop working if an exception is thrown during message processing.
• Lift Actors do not have a running state.  They will always process incoming messages.  This means there's no start or exit methods.  Lift Actors will be garbage collected by the JVM's GC mechanism if there are no more references to it.
• Lift Actor's message processing is defined by the protected def messageHandler: PartialFunction[Any, Unit] method.  Because of Scala's uniform access, this method may be defined as a var that can be changed from message to message.  If you have fixed rules for handling incoming messages (as most apps do), you can define a method (or even a lazy val) for your messageHandler.
• Lift Actors return futures using the !< method rather than the !! method.  Jonas and I chose to use !! for returning an option of the result.

How do you convert from Scala Actors to Lift Actors?

• Remove scala.actors.xxxx imports
• Add import net.liftweb.actor._
• Change "Actor" to "LiftActor"
• Remove all calls to start()/exit()/link()/unlink()  If you have a monitor hierarchy, you'll have to manage running/not running manually.
• change def act = loop { react {msgHandler } } } to protected def messageHandler = { msgHandler }  Note that the substance of your msgHandler does not change.

If you want to mix Scala Actors and Lift Actors in your codebase and you want to have a way of saying "something that accepts the ! method", you can use Scala's Structural Types:


Note that as of Scala 2.7.5, structural types are not thread safe.

Hope this helps.

Thanks,

David

See Jack Die was an awesomely fun read.  A few months back, I got a bit of Twitter spam from Nicholas Black (@NicholasBlack60)... author of See Jack Die.  I followed him (a rare thing for me to do.)  Black tweeted an offer for a PDF review copy of the book.  I took him up on it.  It took me a month to find the right eReader to read it on (a Kindle DX), then I was off to the races.

I has just finished David Balducci's Simple Genius.  It was a less than satisfying book... which seems to be the case for most of Balducci's books... but somehow I can't seem to stop reading them.  Balducci's characters are barely one dimensional.  The dialog is stilted and the plots are plotless... yet (and this is probably my personality defect), but I can't stop reading them once I've started.

So, I get Black's book, See Jack Die, and I start to read it.  It's fun.  It's funny.  It's a little confusing at first... the main character, Jack, has amnesia and as a reader, I feel his amnesia.  During the first part of the book, I really feel like I'm exploring Jack's new-found world with him and trying to make sense of it.  Black really drew me in.

Okay... I'm going to take a break and complain a little:

• Black needs some Twitter lessons.  Yeah, he sucked me in by following me, but his tweets are generally pointless, repetitive and come 3 an hour, 24x7.  I stopped following him after 2 days.  I'd be interested in Black's tweets about actual writing (e.g., "Put Jack in a bad place... how am I going to get him out?"  "Ricky, Ricky, Ricky... I don't think dad's dollars are going to help you with that." "Can't write today... gotta get some transfat into my body.")
• The book needs a couple of more editing passes.  I found a fair number of type-os.  Now, I'm not one to throw too many stones about type-os and grammar checking (just look at some of the reviews of Beginning Scala), but I think Black needs to enlist the community to help him with editing.
• Ricky (Jack's best friend) seems to morph over the course of the book to deliver whatever was needed as plot devices... either that or Black has a deeper plan for why Ricky seems to pull so many rabbits out of his hat.

So, those are my complaints about the book.  On the good side:

• The dialog is really good.  It's not quite Robert Parker (the old stuff, not the newer stuff) good, but it's Sanford good.
• Bonding over transfat is just plain funny.  I can't stop thinking about tard-farmers.
• Black kills characters better than anyone else.  In fact, he's got a dark, twisted and darned funny way of dealing with death... which is a big part of the book.
• The research is really good.  It's Jeffrey Deaver good.  Lots of interesting facts that Black clearly understands and is able to weave into the plot a lot better than Balducci does with quantum computing (the subject of Simple Genius.)

So, I strongly recommend See Jack Die.  Black is right up there with authors like PJ Tracy.  He's new.  He's got interesting and original ideas.  He can write a book that hangs together very well and is very entertaining.  I highly recommend that you buy a PDF copy of See Jack Die at Scribd or buy the dead tree version from Amazon.  I wish Black a lot of luck and look forward to reading more of his books and more of the See Jack ... series.

It's been almost three years since my Web Framework Manifesto.  In those three years, I've founded the Lift Web Framework project, we've shipped Lift 1.0, I've seen a fair number of projects built with Lift and I've had the honor to interact with a fair number of excellent engineers at the largest social networking sites.  While Lift has pushed the envelop and allowed developers to write high performance, highly interactive web sites, there's a lot more than needs to be done.  Specifically, the impedance mismatch between browser-resident data, server-resident data, and the persistance layer.  Additionally, the current crop of persisance mechanisms, relational databases, are ill-suited to tasks including social networking and interactive gaming.

A Lift retrospective

Lift 1.0 is out the door.  The Lift community has grown beyond 1,000 members.  I've built or been part of more than a dozen Lift-related projects.  I'm very happy with how Lift has evolved.  But, what is Lift?

What is Lift?

When I started the Lift project, I envisioned Lift being all things, from soup to nuts, that a web developer would need to build web apps.  That included an OR mapper, interfaces to Facebook, PayPal, OpenID, etc.  As Lift evolved and I heard from the community and other folks started contributing to the Lift code base, it became clear that Lift-style abstractions on top of all the things that people would want to do with web apps was not the right answer.  While there are a bunch of Lift modules, the core of Lift is Lift Utils and Lift WebKit.  These two modules provide developers a way to securely abstract away the HTTP request/response cycle and focus on the business logic of their applications.  In the form of SiteMap as well as pattern matching and extractors, security and access control is primarily handled by declarative code.  Developers can trust that by the time a request gets to the view, the access control rules have been enforced.

Lift's support for Ajax and Comet (a.k.a, Ajax Push or long polling) is unparalleled.  There is no other framework that allows you to build a real time chat app in less than 30 lines of code.  Lift's abstractions of long polling combined with Scala's Actors provide an amazingly simple way to build web sites that update automatically and scale very effectively, and without deadlocks and other hallmarks of multi-threaded code.  Lift's pattern of binding HTML elements to functions provides a secure, unified, and extremely concise mechanism for developers to write standard HTML as well as Ajax apps.

Lift offers unparalleled security

Lift apps are secure by default.  Here is a partial list of Lift's security features:

• The Java Virtual Machine is impervious to buffer overflow attacks
• Scala is a strongly typed language so you will always know the type of the parameter passed to your method (e.g., you will not get a List[String] when you were expecting a String)
• Request parameters are treated as UTF-8 and all Strings are URL decoded before they reach application level methods (unless the developer actively requests earlier access to the data)
• Lift builds all web pages as well formed XHTML rather than a String or a stream of characters or bytes.  This means that the developer doesn't have to think about escaping strings to HTML, it's done correctly and automatically.  If the developer wants to include non-escaped characters in the output, the developer must affirmatively use the Unparsed directive.  This means that the developer must actively do something to open a cross site scripting hole in the application.
• Lift apps are built with Lift's mapper or JPA.  In both persistence frameworks, SQL query parameters are properly escaped.  The developer must actively choose to send a raw query String to the database before the app is vulnerably to an SQL injection attack.
• Lift's forms mechanism associates randomly generated GUIDs with form elements and functions.  When forms are submitted containing the GUID, the function is invoked.  This means that replay attacks are not possible with Lift apps and it means that only the form fields presented to the user can be submitted back and cause code to execute on the server.  An attacker cannot add fields to a POST and have those fields cause code to be executed on the server.  Further, Lift's default select and multi-select generators will reject any options that were not originally presented when the form was generated.  All this means is that parameter tampering attacks are very difficult against Lift apps.
• Lift's SiteMap provides unified menu generation and access control.  All pages have access control rules and if the rules do not evaluate to true, links to the page will not be presented.  When a page is requested, the access control rules are evaluated and if they do not succeed, the request will not be serviced.  The access control rules are defined declaratively and can be easily audited in a code review.

All of the above means that Lift apps are secure.  I've been through a number of penetration tests with Lift apps that I've put into production.  The pen testers have never found a material security vulnerability in any of the apps they tested.  There's not one single vulnerability in the OWASP to 10 that's crept into a Lift app I've worked on.  I've heard similar reports from other folks who have gone through pen tests with their Lift apps.

The other stuff

The other parts of Lift are valuable and helpful, but they are not core to Lift.  Having built in support for XMPP and AMQP as well as PayPal and OpenID makes building apps nice.  There is the Lift Widgets module for stuff like calendars, etc.  This all makes building rich Lift apps easy because so many of the pieces are there.  Additionally, the Lift WebKit has no dependencies on the other Lift modules.  This means that the other Lift modules could have been written by external parties.  This modularity is nice if you want to use your own persistence layer, your own JavaScript libraries (rather than jQuery or YUI, the Lift defaults), etc.

Share nothing is fail

One thing that has become very clear after spending lots of time with Lift and watching other folks adopt Lift: share nothing is fail.  The share nothing architectures are thin layers on top of relational databases.  The amount of work that it takes to build secure, complex apps in share nothing is amazingly high.  Share nothing apps must either be primarily stateless or they must put the state someplace.  Both choices lead to seriously suboptimal results in the goal is to build secure or interactive applications.

The primarily stateless model of share nothing apps is that there's a cookie that represents the current session.  The information in the session itself is usually the primary key of the current user.  All the other information regarding state is based on the URL and any parameters that are passed as part of the request.  The difficultly in building multi-form wizards or shopping carts or other things with state is huge.  Is the state kept in cookies or in hidden fields?  If so, the state is subject to parameter tampering.  Putting aside the security issues, the developer effort required to marshal and unmarshal state on each request is significant.  Even if the developer gets it right, making any changes to the code becomes a seriously difficult issue.

The other option for share nothing is to push state into memcached or the RDBMS.  This is a more secure method because the state cannot be altered changing parameter or cookie values.  However, the issues related to marshaling remain.  Further, there is pressure put on the RDBMS unless memcached is used to store state, but memcached is a cache and subject to cache misses which means that state can go away.  Sure it works to marshal via the RDBMS, but then you have a single point of pressure in your application and when the RDBMS hits the wall, your application will come to a grinding halt... think fail whale here.

There are a number of other success web frameworks, including Wicket, WebObjects, and Seaside, that are highly stateful.  Many high volume sites (including Apple's store) run on these systems and have a very good uptime record.

Frameworks still evolving

At the end of the day, web frameworks are still evolving.  Rails' awesome leap forward in developer productivity provided the catalyst for other folks to critically evaluate the web development process.  While I believe Lift has a bunch of really good concepts built into it, I think there are other better ways to describe web applications.  I expect web frameworks as a whole to be a growing, evolving category over the next 10 years until we've been able to capture the semantics of web development and refined and reduced the semantics into composible structures.

Models

Sometimes, software abstractions grow out of the definitions of the underlying computing systems.  For example, C abstracted assembly language.  C++ abstracted C.  Java abstracted C++ (more or less and in the case of templates, a whole lot less.)  Sometimes, software abstracts the needs of the application developer.  SQL and COBOL are good examples.   But as the application needs change, the tools and frameworks must change to support the application demands.

SQL is an example of an excellent response to the real world needs of business applications.  SQL is great for ERP.  SQL in the abstract is good for more recently popular applications such as social networking.  In practice, SQL databases are not good at dealing with social graphs once the social graphs exceed a certain size.  However, it's become clear that SQL does not solve all the problems on the web.  The rise of CouchDB, Google App Engine/BigTable, Amazon's SimpleDB, etc. indicates that for the web, the relational model is not the right fit.

Just as Rails signaled a shift in web frameworks, the spate of new persistence technologies is signaling that it's time for us to reexamine the way we store data and the way that we reflect the flow of data from the edge of the network, typically the browser, through the business logic (increasingly browser resident), through the server-application into persistence and/or other systems that will trigger the flow of data to other edge devices.  That's a long way of saying, "moving this data is hard and we don't have a clue how to do it right."

The problem is further amplified by the rise of new client technologies: DHTML (the real stuff) and Flash/Flex/Air.  Interactive client apps are pretty darned hard to build and even harder to keep up to date with servers that have different business logic models and different persistence models (not to mention different object models.)  While Adobe is doing a good job of providing data synchronization tools for Air apps, I don't think proprietary systems will have the reach that open systems built on open standards do.

Goat Rodeo

There needs to exist a unified model for building real-time interactive web applications from the persistence layer, through the messaging layer, through the business logic layer, out to the client.  Lift is part of this equation.  Goat Rodeo will become the other part of the equation.

Goat Rodeo will provide:

• A compiler-checked, Scala based unified model for describing data structures that can be consumed by Scala, JavaScript and anything else that speaks JSON... these items are called Qs (quanta of information)
• A distributed transaction model based on the ZooKeeper project.  This will support 20K transactions per second.
• A scalable persistence layer, most likely built on Cassandra
• A transaction model based on Software Transactional Memory that is exposed to the developer using Scala's for comprehension
• References that support the storage of Qs, doubly linked lists of Qs or maps of Qs with persistence definitions of local, ZooKeeper, and long term
• Association of methods with Qs such that the methods can be emitted in native Scala or JavaScript... the code is a subset of Scala and can be pushed to environments (e.g., the browser

With the above building blocks, you get:

• Scalable persistence
• Distributed actors
• Synchronization of data and data model from browser through long term storage (yes, I'll be working to make sure this works with Lee's jsync.)
• Unified interprocess communications, even across heterogeneous processes
• What I believe will be a very scalable system for social networks and other social, interactive web apps to build on top of

Right now, Goat Rodeo is in the noodling and building phase.  I'm working on the Scala compiler plugin that does the enforcement of Q types as well as the generation of serialization and deserialization.  I'm also wiring up ZooKeeper and Cassandra such that the play with transactional niceness with each other.

I am not developing Goat Rodeo in a vacuum.  Goat Rodeo will be a back end for Lift's new Record system (along with JDBC and JPA).  I am working on a real time information sharing system so that people can annotate and link pictures and other information from the Scala Lift Off as a test bed and driver for Goat Rodeo.

I expect that by end of summer, Goat Rodeo and its associated Lift integration will be ready for someone who isn't me to play with.  In the mean time, if you're interested in helping to shape the direction of Goat Rodeo, please drop by the Lift list and share your thoughts.

Oh... and if you're wondering about the name... just think about the job of managing the piles of data that come into your application... it's a...

No, my machine was not hacked. Yes, I do believe that DHH and the Rails community has done far more to advance web development than the Apache Software Foundation. I radically disagree with J Aaron Farr's post comparing the two.

First, I am a fan of DHH. I think he's a wicked smart person and that Rails as a piece of software and as a catalyst for web development is a milestone. DHH built Rails and created the Rails community. He's done a masterful job. He's also become more of a statesman over the years. With that being said, I'm not a huge fan of the Rails community or its culture.

I'm a user of a lot of Apache Software Foundation stuff. It's solid. It's maintained. With the possible exception of Wicket, none of it is groundbreaking. In fact, some of it is poorly designed crap. I'm an Apache commiter (perhaps not for long) on the ESME project. I have seen very little of the meritocracy that Aaron blogs about.

It's not easy being green running a ground-breaking project

I've run my share of groundbreaking software projects in my life. It's really not an easy task. Having vision, managing employees/committers, building community, and satisfying users all takes a bit of attitude that pisses people off. Let's look at what DHH did with Rails:

• He blew away most of the other web frameworks by demonstrating that web development could be concise and that web apps could be built to be visually pleasing and easy to use without a ton of developer time. That, IMHO, puts him in diety territory.
• He built an ecosystem around Rails that had enough critical mass to spawn a whole industry around Rails development.
• And the 37 Signals apps don't suck.

But doing all that requires making choices. DHH made a series of choices about style, community building, etc. Some people think he's a jerk... but the same can be said about Steve Jobs, Bill Gates, Halsey Minor and other folks who fundamentally changed the game (yeah, I'm lumping DHH in league with those guys.) In my opinion, DHH is not a one trick pony (Folks who built the future). He's got direction, attitude, energy, charisma, and the other stuff that it takes to change the game, and he'd done just that.

And in the other corner, weighing 98 pounds...

Now, let's look at the ASF. It's a nice place to house a project. There are lots of reports to write. In fact, I believe more time is spent on the ESME project writing monthly reports than writing code. The ASF has done nothing to help the ESME folks build community or inspire folks to join the project. There's been little in the way of technical guidance. Sure there's brand and visibility in having an ASF project, but there's no meritocracy. There's nothing that I've seen in the ASF that rewards anything other than being nice and writing reports and having statistics get better. Sure, if I'm Yahoo! and I want to open source Hadoop and pay my developers to work on it, Apache is a nice neutral place to house it. CouchDB is high on the doesn't suck scale, but I gotta say, Damien Katz is the personality behind it (no, he's not as acerbic as DHH) and without him, CouchDB would not be (violating the ASF guidelines, but I guess they needed a flashy project).

Further, let's look at the ASF crap pile. Well, there's the HTTP server... so bogged down in commityism that Nginx blows it away. Struts... the ultimate piece of crap web framework. ActiveMQ... junk compared to RabbitMQ. Tomcat, consistently behind Jetty (a personality driven show.) There's no real merit that I can see across the board in Apache's offerings. Put another way, if the ASF were a meritocracy, they'd dump the crap that's in their top-level projects.

Yes, I think that the Rails community could be a nicer place to be. I've tried to build the Lift community so that it's a warmer and more welcoming place to be than the Rails community. But I am sure glad Lift is not part of the ASF. I tip my hat to DHH and what he's built.

Party and (likely) flame on.

Two years ago, today, I launched the Lift Web Framework as an open source project.  Wow... it's been a long and fun experience... and today the dozen plus Lift committers and the whole Lift community together are releasing Lift 1.0.

Lift is an expressive elegant web framework based on the Scala programming language and released under an an Apache 2.0 license.  Lift provides developers the best way to build interactive, high performance web applications.  Lift based applications are deployed as WAR files into J2EE containers such as Jetty, Tomcat, and WebLogic.  Lift based applications are high performance and can make use of your existing Java libraries.

I could wax on for hours about:

• Lift's Comet and Ajax support which allows you to build real-time interactive applications
• Lift's concise code allowing developer productivity normally associated with Rails and TurboGears
• Lift's high performance and scalability
• Lift's built-in support for REST and other web services
• Lift's use of Scala's type-safety so your tests can focus on business logic

But, that's not the most impressive thing about Lift.  Lift is powered by a community of committers and users that cares about building tools for building great web apps.  Lift is impressive because of the people who use, drive, enhance and exchange ideas about Lift.  The Lift community is a warm, welcoming place for people of all backgrounds.  The Lift community and Lift committers strive to learn from others and roll that learning into Lift and their own projects.  That's my take on what makes Lift great, but let's hear what other have to say about Lift:

The interest and excitement about Scala continues to grow.  It's great to see Lift reaching the 1.0 milestone as this is a proof point for the maturity of Scala as a software platform.

Martin Odersky, ACM Fellow, Father of Scala

Lift is the only new framework in the last four years to offer fresh and innovative approaches to web development. It's not just some incremental improvements over the status quo, it redefines the state of the art. If you are a web developer, you should learn Lift. Even if you don't wind up using it everyday, it will change the way you approach web applications.

Michael Galpin, Developer, eBay

The slight added complexity of static typing is more than offset by the performance, scalability, and the benefits of type safety. Lift is maturing rapidly and has already proven itself many times over, and it will only get better.

As much as I liked Ruby and Rails, I like Scala and Lift better. After more than two years of developing software in Ruby/Rails, we've shifted all our development efforts to Scala/Lift. And we are not looking back.

Charles Munat, Lightsource Interactive

Lift's excellent 'Comet made easy' philosophy made it an absolute no-brainer as the choice of framework for the Apache ESME project. Additionally, the fact that Lift-based applications run unchanged on the SAP's NetWeaver CE Java application server makes this an intriguing approach for enterprise applications in the SAP world.

Darren Hague, SAP Mentor, ESME team lead

When I decided to put Innovation Games® online, I knew that I couldn't afford a massive development effort. I needed a small, sharp team who could leverage best-in-class tools to help us solve the problems we knew that we'd have to solve in creating a new kind of collaborative gaming experience on the web. David suggested Lift and Scala and initial testing proved that we could realize the developer efficiency and backend scalability that we felt was required to efficiently support thousands of simultaneous games. We're now very comfortable with Lift and Scala and are pleased with how the solution framework continues to evolve to meet our needs. While we've used lift to push the boundaries of interactive web design, I strongly recommend anyone who wants to build a compelling web experience using an elegant framework to consider using Lift.

Luke Hohmann, CEO, Enthiosys

If you're looking for a web framework on a strongly typed functional language and the JVM, Lift is the only game in town. Oh, and it just works, too.

L.G. Meredith, Managing Partner, Biosimilarity LLC

For me it's mainly because Lift represents collective web wisdom - all lessons learned and new to be discovered.

Viktor Klang

I find Lift a very solid piece of software very well designed and written. It is the result of many years of experience of many people. I believe in Lift's utility when developing not only compelling web applications but also other server side applications sitting on top of HTTP stack. I would choose Lift over any other web framework out there without blinking.

Marius Danciu

Lift is like a breath of fresh air: concise, elegant and robust - all on my existing Java infrastructure...

Lift stands on the shoulders of giants, learns from their mistakes and adds a whole new dimension to web application development.

Irrespective of the technology, what makes Lift really special is the community - well read, intelligent and welcoming.

Tim Perrett

Lift combines simplicity, flexibility and power better than any other web framework in my experience.

Derek Chen-Becker

Lift allows a single person to accomplish what would have previously taken an entire team.

Tyler Weir

You can get started with Lift or join the Lift community right now.  But, it's time for me to thank a whole bunch of folks that led to Lift and keep Lift going:

• The Lift committers who are a totally awesome group of folks that I'm honored to work with.
• The Lift community as a whole.
• Dani, Jon and Brion for doing the SmartMode thing back in 2000-2001.  SmartMode inspired Lift.
• Martin Odersky, Lex Spoon, Burak Emir, Philipp Haller and the other awesome people that build Scala and the Scala community.
• Jamie and Jon who, along with Burak, taught me Scala.
• The Scala community as a whole which is a great place.
• Roger Rohrbach for the name Lift and the rest of the Gabblists for a lot of support and feedback.
• Matthew and Walt for taking the first chance with Lift and helping me understand how to teach Scala.
• Tim O'Reilly for raising awareness about Lift.
• Luke Hohmann for betting Buy a Feature on me and Lift.
• SteveJ, Jorge, and DavidB for being the early committers and the guys that turned Lift from my project into the community's project.
• Aaron Williams for the putting Buy a Feature into SAP's Collaboration Workspace.
• Lee Mighdoll for making repeated bets on me and Lift, not to mention crafting the phrase "expressive elegant web framework".
• Kaliya who builds the best communities.
• Jack, an awesome CEO, for reminding me what focused leadership means.
• Darren and Dick for choosing Lift for ESME and for including me in the ESME project.
• The whole ESME team for exposing 10,000+ SAP developers to a Lift application at three DemoJams.
• Greg who keeps on asking me the hard questions that drive Lift in a more functional direction.
• Debby who has been herding the Lift project towards 1.0.
• My wife and father and kids who provide(d) the tools for taking the risks of thinking beyond the norm.

Lift is 1.0.  Lift is ready and able to power your interactive web applications.  The Lift community is waiting to welcome you, your questions and your feedback.  Please join us.

I recently did a pretty negative review of the IntelliJ and its Scala support.  Based on a lot of feedback and some very polite posts from Ilya, one of the Scala plugin developers, I took another look at IntelliJ.  Based on spending more than a week with IntelliJ, configuring and tuning it, I fail to understand why someone would pay a dime for IntelliJ.  I find it to be a poorly thought out product.  The Scala support has strengths and weaknesses compared to the NetBeans Scala plugin.  If you use and like IntelliJ, you will likely disagree with this review and you'll likely find the Scala plugin to be a good tool.  If you haven't used IntelliJ yet, don't waste your time.

I've been using programming editors since 1983 when I first started doing C programming for the Apple ][.  Despite this, I do not think I'm a power user of programming editors.  I have a few things that I do and don't try to dig deeper.  My favorite editor of all time is Brief, but alas (and partially due to my lawyering work), Brief is not longer available.  Emacs is my default editor of choice.

I write a lot of code, but I do it simply.  I write code.  I compile code.  I test code.  I write more.  I open lots of files and usually have 2 or 3 projects open at the same time.  In rank order, the most important things an editor can do for me are:

1. Allow me to quickly navigate to the file I want to edit
2. Find the lines in the file that I need to change/add to
3. Give me real-time feedback about errors
4. Format code nicely
5. Support a very nice manual find/replace mechanism (Emacs is king of this)
6. Allow me to see more than one part of a file at once
7. Support opening multiple projects and files outside of projects quickly and easily
8. Does not assume that I will be doing all my work inside the editor... I use the command line a lot
9. Open files based on a regular expression (e.g., emacs $(grep -l Foo $(find . -name *.scala)))

While Emacs does most of this stuff very, very well, it doesn't support real-time feedback on program errors, nor is its Scala formatting particularly good.  I was using jEdit for a while, but as the NetBeans plugin matured, I switched to NetBeans.  It does 1 and 2 well enough and is great at 3 and 4.  NetBeans is still beta.  Its got its weaknesses, most notably class navigation is particularly weak.

A couple of random things.  First, I have a vested interest in more better tools for Scala, so posting negative things about IntelliJ is not in my interest.  Second, Scala's object model, implicits, traits, etc. make things particularly difficult for IDE plugin writers.  Third, I do not do automatic refactoring.  I find that it takes away from my understanding about the code.  Even if a refactoring takes me 2 hours rather than 30 seconds, the 2 hours is always well spent learning about how the code works.

I'm going to put aside my installation issues for IntelliJ on 64 bit Linux.  I outlined those issues in my last post.  All my testing was done on a Windows Vista machine.  Installation was flawless and IntelliJ fired right up.  So far, so good.

I installed the 8.1 Diana EAP version of IntelliJ as I was told that this was better for Scala work.  I installed the Git, Maven, and Scala plugins.  I opened Lift and went for the most gnarly file for an IDE: the Lift Textile parser.  This 1,300 line file results in about 2,000 classes.  It's a big hairy mess for an IDE.  Good news, IDEA handles the file.  Bad news, IDEA finds a bunch of errors in the file, but when I edit the file and insert junk, IDEA doesn't flag my typing errors.  I closed the file and try for something less nasty.

I went searching for another file to edit.  I found one, although it seems that browsing in IDEA is by class, not by file.  This is a challenge in Lift when packages often have hundreds of classes.  After a lot of rooting around, I found that you can view a project by "Scope: All" and navigate by file.

I open another file, move to the line I want to edit and click on the line.  My cursor is at the click-point, rather than the end of the line.  This is kinda stupid.  Why doesn't IntelliJ just put the cursor at the end of the line (this and other annoyances are configurable, we'll get to that in a little while)?  The editor is counter-intutive.  Okay, I do some editing.  I do some code navigation.

The code navigation is clearly IntelliJ's strong point.  I navigate quickly through all of my Scala classes.  For library classes, IntelliJ provides me with method signatures that are enough for me to understand what's going on.  I'm starting to warm up to IntelliJ.  I'm liking the Scala plugin.  It's doing something that I haven't seen since Java in Eclipse... seamless class navigation.  I'm starting to understand why people like IntelliJ and the Scala plugin.

I do some find and replace.  Once again, IntelliJ is good at this.  Its editor makes incremental search, changing something, and searching again a very simple task.  It's not quite as nice as Emacs, but it's better than NetBeans.

And this is as good as IntelliJ gets.  The rest is a freaking horror show.

I committed up my changes to Lift.  I monitor Hudson and the build is broken.  It's broken in Textile.  IntelliJ frickin' saved my bogus changes to the Textile file.  What the f**k?  So, I dimly remember that IntelliJ has an autosave mode and apparently the autosave mode also marks the file as "added" to Git.  This is absolute stupidity.  Unless I explicitly save a file, it should not change on disk.  So, I spend an hour working through the preferences in IntelliJ.  I disable auto-save (this should not be the default!!!!)  I set the editor up to work the way most other editors work.  I go back to work.

Even with auto-save disabled, IntelliJ saves file changes automatically when the editor is exited.  This alone is a disqualifier for IntelliJ.  I haven't found a way to disable this behavior.  What the f**k are the IntelliJ folks thinking?  If I want to save a file, I'll save it.  If I want to discard a file, I'll discard it.  The default behavior should be the way most other programs are: files are explicitly saved.

The default editor behaviors are screwed up.  The default should be the way other editors work.  If IntelliJ wants to changed the paradigm, fine.  But the mental cost of forcing me to spend a lot of time navigating preferences makes me think the IntelliJ folks are more interested in foisting their paradigm on me than giving me a tool that works the way I want to work.  At the very least, there should be a simple option "Emacs mode" or some-such that sets the various properties to work the way us old-time Emacs kinda guys work.

As a side note, I did some work with the Talligent C++ class-based editor back in the 90s.  IntelliJ seems to have a paradigm that's part of what Talligent was doing: navigate and edit the class, not the file.  Implicitly save changes, etc.  If there was an obvious change log that could be undone in a single operation and  class-granularity for the editor rather than file granularity (when you open a class, you just see that class), the paradigm might work.  Unfortunately, the way IntelliJ has stradled between file and class-based editing is an amazingly bad balance.

Next thing I did was open two projects.  IntelliJ opens a new window (they call it a frame, because different names are good?) for the new project.  Navigation across classes in the projects worked well.  I was feeling good, until I exited IntelliJ.  I later reopened it and only one of my projects was open.  This is once again a wicked bad design.  I IntelliJ keeps track of stuff, then it should keep track of all the projects I have open, not just the most recently edited project.

Other stuff that I find anoying about IntelliJ:

• Lots of flashing icons.  Once I've clicked on the icon, it should stop flashing.
• What are facets and why should I care?
• The default character set on Windows is the Windows character set.  This is brain damaged for any web-related work (it should be UTF-8) and I haven't figured out how to change this.
• The icons are ugly.  A commercial product, an expensive commercial product, should look better.
• The JDK for a project is not automatically set.  Do what Eclipse does: if the JDK is not defined, prompt the user.
• Sometimes IntelliJ just freezes without a wait cursor.  If IntelliJ is doing something, then let me know... don't make me guess to see if the IDE has frozen.

The Scala plugin is in beta, so my comments are only applicable to the current beta version.  Ilya seems to care a lot about what people think and seems to be working towards having the best Scala plugin.  So, the code navigation and syntax highlighting in the Scala plugin are first rate.  The stuff that doesn't work so well for me are:

• Errors are flagged for default packages until the JDK is defined.  This one tripped me up until Illy told me to define my JDK.
• Errors are not flagged as I type.  It seems that the Scala plugin detects errors about 40% of the time.  This is less than optimal.  Errors should be flagged correctly.
• XML support is marginal.  Yes, this is so hard that the Eclipse plugin punts on it.  I find a lot of pauses, long ones, when I'm typing in XML.
• The code formatting choices are not the right ones.  There are Scala coding guidelines that EPFL uses.  These should be the defaults for the plugin.
• XML is not indented properly.

When I tried IntelliJ for the second time, I really wanted to like it.  I tried very hard to care about the positive feedback that a lot of people I respect have about IntelliJ.  I wanted to say good things so I could answer the "how's the Scala tools support?" question more positively.  I'll even go farther.  If someone wants to send me configuration steps for IntelliJ to make it work like NetBeans or Emacs, I'll spend another week using it.  I want very much to say good things about IntelliJ, but my experiences with it are so overwhealmingly negative that I can't find nice things to say beyond praising Ilya's work and progress with the Scala plugin.

So, I'm not going to use IntelliJ.  I'm betting the Scala plugin will improve, but the overall horribleness of IntelliJ is so overwhelming that I can't see using IntelliJ, even if the Scala plugin becomes the best Scala plugin on the planet.  Yes, I know that a lot of people swear by IntelliJ.  Great, for you guys, go ahead and use it with the Scala plugin.  For the folks in the world who use programming editors the way I do, stay as far away from IntelliJ as you can.  It's a poorly designed piece of junk.
 

Scala IDEs are not up to the same level of fit and polish as as Java IDEs.  This is something of a bummer.  While I'm an early adopter and I'm willing to put up with a fair amount of rough edges, I do like having good tools.  I've been using and liking NetBeans with the Scala plugin.  I've been generally happy with it, but I've heard a lot of good things about IntelliJ, so I decided to give it a try.  IntelliJ is perhaps the worst IDE I've used in the last 10 years, Scala or no.  I'm wondering if the IntelliJ folks even have a QA department.

Yep, those words about IntelliJ are pretty harsh.  But I was unable to get IntelliJ to work at all on my Linux machine.  On Windows, it works, but is so amazingly clunky that it's not worth installing.

I downloaded IDEA 8 and untarred it on my 64 bit Ubuntu 8.04 machine.  I fired up IDEA... and it complained that JDK_HOME was not defined.  Gosh, JAVA_HOME is.  So, why can't their scripts check for common variations on the location of the JVM... or even just ask Java what it's home is.  Not hard folks.

So, I set the environment variables and fire up IDEA again.  This time, all the dialog windows are blank.  No content.  I keep pressing "Enter" until IDEA opens up.  I install Git, Maven, and Scala plugins and open a Scala project and IDEA and IDEA runs out of memory.  So, I give IDEA 1GB of RAM and try again. IDEA runs out of RAM.  How about 4GB.  Nope... still out of memory.

I tried running IDEA on my Windows box.  It installed and was able to open projects.  Unfortunately, it leaves turds in the project directory after loading the pom.xml file.  Okay, I can live with that.  Then IDEA gives me a bunch of errors while resolving the dependencies in my pom file.  I was able to open my project and start editting files.  However, every time I tried to imbed XML in my Scala code, IDEA would freeze for many seconds at a time while using 100% CPU.  Yeah, I know that handling XML is non-trivial, but NetBeans' Scala plugin does it just fine.

There's no way to browse files in IDEA.  There's only class browsing.  Scala programs have far more classes in a given package than does Java.  So, I was hoping for file-based browsing.  No luck.  I've got 3 or 4 screen-fulls of classes to wade through to open the one I want... but the file name is listed in the tabs.  So, which is it folks?  File name or class name?  If it's only class name, why not just display the class as a stand-alone in editor?

I also tried running IDEA on a 32 bit JVM on Ubuntu.  The blank dialog problem is gone, but in its place, the plugin manager cannot reach IntelliJ's plugin site.

So, maybe I'm missing the glory and beauty of IntelliJ, but I can't see spending a penny on an IDE like IDEA, especially for doing Scala development.  Nope... I'm sticking with NetBeans.

Folks... more information on Beginning Scala, the Scala book I'm working on. It's available for pre-order from APress's web site. Additionally, draft PDFs are available via APress's Alpha program.

For those who are wondering about the Table of Contents, here it is:

1. About Scala and how to install it
2. Scala syntax, scripts and the command line
3. Collections and the joys of immutability
4. Fun with Functions and never having to close that JDBC connection
5. Pattern matching and getting rid of unwanted visitor patterns
6. Actors and concurrency
7. Traits and Types, gnarly stuff for architects
8. Parsers because BNF is not just for academics anymore
9. Scaling your team and your systems

I look forward to your feedback on the book!

I am very excited to announce that I am writing Beginning Scala and it will be published by APress.

You may be asking, "Why is David, the Lift guy, writing a Scala book?"

First... let's talk about APress.

I've been a long-time fan of APress.  It's a great tech publishing house.  Gary Cornell founded APress after a less than optimal experience with other publishers.  He fashioned APress into an author's publisher and has attracked and published some pretty amazing talent and great books over the years.

Despite that Gary is no longer at APress day-to-day, his legacy and vision lives on.  When Steve Anglin approached me to write a Lift book, I was initially hesitant.  I've had sub-optimal writing experiences in the past.  I've talked to most of the tech publishers about a Lift book.  But, Steve went a couple of extra miles in the process.  He found a co-author (which it turns out I didn't need.)  He spent time discussing the book market with me.  He convinced me that Gary's vision was still alive and well at APress.  APress had the environment that I wanted to work in.

Steve worked with me to put together an outline and a proposal for a Lift book.  However, the publishers thought that the market is not yet ripe for a Lift book.  So, Steve proposed that I do a Scala book.  Once again, Steve worked with me to put together an outline and proposal.  The Scala proposal was accepted (woo hoo!)

I started work in mid-December for a January 5th deadline for the first three chapters.  I slipped the deadline by a week or so (writing during the holidays, especially when our nanny was out on maternity leave, is something of a challenge.)

The book will be 9 chapters and it will mostly cover Scala basics.  I'm focusing a lot on the joys of immutability and how the mind-shift to immutability, function passing, and small blocks of code can make composing large scale and defect-reduced programs super-simple.

The book will not be "open source."  It may be available as an early access PDF from APress's site.  We're targetting a publication date at JavaOne.  Maybe APress will even throw a party at JavaOne.

Folks,

After many months of "breaking changes", the Lift team is proud to announce
Lift 0.10.  Lift 0.10 has frozen APIs and barring any material problems, the
0.10 APIs will be the same as the upcoming 1.0 APIs.  We're planning to
release 1.0 at the end of February.

Lift is an expressive and elegant framework for writing web
applications. Lift stresses the importance of security, maintainability,
scalability and performance while allowing for high levels of developer
productivity. Lift is a Scala web framework.

Over the next 6 weeks, the Lift team will be focusing effort on cleaning up
some Lift internals, improving performance, improving documentation, and a
new Lift web site.
There are a ton of people who have contributed to Lift over the last two
years by writing code, by writing documentation, by asking questions, by
building Lift-based apps, and by participating in the Lift community.  I
would like to extend special thanks to some folks who have made Lift 0.10
both possible and awesome:

   - Marius... he's an awesome developer who has amazing vision and
   understanding of what Lift is and should be.  If I got hit by a bus, Marius
   could continue to drive the Lift code base forward.
   - DavidB provides great structure and process to Lift.  He runs the
   scala-tools.org site and keeps all the dependencies and pieces of Lift
   and the greater Scala/Maven ecosystem humming.
   - Jorge's calm, deliberative way helps bring the needs of all users into
   perspective.
   - Derek's JPA contributions have moved Lift into a place where it can
   play in non-greenfield apps.
   - Tyler's energy is infectious.
   - Tim's energy is properly infectious.
   - Derek, Marius, and Tyler have conspired to deliver some excellent Lift
   bookage for those who can convert LyX to PDF.
   - Charles who delivers apps and asks questions.
   - Debby who is bringing her legendary cat-herding skills to the Lift
   committers.

So, as we spirit Lift through the final part of the beta process, thank you
all for participating in the community.

Changes in this version include:

New features:
o A Currency class
o Added a nifty mechanism for stateful form management (Hoot)
o Added fix CSS support
o Added support for other JS libraries
o Added JSON forms support
o Added PayPal Integration module

Changes:
o Consolidate LiftRules
o Added HTTP authentication support
o Upgrade to Scala 2.7.3
o Upgrade to Scalacheck 1.5
o Upgrade to Specs 1.4.0
o Added Record/Field generic support
o Changed Can to Box
o Changed RequestState to Req
o Updated LiftView to be more syntactically pleasing
o Fixed a bug with how RequestVars and traits work
o Enhanced the DateTime inputs
o First pass at complete PayPal ITN and PDT stuff
o Refactoring of the PayPal stuff
o Updated to jQuery 1.2.6
o Redesigned Gavatar widget

Thanks,

David

I had a rant about Ubuntu 8.10 being a buggy release. Turns out that the Canonical people knew about the defect and release Intrepid anyway. The problem impacts about 10% of modern laptops... ones with the most common Intel WiFi chipset. The defect has not been fixed. The workaround doesn't work. What are these people thinking?

I've got lots of things to be Thankful for in my business life this year.

• The Lift community hit 700 people today.  It's a strong, vibrant, growing community full of smart and caring people.
• The Lift committers are awesome to hang with... they're so smart and driven... and they do awesome care and feeding of the Lift community.
• Scala and Lift are joys to work with.  Scala is such awesome technology and I love building cool software with it.
• The first Lift Workshop was great.  The folks who came asked awesome questions helped me understand how other people use Lift.
• Working with Dan, Luke, Kaliya and Jorge is a real joy.  They're stellar and I learn so much from them.
• Jack and Alex gave me an opportunity earlier this year... it was great working with them... and it was cool to work with an excellent CEO.
• The ESME team is a wild thing to be involved with... and I've learned more about SAP than I could have imagined.
• It's cool working with John on the book... he lends a lot of balance... and Steve's a great editor, continuing in the APress tradition.

I hope you have an awesome Thanksgiving and holiday season.

Thanks!

David