A real-world use of lift

Wednesday, March 21. 2007

Well, lift is actually being used in production. I converted a Rails app to lift and it was a very interesting experience.

For those who know me, I've been working on a secret side-project for the last few months. The side project has become my main project and is called CircleShare. The name (and URL) are no longer secret, but the project is still under wraps.

CircleShare is a combination of a desktop application, web site, and web infrastructure. The desktop app connects to the server and sends information or waits for instructions. For the most part, there's not much going on, but there are a lot of clients connecting to the server.

We did the first prototype of CircleShare in Rails. We were able to add new features very, very quickly. The Rails app worked well for the rapid prototyping. However, when we started doing calculations of how much server infrastructure we'd need, we realized that Rails' single-threaded model killed us. We could only run 40 or so connections for each commodity server. This was a problem if, for example, we had 10,000 clients connected at any given time (that's 250 boxes.)

We initially looked to see if JRuby would address the issue. Rails running in JRuby is still single threaded, so we'd need 1 JVM per concurrent connection (or do some very funky things with class loaders.) This was not a solution.

When we read about Jetty Continuations we knew we had to do our system in something that would run in the JVM. I was given the task of seeing how easy a port of our 12 model classes and controllers would be to lift. I estimated that it would take on the order of a month. The actual work took a week, including enhancements to lift.

My first task was to modify our Rails-based controller tests to make actual HTTP requests such that I could test the lift-based version of CircleShare with the same tests we were using for the Rails code. That took a day and a half.

Next, I wrote the data access part of the models. lift models differ from Rails models in that one must declare the database columns in the lift models (but there's no need for migrations files, so LOC wise, it's a wash) Here's a couple of lines from a model definition:

// the primary key for the database

val id = new MappedLongIndex(this)

val friendlyName = new MappedString(this) {override def maxLen =64

override def dbColumnName = "friendly_name"}

val users = new HasManyThrough(this, User, CircleUser, CircleUser.circle, CircleUser.user)

val owner = new MappedLongForeignKey(this, User) {override def dbColumnName = "user_id"}

Verbosity-wise, there's not much difference from Rails. However, in lift the object's field can differ from the database column name (although, but default, they're the same. This is helpful if your column name is 'default' which causes silent failures in Rails.)

The next thing I did was go through the controllers line-by-line, copied and pasted the Rails code into the lift project and changed the syntax from Ruby syntax to Scala syntax. For the most part, there was a literal line-for-line equivalence. With one exception, for the REST controller, lift allows inclusion of the XML return value in the controller itself. One of the sources of many errors in the code with the impedance mismatch between the REST controller and the multitude of REST view files.

Next, I updated the code to use Scala idioms. Here's the completed conversion:

# get salt and nonce for a given user in order to calculate the

# password hash

def get_salt

@nonce = MiscUtils.random_string 20 # generate a nonce

session[:nonce] = @nonce # store it in the session so that there's no way to do a replay attack

@user_name = params[:user_name] # the username

@user = User.find_by_email(@user_name.downcase) unless @user_name.blank? # look up the user

@salt = @user.salt if @user # populate the salt from the user

@success = true

end

// get salt and nonce for a given user in order to calculate the

// password hash

def get_salt = {

val nonce = randomString(20)

set("nonce", nonce) // # store it in the session so that there's no way to do a replay attack

val (success, salt, userName) = (

for (val userName <- params("user_name");

val user <- User.find(ByField(User.email, userName)))

yield (true, user.password.salt, userName)) getOrElse (false, "", "")

XmlResponse(<CircleShare success={success.toString} operation="get_salt">{

if (success) <user_salt user_name={userName} salt={salt} nonce={nonce} />

else Text("")

}</CircleShare>)

}

A couple of things to note about the Scala code:

- There's not mutable state in the Scala code. Once a value is defined, it's not changed. This results in fewer bugs.

- There's no explicit guard to test if the user_name parameter was supplied. The "for" comprehension will bail out (and the expression will result in the other side of the getOrElse) if the user_name is not supplied.

- There was no need to downcase the user_name because the find(ByField) knows that the email address field should always be lower case and does that automatically when the query is constructed.

- There's no explicit test (or exception thrown) if the user isn't found. Just as when there's no "user_name" supplied, the "for" comprehension will stop and the "else" part will be the result of the expression

Once I finished the week-long port, we did a test with our client running against the server. We found one bug that was not covered by the tests.

Then we did some benchmarking. For single request processing, the lift code, running inside Tomcat, ran 4 times faster than the Rails code running inside Mongrel. However, the CPU utilization was less than 5% in the lift version, where it was 100% of 1 CPU (on a dual core machine) for the Rails version. For multiple simultaneous requests being made from multiple machines, we're seeing better than 20x performance of the lift code versus the Rails code with 5 Mongrel instances. Once again, the lift code is not using very much CPU and the Rails code is pegging both CPUs.

In terms of new features, we've been able to add new features to the lift code with fewer defects than with the Rails code. Our Rails code had 70% code coverage. We discovered that anything shy of 95% code coverage with Rails means that type-os turn into runtime failures. We do not have any code coverage metrics for the lift code, but we have seen only 1 defect that's been checked in in the 2 weeks since we started using lift (vs. an average of 1 defect per checkin with the Rails code.)

So, yes, I'm pimping my own framework, and yes, I'm able to do with lift what guys like DHH are able to do with Rails, so the comparison is, in some ways, unfair.

On the other hand, Scala and lift code can be as brief and expressive as Ruby code. lift offers developers amazing productivity gains vs. traditional Java web frameworks, just as Rails does. On the other hand, lift code scales much better than Rails code. lift code is type-safe and the compiler becomes your friend (this does not mean you should not write tests, but it means that your tests can focus on the algorithm rather than making sure there are no type-os in variable and method names.)

So, lift has had its first real-world test and has exceeded my expectations.

Posted by David Pollak in Scala at 11:05 | Comments (18) | Trackbacks (0)

Trackbacks

Trackback specific URI for this entry

No Trackbacks

Comments

Display comments as (Linear | Threaded)

very promising. I'd love to keep reading more real world experiences your having with the port beyond the scalability. How was your teams reactions to using yet another new framework and language? I'll be honest, scala's syntax turns me off, but I haven't given it a honest try, just a skim of the docs.

#1 Alex Soto on 2007-03-21 17:43 (Reply)

The team was initially cranky with "yet another language." However, they're Java guys so they felt much more at home with Scala. One of the team members had working code in a day, so that was a huge win.

On the syntax front, Scala is less ugly (by a long shot) than Java. Once one starts getting "functional" with Scala, being familiar with Scala idioms helps.

One can write Scala code that looks like Ruby code (and not having the annoying @ signs everywhere is a real win.) But that code is much more "imperative" in its style and tends to be more verbose.

I find that Haskell is a much more difficult for me to read as a functional language that Scala.

A lot of it, I think, is experience with particular syntax and idioms.

#1.1 David Pollak (Homepage) on 2007-03-21 20:03 (Reply)

You know, you can have multiple JRuby runtimes in the same VM to handle many Rails requests without spawning another VM. Rails is designed to be single-threaded (or really, single-runtime) but by pooling a few JRuby runtimes you can handle as many requests as you like.

Look at the rails-integration project hosted in the JRuby Extras project on RubyForge or google it to get more information.

#2 Charles Oliver Nutter (Homepage) on 2007-03-21 18:07 (Reply)

Charles,

Thanks for the clarification!

David

#2.1 David Pollak (Homepage) on 2007-03-21 19:57 (Reply)

I'm curious as to why your performance test was behind Tomcat when you cited Jetty continuations as one reason for moving over to Scala/JVM.

At first glance, I have to say that the Scala syntax is not altogether pleasing to my eyes, either. BUT, Scala packs a lot of power into its syntax and it's not so awful looking that I find it unapproachable. Scala balances a fair number of requirements well.

#3 Kevin Dangoor (Homepage) on 2007-03-21 19:40 (Reply)

I haven't finished the magic to get Jetty continuations to work with lift yet. Additionally, our deployment infrastructure is currently Tomcat, although I expect to change that over once the continuations work is done.

#3.1 David Pollak (Homepage) on 2007-03-21 19:55 (Reply)

Makes sense. I'll be curious to see what the Jetty continuations part looks like.

#3.1.1 Kevin Dangoor (Homepage) on 2007-03-21 20:42 (Reply)

It seems to me that Jetty's continuations only provide a subset of what Scala's event-based actors can do for you. For example, when an actor waits for a message using react { ... }, it also frees the underlying thread, but it also saves all the state that is accessed from inside the message handling cases (because it constructs a closure). So, why not wrap all the asynchronous requests in thread-less actors?

#3.1.2 Philipp Haller (Homepage) on 2007-03-24 14:36 (Reply)

Lift looks lovely but for the inlined XmlResponses, which has that whole "code/presentation mixing" problem. Might be appropriate for this XML-returning RESTful app, but web apps that run on xhtml really do need designer-friendly external templates. Well, at least mine do.

Is there a plan (or something already existing) for lift to support an attribute language, or perhaps some kind of Scala HaXML workalike?

#4 chuck on 2007-03-22 10:00 (Reply)

Yes. "Controller" methods can return a lot of different things.
A controller can return a map of name/value pairs that's passed into a view. A controller can return a function that operates as a view. A controller can return nothing and an appropriate view is located. Or a controller can compose and return XML or XHTML. So, there's maximum flexibility.

In this example, I found that integrating the view and controller was optimal. For XHTML page composition, the view/controller separation is the optimal way to do things.

#4.1 David Pollak (Homepage) on 2007-03-22 10:25 (Reply)

I disagree Chuck: in this case, the XML response isn't 'presentation' like HTML might be -> it is truly a typed-contract between the server and the client (albeit, one the compiler can't check). So to me it is very appropriate that it appear there in code.

Btw, this is great work!

#4.2 Aaron White on 2007-03-22 11:29 (Reply)

One of my biggest annoyances with developing webapps in Java (on Tomcat), is that I can't simply change the code and reload the app. Sometimes I might be able to reload the app using for example the tomcat manager, but in my current project I can't do that due to various other libraries being used. So I have to restart the entire server every time I make the smallest change. Very frustrating. How is the Scala/lift combination in this respect?

#5 André Næss on 2007-03-22 10:42 (Reply)

Right now, the results are mixed, but it's a priority for me.

For "long lived controllers" that use Scala Actors (that's how lift implements the chat and clock), there's less than optimal interaction with Tomcat's reload mechanism. I'm working with the folks who wrote Scala's Actor library and the issue will be resolved either this week or in the next release of the Scala Actor package.

For short-lived controllers (like the ones used in the real world work and like Rails controllers), there are very few issues. Basically, change a class, wait 5 seconds, load the web page and all is good. Every once in a while, if the change is global (e.g., many or all classes are rebuilt), Tomcat's classloader picks up a bad combination of old and new classes. This happens to me once or twice a day.

#5.1 David Pollak (Homepage) on 2007-03-22 10:52 (Reply)

I agree that in this case, the XML response is quite strongly typed and perfectly appropriate for this particular app. Some apps demand looser coupling however. My fault though: I failed to notice http://liftweb.googlecode.com/svn/trunk/liftweb/example/src/main/webapp/ which has exactly the sort of XHTML template scheme I was asking about.

I wish I'd come across lift sooner -- might have been able to avoid so much of the pain I've experienced using JSF (Seam makes it quite tolerable, but really only in a relative sense).

#6 Chuck Adams on 2007-03-22 11:53 (Reply)

Hello this has nothing to do with your blog or post, but I just wanted to ask if you own the domain www.athena.com, and if so, why did you choose that name?

Thanks!

#7 Athena on 2007-04-05 18:31 (Reply)

I just wanted to point out that your original premise (concern about the scalability of Rails) is incorrect. Rails scales using LAMP style architectures, not by language-based multi-threading. You could have made Rails do what you wanted.

That being said, Scala and lift look very interesting, and I am investigating them myself.

#8 Curt Hibbs (Homepage) on 2007-04-09 10:27 (Reply)

Curt,

Please explain how you can have 10,000 simultaneous clients connected to a single machine running Rails. We could not figure a way that Rails could service more than 40-50 simultaneous (open) connections on the same box.

Please explain to me how Rails/Ruby can listen for "events" beyond just querying the database asking "should I do something now?" I looked into a Rinda-based solution without luck.

Even Rich Kilmer uses Erlang for part of the indi infrastructure (or so he claimed at RubyConf.) There are places where event based/Actor based architectures work better than Rails request/response architecture. We were unable to solve the scalability issues that our application presented exclusively within Rails/Ruby. Short of going outside Rails/Ruby, how can one solve the massive simultaneous connection issue?

I look forward to any comments and feedback you have on this issue as well as on lift.

#8.1 David Pollak (Homepage) on 2007-04-09 11:06 (Reply)

David,

I am curious about that 100% utilization on Rails. Sound a bit suspicious.

Did you try to profile it? Where was all the time spent?
What was the memory utilization in both cases?

I was wondering if you might have had some bad code in Rails in maybe a model that was then fixed during the code to Lift. Maybe some plugin, or bad architectural decision.

I developed many rails apps and never saw cpu going to 100%. Were you pounding the crap out of the app during benchmarking?

If so, 100% cpu utilization might be what you want. If you are hitting the app full throttle and the cpu is only going 5% it might mean that you are either not hitting it hard enough or the bottleneck is somewhere else, like memory or disk IO.
Maybe then fixing that bottleneck should give you higher cpu usage and higher throughput.

Just curious.

p.s.
I am also been investigating Lift and Scala for a project that I am currently working on.

#9 Diego Scataglini (Homepage) on 2008-01-23 12:12 (Reply)

Add Comment

Name
Email
Homepage
In reply to
Comment	Enclosing asterisks marks text as bold (word), underscore are made via _word_. Standard emoticons like :-) and ;-) are converted to images. To prevent automated Bots from commentspamming, please enter the string you see in the image below in the appropriate input box. Your comment will only be submitted if the strings match. Please ensure that your browser supports and accepts cookies, or your comment cannot be verified correctly. Enter the string from the spam-prevention image above:
	Remember Information? Subscribe to this entry