Study notes on Cobit: Overview and strategic IT planning

November 19th, 2010

What is strategic IT planning?

Simply put, strategic IT planning is to suggest various IT solutions to support business objectives or strategies. For example, suppose that you are the CIO of a school and that the school has an objective of improving education quality. It may have defined business strategies to achieve that objective like:

  • Making learning fun
  • Engaging parents

As the CIO of the school, you could suggest various IT solutions to support one of those strategies. For example, for the first strategy (“Making learning fun”), you may suggest:

  • Provide educational games to the teachers.
  • Provide training to enable teachers to develop interactive courseware.
  • Locate and provide the best student-engaging lecture videos to the teachers.

Note that each IT strategy must have a clear linkage to the business strategy so that it creates business value (“Value” is one of the three major concerns of Cobit). For example, it is assumed that educational games will make learning fun because students like playing games.

Cost and risk

After coming up with these IT strategies, you will explain them to the CEO (the principal) and business executives (the vice principals) to let them choose which ones to proceed. However, without the information regarding the cost (“Cost” is the second major concern of Cobit) and other needed resources for each IT strategy, it is just impossible to choose. So, you have to provide such information. For example, for the IT strategy of providing educational games, you may need:

  • MOP1,000 (cost) for a site license for each game (software).
  • Teachers (people) to attend a training session.
  • A server (hardware) to host the games.
  • A certain internal network bandwidth (network).

These needed resources (people, software, hardware, network) are called “Enterprise architecture for IT” in Cobit.

In addition, what if some students become addicted to the games? What if some parents don’t want their kids to play games? These risks must be communicated to the CEO and business executives to help them choose which IT strategies to pursue (“Risk” is the third major concern of Cobit).

Control objective

So, you can perform strategic IT planning as described above. But how to ensure the desired output (IT strategies as selected by the CEO will have good value, low cost and low risk) will be produced? For example, there are some success factors:

  • You (the CIO) knows the business objectives and strategies well.
  • The CEO is willing to make clear decisions.
  • You, the CEO and business executives can discuss and negotiate well.
  • You can plan (estimate) the costs, risks and other needed resources well.

Each process has its own such success factors and they’re key to ensure that the process is effective (producing the desired output) and performance (producing the output efficiently). Such success factors are called “control objectives” in Cobit and is a key concept in Cobit (the term “Cobit” simply stands for control objectives for IT).

Maturity level

However, even if you’re doing the strategic IT planning process very well, it doesn’t mean that the process is embedded in the DNA of the organization. For example, if you leave the organization and your successor doesn’t do it or does it poorly, then it is not in the DNA of the organization. This is reflected in the “maturity level” of the process:

  • Level 0 (No process): The organization has no process at all.
  • Level 1 (Ad-hoc): Different people in the organization have different processes to do the same thing.
  • Level 2 (Repeatable): Different people in the organization follow the same process, but the process is informal (not written down), so it is not formally reviewed, approved and used for training.
  • Level 3 (Defined): The process is formally defined.
  • Level 4 (Managed): The process is continuously improved.
  • Level 5 (Optimized): The process is considered among the best in that  industry.

Scala exercise 6: Tackling the Wicket hierarchy mismatch problem

November 7th, 2010

Introduction

This is the sixth exercise in my Scala exercises series. If you haven’t seen it before, you may want to start from the previous exercises. Below is exercise 6: Tackling the Wicket hierarchy mismatch problem.

Some people have alleged that a weakness in Wicket is that you have to keep the hierarchy of the component tree in sync with that of the HTML elements. For example, for the HTML code:

<html>
    <form wicket:id="f">
        <input type="text" wicket:id="num">
        <input type="submit" value="OK">
    </form>
    <span wicket:id="result">10</span>
</html>

You would construct a component tree (in Scala) like:

class MyPage extends WebPage {
    val f = new Form[MyPage]("f") {
      override def onSubmit() {
        ...
      }
    }
    add(f)  //add the form to the page
    val numField = new TextField[Int]("num")
    f.add(numField)  //add the text field to the form
    val r = new Label("result")
    add(r) //add the label to the page
}

The problem is that, if, say, you’d like to move the result span into the form, you must change the code accordingly:

class MyPage extends WebPage {
    ...
    val r = new Label("result")
    f.add(r) //you must add the label to the form, not to the page!
}

and a common problem is that we may forget to do so. While there is no easy way to solve this problem, you could make the Scala code reflect the HTML structure visually (and look more like a declarative UI):

class MyPage extends WebPage {
  //the "ctx" represents a surrounding container context for the
  //construction of child components
  this containing { ctx =>
    val f = ...
    ctx.add(f)  //add the form to the context (the page)
    f containing { ctx =>
      val numField = ...
      ctx.add(numField) //add the text field to the context (the form)
    }
    val r = new Label("result")
    ctx.add(r)  //add the label to the context (the page)
  }
}

Note that the Scala code reflects the structure of the HTML code so you can compare them visually. In addition, if you need to move the label into the form, you can simply cut and paste the construction code of the label into that context of the form:

class MyPage extends WebPage {
  //the "ctx" represents a surrounding container context for the
  this containing { ctx =>
    val f = ...
    ctx.add(f)
    f containing { ctx =>
      val numField = ...
      ctx.add(numField)
      //the following lines were simply cut and pasted into here
      val r = new Label("result")
      ctx.add(r)
    }
  }
}

Now, your task is to implement this solution by creating the necessary code. Try to do it now.

See the answer here.

Scala exercise 5: monitoring pattern

November 2nd, 2010

Introduction

This is the fifth exercise in my Scala exercises series. If you haven’t seen it before, you may want to start from the previous exercises. Below is exercise 5: monitoring pattern.

Suppose that you have created a web application displaying product information to and taking order from customers. From your testing, let’s say you have determined that it can at most display the product page 100 times per minute, otherwise it will get painfully slow or even crash. Now, you’d like to monitor the application in production to make sure its loading is not approaching that limit. If yes, you’ll promptly allocate more computing resources to it.

In order to do that, it would be great if the application would support something like SNMP and provide a variable named “display counter” for you to access. Then you could use a monitoring system like zabbix or nagios to monitor it. The good news is, there is something similar for Java: it is called JMX (Java Management eXtension).

Next, you’ll do it step by step. Your job is to fill in the missing code.

First, create a Maven project. Use the following pom.xml file:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>scala-jmx</groupId>
    <artifactId>scala-jmx</artifactId>
    <version>1.0</version>
    <packaging>war</packaging>
    <properties>
        <spring.version>3.0.5.RELEASE</spring.version>
    </properties>
    <repositories>
        <repository>
            <id>java.net2</id>
            <name>Repository hosting the jee6 artifacts</name>
            <url>http://download.java.net/maven/2</url>
        </repository>
        <repository>
            <id>scala-tools.org</id>
            <name>Scala-tools Maven2 Repository</name>
            <url>http://scala-tools.org/repo-releases</url>
        </repository>
    </repositories>
    <pluginRepositories>
        <pluginRepository>
            <id>scala-tools.org</id>
            <name>Scala-tools Maven2 Repository</name>
            <url>http://scala-tools.org/repo-releases</url>
        </pluginRepository>
    </pluginRepositories>
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework</groupId>
                <artifactId>spring-context</artifactId>
                <version>${spring.version}</version>
            </dependency>
            <dependency>
                <groupId>org.springframework</groupId>
                <artifactId>spring-core</artifactId>
                <version>${spring.version}</version>
            </dependency>
        </dependencies>
    </dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>javax</groupId>
            <artifactId>javaee-web-api</artifactId>
            <version>6.0</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>2.8.0</version>
        </dependency>
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-core</artifactId>
            <version>${spring.version}</version>
        </dependency>
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-web</artifactId>
            <version>${spring.version}</version>
        </dependency>
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-context</artifactId>
            <version>${spring.version}</version>
        </dependency>
    </dependencies>
    <build>
        <pluginManagement>
            <plugins>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-compiler-plugin</artifactId>
                    <version>2.0.2</version>
                </plugin>
                <plugin>
                    <groupId>org.scala-tools</groupId>
                    <artifactId>maven-scala-plugin</artifactId>
                    <version>2.9.1</version>
                </plugin>
            </plugins>
        </pluginManagement>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <executions>
                    <execution>
                        <phase>compile</phase>
                        <goals>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <source>1.6</source>
                    <target>1.6</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-war-plugin</artifactId>
                <version>2.1-beta-1</version>
                <configuration>
                    <failOnMissingWebXml>false</failOnMissingWebXml>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.scala-tools</groupId>
                <artifactId>maven-scala-plugin</artifactId>
                <executions>
                    <execution>
                        <id>scala-compile-first</id>
                        <phase>process-resources</phase>
                        <goals>
                            <goal>add-source</goal>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                    <execution>
                        <id>scala-test-compile</id>
                        <phase>process-test-resources</phase>
                        <goals>
                            <goal>testCompile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

There is nothing special except that you’ll use Spring. Next, create a Scala trait PerfCountersMBean in the com.foo package:

trait PerfCountersMBean {
  def getDisplayCounter(): Int
}

It says that you’ll have an “MBean” that has a property named displayCounter. MBean stands for managed bean, which is just a Java bean that can be inspected or called from a remote management console. Next, create a Scala class to implement the MBean in the com.foo package (fill in the code later):

...
class PerfCounters extends PerfCountersMBean {
  ...
}

In order to initialize the MBean, you’ll turn it into a Spring bean (you’ll do it later by annotating the PerfCounters class). Then, to register it with the JVM, you’ll create a Spring bean that performs the registration. So, create a beans.xml file in the src/main/resources folder:

<?xml version="1.0" encoding="UTF-8" ?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:context="http://www.springframework.org/schema/context"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://www.springframework.org/schema/beans

http://www.springframework.org/schema/beans/spring-beans-3.0.xsd

http://www.springframework.org/schema/context

http://www.springframework.org/schema/context/spring-context-3.0.xsd">

    <context:component-scan base-package="com.foo"/>
    <!-- init this exporter eagerly as nobody will ask it to export the MBeans -->
    <bean id="mbeansExporter" class="org.springframework.jmx.export.MBeanExporter" lazy-init="false">
        <property name="beans">
            <map>
                <!--
                   the key is the name of the MBean, which starts with the domain
                   name (foo.com) and then some name=value pairs.

                   the value is the Spring bean acting as the MBean
                -->
                <entry key="com.foo:name=performance-counters" value-ref="perfCounters"/>
            </map>
        </property>
    </bean>
</beans>

Of course, you need to increment the counter whenever the product page is displayed. So, create a servlet which should display something as the product information and more importantly, get access to your MBean (a Spring bean) and increment the counter (you’ll fill in the code later):

class DisplayServlet extends HttpServlet {
  ...
}

Finally, create the web.xml file the src/main/webapp/WEB-INF folder:

<?xml version="1.0" encoding="UTF-8"?>
<web-app version="3.0" xmlns="http://java.sun.com/xml/ns/javaee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd">
    <servlet>
        <servlet-name>s1</servlet-name>
        <servlet-class>com.foo.DisplayServlet</servlet-class>
    </servlet>
    <servlet-mapping>
        <servlet-name>s1</servlet-name>
        <url-pattern>/product</url-pattern>
    </servlet-mapping>
    <context-param>
        <param-name>contextConfigLocation</param-name>
        <param-value>classpath:/beans.xml</param-value>
    </context-param>
    <!-- initialize Spring -->
    <listener>
        <listener-class>org.springframework.web.context.ContextLoaderListener</listener-class>
    </listener>
</web-app>

Run “mvn package” to build the war file. Then deploy it into Tomcat. To enable JMX in the JVM on which Tomcat runs, you need to pass some JVM arguments by setting the CATALINA_OPTS environment variable before running startup.sh (or startup.bat). On Linux:

export CATALINA_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=1234 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false"

On Windows:

set CATALINA_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=1234 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false"

Now, fill in all the missing code above.

To verify that it is working, run Tomcat with startup.sh (or startup.bat). Use netstat to check if it is listening on port 1234 (the port you specified above). Finally, run jconsole and connect to localhost:1234. Choose the “MBeans” tab and you should see your MBean in the com.foo folder. Check the value of the DisplayCounter. Access the product page at http://localhost:8080/scala-jmx/product. Then check the value again and it should have been incremented.

See the answer here.

Scala exercise 4: circuit breaker pattern

October 24th, 2010

Introduction

This is the fourth exercise in my Scala exercises series. If you haven’t seen it before, you may want to start from exercise 1: template method, exercise 2: observer design pattern and exercise 3: decorator and composite design patterns. Below is exercise 4: circuit breaker pattern.

Problem: Create a console program that reads commands from the user. If the command is “q”, it will query a hard-coded URL (e.g., http://localhost:1234) and print the result to the console. If the command is “x”, it will exit. Below is a sample session:

Enter: q
Hello <-- this is the content at the URL
Enter: q
Hello <-- ditto
Enter: x
Exiting

In practice, if the remote server is overloaded, out of order or the firewall is mis-configured, every time you try to access the URL, it may take up to several minutes for your client to timeout or receive an error. It would also place unnecessary load on that server which possibly is already overloaded. Therefore, a better way to do it is, if the server returns an error (or does that for a consecutive number of times), your client will not try to access it anymore (treat it as an error immediately), until the administrator resets it. This is called the “circuit breaker” pattern.

Now, your task is to implement this behavior by filling in the code below. There is also a new command “r” to reset the circuit breaker. For simplicity, you’ll stop calling the server as long as one exception is caught.

//it extends the CircuitBreaker trait to obtain the functionality. Unit is the return
//type of the code block to be protected by the circuit breaker.
object HttpClient extends CircuitBreaker[Unit] {
  def main(args: Array[String]) {
    while (true) {
      print("Enter: ")
      try {
        readLine match {
          case "q" => queryHttpServer
          case "r" => reset
          case "x" => {
            println("Exiting")
            return
          }
          case _ => println("Unknown command")
        }
      } catch {
        case e: Exception => e.printStackTrace
      }
    }
  }

  def queryHttpServer {
    //run the code block in the protect() method
    protect {
      ... //get the content at http://localhost:1234
    }
  }
}

trait CircuitBreaker[T] {
  var isOpen = false

  //run the code block only if the circuit breaker is not open. If it is,
  //thrown an CircuitOpenException immediately.
  def protect(codeBlock: ...): T = {
    ...
  }

  def reset {
    println("Resetting the circuit")
    isOpen = false
  }
}

class CircuitOpenException extends Exception

In addition, for your own testing, you can create a simple HTTP server with netcat. Just run:

  echo -e "HTTP/1.0 200 OK\n\nHello\n" | nc -l 1234

It will quit after serving one request. So, you need to run it again to serve multiple requests.

Try to do it now! Then, click here to see the answer.

Scala exercise 3: decorator and composite design patterns

October 16th, 2010

Introduction

This is the third  exercise in my Scala exercises series. If you haven’t seen it before, you may want to start from exercise 1: template method and exercise 2: observer design pattern. Below is exercise 3: decorator and composite design patterns.

Problem: In most UI frameworks including JSF, Wicket or Swing, you will need to provide a callback/listener object to handle requests from the user. Typically in such a callback, if there are some errors, you’d like to display a specific error message instead of propagating it to the framework, otherwise the framework would simply display a generic error to the user.

To hand code such a callback, you may do it like:

new Callback() {
  def onCallback(ev: Any) {
    try {
      //perform the business logic here
    } catch {
      case e: LoginException => {
        //assuming that error() will display the error
        error("failed to login")
      }
      case e: SQLException => {
        error("error accessing the database")
      }
    }
  }
}

The problem with this approach is that there is a lot of boilerplate code there, while most usually we only want to say for exception class E1, display some error message M1:

new ErrorHandlingCallback(
  //classOf[Foo] is the same as Foo.class in Java
  classOf[LoginException], "failed to login",
  classOf[SQLException], "...") {
  def performBusinessLogic(ev: Any) {
      //perform the business logic here
  }
}

But what if you’d like to extract some information from the exception and include it into the error message or would like to do something special? Then, ideally, you should be able to specify a function as the error handler:

new ErrorHandlingCallback(
  classOf[LoginException], "failed to login",
  classOf[SQLException], "...",
  (e: Exception) => doSomething(e)) {
  def performBusinessLogic(ev: Any) {
      //perform the business logic here
  }
}

Finally, you should be able to pre-define an object to handle the commonly seen exceptions:

//ideally you should be able to "add" the error
//handlers together to get a compound error
//handler
val defaultErrorHandler =
  (classOf[IOException], "I/O error") +
  (classOf[Exception], "Unknown catch all error")

new ErrorHandlingCallback(
  classOf[LoginException], "failed to login",
  ...,
  defaultErrorHandler) {
  def performBusinessLogic(ev: Any) {
      //perform the business logic here
  }
}

Your task is to complete the code below and create the other necessary classes as needed:

trait Callback {
  def onCallback(ev: Any)
}

abstract class ErrorHandlingCallback(errorHandler: ErrorHandler) extends Callback {
  //overload the constructor to take multiple error handlers (the star does that)
  def this(errorHandlers: ErrorHandler*) = ...

  def performBusinessLogic(ev: Any)

  def onCallback(ev: Any) {
    ...
  }
}

object ErrorHandlerUtil {
  //allow you to use a function as an error handler
  implicit def fromFunc(f: Exception => Boolean): ErrorHandler = ...
  //allow you to use a pair (error class, error message) as an error handler
  implicit def fromPair(p: (Class[_ <: Exception], String)): ErrorHandler = ...
}

Then, the following code should compile and run:

object ErrorHandlerTest {
  //in order to use the implicit conversion methods, you
  //must import these objects so that those methods can
  //be invoked without a prefix.
  import ErrorHandlerUtil._

  def main(args: Array[String]) {
    //assume that this is the default error handler in this context
    //classOf[Foo] is the same as Foo.class in Java
    val defaultErrorHandler = (classOf[IOException], "I/O error") + (classOf[Exception], "Unknown catch all error")
    //create a decorator to handle additional errors
    val decorator = new ErrorHandlingCallback(
      //convert a pair to an error handler
      (classOf[IndexOutOfBoundsException], "index out of bound"),
      //ditto
      (classOf[NullPointerException], "hit a null pointer"),
      //you can define a custom error handler using a function to, say,
      //access the info in the exception (not just its class).
      (e: Exception) => if (e.getMessage.contains("xyz")) {
        println(e.getMessage)
        true //indicate that it has been handled
      } else false,
      //specify the default error handler here
      defaultErrorHandler) {
      def performBusinessLogic(ev: Any) {
        println("called")
        ev match {
          //do nothing. No error.
          case "foo" =>
          //try to access the 100th element of an array which has only 3 elements
          case "bar" => Array[Int](1, 2, 3).apply(100)
          //Try to call a method on null
          case "baz" => null.equals("oops!")
          //throw a custom exception
          case "baz2" => throw new RuntimeException("I am xyz!")
          //divided by zero (something unexpected to test the ultimate fallback)
          case "baz3" => 100 / 0
        }
      }
    }
    decorator.onCallback("foo")
    decorator.onCallback("bar")
    decorator.onCallback("baz")
    decorator.onCallback("baz2")
    decorator.onCallback("baz3")
  }
}

Try to do it now! Then, click here to see the answer.

Scala exercise 2: observer design pattern

October 10th, 2010

Introduction

This is the second exercise in my Scala exercises series. If you haven’t seen it before, you may want to start from exercise 1: template method. Below is exercise 2: observer design pattern.

Problem: Complete a Scala trait Observed (shown below) to represent the subject being observed and the Scala trait Observer to represent an observer. The Observed object allows one or more Observers to register with it. Later, it can fire an event and notify all such Observers. The code is like (where E is the type of the event to be fired):

trait Observed[E] {
  def addObserver(o: Observer[E]) ...
  def notifyObservers(ev: E) ...
}

trait Observer[E] {
  def eventOccurred(ev: E)
}

Then use these traits to implement Java bean “bounded properties”, e.g., to allow others to get notified when properties of a Book instance is changed:

//Let others observe changes to its properties
case class Book(var title: String, var price: Double) extends Observed[PropertyChangeEvent] {
  def setTitle(title: String) {
    val oldTitle = this.title
    this.title = title
    //Notify the observers
    ...
  }
  def setPrice(price: Double) {
    val oldPrice = this.price
    this.price = price
    //Notify the observers
    ...
  }
}

//A sample observer class
class Foo extends Observer[PropertyChangeEvent] {
  //Just print some info after a property has been changed
  def eventOccurred(ev: PropertyChangeEvent) = {
    printf("Foo: %s of %s has changed from %s to %s\n", ev.getPropertyName, ev.getSource, ev.getOldValue, ev.getNewValue)
  }
}

object BeanTest {
  def main(args: Array[String]) {
    val b1 = new Book("Scala programming", 35.95)
    val foo = new Foo
    b1.addObserver(foo) //Register the observer
    b1.setTitle("Thinking in Scala")  //foo should get an event
    b1.setPrice(39.95) //ditto
    b1.setTitle("Effective Scala") //ditto
  }
}

The above code should print:

Foo: title of Book(Thinking in Scala,35.95) has changed from Scala programming to Thinking in Scala
Foo: price of Book(Thinking in Scala,39.95) has changed from 35.95 to 39.95
Foo: title of Book(Effective Scala,39.95) has changed from Thinking in Scala to Effective Scala

Try to do it now! Then, click here to see the answer.

Scala exercise 1: template method design pattern

October 3rd, 2010

Introduction

As part of my studying with Scala, I have tried to find some exercises to do but most are either too simple or too “academic”. So, I decided to create a series of Scala exercises to implement the well known design patterns in an industrial context. Here is the first one: template method.

Problem: Create a Scala class JdbcTemplate that has two template methods: execute() and load(). They will execute a SQL statement and load the query results using JDBC respectively. The methods should open and close the connection, create and close the statement and etc. so that the caller doesn’t need to worry about those. For example, the client could use your code like this:

object Test {
  def main(args: Array[String]) {
    Class.forName("org.h2.Driver")
    val t = new JdbcTemplate
    val ds = () => DriverManager.getConnection("jdbc:h2:~/test")
    t.execute(ds, "drop table products if exists")
    t.execute(ds, "create table products (id long primary key, name varchar(128))")
    val insertSql = "insert into products values(?, ?)"
    t.execute(ds, insertSql, List(100, "p1")) //the list is the parameters
    t.execute(ds, insertSql, List(101, "p2")) //ditto
    val ps = t.load[Product](ds, "select * from products", Nil,
      { //this function maps a record into an object
        (rs) => new Product(rs.getLong("id"), rs.getString("name"))
      })
    println(ps)
  }
}

//make it a case class so that it can be printed to the console nicely
case class Product(val id: Long, val name: String) {
}

To make it run, you need to include the h2 database into your classpath. If you use Maven, just add the following to pom.xml:

       <dependency>
             <groupId>com.h2database</groupId>
             <artifactId>h2</artifactId>
             <version>1.2.143</version>
       </dependency>

Note that I am not suggesting that you do this in production code; you’d probably want to use an ORM framework.

Click here to see the answer.

Learn the basic concepts and significance of functional programming in 10 minutes

July 4th, 2010

Introduction

If you’re a Java programmer, this short article will show you the basic concepts of functional programming and the reason why it may be getting more and more important in the near future. All sample code is in Java so it should be very easy to understand.

What is functional programming?

Functional programming is just like our programming in Java or C#, but without assignments. You may wonder is this possible at all? For example, to calculate the total price of a list of product prices, one might write in functional programming style (the code is still in Java):

abstract class List {
}

class EmptyList extends List {
}

class NonEmptyList extends List {
  int head;
  List tail;
}

class Foo {
  int getTotalPrice(List ps) {
    if (ps instanceof EmptyList) {
      return 0;
    } else {
      final NonEmptyList nl = (NonEmptyList) ps; //initialization, NOT assignment
      return nl.head + getTotalPrice(nl.tail);
    }
  }
}

Note that in line 17 there seems to be an assignment, but it is not: It is an initialization and the variable is declared as final, so no modification can be made. This is perfectly allowed in functional programming.

So you can see that assignment is not really required. But why the lack of assignment is a good thing?

Why functional programming may be huge in the near future?

To see the benefit, let’s assume that if a price is >= 100 then you’ll give a 20% off discount. So, you may modify the code as:

class Foo {
  int getTotalPrice(List ps) {
    if (ps instanceof EmptyList) {
      return 0;
    } else {
      final NonEmptyList nl = (NonEmptyList) ps;
      final int headDiscountedPrice = nl.head >= 100 ? (int) (nl.head * 0.8) : nl.head;
      final int tailPrice = getTotalPrice(nl.tail);
      return headDiscountedPrice + tailPrice;
    }
  }
}

Note the three initializations. Because all the variables and fields can’t be modified, their order is unimportant and, in a real functional programming language, the order can be changed at wish without changing the return value. For example, you could change it like:

class Foo {
  int getTotalPrice(List ps) {
    if (ps instanceof EmptyList) {
      return 0;
    } else {
      final int tailPrice = getTotalPrice(nl.tail); //nl is not initialized yet, how it works?
      final int headDiscountedPrice = nl.head >= 100 ? (int) (nl.head * 0.8) : nl.head;
      final NonEmptyList nl = (NonEmptyList) ps;
      return headDiscountedPrice + tailPrice;
    }
  }
}

Note that when initializing the tailPrice variable, the nl variable hasn’t been initialized yet. Will this cause a problem? No. In a real functional programming language, each of three variables will be initialized at the same time with a lazy expression. When the value is really needed, the lazy expression will be evaluated. So, if the value of tailPrice is needed but nl hasn’t been evaluated yet, it will be evaluated and the calculation will proceed. No matter what execution order is, the final total price will be the same.

Now, let’s get to the core issue of why this is important. As the order of these expressions are unimportant, they can be evaluated concurrently. As nowadays we’re getting more CPU cores instead of speedier single CPU, this programming model may become the mainstream in the future as the evaluations of different expressions can be done in different cores.

Note that the semicolon in the code now has a different meaning: In Java, it means sequential execution, but in a real functional programming language, it only separates the expressions but there is no ordering at all. That is, in a real functional programming language, there is no obvious concept of sequential execution.

How to maintain states or write to a database?

How to write to a database in functional programming? Let’s say there is special built-in methods to perform reading and writing to a certain database record. Let’s try to increment the value of the record twice with the code below:

class Foo {
  int read() {
    //...
  }
  void write(int v) {
    //...
  }
  void inc() {
    final int v = read();
    final int w = v+1;
    write(w);
    final int t = w+1;
    write(t); //Problem: It may occur before write(w)!
  }
}

The problem with the code is that, because there is no ordering, write(t) may be evaluated before write(w), so finally the value will have been increased by one, not two! The problem here is that, once we have assignment (the write operation to the record), the ordering becomes important, but there is no obvious ordering in functional programming.

Creating sequential ordering in functional programming

In fact, it is possible to create the effect of sequential evaluation in functional programming. For the above example, you can restructure the code as:

interface Function {
  Object eval(Object arg);
}

class Foo {
  int write(int v) {
    ...
  }
  void inc() {
    final int v = read();
    new Function() {

      public Object eval(Object arg) {
        final int w = v+1;
        final int r = write(w);
        return new Function() {

          public Object eval(Object arg) {
            final int t = w+1;
            return write(t);
          }
        }.eval(r); //Force r to be evaluated before going into eval()
      }
    }.eval(v); //Force v to be evaluated before going into eval()
  }

Now, you’re putting the logic into three steps: step 1: read the record. step 2: write w to the record. step 3: write t to the record. To make sure step 1 is executed before step 2, you wrap step 2 into a Function object and pass the result of step 1 (the “v” variable) as an argument to its eval() method. In a real functional programming language, when calling a method, its arguments will be fully evaluated. So, this guarantees that step 1 is executed before step 2. You use the same trick to put step 3 into a Function object inside step 2 and pass the result of step 2 (the “r” variable) as an argument to its eval() method. For this to work, the write() method must return something instead of void. So, I changed it to return an int.

It works, but the code looks complex. To make it simpler, you can introduce a method like:

class Foo {
  Object chain(Object arg, Function f) {
    return f.eval(arg);
  }
  void inc() {
    final int v = read();
    chain(v, new Function() {

      public Object eval(Object arg) {
        final int w = v+1;
        final int r = write(w);
        return chain(r, new Function() {

          public Object eval(Object arg) {
            final int t = w+1;
            return write(t);
          }
        });
      }
    });
  }

Once you have such a chain() method, you can perform more interesting things. For example, instead of passing the previous result directly as the argument, you could expect it to be an instruction to access the database so that you perform the database access in the chain() in the top level of your program, so that all the code inside is free of any side-effect:

class Read {
}

class Write {
  int v;

  public Write(int v) {
    this.v = v;
  }
}
class Foo {
  Object chain(Object arg, Function f) {
    if (arg instanceof Read) { //arg is now the instruction
      return f.eval(read()); //Perform the side effect here
    }
    if (arg instanceof Write) {
      final int v = ((Write)arg).v;
      return f.eval(write(v)); //Perform the side effect here
    }
    return null;
  }
  void inc() {
    chain(new Read(), new Function() {

      public Object eval(Object arg) {
        final int w = (Integer)arg+1;
        final Write r = new Write(w);
        return chain(r, new Function() {

          public Object eval(Object arg) {
            final int t = w+1;
            return chain(new Write(t), ...);
          }
        });
      }
    });
  }
}

In addition, you can perform other interesting things in the chain() method. For example, the argument could also contain an integer indicating the number of available resources. Every time you could reduce it by one before passing to the next step. If it is zero, you could abort.

BTW, you’ve just learned one of the most intriguing concepts in functional programming: monad. It is just the way above to achieve sequencing, optionally manipulating the previous result, and combining it with the next step.

Better syntax?

Even though it works, the multiple levels of embedding is still very complicated. In a real functional programming, you won’t need to write so much to create the anonymous Function object, to define the eval() method and you probably can use an infix operator in place of the chain() method. For example, in Haskell, the code can be written something like this:

  inc =
    new Read() >>=
    \v ->
        w = v+1
        new Write(w) >>=
    \r ->
        t = w+1
        new Write(t)

Here the >>= infix operator has replaced for the chain() method. The \x -> expresssion defines an anonymous function and x is the argument. It looks simple, right?

What can you do next?

Now you’ve learned the significance of functional programming. What to do next? If you’d like to learn more about functional programming, if you prefer static typing, I’d recommend that you study Haskell by going through this excellent tutorial. Then, for production, you may check out Scala which supports functional programming and integration with JVM and all existing Java classes. It borrows a lot of concepts from Haskell. It also supports assignments and OO, so it can ease the migration from Java.

If you prefer dynamic typing, for production, you may check out Clojure which also integrates with JVM and existing Java classes.

Getting started with Scala, Spring, Hibernate & Wicket

June 19th, 2010

Introduction

Below is a tutorial to document what I did to create a fully working Scala project utilizing the best or the most popular frameworks or tools in enterprise Java such as Spring, Hibernate/JPA, Wicket, Maven and Intellij IDEA. The purpose is to help other Java programmers get started quickly with a fully working enterprise Scala project.

Setting up the IDE

First, download the Intellij IDEA 9.0.x (community edition) as it is the best scala IDE right now. Then choose File | Settings | Plugins, choose the Available tab to install the Scala plugin.

Creating the Maven project

In IDEA, choose File | New Project and choose the Maven module to create a Maven project. Then modify pom.xml as shown below. This will add all the dependencies you need and set up the compilation of Scala classes in the build processes:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.foo</groupId>
    <artifactId>myapp</artifactId>
    <packaging>war</packaging>
    <version>1.0-SNAPSHOT</version>
    <name>myapp Java EE 6 Webapp</name>
    <url>http://maven.apache.org</url>
    <properties>
        <spring.version>3.0.3.RELEASE</spring.version>
    </properties>
    <repositories>
        <repository>
            <id>java.net2</id>
            <name>Repository hosting the jee6 artifacts</name>
            <url>http://download.java.net/maven/2</url>
        </repository>
        <repository>
            <id>scala-tools.org</id>
            <name>Scala-tools Maven2 Repository</name>
            <url>http://scala-tools.org/repo-releases</url>
        </repository>
        <repository>
            <id>wpt-release</id>
            <url>http://wicketpagetest.sourceforge.net/m2-repo/releases</url>
        </repository>
        <repository>
            <id>wpt-snapshot</id>
            <url>http://wicketpagetest.sourceforge.net/m2-repo/snapshots</url>
        </repository>
    </repositories>
    <pluginRepositories>
        <pluginRepository>
            <id>scala-tools.org</id>
            <name>Scala-tools Maven2 Repository</name>
            <url>http://scala-tools.org/repo-releases</url>
        </pluginRepository>
    </pluginRepositories>
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework</groupId>
                <artifactId>spring-context</artifactId>
                <version>${spring.version}</version>
            </dependency>
            <dependency>
                <groupId>org.springframework</groupId>
                <artifactId>spring-core</artifactId>
                <version>${spring.version}</version>
            </dependency>
            <dependency>
                <groupId>org.springframework</groupId>
                <artifactId>spring-web</artifactId>
                <version>${spring.version}</version>
            </dependency>
            <dependency>
                <groupId>org.springframework</groupId>
                <artifactId>spring-orm</artifactId>
                <version>${spring.version}</version>
            </dependency>
        </dependencies>
    </dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>com.h2database</groupId>
            <artifactId>h2</artifactId>
            <version>1.2.120</version>
        </dependency>
        <dependency>
            <groupId>org.hibernate</groupId>
            <artifactId>hibernate-entitymanager</artifactId>
            <version>3.4.0.GA</version>
        </dependency>
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-orm</artifactId>
        </dependency>
        <dependency>
            <groupId>com.ttdev</groupId>
            <artifactId>wpt-core</artifactId>
            <version>1.5.2-SNAPSHOT</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>com.ttdev</groupId>
            <artifactId>wpt-runtime-spring</artifactId>
            <version>1.5.2-SNAPSHOT</version>
        </dependency>
        <dependency>
            <groupId>javax</groupId>
            <artifactId>javaee-web-api</artifactId>
            <version>6.0</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>3.8.2</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>2.8.0.RC3</version>
        </dependency>
        <dependency>
            <groupId>org.apache.wicket</groupId>
            <artifactId>wicket</artifactId>
            <version>1.4.9</version>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-log4j12</artifactId>
            <version>1.4.2</version>
        </dependency>
    </dependencies>
    <build>
        <pluginManagement>
            <plugins>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-compiler-plugin</artifactId>
                    <version>2.0.2</version>
                </plugin>
                <plugin>
                    <groupId>org.scala-tools</groupId>
                    <artifactId>maven-scala-plugin</artifactId>
                    <version>2.9.1</version>
                </plugin>
            </plugins>
        </pluginManagement>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <executions>
                    <execution>
                        <phase>compile</phase>
                        <goals>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <source>1.6</source>
                    <target>1.6</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-war-plugin</artifactId>
                <version>2.1-beta-1</version>
                <configuration>
                    <failOnMissingWebXml>false</failOnMissingWebXml>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.scala-tools</groupId>
                <artifactId>maven-scala-plugin</artifactId>
                <executions>
                    <execution>
                        <id>scala-compile-first</id>
                        <phase>process-resources</phase>
                        <goals>
                            <goal>add-source</goal>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                    <execution>
                        <id>scala-test-compile</id>
                        <phase>process-test-resources</phase>
                        <goals>
                            <goal>testCompile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
        <finalName>myapp</finalName>
    </build>
</project>

Wait for a while and IDEA will prompt you on whether to import the changes into the project. Say yes.

Setting up web.xml

Next, modify main/webapp/WEB-INF/web.xml as below. This sets up the Wicket filter, the Spring filter to open the JPA entity manager and the Spring listener to initialize Spring itself.

<?xml version="1.0" encoding="UTF-8"?>
<web-app version="3.0" xmlns="http://java.sun.com/xml/ns/javaee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd">
    <filter>
        <filter-name>f2</filter-name>
        <filter-class>org.springframework.orm.jpa.support.OpenEntityManagerInViewFilter</filter-class>
    </filter>
    <filter>
        <filter-name>f1</filter-name>
        <filter-class>org.apache.wicket.protocol.http.WicketFilter</filter-class>
        <init-param>
            <param-name>applicationClassName</param-name>
            <param-value>com.foo.myapp.MyApp</param-value>
        </init-param>
    </filter>
    <filter-mapping>
        <filter-name>f1</filter-name>
        <url-pattern>/app/*</url-pattern>
    </filter-mapping>
    <filter-mapping>
        <filter-name>f2</filter-name>
        <url-pattern>/*</url-pattern>
    </filter-mapping>
    <context-param>
        <param-name>contextConfigLocation</param-name>
        <param-value>classpath:/beans.xml</param-value>
    </context-param>
    <listener>
        <listener-class>org.springframework.web.context.ContextLoaderListener</listener-class>
    </listener>
</web-app>

Setting up the Spring beans for database access and transaction

To define those Spring beans, create main/resources/beans.xml with the content below. This defines the entity manager factory, the transaction manager and etc.

<?xml version="1.0" encoding="UTF-8" ?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:context="http://www.springframework.org/schema/context"
       xmlns:tx="http://www.springframework.org/schema/tx"
       xsi:schemaLocation="http://www.springframework.org/schema/beans

http://www.springframework.org/schema/beans/spring-beans-3.0.xsd

http://www.springframework.org/schema/context

http://www.springframework.org/schema/context/spring-context-3.0.xsd

http://www.springframework.org/schema/tx

http://www.springframework.org/schema/tx/spring-tx-3.0.xsd">

    <context:component-scan base-package="com.foo.myapp"/>
    <bean id="emf" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
        <property name="persistenceXmlLocation" value="classpath:/META-INF/my-persistence.xml"/>
    </bean>
    <bean id="transactionManager" class="org.springframework.orm.jpa.JpaTransactionManager">
        <property name="entityManagerFactory" ref="emf"/>
    </bean>
    <tx:annotation-driven />
</beans>

The entity manager factory will read the database configuration from the my-persistence.xml file. So, create it in main/resources/META-INF with the content below. Here, you’ll access an H2 database named myapp in your home directory.

<persistence xmlns="http://java.sun.com/xml/ns/persistence"
             xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
             xsi:schemaLocation="http://java.sun.com/xml/ns/persistence http://java.sun.com/xml/ns/persistence/persistence_2_0.xsd"
             version="2.0">
    <persistence-unit name="myapp" transaction-type="RESOURCE_LOCAL">
        <provider>org.hibernate.ejb.HibernatePersistence</provider>
        <properties>
            <property name="hibernate.connection.driver_class" value="org.h2.Driver"/>
            <property name="hibernate.connection.url" value="jdbc:h2:tcp://localhost/~/myapp"/>
            <property name="hibernate.connection.username" value="sa"/>
            <property name="hibernate.connection.password" value=""/>
            <property name="hibernate.dialect" value="org.hibernate.dialect.H2Dialect"/>
            <property name="hibernate.hbm2ddl.auto" value="create"/>
        </properties>
    </persistence-unit>
</persistence>

Creating the Wicket page

Here, you’ll create a Wicket page to display some products loaded from the database. So, create the main/scala folder, right click it and choose New | Scala Class. Name the class as MyPage and put it into the com.foo.myapp package. The class is shown below:


package com.foo.myapp

import org.apache.wicket.markup.html._
import basic.Label
import list.{ListItem, ListView}
import org.apache.wicket.spring.injection.annot.SpringBean
import org.apache.wicket.model.CompoundPropertyModel

class MyPage extends WebPage {
  @SpringBean
  var ps: ProductService = _
  val productListView = new ListView[Product]("productListView", ps.getAll) {
    def populateItem(item: ListItem[Product]) = {
      item.setModel(new CompoundPropertyModel[Product](item.getDefaultModelObject))
      item.add(new Label("name"))
      item.add(new Label("price"))
    }
  }
  add(productListView)
}

Note that it is using a ProductService object to load the products. You’ll create it later. In addition, note that the field is assigned to an underscore (_), which tells the Scala compile to NOT initialize, but leave it at the default state (null in this case). This is required for the injection to work. If you assign it to null explicitly, you
will overwrite the Spring bean as the injection will occur before the constructor of MyPage is executed.

Now, create the MyPage.html file in src/main/resources/com/foo/myapp:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
    <table border="1">
        <tr wicket:id="productListView">
            <td wicket:id="name"></td>
            <td wicket:id="price"></td>
        </tr>
    </table>
</html>

Creating the ProductService

Right click the com.foo.myapp package in the src/main/scala folder and choose New | Scala Class, then choose to create a trait named ProductService:

package com.foo.myapp

import java.util.List

trait ProductService {
  def getAll: List[Product]
}

This is the interface. To create the implementation, create a DefaultProductService Scala class in the same package:

package com.foo.myapp

import javax.persistence.{PersistenceContext, EntityManager}
import org.springframework.stereotype.{Service}
import org.springframework.transaction.annotation.Transactional
import org.springframework.beans.factory.annotation.Autowired

@Service
@Transactional
class DefaultProductService extends ProductService {
  @Autowired
  var products: Products = _

  def getAll = products.getAll
}

Note that it is using a DAO named of type Products to do the work.

Creating the Products DAO

To create the DAO, create a trait named Products in the same package:

package com.foo.myapp

import java.util.List

trait Products {
   def getAll: List[Product]
}

Then create the implementation Scala class DefaultProducts in the same package:

package com.foo.myapp

import javax.persistence.{PersistenceContext, EntityManager}
import org.springframework.stereotype.Repository
import java.util.List

@Repository
class DefaultProducts extends Products {
  @PersistenceContext
  var em: EntityManager = _

  def getAll = {
    em.createQuery("select p from Product p").getResultList.asInstanceOf[List[Product]]
  }
}

Creating the entity class

Create the Product class and map it to the database:

package com.foo.myapp

import javax.persistence.{GeneratedValue, Id, Entity}

@Entity
class Product {
  @Id
  @GeneratedValue
  var id: Long = _
  var name: String = _
  var price: Double = _
}

Creating the Wicket application class

Finally, create MyApp Scala class in the same package:


package com.foo.myapp

import org.apache.wicket.protocol.http.WebApplication
import com.ttdev.wicketpagetest.MockableSpringBeanInjector

class MyApp extends WebApplication {
  def getHomePage = classOf[MyPage]

  override def init = {
    MockableSpringBeanInjector.installInjector(this)
  }
}

Here you specify MyPage as the home page and install an injector that can inject Spring beans (as well as mock objects, even though you won’t use this capability here).

Running the application

As the Intellij IDEA community edition doesn’t include the integration with app servers, you’ll embed Jetty to run your application. This is easy. Just create a Scala class ManualTest in the com.foo.myapp package in the test/scala folder (NOT the main/scala folder). The content is below:

package com.foo.myapp

import com.ttdev.wicketpagetest.{WebAppJettyConfiguration, WicketAppJettyLauncher}

object ManualTest {
  def main(args: Array[String]) {
    val l = new WicketAppJettyLauncher
    l.startAppInJetty(new WebAppJettyConfiguration)
  }
}

To run the application, you need to have the H2 database server running first. So, go to http://www.h2database.com to download and unpack it. Then change into h2/bin and run h2.bat (or h2.sh on Linux).

While the ManualTest class is the active editor in IDEA, choose Run | Run in IDEA. It will run ManualTest which will launch Jetty to run your application.

To test it, try accessing http://localhost:8888/app in a browser. It should display nothing as there is no product in the database. However, it will create the table required.

To add some products, go to http://localhost:8082 to access the H2 web client. Enter jdbc:h2:tcp://localhost/~/myapp as the JDBC URL. Click Connect. Then issue the SQL statement:

SELECT * FROM PRODUCT

to select the product. Nothing should be there. That’s fine. In the result display, click the plus sign to create a product record. Feel free to add more.

Finally, reload http://localhost:8888/app and you should see the products displayed in the Wicket page.

Updated: You can download the project folder.

Scala IDE has matured for the mass

June 18th, 2010

For a language to be readily used by the mass, in addition to the language itself, it needs a production quality runtime, rich libraries and IDE (I know superstar programmers can use Emacs or vi to program, but I believe the mass including myself really need an IDE that works). As Scala is using the JVM and can use all the libraries written in Java, the first two conditions are already checked. What has been holding it back is the IDE. The good news is, the free intellij IDEA community edition now meets all the needs of the majority of web programmers.

Why? First of all, code complete actually works. Second, it supports Maven project mixing Java and Scala. Third, even though the community edition doesn’t support integration with app servers, we can easily launch an embedded Jetty to run the webapp right in the IDE (to see an example, see the example given by Larry H there). The effect is no different from having an integration with Jetty.

Before trying intellij IDEA, I also tried Eclipse and NetBeans. The Scala plugin for Eclipse is of alpha quality (and has always been so for the past several years). Every time I tried it always had all kinds of obvious problems like code-completion not working while it should, showing errors while the code compiles fine with scalac.

The Scala plugin for NetBeans is much better than the Eclipse one and is quite close to being production ready. Basically code complete works. However, when auto-completing a class name, it won’t add the import statement for you. In addition, it can’t generate the method signature for you when you want to override a method or implement an abstract method. In my opinion these shortcomings must be fixed before it can be used in day-to-day work.