Zu - A simple cluster management system based on Twitter’s ServerSets api

What is it?

A simple cluster management system for partitioned data. E.g. a distributed search engine.

Zu is built on top of Twitter’s Zookeeper based server management api called ServerSets.

Where does the name mean?

Zu comes from the Chinese character: 组, which means a group or a set.

What are some of Zu’s features?

• Simple api
• Manages partitioned data
• Integration with Twitter’s Finagle
• Multi-tenancy for Finagle
• Custom serialization
• Scatter-gather support
• Customizable requesting routing algorithm

Build it:

mvn clean package

2 artifacts:

• zu-core - core library
• zu-finagle - Finagle support

Maven:

Core:

<groupid>com.senseidb.zu</groupid>
<artifactId>zu-core</artifactId>

Finagle extension:

<groupid>com.senseidb.zu</groupid>
<artifactId>zu-finagle</artifactId>

Zu Core:

• Cluster management
• Routing algorithm API

Code snippet:

// Zookeepr location
String zkHost = ...
int zkPort = ...

ZuCluster cluster = new ZuCluster(new InetSocketAddress(zkHost,zkPort), "mycluster");

cluster.addClusterEventListener(new ZuClusterEventListener() {

      @Override
      public void clusterChanged(Map<Integer, ArrayList<InetSocketAddress>> clusterView) {
	    // a shard -> host mapping
	    System.out.println("cluster changed");
      }
 });

// join a cluster
List<EndPointStatus> endpoints = cluster.join(new InetSocketAddress(host,port), new HashSet<Integer>)(Arrays.asList(1,2)));

// leaving a cluster
cluster.leave(endpoints);

For more details, checkout the unit tests here.

Zu Finagle

Zu finagle is an extension to Twitter’s Finagle with the following enhancements:

(To learn more about Finagle, checkout this great example)

• Multi-tenancy - add handlers for different APIs without rebuilding the service IDL
• Custom serializer - define different rpc serializer for each handler
• Scatter-gather support

Example 1:

This example defines an API that returns the length of a string and is then deployed as a server.

See code listing here.

Example 2:

This example defines a set of 3 finagle servers, each handles different shards. They all join a cluster with scatter-gather logic defined.

We construct a broker based on the provided scatter-gather logic with random request routing logic for each shard.

See code listing here.

Example 3:

We have the same setup as example 2, but instead of starting the finagle servers as finagle services, we wrap them up as ZuFinagleServers.

We then create the broker both as a finagle service as well a ZuFinagleServer in different tests.

See code listing here.

Example 4:

Same as example 3, but instead of starting the ZuFinagleServers the wraps finagle services, we register our API as a ZuTransportService. This allows us to add different handlers while abstracting finagle’s inner details as well as without having to re-compile the underlying thrift IDLs.

See code listing here.