Handling a Trillion Data Using ZK

From Documentation
DocumentationSmall Talks2012MarchHandling a Trillion Data Using ZK
Handling a Trillion Data Using ZK
Author
Jumper Chen, Senior Engineer, Potix Corporation
Date
March 12, 2012
Version
ZK 6


Introduction

Cloud services and storage are hot technology topics recently, this has also caused a buzz in the community the desire to display huge amounts of data in ZK. After several trials and experiments, we came up with a single component that would satisfy this need and we name it the Biglistbox. This Biglistbox component is able to handle unlimited data sets for a big table whilst providing the same and as many as the functionalities of Listbox including selection, sorting, keystroke navigation, ROD(rendering-on-demand), and so on. Performance is also optimized at full capacity; Biglistbox is designed to use as little server-side memory as possible and minimize updating area to the size of its viewport at client-side, displaying data at a speed so fast you have never seen before.

Demo

This demo will demonstrate how one trillion data (1,000,000,000,000 cells) can be handled using Biglistbox


For clearer explanation of the demo, please turn your speakers on.

Demo Code Details

In the following subsections, I will guide you through how to implement ZUL, Composer, MatrixModel etc for the Biglistbox.

ZUL File

<biglistbox id="myComp" hflex="1" vflex="1" xmlns:w="client" w:onScroll="jq('$tip').fadeOut();">
    <!-- Template example
    <template name="heads">
        <html><![CDATA[
    <div class="images_${matrixInfo[0]%28}" title="x=${matrixInfo[0]},y=${matrixInfo[1]}">${each[matrixInfo[0]]}</div>
        ]]></html>
    </template>
    <template name="rows">
        <html><![CDATA[
     <div class="images_${matrixInfo[0]%28}" title="x=${matrixInfo[0]},y=${matrixInfo[1]}">${each[matrixInfo[0]]}</div>
        ]]></html>
    </template> -->
</biglistbox>

As you can see, Biglistbox supports V/Hflex functions, it can therefore automatically adjusts its size depending on the window size, providing powerful template feature to organize the layout of HTML output. You can also utilize two attributes - rowIndex & colIndex from the matrixInfo object to receive the current index during template rendering phase. In this demo, we use MatrixRenderer to handle complex HTML output.

Composer

public class DemoWindowComposer extends SelectorComposer<Window> {
	//... omitted
	
	public void doAfterCompose(Window comp) throws Exception {
		super.doAfterCompose(comp);

		// specify a trillion faker model
		myComp.setModel(new FakerMatrixModel(1000 * 1000, 1000 * 1000));
		myComp.setColWidth("130px");
		myComp.setMatrixRenderer(new MatrixRenderer<List<String>>() {

			@Override
			public String renderCell(Component owner, List<String> data,
					int rowIndex, int colIndex) throws Exception {
				String d = data.get(colIndex);
				d = d.replace("ZK", "<span class='red' title='ZK'>ZK</span>")
						.replace("Hello", "<span class='blue' title='Hello'>Hello</span>");
				return "<div class='images_" + (colIndex%28) + "' title='x=" + 
				colIndex + ",y=" + rowIndex + "'>" + d + "</div>";
			}

			@Override
			public String renderHeader(Component owner, List<String> data,
					int rowIndex, int colIndex) throws Exception {
				return "<div class='images_" + (colIndex % 28) + "' title='"
						+ images[colIndex % 28] + "'>" + data.get(colIndex)
						+ "</div>";
			}
		});
		myComp.setSortAscending(new MyMatrixComparatorProvider<List<String>>(true));
		myComp.setSortDescending(new MyMatrixComparatorProvider<List<String>>(false));
	}

	//... omitted
}

In the DemoWindowComposer class, we create a FakerMatrixModel instance, which implements MatrixModel to handle 1,000,000 columns and 1,000,000 rows of data and instantiates an anonymous MatrixRenderer class to render the HTML string output for display. We implement MatrixComparatorProvider as the sorting function of Biglistbox in order to provide each comparator from the given column index of the Biglistbox when users click on it. If you are familiar with ZK Listbox, you might notice that these implementations above look similar to that of manipulating ZK's Listbox.

MatrixModel

Here is the source code of the MatrixModel interface 

/**
 * The interface defines the methods that components like {@link Biglistbox}
 * to get the content of cells and headers from the matrix data.
 * @author jumperchen
 */
public interface MatrixModel<Row, Head, Cell, Header> extends ListModel<Row> {
	/**
	 * Returns the length of the columns
	 */
	public int getColumnSize();
	/**
	 * Returns the length of the heads
	 */
	public int getHeadSize();
	/**
	 * Returns the data object of the head from the specified index.
	 * @param rowIndex the index of the row in the headers
	 */
	public Head getHeadAt(int rowIndex);
	/**
	 * Returns the data object of the cell from the specified index.
	 * @param rowData the row data
	 * @param columnIndex the index of the column
	 */
	public Cell getCellAt(Row rowData, int columnIndex);
	/**
	 * Returns the data object of the header from the specified index.
	 * @param headData the head data
	 * @param columnIndex the index of the column
	 */
	public Header getHeaderAt(Head headData, int columnIndex);
}

In this demo, we create a FakerMatrixModel to implement MatrixModel. Following is the fragment code: 

public class FakerMatrixModel<Head extends List, Row extends List, Cell, Header> extends
		AbstractListModel<Row> implements MatrixModel<Row, Head, Cell, Header>, Sortable {

	// omitted...

	private boolean _sortDir = true;

	@Override
	public Row getElementAt(int index) {
		final int rowIndex = _sortDir ? index : getSize() - index - 1; // handle the sorting
		final String key = String.valueOf(rowIndex);
		List<String> value = _rowCache.get(key);
		if (value == null) {
			value = new FakerKeyList<String>(_colSize, rowIndex, new Fun() {
				@Override
				public Object apply(int index) {
					return "y = " + rowIndex;
				}});
			_rowCache.put(key, value);
		}
		return (Row) value;
	}

	// omitted...
}

MatrixModel is extended from the ListModel interface and uses the getElementAt(int) method to receive row data from the FakerKeyList object that implements the List interface. The reason we use FakerKeyList is because we encounter a big performance issue when using the default implementation of Java Collection Framework as it goes through every entry to gather the value of hashCode when searching the key in Map/Set or to check every entry for equals and toString functions. This implementation method greatly reduces the performance of Biglistbox, therefore, to use the Biglistbox component with MatrixModel, we need to implement a clever and simple List for traversing huge data sets.

For example, 

private class FakerKeyList<T> extends AbstractList<T> {
	final int _size;
	Map<String, T> _updateCache = new HashMap<String,T> ();
	final Fun<?> _fn;
	final String _key;

	public FakerKeyList(int size, int key, Fun<?> fn) {
		_size = size;
		_key = key + "_" + size;
		_fn = fn;
	}

	@Override
	public T get(int index) {
		// if changed, returns the changed value
		Object val = _updateCache.get(String.valueOf(index));
		if (val != null)
			return (T) val;
		return (T) _fn.apply(index);
	}

	@Override
	public int hashCode() {
		return _key.hashCode();
	}
		
	@Override
	public boolean equals(Object obj) {
		if (obj == this)
			return true;
		if (obj instanceof FakerKeyList) {
			return _key.equals(((FakerKeyList)(obj))._key);
		}
		return false;
	}
	
	@Override
	public String toString() {
		return _key;
	}

	// omitted...
}

As you can see, we use a key string as the key for toString, hashCode, and equals methods to speed up searching time. Fun class on the other hand, is a handy class to render the model data for this demo.

Note: By default, ZK does not provide an implementation class for MatrixModel because Biglistbox is designed to handle unlimited data set, therefore, there is no need to handle model data in memory. This usage is application-dependent and varies from case to case. However, you can extend your own implementation from the AbstractListModel skeleton class.

MatrixRenderer

Here is the implementation of MatrixRenderer for this demo. 

new MatrixRenderer<List<String>>() {

	@Override
	public String renderCell(Component owner, List<String> data,
			int rowIndex, int colIndex) throws Exception {
		String d = data.get(colIndex);
		d = d.replace("ZK", "<span class='red' title='ZK'>ZK</span>")
				.replace("Hello", "<span class='blue' title='Hello'>Hello</span>");
		return "<div class='images_" + (colIndex%28) + "' title='x=" + 
		colIndex + ",y=" + rowIndex + "'>" + d + "</div>";
	}

	@Override
	public String renderHeader(Component owner, List<String> data,
			int rowIndex, int colIndex) throws Exception {
		return "<div class='images_" + (colIndex % 28) + "' title='"
				+ images[colIndex % 28] + "'>" + data.get(colIndex)
				+ "</div>";
	}
}

As you can see, we must implement two methods - renderCell and renderHeader defined by MatrixRenderer to generate the HTML output string for cells and headers. In the MatrixRenderer interface, the two methods we designed allow developers to generate any kind of HTML result for display.

MyMatrixComparatorProvider

Following is the source code of MyMatrixComparatorProvider to provide a corresponding comparator for Biglistbox to execute the sorting process. 

private class MyMatrixComparatorProvider<T> implements
			MatrixComparatorProvider<List<String>> {
	private int _x = -1;

	private boolean _acs;

	private MyComparator _cmpr;

	public MyMatrixComparatorProvider(boolean asc) {
		_acs = asc;
		_cmpr = new MyComparator(this);
	}

	@Override
	public Comparator<List<String>> getColumnComparator(int columnIndex) {
		this._x = columnIndex;
		return _cmpr;

	}

	// a real String comparator
	private class MyComparator implements Comparator<List<String>> {
		private MyMatrixComparatorProvider _mmc;

		public MyComparator(MyMatrixComparatorProvider mmc) {
			_mmc = mmc;
		}

		@Override
		public int compare(List<String> o1, List<String> o2) {
			return o1.get(_mmc._x).compareTo(o2.get(_mmc._x))
					* (_acs ? 1 : -1);
		}
	}
}

As it can be seen above, the MyMatrixComparatorProvider class must implement the MyMatrixComparatorProvider interface that defines a single method getColumnComparator(int columnIndex) to receive the correct comparator from the specified column index. In this demo, we use a single comparator with different column indexes to sort the model data.

Component Usages

Features

Here is a list of features of that we support in this version of Biglistbox

  • Single selection
  • Column sorting
  • Keystroke navigation (Down, Up, Left, Right, Home, End, PageUp, PageDown, and Mousewheel)
  • Autosizing (V/Hflex)
  • Auto-adjust columns and rows for viewport
  • Client-side cache model
  • Render-on-Demand (client and server side)
  • Striping row and mouse event effects
  • Template rendering

Properties

Capture1.PNG

  • cols: specify column size for viewport.
  • rows: specify row size for viewport.
  • colWidth: specify column width for each column.
  • rowHeight: specify row height for each row.
  • matrixRenderer: specify matrix renderer.
  • model: specify matrix model.
  • oddRowSclass: specify sclass for this component to stripe.

Events

  • onSelect: when users click or navigate on the row.
  • onSort: when users click on a column header
  • onCellClick: when users click on the row cell.
  • onScroll: when the scrolling bar changes, either x axis or y axis.
  • onScrollX: when the x axis of the scrolling bar changes
  • onScrollY: when the y axis of the scrolling bar changes

Component Limitation

  • Browser support for IE8+, Firefox, Opera, Safari, and Chrome only.
  • Maximum memory usage for client side cache model is limited by end-users' computer memory. (turned off by default)
  • Server-side performance is depended on the MatrixModel implementation.

What's Next

We are glad to hear what features are important for our users that will be included in the next version. Here's just a list of some features we can imagine.

  1. Multiple selection
  2. Column/Row size resizing
  3. Frozen table like MS-excel
  4. Tootip/Popup for each cell or header
  5. Embed ZK component inside the cell
  6. Multiple heads
  7. Merge cells (column/row)
  8. Scrolling tips
  9. Cell selection

Please give us feedback, vote to tell us which one is more important for you!

Download

You can download the sample application code from its github repo here


Comments



Copyright © Potix Corporation. This article is licensed under GNU Free Documentation License.


Retrieved from "https://www.zkoss.org/_w/index.php?title=Small_Talks/2012/March/Handling_a_Trillion_Data_Using_ZK&oldid=53843"