Handling a Trillion Data Using ZK
This page is under construction, so we cannot guarantee the accuracy of the content!
Jumper Chen, Senior Engineer, Potix Corporation
March 12, 2012
ZK 6
Introduction
Cloud services and storage are the hot IT topics recently, and the desires of displaying a huge data in ZK are buzz from our users. After doing some experiments, we brew a single component with model named Biglistbox, a Big Listbox component, to handle an unlimited data set for a big table and provide the same as many as Listbox functions, selection, sorting, keystroke navigation, ROD(rendering-on-demand), and so on. The best features in the Biglistbox are that it uses the server-side memory as less as it can and optimizes the updating area as minimum as its viewport at client-side, so that it can display as fast as you never seen before.
In the following demo, we will demonstrate how to use Biglistbox to handle a trillion data (1,000,000,000,000 cells).
Demo
Demo Code Details
In the following subsections, we will guide you how to implement those stuffs (ZUL, Composer, MatrixModel, ...) for the Biglistbox.
ZUL File
<biglistbox id="myComp" hflex="1" vflex="1" xmlns:w="client" w:onScroll="jq('$tip').fadeOut();">
<!-- Template example
<template name="heads">
<html><![CDATA[
<div class="images_${matrixInfo[0]%28}" title="x=${matrixInfo[0]},y=${matrixInfo[1]}">${each[matrixInfo[0]]}</div>
]]></html>
</template>
<template name="rows">
<html><![CDATA[
<div class="images_${matrixInfo[0]%28}" title="x=${matrixInfo[0]},y=${matrixInfo[1]}">${each[matrixInfo[0]]}</div>
]]></html>
</template> -->
</biglistbox>
As you can see, the Biglistbox can support V/Hflex function to adjust its size automatically and provide the powerful template feature to organize the layout of the HTML output, and then you can utilize two attributes the rowIndex and colIndex from the matrixInfo object to receive the current index while template rendering phase. In our demo, we use MatrixRenderer to handle a complex HTML output.
Composer
public class DemoWindowComposer extends SelectorComposer<Window> {
//... omitted
public void doAfterCompose(Window comp) throws Exception {
super.doAfterCompose(comp);
// specify a trillion faker model
myComp.setModel(new FakerMatrixModel(1000 * 1000, 1000 * 1000));
myComp.setColWidth("130px");
myComp.setMatrixRenderer(new MatrixRenderer<List<String>>() {
@Override
public String renderCell(Component owner, List<String> data,
int rowIndex, int colIndex) throws Exception {
String d = data.get(colIndex);
d = d.replace("ZK", "<span class='red' title='ZK'>ZK</span>")
.replace("Hello", "<span class='blue' title='Hello'>Hello</span>");
return "<div class='images_" + (colIndex%28) + "' title='x=" +
colIndex + ",y=" + rowIndex + "'>" + d + "</div>";
}
@Override
public String renderHeader(Component owner, List<String> data,
int rowIndex, int colIndex) throws Exception {
return "<div class='images_" + (colIndex % 28) + "' title='"
+ images[colIndex % 28] + "'>" + data.get(colIndex)
+ "</div>";
}
});
myComp.setSortAscending(new MyMatrixComparatorProvider<List<String>>(true));
myComp.setSortDescending(new MyMatrixComparatorProvider<List<String>>(false));
}
//... omitted
}
In the DemoWindowComposer class, we create a FakerMatrixModel instance, which implements MatrixModel, to handle 1,000,000 columns and 1,000,000 rows data and instantiates an anonymous MatrixRenderer class to render the HTML string output for display. The sorting function of the Biglistbox we used is to implement the MatrixComparatorProvider to provide each comparator from the given column index of the Biglistbox when user clicks on it. Those implementations above look similar to manipulate ZK Listbox, if you ever used it.
MatrixModel
Here is the source code of the MatrixModel interface
/**
* The interface defines the methods that components like {@link Biglistbox}
* to get the content of cells and headers from the matrix data.
* @author jumperchen
*/
public interface MatrixModel<Row, Head, Cell, Header> extends ListModel<Row> {
/**
* Returns the length of the columns
*/
public int getColumnSize();
/**
* Returns the length of the heads
*/
public int getHeadSize();
/**
* Returns the data object of the head from the specified index.
* @param rowIndex the index of the row in the headers
*/
public Head getHeadAt(int rowIndex);
/**
* Returns the data object of the cell from the specified index.
* @param rowData the row data
* @param columnIndex the index of the column
*/
public Cell getCellAt(Row rowData, int columnIndex);
/**
* Returns the data object of the header from the specified index.
* @param headData the head data
* @param columnIndex the index of the column
*/
public Header getHeaderAt(Head headData, int columnIndex);
}
In this demo, we create a FakerMatrixModel to implement the MatrixModel. Here is the fragment code:
public class FakerMatrixModel<Head extends List, Row extends List, Cell, Header> extends
AbstractListModel<Row> implements MatrixModel<Row, Head, Cell, Header>, Sortable {
// omitted...
private boolean _sortDir = true;
@Override
public Row getElementAt(int index) {
final int rowIndex = _sortDir ? index : getSize() - index - 1; // handle the sorting
final String key = String.valueOf(rowIndex);
List<String> value = _rowCache.get(key);
if (value == null) {
value = new FakerKeyList<String>(_colSize, rowIndex, new Fun() {
@Override
public Object apply(int index) {
return "y = " + rowIndex;
}});
_rowCache.put(key, value);
}
return (Row) value;
}
// omitted...
}
The MatrixModel extends from ListModel interface and uses the getElementAt(int) method to receive the row data from the FakerKeyList object that implements the List interface. The reason we used FakerKeyList is that we encounter a big performance issue come from the default implementation that Java Collection Framework provided, the issue is that Java Collection Framework will go through each entry to gather the value of hashCode for seeking the key in the Map/Set or to check each entry for the equals and toString functions, this implementations will reduce the Biglistbox component's performance. Therefore, using the Biglistbox component with MatrixModel has to implement a clever simple list for traversing the huge data set.
For example,
private class FakerKeyList<T> extends AbstractList<T> {
final int _size;
Map<String, T> _updateCache = new HashMap<String,T> ();
final Fun<?> _fn;
final String _key;
public FakerKeyList(int size, int key, Fun<?> fn) {
_size = size;
_key = key + "_" + size;
_fn = fn;
}
@Override
public T get(int index) {
// if changed, returns the changed value
Object val = _updateCache.get(String.valueOf(index));
if (val != null)
return (T) val;
return (T) _fn.apply(index);
}
@Override
public int hashCode() {
return _key.hashCode();
}
@Override
public boolean equals(Object obj) {
if (obj == this)
return true;
if (obj instanceof FakerKeyList) {
return _key.equals(((FakerKeyList)(obj))._key);
}
return false;
}
@Override
public String toString() {
return _key;
}
// omitted...
}
As you can see, we use a key string as the key for toString, hashCode, and equals methods to speed up the seeking time, and Fun class is a handy class to render the model data for this demo.
Note: By default, ZK do not provide an implementation class for the MatrixModel, because the Biglistbox component is designed to handle unlimited data set, there is no need to handle the model data in memory and most of this usage will be done by application-dependent. However, you can extend your own implementation from the AbstractListModel skeleton class.
MatrixRenderer
Here is the implementation of MatrixRenderer for this demo.
new MatrixRenderer<List<String>>() {
@Override
public String renderCell(Component owner, List<String> data,
int rowIndex, int colIndex) throws Exception {
String d = data.get(colIndex);
d = d.replace("ZK", "<span class='red' title='ZK'>ZK</span>")
.replace("Hello", "<span class='blue' title='Hello'>Hello</span>");
return "<div class='images_" + (colIndex%28) + "' title='x=" +
colIndex + ",y=" + rowIndex + "'>" + d + "</div>";
}
@Override
public String renderHeader(Component owner, List<String> data,
int rowIndex, int colIndex) throws Exception {
return "<div class='images_" + (colIndex % 28) + "' title='"
+ images[colIndex % 28] + "'>" + data.get(colIndex)
+ "</div>";
}
}
As you can see, we must implement two methods renderCell and renderHeader that MatrixRenderer defines to generate the HTML output string. In the MatrixRenderer interface we designed is to allow developer to generate any kind of HTML result to display.
MyMatrixComparatorProvider
Here is the source code of MyMatrixComparatorProvider to provide a corresponding comparator for Biglistbox to sort.
private class MyMatrixComparatorProvider<T> implements
MatrixComparatorProvider<List<String>> {
private int _x = -1;
private boolean _acs;
private MyComparator _cmpr;
public MyMatrixComparatorProvider(boolean asc) {
_acs = asc;
_cmpr = new MyComparator(this);
}
@Override
public Comparator<List<String>> getColumnComparator(int columnIndex) {
this._x = columnIndex;
return _cmpr;
}
// a real String comparator
private class MyComparator implements Comparator<List<String>> {
private MyMatrixComparatorProvider _mmc;
public MyComparator(MyMatrixComparatorProvider mmc) {
_mmc = mmc;
}
@Override
public int compare(List<String> o1, List<String> o2) {
return o1.get(_mmc._x).compareTo(o2.get(_mmc._x))
* (_acs ? 1 : -1);
}
}
}
As you can see, the MyMatrixComparatorProvider class must implement the MatrixComparatorProvider interface that defines a single method getColumnComparator(int columnIndex) to receive the correct comparator from the specified columnIndex. In this demo, we use a single comparator with different column index to sort the model data.
Component Usages
Features
Here lists the features of this Biglistbox component we supported in this version.
- Single selection
- Column sorting
- Keystroke navigation (Down, Up, Left, Right, Home, End, PageUp, PageDown, and Mousewheel)
- Autosizing (V/Hflex)
- Auto-adjust columns and rows for viewport
- Client-side cache model
- Render-on-Demand (client and server side)
- Stripe and mouse effects
- Template supports
Properties
- cols: specify the column size for the viewport.
- rows: specify the row size for the viewport.
- colWidth: specify the column width for each column.
- rowHeight: specify the row height for each row.
- matrixRenderer: specify the matrix renderer.
- model: specify the matrix model.
- oddRowSclass: specify the sclass for this component to stripe.
Events
- onSelect: when users clicks or navigates on the row.
- onSort: when users clicks on a column header
- onCellClick: when users clicks on the row cell.
- onScroll: when the scrolling bar is changed either x axis or y axis.
- onScrollX: when the x axis of the scrolling bar changed
- onScrollY: when the y axis of the scrolling bar changed
Component Limitation
- Browsers support for IE8+, Firefox, Opera, Safari, and Chrome only.
- The max memory usage for client side cache model is limited by end-users computer memory. (turn off by default)
- Server side performance is depended on the MatrixModel implementation.
What's the Next
We are glad to hear which features are important for our users, that will be part of the next version. Here just lists a few features we can imagine.
- Multiple selection
- Column/Row size resizing
- Frozen table like MS-excel
- Tootip/Popup for each cell or header
- Embed ZK component inside the cell
- Multiple heads
- Merge cells (column/row)
- Scrolling tips
- Cell selection
Please give us feedback to vote which one is important for us.
Download
You can download the sample application code from its github repo here
Comments
| Copyright © Potix Corporation. This article is licensed under GNU Free Documentation License. |