Displaying Huge Amount of Data

The Huge Amount of Data Problem

When we display data with data components (Listbox or Grid), we may query all data at one time and store them in a model object. But sometimes the time to query all data from a database is unbearable long. To cope with this problem, we demonstrate a custom ListModel implementation which only queries a small size of data a time instead of all data in this section.

Cache One Page Size of Data

Assume we are going to display large amount of personal information in a Listbox. Because there are too many of them, they cannot be displayed at one time. One way is set "rows" to limit the visible rows, and another way is to use "paging" mold.

	<window  width="400px" apply="org.zkoss.bind.BindComposer"
		viewModel="@id('vm') @init('org.zkoss.reference.developer.mvvm.advance.HugeDataVM')">
		 Use custom ListModel:
		<listbox model="@bind(vm.personListModel)" selectedItem="@bind(vm.selectedPerson)" rows="10">
			<listhead>
				<listheader label="ID" />
				<listheader label="First Name" />
				<listheader label="Last Name" />
				<listheader label="Age" />
			</listhead>
			<template name="model">
				<listitem>
					<listcell label="@bind(each.id)" />
					<listcell label="@bind(each.firstName)" />
					<listcell label="@bind(each.lastName)" />
					<listcell label="@bind(each.age)" />
				</listitem>
			</template>
		</listbox>
...
	</window>

As querying all data is time-consuming, we need to implement a custom ListModel for the Listbox. It only queries a page size of data during one request execution and store it as a cache.

package org.zkoss.reference.developer.mvvm.advance;

import java.util.*;
import org.zkoss.reference.developer.mvvm.advance.domain.*;
import org.zkoss.zk.ui.*;
import org.zkoss.zul.AbstractListModel;

public class LivePersonListModel extends AbstractListModel<Person>{

	private static final long serialVersionUID = -7982684413905984053L;
	
	private PersonDao personDao;
	private int pageSize = 10;
	private final String CACHE_KEY= LivePersonListModel.class+"_cache";
	
	public LivePersonListModel(PersonDao personDao){
		this.personDao = personDao;
	}
	
	public Person getElementAt(int index) {
		Map<Integer, Person> cache = getCache();

		Person targetPerson = cache.get(index);
		if (targetPerson == null){
			//if cache doesn't contain target object, query a page size of data starting from the index
			List<Person> pageResult = personDao.findAll(index, pageSize);
			int indexKey = index;
			for (Person o : pageResult ){
				cache.put(indexKey, o);
				indexKey++;
			}
		}else{
			return targetPerson;
		}

		//get the target after query from database
		targetPerson = cache.get(index);
		if (targetPerson == null){
			//if we still cannot find the target object from database, there is inconsistency between memory and the database
			throw new RuntimeException("Element at index "+index+" cannot be found in the database.");
		}else{
			return targetPerson;
		}
	}
	
	private Map<Integer, Person> getCache(){
		Execution execution = Executions.getCurrent();
		//we only reuse this cache in one execution to avoid accessing detached objects.
		//our filter opens a session during a HTTP request
		Map<Integer, Person> cache = (Map)execution.getAttribute(CACHE_KEY);
		if (cache == null){
			cache = new HashMap<Integer, Person>();
			execution.setAttribute(CACHE_KEY, cache);
		}
		return cache;
	}	

	public int getSize() {
		return personDao.findAllSize();
	}


	public int getPageSize() {
		return pageSize;
	}

	public void setPageSize(int pageSize) {
		this.pageSize = pageSize;
	}
}

It works fine under paging or default mold.