JavaScript Chart with Virtualized Data 10 Million Points on server

One of the challenges of big-data JavaScript Charts is that data is often located on the server.

In this JavaScript Chart Example we show how to dynamically load a dataset with 10 million points as the chart is zoomed or panned, virtualizing and reducing the data on the server side.

How to Virtualize Data in SciChart.js

Virtualising data is quite simple in SciChart.js. First, you need to create your chart as normal, but keep references to dataSeries that you want to update and the xAxis as we’re going to listen to axis.visibleRange changes.. We’re going to update the data later when the user zooms and pans the chart.

const { wasmContext, sciChartSurface } = await SciChartSurface.create(divElementId, {theme: appTheme.SciChartJsTheme});
const xAxis = new NumericAxis(wasmContext, {
	axisAlignment: EAxisAlignment.Bottom,
	visibleRange: new NumberRange(4000000, 5000000),
	autoRange: EAutoRange.Never,
	labelPrecision: 0,
	useNativeText: true
});

sciChartSurface.xAxes.add(xAxis);
const yAxis = new NumericAxis(wasmContext, {
	axisAlignment: EAxisAlignment.Right,
	visibleRange: new NumberRange(-5000, 5000),
	autoRange: EAutoRange.Never,
	labelPrecision: 0,
	useNativeText: true
});
sciChartSurface.yAxes.add(yAxis);

// Keep a reference to the dataSeries. We will update this later to virtualize the chart
const dataSeries = new XyDataSeries(wasmContext, { containsNaN: false, isSorted: true });
const rendSeries = new FastLineRenderableSeries(wasmContext, { dataSeries, strokeThickness: 2, stroke: appTheme.VividOrange });
sciChartSurface.renderableSeries.add(rendSeries);
rendSeries.rolloverModifierProps.tooltipTextColor = "black";
rendSeries.rolloverModifierProps.showRollover = true;

// We also need zoom and pan ChartModifiers (behaviours)
sciChartSurface.chartModifiers.add(
	new ZoomExtentsModifier({ xyDirection: EXyDirection.YDirection }),
	new XAxisDragModifier(),
	new YAxisDragModifier(),
	new ZoomPanModifier(),
	new MouseWheelZoomModifier()
);

Building a Server-Side API to fetch data based on chart-zoom

The server-side API to fetch data is found in src/server/api.ts. 10 Million data-points are created on the server and an API endpoint is created to fetch the data via rest.

const router = express.Router();

// Create a random walk as a server variable
// In practice xValues, yValues should be data in your database or other server-side data store
const POINTS = 10000000;
const xValues = new Array(POINTS);
const yValues = new Array(POINTS);
let prevYValue = 0;
for (let i = 0; i < POINTS; i++) { 
   const curYValue = Math.random() * 10 - 5; 
   xValues[i] = i; 
   yValues[i] = prevYValue + curYValue; 
   prevYValue += curYValue; 
} 

// Create the API for get "data/from-to/pointcount" 
router.get("/data/:xfrom-:xto/:pointCount", (req, res) => {
    let xStart = xValues.findIndex(x => x === Number.parseInt(req.params.xfrom));
    let xEnd = xValues.findIndex(x => x === Number.parseInt(req.params.xto));
    xStart = Math.max(0, xStart);
    xEnd = Math.min(xEnd, POINTS);
    const pointCount = Number.parseInt(req.params.pointCount);
    const xData = xValues.slice(xStart, xEnd);
    const yData = yValues.slice(xStart, xEnd);
    if (pointCount > 0 && pointCount < xData.length / 2) {
	    // Very simple data-reduction algorithm. For info purposes only!
        const interval = Math.floor(xData.length / pointCount);
        const rsX = [];
        const rsY = [];
        for (let i = 0; i < xData.length; i += interval) {
            rsX.push(xData[i]);
            rsY.push(yData[i]);
        }
        res.send({ x: rsX, y: rsY });
    } else {
        res.send({ x: xData, y: yData });
    }
});

In this example we’ve implemented a very basic data-reduction or thinning algorithm on the server when the client requests a new data range.

This algorithm takes every N-th point and should not be used in production! Better methods are available which are lossless and compress bandwidth significantly – please contact us to learn more.

Handling zoom/pan changes and requesting data on the client

Lastly, we want to request the data when the user zooms and pans. We can use SciChart.js’ built-in visibleRangeChange handler on the xAxis to be notified when the user has zoomed or panned in the x-direction.

How the code below works:

1. Use rx-js to create a Subject and push the visibleRange changes into it
2. Subscribe to the subject and debounce events within 250ms to reduce the number of requests to the server.
3. When the xAxis.visibleRange changes we can now load new datapoints from the server using the fetch command.
4. Once new points have been received, call dataSeries.clear() and dataSeries.appendRange() to replace the data on the chart.

// Create an observable stream using Rx-js 
const subject = new Subject();

// Push visible range changes into the observable
xAxis.visibleRangeChanged.subscribe(async args => {
	subject.next(args.visibleRange);
});

// subscribe to the observable with a debounce
subject.pipe(debounceTime(250)).subscribe((r: NumberRange) => {
	// Fetch data and update the dataSeries
	loadPoints(r.min, r.max, sciChartSurface.domCanvas2D.width, dataSeries).then(() => {
		// Update the y axis
		const yRange = yAxis.getWindowedYRange(null);
		yAxis.animateVisibleRange(yRange, 250, easing.outExpo);
	}).catch(err => showError(sciChartSurface, "Server data is unavailable.  Please do npm run build, then npm start and access the site at localhost:3000"));
});
	
	
const loadPoints = async (xFrom: number, xTo: number, chartWidth: number, dataSeries: XyDataSeries) => {
    chartWidth = Math.floor(chartWidth);

    const response = await fetch(`api/data/${xFrom}-${xTo}/${chartWidth}`);
    const data: { x: number[]; y: number[] } = await response.json();
    console.log(`Loaded ${data.x.length} points`);
	// Loading new data by clearing the dataseries and appending new data from server
    dataSeries.clear();
    dataSeries.appendRange(data.x, data.y);
};

A note about the Overview Scrollbar

At the bottom of the example we’ve included a SciChartOverview scrollbar control. The overview shows the entire dataset and allows you to zoom and pan.

To save bandwidth & client-side memory we don’t show the entire dataset on the overview, but a reduced version.

You can just create the overview as normal, then replace its dataSeries.

const overview = await SciChartOverview.create(sciChartSurface, divOverviewId, {theme: appTheme.SciChartJsTheme});

const overviewData = new XyDataSeries(wasmContext, { containsNaN: false, isSorted: true });

// Load the full dataSet

loadPoints(0, 10000000, overview.overviewSciChartSurface.domCanvas2D.width, overviewData).catch(err => {});

overview.overviewSciChartSurface.renderableSeries.get(0).dataSeries = overviewData;

overview.overviewSciChartSurface.zoomExtents();

Improving the performance of Chart loading from Server

The above example is designed to be quite simple to demonstrate how to fetch data from a server and plot on a JavaScript chart, dynamically changing the data on zoom and pan, with a basic server-side data thinning algorithm.

The fetch() command uses REST/http to fetch JSON data which is sent as plain text. This has to be parsed into number arrays before updating SciChart.js. The serialize/fetch/deserialize step takes most of the time in this demo.

Instead of fetching json, we recommend binary serialization using protobuf over gRPC.  This also has excellent server streaming support and the benefit of a strong client server contract. Optimising your data-transport layer will significantly improve the speed and performance of client/server javascript chart applications. SciChart.js itself can handle millions of points updated without a problem!

We also recommend a better data-thinning or reduction algorithm on the server side. SciChart has one built-in for client-side rendering, and it’s possible to implement this on the server. If of interest, please contact us!