`NDWidget` by kushalkolar · Pull Request #971 · fastplotlib/fastplotlib

kushalkolar · 2025-12-25T09:06:01Z

Internals

RangeContinuous

Just a dataclass that stores the start, stop, and step of a reference range. A reference range is usually in scientific units, such as seconds/ms for time. Depth is another example of a reference range. A reference range just specifies the min and max of a dimension that will be used as a slider dimension. The imgui sliders use the reference range to determine the min, max, and step size of the UI elements (sliders, step-one button, etc.).

RangeDiscrete is mostly a placeholder for now, not fully implemented yet, waiting for a proper usecase. Genes might be an example? Session index for multi-session data could be another potential usecase.

Users must define a reference range for every dimension they want to use as a slider dimension.

TODO: Would be useful to have an automatic reference range that's generated for a slider dimension when it's not specified. For example, sometimes you just want to dump a bunch of calcium arrays and look at them, and they're already in the same time-space. Could have a simple auto-reference range mode that just sets the min and max for eacher slider dim based on the current arrays in the NDWidget.

ReferenceIndex

This manages the reference index for an NDWidget. It's the only place where indices should be set.

TODO: pushing/popping reference ranges.

NDProcessor

This manages n-dimensional data of any type. Subclasses must implement a get() method which takes a dict mapping dim names -> index for that dim. If specified, window functions are applied on slider dimensions. _get_slider_dims_indexer takes the indices dict and outputs a dict of dim_names -> slice. _apply_window_functions() uses these slices and applies the window functions. The final output after window functions has the same number of dims as the data, but with 1 element in any dimensions that were sliced (i.e. all slider dims, spatial dims remain untouched). In the end, the get() method must return an array that can be directly mapped to graphic data.

Key properties in an NDProcessor:

data - usually an xarray, but subclasses can manage data that is of any type. The NDProcessor NDPP_Pandas for positional data stored in a pandas dataframe is an example of this.

shape, ndim, dims, spatial_dims, slider_dims - self explanatory, dims are always named

window_funcs - window functions, dict {dim_name: (func, window_size)}. A window function must take axis and keep_dims as kwargs.
slider_dim_transforms - dict of functions for each slider dim that maps a reference range value to the local array index value. If a user doesn't provide a transform for a dim (for example, they just want to look at many arrays which are already in the same time-space, such as calcium movies) the it just uses an identify mapping.

NDPositionsProcessor and NDImageProcessor have logic specific to those types of data.

NDPositionsProcessor has an additional display_window properties which acts on the datapoints dimension. The datapoints dim is also both a slider dim and spatial dim, but it is excluded for window_funcs. datapoints_window_func is a separate property on NDPositionsProcessor that applies window funcs for that dimension since it's a bit different from other slider dims.

NDGraphic

A n-dimensional graphical representation. It has an NDProcessor and fpl.Graphic that it manages. ReferenceIndex sets the indices on an NDGraphic, which then call get() on its NDProcessor to retrieve the new graphic data, and then sets it on the graphic. It has property aliases for the various processor properties, so that users can set window funcs, slider dim transforms, etc.

NDPositions and NDImage are two NDGraphic subclasses that have specific code for positional and image specific representations.

NDPositions also manages a linear selector for timeseries representations, and allows swapping between scatter, line stack, line collection, and heatmap representations. It also manages an "auto x range" mode for timeseries representations

NDImage also manages the HistogramLUTTool and swapping between ImageGraphic and ImageVolumeGraphic. It also sets the camera and controller when these are swapped.

NDWSubplot

The main entry point for users to add NDGraphics to a subplot in an NDWidget. It mainly has add_nd<...> methods for users to provide data and kwargs to add NDGraphics.

Need to cleanup this subclass so the args and kwargs for each add method are properly specified and documented.

NDWidget

Just a very simple class that has a fpl.Figure, an instance of ReferenceIndex, and the NDWSubsplots.

Very simplified diagram:

s1, ... sn are slider dims, d1, d2, ... are spatial dims.

User API

The user mainly interacts via the NDW subplots and NDGraphics. For example with images:

data = np.random.rand(1000, 30, 64, 64)

# must define a reference range for each dim
ref = {
    "time": (0, 1000, 1),
    "depth": (0, 30, 1),
}


ndw = fpl.NDWidget(ref_ranges=ref, size=(700, 560))
ndw.show()

ndi = ndw[0, 0].add_nd_image(
    data,
    ("time", "depth", "m", "n"),  # specify all dim names
    ("m", "n"),  # specify spatial dims IN ORDER, rest are auto slider dims
    name="4d-image",
)

# change spatial dims on the fly
# ndi.spatial_dims = ("depth", "m", "n")

Positional data:

# generate some toy timeseries data
n_datapoints = 100_000  # number of datapoints per line
n_freqs = 20  # number of frequencies
n_ampls = 15  # number of amplitudes
n_lines = 8

xs = np.linspace(0, 1000 * np.pi, n_datapoints)

data = np.zeros(shape=(n_freqs, n_ampls, n_lines, n_datapoints, 2), dtype=np.float32)

for freq in range(data.shape[0]):
    for ampl in range(data.shape[1]):
        ys = np.sin(xs * (freq + 1)) * (ampl + 1) + np.random.normal(
            0, 0.1, size=n_datapoints
        )
        line = np.column_stack([xs, ys])
        data[freq, ampl] = np.stack([line] * n_lines)


# must define a reference range, this would often be your time dimension and corresponds to your x-dimension
ref = {
    "freq": (1, n_freqs + 1, 1),
    "ampl": (1, n_ampls + 1, 1),
    "angle": (0, xs[-1], 0.1),
}

ndw = fpl.NDWidget(ref_ranges=ref, size=(700, 560))

nd_lines = ndw[0, 0].add_nd_timeseries(
    data,
    ("freq", "ampl", "n_lines", "angle", "d"),
    ("n_lines", "angle", "d"),
    slider_dim_transforms={
        "angle": xs,
        "ampl": lambda x: int(x + 1),
        "freq": lambda x: int(x + 1),
    },
    x_range_mode="view-range",
    name="nd-sine"
)

nd_lines.graphic.cmap = "tab10"

subplot = ndw.figure[0, 0]
subplot.controller.add_camera(subplot.camera, include_state={"x", "width"})

ndw.show(maintain_aspect=False)

Main things left todo

implements #951

kushalkolar · 2025-12-27T11:54:31Z

Got basic timeseries with linestack working. I've also got some code snippets for interpolating to display heatmap with non-uniformly sampled timeseries data. I should be able to have this fully working with time series very soon :D

Kooha-2025-12-27-03-50-45.mp4

kushalkolar · 2025-12-28T01:49:08Z

Got heatmap to display timeseries working. It should also work with non-uniformly sampled data by interpolating, need to test.

Also need to implementing switching between heatmap and line representations, need to delete the graphic when switching.

Kooha-2025-12-27-17-47-16.mp4

kushalkolar · 2025-12-29T06:38:19Z

So timeseries can be represented with arrays of one of the following shapes (let's ignore x-axis values for now).

If we have:

l: number of timeseries
p: number of datapoints in a timeseries

We can have the following shapes:

p: only y-values
p, 2: yz vals
l, p: l-timeseries with y values
l, p 2: l-timeseries with yz values

Extended to n-dimensional arrays (for example, trajectories projected onto principal components?). If each non-timeseries dim is $d_1, d_2, ... d_n$, then the above becomes:

$$\begin{align*} d_1, ... d_n, p\\\ d_1, ... d_n, p, 2\\\ d_1, ... d_n, l, p\\\ d_1, ... d_n, l, p, 2\\\ \end{align*}$$

I don't think we can auto-detect if l is present or not and the user should specify, something like:

multi_timeseries: bool = True

Scatters can be similar to some cases of nd-lines 🤔 , but x values would be directly specified and the current index is parametric (example with time indicating position in a low dim space). This would actually be true for lines as well if representing trajectories.

So for nd-line maybe we have two versions, parametric (y and z are not functions of x, but x, y, z are a function of some other dim) and non-parametric (simple timeseries lines where y and z are functions of x).

kushalkolar · 2025-12-29T09:52:33Z

I made a more generalist NDPositions which can map data that is:

[s1, s2, ... sn, l, p, 2 | 3]

where:
s1, s2, .... sn are slider dims
l is number of lines or number of scatters, this dimension is optional and the user must specify whether or not it exists
2 | 3 is the last dim, indicating xy or xyz positions.

It can map arrays of these dims to a line, line collection, line stack, scatter, or list of scatters (similar to multi-line).

I think this is a much more elegant way to deal with things, and NDTimeSeries is not necessary. The user can provide a slider mapping (to map from reference units to array index) for the p dimension which is the same as the "x-axis" for time series data!

Example if we have data that is [n_timepoints, 2], and the x-positions here (in the last dim) are in seconds. The NDWidget reference units for the slider can also be in seconds, and we can provide a mapping function that goes from the slider reference units to the n_timepoints index.

I think we can also use this for heatmaps and interpolation. Use the reference units to determine a uniform x-range for the current display window, and we can interpolate using [n_timepoints, 2] data.

EDIT: I think that the NDPositions will also work for PolygonGraphic ! Can think about meshes in general later.

kushalkolar · 2025-12-29T10:31:22Z

For positions graphics, I should actually do [n_datapoints, n_lines, 2 | 3] so everything before the last 1 or 2 dims is always sliders.

kushalkolar · 2026-01-27T05:58:42Z

Some more ideas:

Allow any 2-3 dims to be used as the graphic dimensions and specify the slider dims. This would also allow using named dims (such as those used in xarray).
API like:

add_nd_<scatter|lines|heatmap>(
  data=<array>, # array like, defaults to last 2-3 dims are graphical, first (n - 2 | 3) dims are sliders and in order
  x=<int | str | None>  # optionally specify x graphical dim using the integer dim index or dim name
  y = ... # same for other graphical dims
  slider_dims=<None | tuple[int | str]> # specify slider dims, None is auto first (n - 2 | 3), or specify a tuple of slider dims as int or str, examples: (0, 1, 2, 3), ("time", "plane")
  ... # other args
)

We interpret the given order of the slider_dims passed as $$s_1, s_2, ... s_n$$, regardless of their order within the actual array. This will make it clear to users which dims they are syncing when they use the sliders.

EDIT: A limitation of the above is that a user can't collapse multiple "graphic/display dimensions" into "final graphic/display dimensions" if they're hard-coded this like. So something like:

add_nd_<...>(
  data=<array>,
  display_dims=<tuple[int | str]>, # specify display dims in [x, y, z] order, OR display dims that collapse to xyz after the finalizer function
...
)

An example for images would be collapsing [z, m, n] to display a projection over z as slider dims are moved. But we could also have > 3 dims that are used, and then collapsed to 3 or fewer dims for the "final graphic/display dims".

kushalkolar · 2026-01-30T00:46:22Z

We can use LineCollections to display multiple lines, like behavior tracks of keypoints, with shape [n_keypoints, n_timepoints, 2]. This works well with NDPositions and display windows.

I was thinking of what's the best way to show a scatter for each keypoint, and I think I should make a ScatterCollection that behaves like a LineCollection so the same array with the same shape can be given, the only difference is the graphical representation would be a scatter instead of lines. For typical behavior keypoint viz, the display_window would usually be just 1, but it can be greater than 1 for any viz that needs to show a window of scatter points.

…w funcs

kushalkolar · 2026-01-30T03:58:08Z

ok I think stuff is working

ndpositions-2026-01-29_22.55.54.mp4

kushalkolar · 2026-01-30T04:04:09Z

I think I need to make a PolygonCollection too 🤔 . Would be very similar to the ScatterCollection.

kushalkolar · 2026-01-30T05:33:39Z

so the basics are all working 🥳

LineStack and heatmap representations are swappable:

nd_positions_swap_linestack_heatmap-2026-01-30_00.19.27.mp4

Scatter and line collection to show behavior trajectories:

nd_positions_behavior-2026-01-30_00.29.15.mp4

kushalkolar · 2026-01-30T05:36:25Z

A set of imgui UIs that allow controlling some aspects of the "nd graphics" could be useful, such as:

graphical representation, dropdown menu to choose scatter, line, heatmap
display_window
multi bool

kushalkolar · 2026-01-30T07:28:33Z

Stuff I should finish before implementing the orchestrator:

merge auto-replace buffers #974 , so I can implement changing the display_window
~~implement logic to allow the display_window to centered, on the left or on the right~~
adding a linear selector, it's "default" positions and its position when the main slider for the n_datapoints dimension changes should correspond whether the display_window is in "center", "start" or "end" mode. Moving the linear selector should probably change the index in all other NDobjects but not its "parent" object. So that the visual representation of the parent object doesn't change, ex: if the linear selector is moved we don't want a linestack to also "move"
spatial function
implement mapping from a slider reference index with units (such as time) to array index.
figure out how to implement stuff like colors, cmaps given that we're essentially performing out of core rendering with these graphics
auto display_window? Zoom in/out and set graphic data at different subsample levels

Things that make me "uncomfortable" that I need to settle:

dim shapes

Dim shape for nd-positions is [s1, s2, ... sn, n, p, 2 | 3] where:

n: number of lines, scatters, or heatmap rows (optional, can be 1)
p: number of datapoints, often corresponds to a time index.

p would be a slider dim, but there's this n dim that's in between p and all the other slider dims. We could instead use a shape of [s1, ... sn, p, n, 2 | 3] but that feels weird and we'd have to do a transpose to get the array for the graphical representation, i.e. [p, n, 2 | 3] -> [n, p, 2 | 3] is required for the graphical representation.

Do we just document this well, that the n dimension is not a slider dim but p is?

When using nd-positions data in conjunction with nd-image data, we'd have something like this:

# nd-positions
[s1, ... sn, n, p, 2 | 3]

# nd-image
[s1, ... sn, p, r, c, 1 | 3 | 4]

Where r: rows, cols: columns. 1 | 3 | 4 denotes grayscale or RGB(A).

The p dimension in the nd-image array above would correspond to the p in the nd-positions. nd-images don't have any n dimension, it doesn't make sense there.

kushalkolar · 2026-01-31T02:16:25Z

Working on "implement mapping from a slider reference index with units (such as time) to array index.", which requires proper implementation of slider_dims and n_slider_dims properties. Need to figure out how to properly separate the p n_datapoints dimension from other slider dims, and also apply the window funcs on the p dim. Will do tomorrow.

kushalkolar · 2026-02-01T09:02:19Z

Window funcs working for p (n_datapoints) dim and all graphical representations

ndp_windows-2026-02-01_03.39.52.mp4

* refactor cmap and cmap_transform for positional graphics * color mode stuff * WIP * more WIP * better add_graphic autogen * more progress * mostly done refactoring positions graphics cmap stuff * forgot to update mixin * cmap_transform tweaks * wip, JaggedArray * mostly works, writing tests * remove a print * image collection examples * fix * feature inheritance in PositionsGraphic and cmap_range * cmap handling * cmap tweaks * remove parse_cmap_values, rely on cmap lib * fix example, remove unused line of code * Image just uses cmap lib now too * delete_ndgraphic() * ndtimeseries cmap example * steps for stacks * ndtimeseries sets stack steps * docstrings * docstrings * comments * renames * docstring * fixes * update tests/test_colors_buffer_manager.py * update test_markers_buffer_manager.py * update plot_helpers test * update test_point_rotations_buffer_manager.py * update test_positions_graphics.py * update another test * more tests * update yet more tests * remove unused * infline tests * collections tests * updating examples, wip * more examples updates * more examples updates * more example updates * more example updates * example and docstring * docstring * remove complex line collection slicing example, stick with real usecases * better line stack example * update example * update example * update link * update more examples * examples and fixes * fix * more fixes * fix * updates * Fix * update docs * docs * add image collection to docs conf.py * update docs

* rename for consistency * more consistency * more * final renames * fix

…ims and no appended UIs

* basic scaffold done * inheritance * done * config works! * config on graphics * axes config * full config implementation basically works * fix * mixins call Graphic construtors with kwargs nothing is positional * print * config presets * comments, docstrings * docstrings * remove ConfigValue * comments * much better add graphics mixin using descriptors, examples, fix a test * anotehr example * GlobalConfig.to_dict() * docs * reset to default config after each screenshot test * add_<graphics>() stub generator, fix maintain_aspect logic w.r.t. config stuff * change so maintain_aspect can be tested better * docstring * reset config after running each docs gallery examle * better example * docs

start ndprocessors

9a01cd5

kushalkolar changed the title ~~NWidget~~ NDWidget Dec 25, 2025

kushalkolar added 2 commits December 27, 2025 03:52

basic timeseries

c46455f

add __init__

d93fa5d

heatmap for timeseries works!

fddefb8

NDPositions, basics work, reorganize, increase default scatter size

d5e4c7d

black

074669b

kushalkolar added 2 commits January 29, 2026 20:27

NDPositions working with multi-dim stack of lines, need to test windo…

ff5c578

…w funcs

scatter collection

3f412c5

kushalkolar added 2 commits January 30, 2026 00:06

progress, need to change to other branch so committing

dc30151

Merge branch 'main' into ndwidget

2b3c9b6

better

db98bde

kushalkolar added 2 commits January 30, 2026 20:45

interpolation for heatmap

3629f70

better place for check

87ea418

kushalkolar added 2 commits February 1, 2026 03:43

window functions working on n_datapoints dim

e5a8d40

p dim window funcs working for single and multiple dims I think

8d050a7

kushalkolar added 30 commits September 1, 2026 20:15

clamp vmin, vmax in colorbar histogram LUT tool (#1076)

197b759

Ndw rename consistency (#1078)

7008c79

* rename for consistency * more consistency * more * final renames * fix

cursor fix

ca3d489

missed renames

52c42ee

pandas fix

83e0918

restore ImageWidget, auto-collapse NDWidget sliders UI if no slider d…

8bc1736

…ims and no appended UIs

imagewidget stuff

a18f06a

ImageVolumeGraphic args fix

0faa5ce

NDWSubplot dims check only for array-like

6d9c32a

collapsed property on ImaguiWindoW

f714da2

update iw examples

9462748

colorbar fix

e1d996d

lingering rename

d703c59

fix

10364d6

type

0317208

clim_quantile for NDImage, other fixes

f079242

public 'max_display_datapoints' property

1a980ce

dw setter

c483f59

imgui right click for NDGraphics

19e7638

axes label tweaks

9ec521c

remove unused zarr placeholder

7ccbf44

fix docstring

63bbffc

remove randon shit

d6e73f3

ndw stuff is all top level importable, move types to utils

7e019b2

fix black

12cf662

fix import

efd3104

matplotlib migration docs

190497e

wording

5a15204

Conversation

kushalkolar commented Dec 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Internals

RangeContinuous

ReferenceIndex

NDProcessor

NDGraphic

NDWSubplot

NDWidget

User API

Main things left todo

Uh oh!

kushalkolar commented Dec 27, 2025

Uh oh!

kushalkolar commented Dec 28, 2025

Uh oh!

kushalkolar commented Dec 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kushalkolar commented Dec 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kushalkolar commented Dec 29, 2025

Uh oh!

kushalkolar commented Jan 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kushalkolar commented Jan 30, 2026

Uh oh!

kushalkolar commented Jan 30, 2026

Uh oh!

kushalkolar commented Jan 30, 2026

Uh oh!

kushalkolar commented Jan 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kushalkolar commented Jan 30, 2026

Uh oh!

kushalkolar commented Jan 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Stuff I should finish before implementing the orchestrator:

Things that make me "uncomfortable" that I need to settle:

dim shapes

Uh oh!

kushalkolar commented Jan 31, 2026

Uh oh!

kushalkolar commented Feb 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kushalkolar commented Dec 25, 2025 •

edited

Loading

kushalkolar commented Dec 29, 2025 •

edited

Loading

kushalkolar commented Dec 29, 2025 •

edited

Loading

kushalkolar commented Jan 27, 2026 •

edited

Loading

kushalkolar commented Jan 30, 2026 •

edited

Loading

kushalkolar commented Jan 30, 2026 •

edited

Loading