Here are our evolving thoughts on the evolution of the streaming file formats.

Streaming File format changes

Currently, spfinalize and sp2sp play some similar roles and could possibly be merged.

Goals:

1. Be able to stream a mesh and a point set together, if they share the some quad/oct tree
2. Files with more than 2^32 points should be supported by all tools.
3. Readers and writers should support F32,F64, U8-U64, I8-I64 and both big and little endian; internally, tools will use mostly single and double, or possibly integer for vertex/point positions (xyz) and integer for triangle vertex indices (v0,v1,v2).
4. Use relative or reused indices for all vertices and triangles so that index size depends on stream width rather than total number of elements.
5. Scale and offset should be supported by all point types, so that UTM coordinates can be handled in single precision.
6. All formats will have Binary representations; most will have ASCII, and important ones will have compressed representations.
7. The streaming encoder should be able to finalize many point types (including mesh files by ignoring triangles)
8. Give access to underlying finalization grid for points so that neighborhood-based algorithms can use it.
9. Be able to write header files separate from data so that the original file can be used as the body without having to duplicate it, if it has enough spatial coherence. (This means that the header may need to contain information on skipping original header and selected data elements.)

Stream types

• SM - streaming mesh : Contains vertices and triangles, with possibly aux information on each
• SP - streaming point : Contains points and finalization tags for cells, with possible aux information on each
• SL - streaming line : Contains vertices and line segments; not as important as the others, so we have not implemented all the aux possibilities.
• SV - streaming volumes : Not used in this project

Compatible Streams

Points and meshes that have the same quadtree and ordering could be considered compatible if they can be simultaneously streamed without requiring a large buffer. We could have filters to make steams compatible. The steams with aux data are the extreme way to make things compatible.

Header contents

Current

• Bounding box
• vertex, triangle, point counts
• Quadtree level - for SP
• Quadtree cell counts - for SP

Needed

bounding box (min/max values for all data columns) scale/offset auxiliary information type descriptors quadtree/grid information & counts

Desired

Geographical metadata - ellipsoid, UTM zone, etc.

Statistics per cell Global stastistics (may not be useful, may have overflow problems)

Scenarios for using the streaming format encoding tool

Header separate from file

To avoid duplicating a large file, we can write the header into a separate file for use by any stream reader. Separately editing the header can allow aux information in the file to be skipped or used without having to refinalize.

Read in and finalize points

Typical use will be to read in points in whatever format and write out (or pipe) a file with header and finalized body.

Read sm as sp

We should be able to downgrade -- read any file that contains points as a streaming point file, for example. (We could write out the points, or just write out the header, as above...)

-- JackSnoeyink
to top