Here are our evolving thoughts on the evolution of the streaming file formats.
Streaming File format changes
Currently, spfinalize and sp2sp play some similar roles and could possibly be merged.
Goals:
- Be able to stream a mesh and a point set together, if they share the some quad/oct tree
- Files with more than 2^32 points should be supported by all tools.
- Readers and writers should support F32,F64, U8-U64, I8-I64 and both big and little endian; internally, tools will use mostly single and double, or possibly integer for vertex/point positions (xyz) and integer for triangle vertex indices (v0,v1,v2).
- Use relative or reused indices for all vertices and triangles so that index size depends on stream width rather than total number of elements.
- Scale and offset should be supported by all point types, so that UTM coordinates can be handled in single precision.
- All formats will have Binary representations; most will have ASCII, and important ones will have compressed representations.
- The streaming encoder should be able to finalize many point types (including mesh files by ignoring triangles)
- Give access to underlying finalization grid for points so that neighborhood-based algorithms can use it.
- Be able to write header files separate from data so that the original file can be used as the body without having to duplicate it, if it has enough spatial coherence. (This means that the header may need to contain information on skipping original header and selected data elements.)
Stream types
- SM - streaming mesh : Contains vertices and triangles, with possibly aux information on each
- SP - streaming point : Contains points and finalization tags for cells, with possible aux information on each
- SL - streaming line : Contains vertices and line segments; not as important as the others, so we have not implemented all the aux possibilities.
- SV - streaming volumes : Not used in this project
Compatible Streams
Points and meshes that have the same quadtree and ordering could be considered compatible if they can be simultaneously streamed without requiring a large buffer. We could have filters to make steams compatible. The steams with aux data are the extreme way to make things compatible.
Header contents
Current
- Bounding box
- vertex, triangle, point counts
- Quadtree level - for SP
- Quadtree cell counts - for SP
Needed
bounding box (min/max values for all data columns)
scale/offset
auxiliary information type descriptors
quadtree/grid information & counts
Desired
Geographical metadata - ellipsoid, UTM zone, etc.
Statistics per cell
Global stastistics (may not be useful, may have overflow problems)
Scenarios for using the streaming format encoding tool
Header separate from file
To avoid duplicating a large file, we can write the header into a separate file for use by any stream reader. Separately editing the header can allow aux information in the file to be skipped or used without having to refinalize.
Read in and finalize points
Typical use will be to read in points in whatever format and write out (or pipe) a file with header and finalized body.
Read sm as sp
We should be able to downgrade -- read any file that contains points as a streaming point file, for example. (We could write out the points, or just write out the header, as above...)
--
JackSnoeyink
to top