After our first rejection at SIGCOMM, we decided to continue our work by reflecting on the reviewers’ comments. The reviewers mostly commented on the missed opportunity of multi-view fusion, which provides 6DoF, a denser set of points, and thus more content. Besides the comments, we knew specific areas that needed corrected: edge detection (post processing to remove flying pixels).
Sender Side: The sender side’s jobs (depth filling and packing) are important components that provide reasons to why efficient 2.5D/3D streaming is possible with BlenDR. Because we use 2D video codecs (H.26x) it is best to capitalize on smooth/consecutive data that prevents unnecessary motion prediction or compression calculations for noise. Noisy data is usually present as depth holes in depth data, which occurs due to inaccuracies of ToF measurements.
Receiver Side: The receiver side’s responsibility is to unpack the data to recover (or at least recreate) the original depth/color data. The depth unpacking mechanism is the inverse of the depth packing process, and it gives an accurate decoding scheme, retrieving 99% of the original data. This, however, is still the depth-filled result and needs post processing:
Multi-view Fusion (End-to-End)
Added two new steps in the initial design system: the iterative closest point algorithm (ICP) to refine calibrated point cloud and the statistical point removal to further remove flying pixels from 3D point reconstruction
Designed a new system to include for new streams of data, assigning ICP to the sender side (which is usually the server) to lessen the burden on the receiver and allow for accurate refine transform measurement
Flying Pixel Problem — originally dealt with the edge detection algorithm
Bleeding Effect in effect
BlenDR
No Post Processing
Edge Detection Only
Edge Detection + Point Remov.
Triangle Method
No Post Processing
Edge Detection Only
Edge Detection + Point Remov.