RTMP Server with Fusion Capabilities and Removing Flying-Pixels

Plan (Mar 28, 29 2024): - Delayed due to exams + Mid Presentation to Prof. Han

• device-to-device transformation
• slave-depth-to-slave-color
• main-depth-to-main-color~~ | | | 3. Send calibration data before streaming (Socket Programming) | 1. Receive calibration data | | | 2. Make necessary transformation matrices before receiving actual color and depth data | | 4. Start streaming: main 2 streams, slave 2 streams | 3. Receive stream and make point cloud |

• APR 24 - MAY 3: Fix NGINX Server taking few streams at a time (instead of all 4)
• MAY 6 - : Fix ICP Problem + Check for Depth Packing Improvement
• June 25 - : Make point cloud generation real-time
• July 2: Try CUDA Version of Point Cloud Creation
• July 3: Try MultiThread (v1 and v2)

July 9-10: Organize Code (Check github push)

• July 10: Meeting + Flying Pixel

**Big Solution!!! (**July 15-July 18)

1. Found a problem with our initial design → edge detection was done in a wrong way

   depth_edge_removed = depth - threshold_result 
   

   This was wrong because threshold_result is a hot-encoding matrix that just shows the pixels where edges are found. Above is changed to :

   new_threshold_result = 1 - threshold_result (get the inverse)
   depth_edge_removed = depth * new_threshold_result
   

2. Ablation Study

Triangle (Comparisons)

Triangle (Comparisons)

BlenDR (Comparisons)

BlenDR (Comparisons)

1. I wanted to try out how DRACO GROOT performs when the generated point cloud is transmitted.

2. DRACO GROOT (For fair evaluation purposes) → stuck for a day due to incomplete code

   GROOT.pdf

   1. Changes Made:
      1. Allowed for Morton code Re-ordering
      2. JPEG Decompression newly added
      3. Point Cloud Generation code newly added

SERIAL REP. of RGB of Point Cloud

MORTON Code of RGB of Point Cloud