Project Overview:

This research and development internship aims to explore and prototype real-time 3D scene reconstruction and rendering pipelines based on the latest advancements in Gaussian Splatting (GS) techniques. Interns will investigate and integrate modern methods such as MOsTR3R (for dynamic scene modeling), 4D-TAM (for dynamic temporal fusion), and others (e.g., Neuralangelo, 3DGS, Gaudi, Splatting Transformers).

Interns will capture both static and dynamic world geometry (using RGB-D or video input) and render these using multi-view, photo-realistic, and time-coherent Gaussian Splatting representations, ultimately generating interactive 3D experiences.

Goals and Deliverables:

1. Literature Review & Benchmarking (Month 1–2)

• Study and summarize core principles of Gaussian Splatting.
• Analyze and compare recent relevant techniques:
• MOsTR3R (Multi-Object Spatio-Temporal Reconstruction and Rendering)
• 4D-TAM (4D Tracking and Mapping)
• Other relevant works (e.g., 3DGS, Neuralangelo, Gaudi)
• Create a benchmarking document with tradeoffs, input requirements, and rendering quality/time for each method.

2. Static World Geometry Capture & Rendering (Month 2–3)

• Capture static scenes using off-the-shelf sensors (RGB-D, LiDAR, etc.)
• Implement Gaussian Splatting rendering based on 3DGS and/or Neuralangelo.
• Integrate view-dependent shading and optimize for rendering speed.

3. Dynamic Scene Representation with Gaussian Splatting (Month 3–5)

• Use datasets or live capture (e.g., synthetic or real-world motion data).
• Implement and test dynamic scene modeling via:
• MOsTR3R-style multi-object tracking and rendering
• 4D-TAM-style temporal mapping
• Develop tools to visualize the evolution of scenes over time.
• Prototype a time-aware viewer that lets users scrub through 4D reconstructions.

4. Integration with WorldEngine (optional stretch goal)

• Explore integration of selected rendering pipeline(s) into WorldEngine.
• Create a bridge module or plugin system for live Gaussian Splat playback.

5. Final Presentation and Demonstration (Month 6)

• Functional demo of both static and dynamic Gaussian Splat reconstructions.
• Technical documentation, final report, and future work suggestions.
• Presentation to the ARCortex team including a live walkthrough and Q&A.

Learning Objectives:

• Understand cutting-edge 3D reconstruction and rendering techniques.
• Gain hands-on experience with multi-modal 3D data (RGB, depth, video).
• Learn about time-varying geometry modeling, and real-time rendering constraints.
• Practice software architecture for modular 3D systems integration.

Required Skills:

• Proficient in Python and C++ (at least one is required).
• Familiarity with 3D geometry and computer vision fundamentals.
• Experience with PyTorch or TensorFlow.
• Basic experience with Unity, Three.js, or OpenGL/WebGL.
• (Optional) Prior exposure to SLAM, NeRF, or Gaussian Splatting is a plus.

Technologies & Tools:

• PyTorch / JAX (for neural reconstruction models)
• Blender + COLMAP (for dataset generation and debugging)
• Unity (for optional WorldEngine integration)
• Meshroom, Open3D, Kaolin, or Instant-NGP (for comparative baselines)

Team Structure:

• 2–3 interns working collaboratively
• Weekly stand-ups with ARCortex mentor
• Monthly milestones and reviews