Xingxing Zuo

I’m a Postdoctoral Research Associate at the Department of Computing + Mathematical Sciences, Caltech since January, 2024, working with Prof. Soon-Jo Chung. Prior to this, I worked at Google in USA as a full-time visiting faculty researcher (Scientist) in 2023. I was an Postdoc Researcher (2021 - 2023) at Technical University of Munich, Germany, working with Prof. Stefan Leutenegger. I was an academic guest (2019 - 2021) in the Computer Vision and Geometry Group (CVG), at ETH Zurich, Switzerland, working with Prof. Marc Pollefeys. I was also a visiting scholar (2018) in the Robot Perception and Navigation Group (RPNG) of Prof. Guoquan (Paul) Huang at University of Delaware, USA, and a visiting scholar (2017) in the UTS Robotics Institute, University of Technology Sydney, Australia, working with Prof. Teresa Vidal Calleja. I received my doctoral degree in 2021, from the Department of Control Science and Engineering at Zhejiang University, China, under the co-supervision of Prof. Guoquan (Paul) Huang and Prof. Yong liu, and graduated with honors. Previously, I graduated with honors and obtained my Bachelor’s degree in Mechanical Engineering at University of Electronic Science and Technology of China (UESTC), Chengdu, China, in 2016.

I’m on the faculty job market now. If you know of any good open positions, I would appreciate it if you could reach out to me via email!

---

research interests

My research interests include robotics perception, 3D computer vision, multi-sensor fusion, state estimation, and foundation models.

I’m conducting research on integrating the intelligence of the deep neural networks and the traditional probabilistic methods for real-time robotic perception tasks.

---

---

academic services

Associate Editor: IEEE Robotics and Automation Letters (RA-L, since March 2022), IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS, 2022&2023&2024), IEEE International Conference on Robotics and Automation (ICRA, 2023&2024&2025).

Conference Session Chair: IROS 2023.

Guest Editor: Sensors [Special Issue 2023].

Journal Reviewer: IEEE Transactions on Robotics (T-RO), The International Journal of Robotics Research (IJRR), Journal of Field Robotics (J-FR), IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI), IEEE Transactions on Automation Science and Engineering (T-ASE), IEEE Robotics and Automation Letters (RA-L), IEEE Transactions on Intelligent Transportation Systems (T-ITS), IEEE Transactions on Industrial Electronics (T-IE), Neurocomputing.

Conference Reviewer: ICRA, IROS, CVPR, ICCV, ECCV, IJCAI

---

representative talks

• 2023.08 “Towards the Era of Robust 3D Perception”. The Chinese University of Hong Kong (ShenZhen).
• 2023.05 “Towards the Era of Robust 3D Perception for Mobile Robots”. Upenn Grasp Lab.
• 2022.06 “Robut 3D Perception with Multi-source Information Fusion”. ETH CVG group.
• 2022.03 Invited speaker at conference, “Towards the Era of Robust Pose Estimation for Robots”. UPLINLIBS 2022.
• 2022.03 “Towards the Era of Robust Spatial Perception”. Peking University.

---

selected publications

Find out my full publication list via Google Scholar.
^* co-first author with equal contribution; ^# Corresponding author.

1. 

   FMGS: Foundation Model Embedded 3D Gaussian Splatting for Holistic 3D Scene Understanding Xingxing Zuo, Pouya Samangouei, Yunwen Zhou, Yan Di, and Mingyang Li International Journal of Computer Vision (IJCV) 2024 [Abs] [arXiv] [Webpage] [Code]

   Precisely perceiving the geometric and semantic properties of real-world 3D objects is crucial for the continued evolution of augmented reality and robotic applications. To this end, we present Foundation Model Embedded Gaussian Splatting (FMGS), which incorporates vision-language embeddings of foundation models into 3D Gaussian Splatting (GS).The key contribution of this work is an efficient method to reconstruct and represent 3D vision-language models. This is achieved by distilling feature maps generated from image-based foundation models into those rendered from our 3D model. To ensure high-quality rendering and fast training, we introduce a novel scene representation by integrating strengths from both GS and multi-resolution hash encodings (MHE). Our effective training procedure also introduces a pixel alignment loss that makes the rendered feature distance of same semantic entities close, following the pixel-level semantic boundaries. Our results demonstrate remarkable multi-view semantic consistency, facilitating diverse downstream tasks, beating state-of-the-art methods by 10.2 percent on open-vocabulary language-based object detection, despite that we are 851X faster for inference. This research explores the intersection of vision, language, and 3D scene representation, paving the way for enhanced scene understanding in uncontrolled real-world environments.

2. 

   NeRF-VO: Real-Time Sparse Visual Odometry With Neural Radiance Fields Jens Naumann, Binbin Xu, Stefan Leutenegger, and Xingxing Zuo^# IEEE Robotics and Automation Letters (RA-L) 2024 [Abs] [arXiv] [Video] [Webpage] [Code]

   We introduce a novel monocular visual odometry (VO) system, NeRF-VO, that integrates learning-based sparse visual odometry for low-latency camera tracking and a neural radiance scene representation for fine-detailed dense reconstruction and novel view synthesis. Our system initializes camera poses using sparse visual odometry and obtains view-dependent dense geometry priors from a monocular prediction network. We harmonize the scale of poses and dense geometry, treating them as supervisory cues to train a neural implicit scene representation. NeRF-VO demonstrates exceptional performance in both photometric and geometric fidelity of the scene representation by jointly optimizing a sliding window of keyframed poses and the underlying dense geometry, which is accomplished through training the radiance field with volume rendering. We surpass SOTA methods in pose estimation accuracy, novel view synthesis fidelity, and dense reconstruction quality across a variety of synthetic and real-world datasets while achieving a higher camera tracking frequency and consuming less GPU memory.

3. 

   Dynamic LiDAR Re-simulation using Compositional Neural Fields Hanfeng Wu, Xingxing Zuo^#, Stefan Leutenegger, Or Litany, Konrad Schindle, and Shengyu Huang^# IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Highlight, 2024 [Abs] [arXiv] [Video] [Webpage] [Code]

   We introduce DyNFL, a novel neural field-based approach for high-fidelity re-simulation of LiDAR scans in dynamic driving scenes. DyNFL processes LiDAR measurements from dynamic environments, accompanied by bounding boxes of moving objects, to construct an editable neural field. This field, comprising separately reconstructed static backgrounds and dynamic objects, allows users to modify viewpoints, adjust object positions, and seamlessly add or remove objects in the re-simulated scene. A key innovation of our method is the neural field composition technique, which effectively integrates reconstructed neural assets from various scenes through a ray drop test, accounting for occlusions and transparent surfaces. Our evaluation with both synthetic and real-world environments demonstrates that DyNFL substantial improves dynamic scene simulation based on LiDAR scans, offering a combination of physical fidelity and flexible editing capabilities.

4. 

   RadarCam-Depth: Radar-Camera Fusion for Depth Estimation with Learned Metric Scale Han Li, Yukai Ma, Yaqing Gu, Kewei Hu, Yong Liu^#, and Xingxing Zuo^# IEEE International Conference on Robotics and Automation (ICRA) 2024 [Abs] [arXiv] [Video] [Code]

   We present a novel approach for metric dense depth estimation based on the fusion of a single-view image and a sparse, noisy Radar point cloud. The direct fusion of heterogeneous Radar and image data, or their encodings, tends to yield dense depth maps with significant artifacts, blurred boundaries, and suboptimal accuracy. To circumvent this issue, we learn to augment versatile and robust monocular depth prediction with the dense metric scale induced from sparse and noisy Radar data. We propose a Radar-Camera framework for highly accurate and fine-detailed dense depth estimation with four stages, including monocular depth prediction, global scale alignment of monocular depth with sparse Radar points, quasi-dense scale estimation through learning the association between Radar points and image patches, and local scale refinement of dense depth using a scale map learner. Our proposed method significantly outperforms the state-of-the-art Radar-Camera depth estimation methods by reducing the mean absolute error (MAE) of depth estimation by 25.6% and 40.2% on the challenging nuScenes dataset and our self-collected ZJU-4DRadarCam dataset, respectively.

1. 

   Coco-LIC: Continuous-Time Tightly-Coupled LiDAR-Inertial-Camera Odometry using Non-Uniform B-spline Xiaolei Lang, Chao Chen, Kai Tang, Yukai Ma, Jiajun Lv, Yong Liu^#, and Xingxing Zuo^# IEEE Robotics and Automation Letters 2023 [Abs] [arXiv] [Video] [Code]

   In this paper, we propose an efficient continuous-time LiDAR-Inertial-Camera Odometry, utilizing non-uniform B-splines to tightly couple measurements from the LiDAR, IMU, and camera. In contrast to uniform B-spline-based continuous-time methods, our non-uniform B-spline approach offers significant advantages in terms of achieving real-time efficiency and high accuracy. This is accomplished by dynamically and adaptively placing control points, taking into account the varying dynamics of the motion. To enable efficient fusion of heterogeneous LiDAR-Inertial-Camera data within a short sliding-window optimization, we assign depth to visual pixels using corresponding map points from a global LiDAR map, and formulate frame-to-map reprojection factors for the associated pixels in the current image frame. This way circumvents the necessity for depth optimization of visual pixels, which typically entails a lengthy sliding window with numerous control points for continuous-time trajectory estimation. We conduct dedicated experiments on real-world datasets to demonstrate the advantage and efficacy of adopting non-uniform continuous-time trajectory representation. Our LiDAR-Inertial-Camera odometry system is also extensively evaluated on both challenging scenarios with sensor degenerations and large-scale scenarios, and has shown comparable or higher accuracy than the state-of-the-art methods.

2. 

   SimpleMapping: Real-Time Visual-Inertial Dense Mapping with Deep Multi-View Stereo Yingye Xin*, Xingxing Zuo*^#, Dongyue Lu, and Stefan Leutenegger IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 2023 [Abs] [arXiv] [Video] [Webpage]

   We present a real-time visual-inertial dense mapping method capable of performing incremental 3D mesh reconstruction with high quality using only sequential monocular images and inertial measurement unit (IMU) readings. 6-DoF camera poses are estimated by a robust feature-based visual-inertial odometry (VIO), which also generates noisy sparse 3D map points as a by-product. We propose a sparse point aided multi-view stereo neural network (SPA-MVSNet) that can effectively leverage the informative but noisy sparse points from the VIO system. The sparse depth from VIO is firstly completed by a single-view depth completion network. This dense depth map, although naturally limited in accuracy, is then used as a prior to guide our MVS network in the cost volume generation and regularization for accurate dense depth prediction. Predicted depth maps of keyframe images by the MVS network are incrementally fused into a global map using TSDF-Fusion. We extensively evaluate both the proposed SPA-MVSNet and the entire visual-inertial dense mapping system on several public datasets as well as our own dataset, demonstrating the system’s impressive generalization capabilities and its ability to deliver high-quality 3D mesh reconstruction online. Our proposed dense mapping system achieves a 39.7% improvement in F-score over existing systems when evaluated on the challenging scenarios of the EuRoC dataset.

3. 

   Incremental Dense Reconstruction from Monocular Video with Guided Sparse Feature Volume Fusion Xingxing Zuo, Nan Yang, Nathaniel Merrill, Binbin Xu, and Stefan Leutenegger IEEE Robotics and Automation Letters 2023 [Abs] [arXiv] [Video]

   Incrementally recovering 3D dense structures from monocular videos is of paramount importance since it enables various robotics and AR applications. Feature volumes have recently been shown to enable efficient and accurate incremental dense reconstruction without the need to first estimate depth, but they are not able to achieve as high of a resolution as depth-based methods due to the large memory consumption of high-resolution feature volumes. This letter proposes a real-time feature volume-based dense reconstruction method that predicts TSDF (Truncated Signed Distance Function) values from a novel sparsified deep feature volume, which is able to achieve higher resolutions than previous feature volume-based methods, and is favorable in outdoor large-scale scenarios where the majority of voxels are empty. An uncertainty-aware multi-view stereo (MVS) network is leveraged to infer initial voxel locations of the physical surface in a sparse feature volume. Then for refining the recovered 3D geometry, deep features are attentively aggregated from multi-view images at potential surface locations, and temporally fused. Besides achieving higher resolutions than before, our method is shown to produce more complete reconstructions with finer detail in many cases. Extensive evaluations on both public and self-collected datasets demonstrate a very competitive real-time reconstruction result for our method compared to state-of-the-art reconstruction methods in both indoor and outdoor settings.

4. 

   Continuous-Time Fixed-Lag Smoothing for LiDAR-Inertial-Camera SLAM Jiajun Lv, Xiaolei Lang, Jinhong Xu, Mengmeng Wang, Yong Liu^#, and Xingxing Zuo^# IEEE/ASME Transactions on Mechatronics 2023 [Abs] [arXiv] [Code]

   Localization and mapping with heterogeneous multisensor fusion have been prevalent in recent years. To adequately fuse multimodal sensor measurements received at different time instants and different frequencies, we estimate the continuous-time trajectory by fixed-lag smoothing within a factor-graph optimization framework. With the continuous-time formulation, we can query poses at any time instants corresponding to the sensor measurements. To bound the computation complexity of the continuous-time fixed-lag smoother, we maintain temporal and keyframe sliding windows with constant size, and probabilistically marginalize out control points of the trajectory and other states, which allows preserving prior information for future sliding-window optimization. Based on continuous-time fixed-lag smoothing, we design tightly coupled multimodal SLAM algorithms with a variety of sensor combinations, like the LiDAR-inertial and LiDAR-inertial-camera SLAM systems, in which online time offset calibration is also naturally supported. More importantly, benefiting from the marginalization and our derived analytical Jacobians for optimization, the proposed continuous-time SLAM systems can achieve real-time performance regardless of the high complexity of continuous-time formulation. The proposed multimodal SLAM systems have been widely evaluated on three public datasets and self-collect datasets. The results demonstrate that the proposed continuous-time SLAM systems can achieve high-accuracy pose estimations and outperform existing state-of-the-art methods.

1. 

   Ctrl-VIO: Continuous-Time Visual-Inertial Odometry for Rolling Shutter Cameras Xiaolei Lang, Jiajun Lv, Jianxin Huang, Yukai Ma, Yong Liu^#, and Xingxing Zuo^# IEEE Robotics and Automation Letters 2022 [Abs] [arXiv] [Code]

   In this letter, we propose a probabilistic continuous-time visual-inertial odometry (VIO) for rolling shutter cameras. The continuous-time trajectory formulation naturally facilitates the fusion of asynchronized high-frequency IMU data and motion-distorted rolling shutter images. To prevent intractable computation load, the proposed VIO is sliding-window and keyframe-based. We propose to probabilistically marginalize the control points to keep the constant number of keyframes in the sliding window. Furthermore, the line exposure time difference (line delay) of the rolling shutter camera can be online calibrated in our continuous-time VIO. To extensively examine the performance of our continuous-time VIO, experiments are conducted on publicly-available WHU-RSVI, TUM-RSVI, and SenseTime-RSVI rolling shutter datasets. The results demonstrate the proposed continuous-time VIO significantly outperforms the existing state-of-the-art VIO methods.

2. 

   Observability-Aware Intrinsic and Extrinsic Calibration of LiDAR-IMU Systems Jiajun Lv*, Xingxing Zuo*, Kewei Hu, Jinhong Xu, Guoquan Huang, and Yong Liu IEEE Transactions on Robotics 2022 [Abs] [arXiv] [Code]

   Accurate and reliable sensor calibration is essential to fuse LiDAR and inertial measurements, which are usually available in robotic applications. In this paper, we propose a novel LiDAR-IMU calibration method within the continuous-time batch-optimization framework, where the intrinsics of both sensors and the spatial-temporal extrinsics between sensors are calibrated without using calibration infrastructure such as fiducial tags. Compared to discrete-time approaches, the continuous-time formulation has natural advantages for fusing high rate measurements from LiDAR and IMU sensors. To improve efficiency and address degenerate motions, two observability-aware modules are leveraged: (i) The information-theoretic data selection policy selects only the most informative segments for calibration during data collection, which significantly improves the calibration efficiency by processing only the selected informative segments. (ii) The observability-aware state update mechanism in nonlinear least-squares optimization updates only the identifiable directions in the state space with truncated singular value decomposition (TSVD), which enables accurate calibration results even under degenerate cases where informative data segments are not available. The proposed LiDAR-IMU calibration approach has been validated extensively in both simulated and real-world experiments with different robot platforms, demonstrating its high accuracy and repeatability in commonly-seen human-made environments.

3. 

   Symmetry and Uncertainty-Aware Object SLAM for 6DoF Object Pose Estimation Nathaniel Merrill, Yuliang Guo, Xingxing Zuo, Xinyu Huang, Stefan Leutenegger, Xi Peng, Liu Ren, and Guoquan Huang IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2022 [Abs] [arXiv] [Code]

   We propose a keypoint-based object-level SLAM framework that can provide globally consistent 6DoF pose estimates for symmetric and asymmetric objects alike. To the best of our knowledge, our system is among the first to utilize the camera pose information from SLAM to provide prior knowledge for tracking keypoints on symmetric objects – ensuring that new measurements are consistent with the current 3D scene. Moreover, our semantic keypoint network is trained to predict the Gaussian covariance for the keypoints that captures the true error of the prediction, and thus is not only useful as a weight for the residuals in the system’s optimization problems, but also as a means to detect harmful statistical outliers without choosing a manual threshold. Experiments show that our method provides competitive performance to the state of the art in 6DoF object pose estimation, and at a real-time speed. Our code, pre-trained models, and keypoint labels are open-sourced.

1. 

   CodeVIO: Visual-inertial odometry with learned optimizable dense depth Xingxing Zuo*, Nathaniel Merrill*, Wei Li, Yong Liu, Marc Pollefeys, and Guoquan Huang Nominated for the "Best Paper Award in Robot Vision (Finalist) of ICRA 2021", IEEE International Conference on Robotics and Automation (ICRA) 2021 [Abs] [arXiv] [Video]

   In this work, we present a lightweight, tightly-coupled deep depth network and visual-inertial odometry (VIO) system, which can provide accurate state estimates and dense depth maps of the immediate surroundings. Leveraging the proposed lightweight Conditional Variational Autoencoder (CVAE) for depth inference and encoding, we provide the network with previously marginalized sparse features from VIO to increase the accuracy of initial depth prediction and generalization capability. The compact encoded depth maps are then updated jointly with navigation states in a sliding window estimator in order to provide the dense local scene geometry. We additionally propose a novel method to obtain the CVAE’s Jacobian which is shown to be more than an order of magnitude faster than previous works, and we additionally leverage First-Estimate Jacobian (FEJ) to avoid recalculation. As opposed to previous works relying on completely dense residuals, we propose to only provide sparse measurements to update the depth code and show through careful experimentation that our choice of sparse measurements and FEJs can still significantly improve the estimated depth maps. Our full system also exhibits state-of-the-art pose estimation accuracy, and we show that it can run in real-time with single-thread execution while utilizing GPU acceleration only for the network and code Jacobian.

2. 

   Pose Estimation for Ground Robots: On Manifold Representation, Integration, Reparameterization, and Optimization Mingming Zhang*, Xingxing Zuo*, Yiming Chen, Yong Liu, and Mingyang Li IEEE Transactions on Robotics 2021 [Abs] [arXiv] [Video]

   In this article, we focus on pose estimation dedicated to nonholonomic ground robots with low-cost sensors, by probabilistically fusing measurements from wheel odometers and an exteroceptive sensor. For ground robots, wheel odometers are widely used in pose estimation tasks, especially in applications in planar scenes. However, since wheel odometer only provides two-dimensional (2D) motion measurements, it is extremely challenging to use that for accurate full 6-D pose (3-D position and 3-D orientation) estimation. Traditional methods for 6-D pose estimation with wheel odometers either approximate motion profiles at the cost of accuracy reduction, or rely on other sensors, e.g., inertial measurement unit, to provide complementary measurements. By contrast, we propose a novel motion-manifold-based method for pose estimation of ground robots, which enables to utilize wheel odometers for high-precision 6-D pose estimation. Specifically, the proposed method, first, formulates the motion manifold of ground robots by a parametric representation, second, performs manifold-based 6-D integration with wheel odometer measurements only, and third, reparameterizes manifold representation periodically for error reduction. To demonstrate the effectiveness and applicability of the proposed algorithmic module, we integrate that into a sliding-window pose estimator by using measurements from wheel odometers and a monocular camera. Extensive simulated and real-world experiments are conducted for evaluation, and the proposed algorithm is shown to outperform competing the state-of-the-art algorithms by a significant margin in pose estimation accuracy, especially when deployed in complex, large-scale real-world environments.

3. 

   CLINS: Continuous-Time Trajectory Estimation for LiDAR-Inertial System Jiajun Lv, Kewei Hu, Jinhong Xu, Yong Liu, Xiushui Ma, and Xingxing Zuo IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2021 [Abs] [arXiv] [Video] [Code]

   In this paper, we propose a highly accurate continuous-time trajectory estimation framework dedicated to SLAM (Simultaneous Localization and Mapping) applications, which enables fuse high-frequency and asynchronous sensor data effectively. We apply the proposed framework in a 3D LiDAR-inertial system for evaluations. The proposed method adopts a non-rigid registration method for continuous-time trajectory estimation and simultaneously removing the motion distortion in LiDAR scans. Additionally, we propose a two-state continuous-time trajectory correction method to efficiently and efficiently tackle the computationally-intractable global optimization problem when loop closure happens. We examine the accuracy of the proposed approach on several publicly available datasets and the data we collected. The experimental results indicate that the proposed method outperforms the discrete-time methods regarding accuracy especially when aggressive motion occurs. Furthermore, we open source our code to benefit research community.

4. 

   MBA-VO: Motion Blur Aware Visual Odometry Peidong Liu, Xingxing Zuo, Viktor Larsson, and Marc Pollefeys International Conference on Computer Vision (ICCV), selected as Oral, 2021 [Abs] [arXiv] [Code]

   Motion blur is one of the major challenges remaining for visual odometry methods. In low-light conditions where longer exposure times are necessary, motion blur can appear even for relatively slow camera motions. In this paper we present a novel hybrid visual odometry pipeline with direct approach that explicitly models and estimates the camera’s local trajectory within the exposure time. This allows us to actively compensate for any motion blur that occurs due to the camera motion. In addition, we also contribute a novel benchmarking dataset for motion blur aware visual odometry. In experiments we show that by directly modeling the image formation process, we are able to improve robustness of the visual odometry, while keeping comparable accuracy as that for images without motion blur.

1. 

   LIC-Fusion 2.0: LiDAR-Inertial-Camera Odometry with Sliding-Window Plane-Feature Tracking Xingxing Zuo, Yulin Yang, Patrick Geneva, Jiajun Lv, Yong Liu, Guoquan Huang, and Marc Pollefeys IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2020 [Abs] [arXiv] [Video] [Video2] [Supp]

   Multi-sensor fusion of multi-modal measurements from commodity inertial, visual and LiDAR sensors to provide robust and accurate 6DOF pose estimation holds great potential in robotics and beyond. In this paper, building upon our prior work (i.e., LIC-Fusion), we develop a sliding-window filter based LiDAR-Inertial-Camera odometry with online spatiotemporal calibration (i.e., LIC-Fusion 2.0), which introduces a novel sliding-window plane-feature tracking for efficiently processing 3D LiDAR point clouds. In particular, after motion compensation for LiDAR points by leveraging IMU data, low-curvature planar points are extracted and tracked across the sliding window. A novel outlier rejection criterion is proposed in the plane-feature tracking for high-quality data association. Only the tracked planar points belonging to the same plane will be used for plane initialization, which makes the plane extraction efficient and robust. Moreover, we perform the observability analysis for the LiDAR-IMU subsystem and report the degenerate cases for spatiotemporal calibration using plane features. While the estimation consistency and identified degenerate motions are validated in Monte-Carlo simulations, different real-world experiments are also conducted to show that the proposed LIC-Fusion 2.0 outperforms its predecessor and other state-of-the-art methods.

2. 

   Multimodal localization: Stereo over LiDAR map Xingxing Zuo, Wenlong Ye, Yulin Yang, Renjie Zheng, Teresa Vidal-Calleja, Guoquan Huang, and Yong Liu Journal of Field Robotics 2020 [Abs]

   In this paper, we present a real-time high-precision visual localization system for an autonomous vehicle which employs only low-cost stereo cameras to localize the vehicle with a priori map built using a more expensive 3D LiDAR sensor. To this end, we construct two different visual maps: a sparse feature visual map for visual odometry (VO) based motion tracking, and a semidense visual map for registration with the prior LiDAR map. To register two point clouds sourced from different modalities (i.e., cameras and LiDAR), we leverage probabilistic weighted normal distributions transformation (ProW-NDT), by particularly taking into account the uncertainty of source point clouds. The registration results are then fused via pose graph optimization to correct the VO drift. Moreover, surfels extracted from the prior LiDAR map are used to refine the sparse 3D visual features that will further improve VO-based motion estimation. The proposed system has been tested extensively in both simulated and real-world experiments, showing that robust, high-precision, real-time localization can be achieved.

3. 

   Targetless calibration of lidar-imu system based on continuous-time batch estimation Jiajun Lv, Jinhong Xu, Kewei Hu, Yong Liu, and Xingxing Zuo IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2020 [Abs] [arXiv] [Video] [Code]

   Sensor calibration is the fundamental block for a multi-sensor fusion system. This paper presents an accurate and repeatable LiDAR-IMU calibration method (termed LI-Calib), to calibrate the 6-DOF extrinsic transformation between the 3D LiDAR and the Inertial Measurement Unit (IMU). % Regarding the high data capture rate for LiDAR and IMU sensors, LI-Calib adopts a continuous-time trajectory formulation based on B-Spline, which is more suitable for fusing high-rate or asynchronous measurements than discrete-time based approaches. % Additionally, LI-Calib decomposes the space into cells and identifies the planar segments for data association, which renders the calibration problem well-constrained in usual scenarios without any artificial targets. We validate the proposed calibration approach on both simulated and real-world experiments. The results demonstrate the high accuracy and good repeatability of the proposed method in common human-made scenarios. To benefit the research community, the code of this useful toolbox is also open-source.

1. 

   LIC-Fusion: LiDAR-Inertial-Camera Odometry Xingxing Zuo, Patrick Geneva, Woosik Lee, Yong Liu, and Guoquan Huang IEEE/RSJ International Conference on Intelligent Robots and Systems 2019 [Abs] [arXiv] [Video]

   This paper presents a tightly-coupled multi-sensor fusion algorithm termed LiDAR-inertial-camera fusion (LIC-Fusion), which efficiently fuses IMU measurements, sparse visual features, and extracted LiDAR points. In particular, the proposed LIC-Fusion performs online spatial and temporal sensor calibration between all three asynchronous sensors, in order to compensate for possible calibration variations. The key contribution is the optimal (up to linearization errors) multi-modal sensor fusion of detected and tracked sparse edge/surf feature points from LiDAR scans within an efficient MSCKF-based framework, alongside sparse visual feature observations and IMU readings. We perform extensive experiments in both indoor and outdoor environments, showing that the proposed LIC-Fusion outperforms the state-of-the-art visual-inertial odometry (VIO) and LiDAR odometry methods in terms of estimation accuracy and robustness to aggressive motions.

2. 

   Visual-Inertial Localization for Skid-Steering Robots with Kinematic Constraints Xingxing Zuo*, Mingming Zhang*, Yiming Chen, Yong Liu, Guoquan Huang, and Mingyang Li International Symposium on Robotics Research (ISRR) 2019 [Abs] [arXiv] [Supp]

   While visual localization or SLAM has witnessed great progress in past decades, when deploying it on a mobile robot in practice, few works have explicitly considered the kinematic (or dynamic) constraints of the real robotic system when designing state estimators. To promote the practical deployment of current state-of-the-art visual-inertial localization algorithms, in this work we propose a low-cost kinematics-constrained localization system particularly for a skid-steering mobile robot. In particular, we derive in a principle way the robot’s kinematic constraints based on the instantaneous centers of rotation (ICR) model and integrate them in a tightly-coupled manner into the sliding-window bundle adjustment (BA)-based visual-inertial estimator. Because the ICR model parameters are time-varying due to, for example, track-to-terrain interaction and terrain roughness, we estimate these kinematic parameters online along with the navigation state. To this end, we perform in-depth the observability analysis and identify motion conditions under which the state/parameter estimation is viable. The proposed kinematics-constrained visual-inertial localization system has been validated extensively in different terrain scenarios.

3. 

   Visual-Inertial Localization With Prior LiDAR Map Constraints Xingxing Zuo*, Patrick Geneva*, Yulin Yang, Wenlong Ye, Yong Liu, and Guoquan Huang IEEE Robotics and Automation Letters 2019 [Abs] [Video]

   In this letter, we develop a low-cost stereo visual-inertial localization system, which leverages efficient multi-state constraint Kalman filter (MSCKF)-based visual-inertial odometry (VIO) while utilizing an a priori LiDAR map to provide bounded-error three-dimensional navigation. Besides the standard sparse visual feature measurements used in VIO, the global registrations of visual semi-dense clouds to the prior LiDAR map are also exploited in a tightly-coupled MSCKF update, thus correcting accumulated drift. This cross-modality constraint between visual and LiDAR pointclouds is particularly addressed. The proposed approach is validated on both Monte Carlo simulations and real-world experiments, showing that LiDAR map constraints between clouds created through different sensing modalities greatly improve the standard VIO and provide bounded-error performance.

1. 

   Robust visual SLAM with point and line features Xingxing Zuo, Xiaojia Xie, Yong Liu, and Guoquan Huang IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2017 [Abs] [arXiv] [Video]

   In this paper, we develop a robust efficient visual SLAM system that utilizes heterogeneous point and line features. By leveraging ORB-SLAM [1], the proposed system consists of stereo matching, frame tracking, local mapping, loop detection, and bundle adjustment of both point and line features. In particular, as the main theoretical contributions of this paper, we, for the first time, employ the orthonormal representation as the minimal parameterization to model line features along with point features in visual SLAM and analytically derive the Jacobians of the re-projection errors with respect to the line parameters, which significantly improves the SLAM solution. The proposed SLAM has been extensively tested in both synthetic and real-world experiments whose results demonstrate that the proposed system outperforms the state-of-the-art methods in various scenarios.