Mapping RGB and depth (Kinect v2)

Illustration
alheka - 2022-01-10T10:41:01+00:00
Question: Mapping RGB and depth (Kinect v2)

Hi everyone, for my project, I'm working with the Kinect v2 and I have to map the depth information onto the RGB images to process them: in particular, I need to know which pixels in the RGB images are in a certain range of distance (depth) along the Z axis; I'm acquiring all the data with a C# program and saving them as images.     I had a look on the internet and I found some equations to map depth and RGB info here. I am actually able to get the intrinsic parameters for the single camera calibration using a checkerboard recorded from both the RGB camera (1920x1080) and the IR sensor (512x424) and processing them with MatLab Camera Calibrator app, but I'm stuck doing the stereo calibration (to find the rotation and translation matrices), because MatLab Stereo Camera Calibrator doesn't accept pair of images with different resolution.     I've also tried some other method with no results; It would be great if anyone could help me figuring out what I have to do in order to solve my problem. Thanks very much!

Related Questions

  • How to connect microsoft kinect studio as kinect v2
  • Mapping RGB and depth (Kinect v2)

    • Expert Answer

    Profile picture of Kshitij Singh Kshitij Singh answered . 2025-11-20

    Hi alheka,
    I used to have the same problem. It's a pity that Matlab still hasn't solved this issue. But there's a solution to that. It took me quite a while to figure it out. Here you have the steps you need:
     
    1. Stereo Calibration:
     
    a) Find checkerboard points:
        [pointsRGB, boardSizeRGB, imagesRGBused] =detectCheckerboardPoints(imagesRGB);
    [pointsIR, boardSizeIR, imagesIRused] = detectCheckerboardPoints(imagesIR);
    b) Remove the pairs where in one of the sets the points were not detected. Matlab does that automatically for the stereo calibration. But you need to write your code. I did make such code, but it's too unreadable to share here !
     
     
    c) Now you need to manually combine the two sets.
        imagePoints = cat(4, poitnsRGB, pointsIR);

    d) Now the word points: where squaresize is the width of each of the squares in your calibration target (unit is mm).

        worldPoints = generateCheckerboardPoints(boardSizeRGB, squaresize);

    e) Calibration:

        [stereoParams,pairsUsed,estimationErrors] = estimateCameraParameters(imagePoints,...
    worldPoints,'EstimateSkew', false, 'EstimateTangentialDistortion', false, ...
    'NumRadialDistortionCoefficients', 2, 'WorldUnits', 'mm');
figure; showReprojectionErrors(stereoParams);
figure; showExtrinsics(stereoParams);
    f) If needed, remove pairs of images with high error and re-calibrate until you're satisfied with the error.
     
    2) Map the images: Actually the link you have is correct for this part. Just follow those instructions.
     
    3) Remember that some artifacts (repeated pixels) can be caused in the mapped image due to parallax and the fact that some pixels are visible in one camera and not in the other. There are ways to fix that.


    Not satisfied with the answer ?? ASK NOW

    Get a Free Consultation or a Sample Assignment Review!

    Upload Assignment Talk to Expert