Indoor Scene Segmentation using a Structured Light Sensor

Indoor Scene Segmentation using a Structured Light Sensor
http://cs.nyu.edu/~silberman/papers/indoor_seg_struct_light.pdf

Pre-processing:
1)remove holes in the depth image <- filter image by the cross-bilateral filter of pairs

2)rotate RGB image and depth map and labels to eliminate any pith and roll< - 3-axis accelerometer provided by Kinect

There exists some offset between RGB image and Depth image provided by Kinect. So, calibration is carried out to obtain precise spatial alignment between the depth and RGB images.
One method: using a set of checkerboard images in conjunction with the calibration tool of Burrus. This also provided the homography between the two cameras.

The energy function E(y) measures the cost of a latent label yi over each pixel in the image N. The energy is composed of three potential terms: 1) unary cost function, which depends on the pixel location I, local descriptor xi and learned parameters theta; 2) a class transition cost between pairs of adjacent pixels I and j; and 3) a spatial smoothness term (inhibit or encourage a class transition).

In this paper, the class transition penalty is very simple, which is as follows:

  

In some applications, the penalty can be learned from training set, and different penalty should be assigned among different class transitions.