Data-driven techniques can reliably build semantic correspondence among images. In this paper, we present a new regularization model for stereo or flow through transferring the shape information of the disparity or flow from semantically matched patches in the training database. Compared to previous regularization models based on image appearance alone, we can better resolve local ambiguity of the disparity or flow by considering the semantic information without explicit object modeling. We incorporate this data-driven regularization model into a standard Markov Random Field (MRF) model, inferred with a gradient descent algorithm and learned with a discriminative learning approach. Compared to prior state-of-the-art methods, our full model achieves comparable or better results on the KITTI stereo and flow datasets, and improves results on the Sintel Flow dataset under an online estimation setting.