Single image super resolution (SISR) is an ill-posed problem aiming at estimating plausible high resolution (HR) image from a single low resolution (LR) image. Current state-of-the-art SISR methods are patch-based. They use either external data or internal self-similarity to learn a prior for a HR image. External data based methods utilize large number of patches from the training data, while self-similarity based approaches use a highly relevant matching patch from the input image as a prior. In this paper, we aim at combining the ideas from both paradigms, i.e. we learn a prior for a patch using a large number of patches collected from the input image. We show that this results in a strong prior. The performance of the proposed algorithm, which is based on iterative collaborative filtering with back-projection, is evaluated on a number of benchmark super-resolution image datasets. Without using any external data, the proposed approach outperforms the current non-CNN based methods on tested standard datasets for various scaling factors. On certain datasets a gain is over 1 dB compared to the recent method A+. For high sampling rates (x4 and higher) the proposed method performs similar to very recent state-of-the-art deep convolutional network-based approaches.