Optical coherence tomography (OCT) is extensively used for diagnosis of diabetic macular edema due to its non-invasive imaging based assessment of the retinal layers. In this paper, we propose a new fully convolutional deep learning architecture, termed ReLayNet, for segmentation of retinal layers and fluid masses in eye OCT scans. ReLayNet uses a contracting path of convolutional blocks (encoders) to learn a heirarchy of contextual features, followed by an expansive path of convolutional blocks (decoders) for semantic segmentation. Additionally, skip connections relaying encoder outputs to matched decoder inputs are introduced to recover spatial information lost during downsampling. ReLayNet is trained with stochastic gradient descent to optimize a joint loss function comprising of both weighted logistic regression and Dice overlap loss. The framework is validated on a publicly available benchmark dataset with comparisons against five state-of-the-art segmentation methods which includes two deep learning based approaches. Additionally, eight incremental baselines are defined and compared with, to validate the individual contributions of the proposed framework. We demonstrate that ReLayNet can reliably segment the retinal layers and accumulated fluids with improved performance in retinal thickness estimation and contour delineation. With a segmentation time of 5s per volume, it is well suited for clinical applications.