Penetration of variable renewable energy sources (RES) significantly changes processes in electric power systems (EPS), especially in case of faults. Such changes can trigger incorrect operation of relay protection (RP), the principles of functioning and settings determination of which were formed without taking into account the noted RES influence. It is necessary to study in detail the processes under all kinds of normal and abnormal conditions both in the EPS, in the transducers, and in the RP devices themselves to take any preventive measures. For that reason, comprehensive studies were carried out in EPS with RES using Hybrid Real-Time Power System Simulator - the EPS simulator developed by the authors, and detailed mathematical model of one of the major type of RP in EPS - numerical transformer differential protection. The main fragments of these studies are presented in the paper. The obtained data can be used, in particular, in the development of new methods and tools for RP setting up.