Check in your task manager on your desktop and go to the performance tab and check the CPU and memory i think the memory is reached to peak thats the reason its not able to merge the RPD as it requires enough memory to load. Try to close the other applications which are not required.
Did you try patch merge .? I feel this is more trustworthy than full merge. :)
Usually when you try to merge two different huge rpds like the OOTB one's with many changes in both and your less available system resources its often gives you that out of memory error.
I suggest you bring down all services while performing this merge and its always better to equalize the objects first and then do a merge either three or two way.! instead of equalizing during the merge.
More about equalization could be found here : http://docs.oracle.com/cd/E14571_01/bi.1111/e10540/mngreposfiles.htm#autoId2