There is no built-in functionality that lets you tell a GUILayout.Window to not draw itself on top of everything. That would go against the entire purpose of having a GUILayout.Window function in the first place - It's on top of everything because it's designed to do exactly that. ;)

That's why GUILayout.Window takes the name of another function, which is then responsible for drawing the window contents. It's designed this way to help Unity's rendering system schedule the actual drawcalls that render the window to the end of everything else, to make sure it gets drawn on top. It is also for this reason that it can be tricky to work with the ToolTip inside a GUILayout.Window (or GUI.Window, for that matter).

Therefore, I don't think it's possible for you to dictate the rendering of a RenderTexture on top of the window. What you can do instead, is to shrink the rectangle within which you allow your GUILayout.Window to be dragged. Would that work for you? You call GUI.DragWindow with a Rect that defines the screen space minus the RenderTexture. Then the user can't drag the window onto the RenderTexture.

Another workaround is to put the RenderTexture in a GUILayout.Window of its own, make that window non-draggable, and then continually call GUI.BringWindowToFront on it to shoot it to the top of everything. That's not particularly elegant, though.