Map Fragment shows as visible despite android:visibility="gone"

I'm working on an alternative simplified layout for an app that is used for smaller screens. This layout should not include a map fragment that is used in the larger layout, however, I'm supposed to avoid programmatic changes if at all possible. The activity that shows the layout also does stuff with the fragment using its ID, so I can't just delete it. I tried making the fragment invisible, like this:

<fragment
            android:id="@+id/map"
            android:name="com.google.android.gms.maps.MapFragment"
            android:layout_width="match_parent"
            android:layout_height="match_parent"
            android:visibility="gone" />

But the fragment remains visible. The same happens when I set the visibility to "invisible". The fragment's default position is partially hidden by a view, but when I move the fragment around the layout, it still shows up as visible wherever I put it.

Is there a reason the fragment ignores the visibility parameter? Is there another non-programmatic solution?