Are you sure it is gdm that you are using in Xubuntu 12.04? The default now is lightdm, but if you updated you may still have gdm. Either way you can edit the /etc/init/lightdm.conf (or gdm.conf) and comment out the section shown in red below.
There is a similar section in the gdn.conf file which you can comment out, if gdm is what you are using.
# LightDM - light Display Manager
# The display manager service manages the X servers running on the
# system, providing login and auto-login services
# based on gdm upstart script
description "LightDM Display Manager"
author "Robert Ancell <firstname.lastname@example.org>"
# start on ((filesystem
# and runlevel [!06]
# and started dbus
# and (drm-device-added card0 PRIMARY_DEVICE_FOR_DISPLAY=1
# or stopped udev-fallback-graphics))
# or runlevel PREVLEVEL=S)