If you check your home directory and see a lot of \.serverauth\.\d+ files, they are the files left after your X window has an unexpected exit or system crashes except one is currently being used.

I found a nice explanation of the situation, this doesn't happen to all users, only those who relies on startx. I added a simple code in my ~/.xinitrc because I don't want to touch system file as you read what that article tells you how to get rid of them by modifying /usr/bin/startx, so you can have a singular server authentication file.

Here is what I am using:
# clean up serverauth files, which were left due to unexpected exit of X window
for sa in ~/.serverauth.*; do pid=${sa/*.serverauth./}; [[ ! -d /proc/$pid ]] && rm ~/.serverauth.$pid; :; done
unset sa pid
It checks if the PID associated with the server authentication file exists, leave it alone if so, or remove the file. Of course, there is possibly false confirmations on PID of different process, but it does no harm except the file still remains. Eventually, you will cleaned them up all.