It looks like your college roommate was right: ultimate frisbee is a sport.
At least that's the opinion of the International Olympic Committee, who have decided to officially recognise the World Flying Disc Federation (WFDF).
https://twitter.com/iocmedia/status/627778076475310080
The WFDF manages multiple disc-based games - such as ultimate frisbee and disc golf. Much like regular golf, WFDF games do not have a referee as they rely on the sportsmanship and fair play of the players.
But no, this doesn't mean that ultimate frisbee is an Olympic sport (yet).
https://www.youtube.com/watch?v=Jsg-wvhB3iE
