Author Topic: SDCC 2017 Hotel Lottery - Analysis and reflection  (Read 2433 times)

Offline alyssa

  • Administrator
  • Stan Lee's Hospitality Suite
  • *****
  • Join Date: Nov 2011
  • Posts: 20987
  • Liked: 5335
SDCC 2017 Hotel Lottery - Analysis and reflection
« on: June 21, 2017, 08:14:12 AM »
[member=4270]mark[/member] has gotten the hotel analysis article finished & it's close to being is published! Please use this thread to ask questions about the conclusions, process or anything else to do with the article!

It's a long read but very very insightful. A must read for anyone participating in the SDCC hotel lottery. While we do not know if our data is correct, we can draw general guidelines and get an a strong sense of where the process is & where it's heading.

eta link to article You are not allowed to view links. Register or Login
« Last Edit: June 21, 2017, 09:16:04 AM by alyssa »
Donations gratefully accepted. You are not allowed to view links. Register or Login,
Our Twitters @FoCCBlog @friendsofcci
"Are you willing to give up all you are, to keep all you have?" G'kar Babylon 5

Offline mark

  • Volunteer HQ
  • ******
  • Join Date: Mar 2015
  • Posts: 3760
  • Liked: 1940
Re: SDCC 2017 Hotel Lottery - Analysis and reflection
« Reply #1 on: June 21, 2017, 08:21:25 AM »
Thanks once again to everyone who supplied info about their submission and experience! We can go deeper into certain aspects here, maybe even talk some math (I know right!) All questions will be answered, a few maybe even satisfactorily.

Friends of Comic Cons

  • Guest
Re: SDCC 2017 Hotel Lottery - Analysis and reflection
« Reply #2 on: Today at 10:41:22 PM »

Offline karl clement

  • Ballroom 20
  • ****
  • Join Date: Feb 2013
  • Posts: 948
  • Liked: 136
Re: SDCC 2017 Hotel Lottery - Analysis and reflection
« Reply #2 on: June 21, 2017, 09:04:13 AM »
Mark, i noticed a trend that was not stated in the blog, it involves the hotel circle hotels and airport hotels, i and a friend did not have to do the hotel loto becaused we got SDCC hotels from hotel circle for under SDCC rates, T&C at $40 below rate per night, Hiton airport at $54 below SDCC rate per night, both july 18 -24 and paid, i found 2 how many other people found them, it might only be a small %
SDCC 2013,2014,2015,2016
FanExpo  2011,2012, 2015
Montreal CC  2011, 2013
Ottawa CC 2013
Cornwall CAPE 2015, 2016

Offline Javier

  • Exhibit Hall
  • **
  • Join Date: Mar 2012
  • Posts: 108
  • Liked: 50
Re: SDCC 2017 Hotel Lottery - Analysis and reflection
« Reply #3 on: June 21, 2017, 09:42:21 AM »
Thank you for the hard work Mark. This was excellent.

Offline debster

  • Temp
  • Ballroom 20
  • *
  • Join Date: Aug 2014
  • Posts: 738
  • Liked: 352
Re: SDCC 2017 Hotel Lottery - Analysis and reflection
« Reply #4 on: June 21, 2017, 09:51:23 AM »
Thanks [member=4270]mark[/member] , for spending the time to do this. Interesting and informative!

Offline TardisMom

  • Supporter
  • Volunteer HQ
  • ******
  • Join Date: Aug 2012
  • Posts: 2849
  • Liked: 1440
Re: SDCC 2017 Hotel Lottery - Analysis and reflection
« Reply #5 on: June 21, 2017, 12:18:32 PM »
Thanks [member=4270]mark[/member] , great info!!

Offline vegasndn

  • Ballroom 20
  • ****
  • Join Date: Mar 2012
  • Posts: 771
  • Liked: 160
Re: SDCC 2017 Hotel Lottery - Analysis and reflection
« Reply #6 on: June 21, 2017, 01:08:34 PM »
Thanks for your time in putting this together. Nice read


Sent from my iPhone using Tapatalk

Offline DRWHO42

  • Supporter
  • Volunteer HQ
  • ******
  • Join Date: Jan 2013
  • Posts: 4575
    • FoCC Blog
  • Liked: 1649
Re: SDCC 2017 Hotel Lottery - Analysis and reflection
« Reply #7 on: June 21, 2017, 02:53:37 PM »
Fantastic article. Thanks to all involved. You
Time and Relative Dimension in Space - 'TARDIS' for short

Editor-in-Chief of the FoCC Blog

Offline mark

  • Volunteer HQ
  • ******
  • Join Date: Mar 2015
  • Posts: 3760
  • Liked: 1940
Re: SDCC 2017 Hotel Lottery - Analysis and reflection
« Reply #8 on: June 21, 2017, 06:19:50 PM »
Need to also mention [member=1]alyssa[/member] [member=1060]DRWHO42[/member] and [member=314]Transmute Jun[/member] for tons of help wrangling it into something more readable, also [member=3724]jristen[/member] and [member=3710]SteveD[/member] for suggestions and ideas that ended up in the article.

You are not allowed to view links. Register or Login
Mark, i noticed a trend that was not stated in the blog, it involves the hotel circle hotels and airport hotels, i and a friend did not have to do the hotel loto becaused we got SDCC hotels from hotel circle for under SDCC rates, T&C at $40 below rate per night, Hiton airport at $54 below SDCC rate per night, both july 18 -24 and paid, i found 2 how many other people found them, it might only be a small %

That's a good point, and you're right, we only looked at data from forms that were submitted, and we didn't ask about people that purchased outside the lottery. I wonder if more people will participate in the lottery if the process stabilizes and people trust it more. I know a number of people who avoided the old sale because it was so intense and stressful.

Offline SteveD

  • Administrator
  • Fulfillment Room
  • *****
  • Join Date: Nov 2014
  • Posts: 1717
  • Liked: 1063
Re: SDCC 2017 Hotel Lottery - Analysis and reflection
« Reply #9 on: June 21, 2017, 06:57:34 PM »
Well done on the article.  Lots of great data collected.


Offline Michael M

  • Supporter
  • Volunteer HQ
  • ******
  • Join Date: Apr 2014
  • Posts: 2599
  • "Asps...very dangerous...you go first"
    • LinkedIn
  • Liked: 1082
Re: SDCC 2017 Hotel Lottery - Analysis and reflection
« Reply #10 on: June 21, 2017, 08:00:22 PM »
Great f'n read!  What will be interesting is, with this data, and us ideally collecting the same info next year, we might be able to start seeing trends and such.  Great article and great work!
"you're makin' - me - beat - up - grass!!"

Offline Imogen16

  • Freebie Table
  • *
  • Join Date: Apr 2014
  • Posts: 31
  • Liked: 7
Re: SDCC 2017 Hotel Lottery - Analysis and reflection
« Reply #11 on: June 22, 2017, 06:47:39 PM »
Thanks for taking the time to do this! It was a really interesting read  :)

Offline mark

  • Volunteer HQ
  • ******
  • Join Date: Mar 2015
  • Posts: 3760
  • Liked: 1940
Re: SDCC 2017 Hotel Lottery - Analysis and reflection
« Reply #12 on: June 23, 2017, 05:49:17 PM »
You are not allowed to view links. Register or Login
Great f'n read!  What will be interesting is, with this data, and us ideally collecting the same info next year, we might be able to start seeing trends and such.  Great article and great work!

It was tricky to compare with so many changes this year, also we asked more questions this year than the last. For example how many people got their #1 pick. Last year CCI put out a statement with a breakdown of results, which matched ours quite well, but they haven't released a similar breakdown this year.

There are 2 things that we can compare pretty well, one is the speed of the lottery, sessions were taken to the form faster than last year. Also, a smaller percentage of requests were waitlisted compared to last year, no doubt due to having the multiple notification groups. This makes me think that the number of requests that were offered a hotel but where they never booked it has been significant, and was a main reason for the new system. If I was involved I think I would make that a primary goal, get as many people into their final room as possible early on, and reduce the amount of bookings that happen outside of the lottery.

Thinking I'll post a little math stuff this weekend ;)

Offline mark

  • Volunteer HQ
  • ******
  • Join Date: Mar 2015
  • Posts: 3760
  • Liked: 1940
Re: SDCC 2017 Hotel Lottery - Analysis and reflection
« Reply #13 on: June 25, 2017, 10:18:24 PM »
I did promise some math so here we go, (don't worry, no equations just some talk and I'll skip a lot of details, especially since Preacher season 2 just started and I'm not going to be able to keep away from the tv for much longer.)

The biggest challenge with this data is what to do with times that are approximate. If you just use them as is then you would get all of these artificial spikes at 1 minute, 1:30, 2 minutes, etc., since that is what people will focus on. If you just look at browser timestamps, especially ones earlier in the lottery, then you see what you would expect to see if access times were truly random. It just looks like noise, and is consistent with the times being uniformly distributed. Now if we could somehow magically see the timestamps for the people who guessed, then those times should be distributed in a similar manner to the timestamps we did see since the underlying process that generates the times is the same. If we assume that everyone who estimated their time was off from the actual time by another random process then we can estimate the parameters of that process, and it seems that the average person tends to be somewhat overconfident and have an error range of about 30 seconds when they guess. (We actually went a bit further than this and made multiple categories based on how much confidence people expressed.)

Next we'll use a very powerful statistical technique called the bootstrap. Basically we make a new data set based on randomly sampling from the actual data, but with replacement so that one of the original records may show up multiple times. If that record had an estimated time then each time it is selected we will randomly dither it according to the parameters mentioned above. So if an estimated time was 1:00 the some of the time it would be included with a timestamp under a minute, sometimes over. If it had been estimated at 0:50 then more of those times would be under a minute, and so on.

So now we have this new data set, and we can ask it questions. It's fuzzier than the original data but that's because it incorporates the uncertainty due to the estimated times. We can still look for trends and as it turns out the trends for things like results and notification groups are still quite clear. To put it another way, combining the two types of data requires adding some noise, but the signal is strong enough to shine through it.

(Note, there are lots of other ways to approach this problem, one nice thing about the bootstrap is that is doesn't assume much and it can be adapted to try out lots of other things.)

There are some other things we looked at but which didn't end up in the write up. The timestamped data shows even clearer trends when looked at on its own. Situations where someone had a surprising result are much more likely when the times are approximated. There was also an asymmetry in the surprises, people were more likely to have a good result with a late time than to have a bad result with an early time, (taking into account what they requested and what options they used.) Even though we didn't publish these, they did lend some credence to the assumption that the process was fundamentally as it was described. If these had turned out differently or if the trends we looked for hadn't been clear then the analysis would have been much different.

All for now, it's Preacher time :)



Offline TardisMom

  • Supporter
  • Volunteer HQ
  • ******
  • Join Date: Aug 2012
  • Posts: 2849
  • Liked: 1440
Re: SDCC 2017 Hotel Lottery - Analysis and reflection
« Reply #14 on: June 26, 2017, 08:13:46 AM »
I was horrible at Stats in college.  And I don't understand any of what you just said ???.  But thanks for trying!