Friends of Comic Cons

Comic-Con International => SDCC Housing, Parking, and Transportation => Topic started by: alyssa on June 21, 2017, 08:14:12 AM

Title: SDCC 2017 Hotel Lottery - Analysis and reflection
Post by: alyssa on June 21, 2017, 08:14:12 AM
[member=4270]mark[/member] has gotten the hotel analysis article finished & it's close to being is published! Please use this thread to ask questions about the conclusions, process or anything else to do with the article!

It's a long read but very very insightful. A must read for anyone participating in the SDCC hotel lottery. While we do not know if our data is correct, we can draw general guidelines and get an a strong sense of where the process is & where it's heading.

eta link to article http://wp.me/p7hNnV-2lQ
Title: Re: SDCC 2017 Hotel Lottery - Analysis and reflection
Post by: mark on June 21, 2017, 08:21:25 AM
Thanks once again to everyone who supplied info about their submission and experience! We can go deeper into certain aspects here, maybe even talk some math (I know right!) All questions will be answered, a few maybe even satisfactorily.
Title: Re: SDCC 2017 Hotel Lottery - Analysis and reflection
Post by: karl clement on June 21, 2017, 09:04:13 AM
Mark, i noticed a trend that was not stated in the blog, it involves the hotel circle hotels and airport hotels, i and a friend did not have to do the hotel loto becaused we got SDCC hotels from hotel circle for under SDCC rates, T&C at $40 below rate per night, Hiton airport at $54 below SDCC rate per night, both july 18 -24 and paid, i found 2 how many other people found them, it might only be a small %
Title: Re: SDCC 2017 Hotel Lottery - Analysis and reflection
Post by: Javier on June 21, 2017, 09:42:21 AM
Thank you for the hard work Mark. This was excellent.
Title: Re: SDCC 2017 Hotel Lottery - Analysis and reflection
Post by: debster on June 21, 2017, 09:51:23 AM
Thanks [member=4270]mark[/member] , for spending the time to do this. Interesting and informative!
Title: Re: SDCC 2017 Hotel Lottery - Analysis and reflection
Post by: TardisMom on June 21, 2017, 12:18:32 PM
Thanks [member=4270]mark[/member] , great info!!
Title: Re: SDCC 2017 Hotel Lottery - Analysis and reflection
Post by: vegasndn on June 21, 2017, 01:08:34 PM
Thanks for your time in putting this together. Nice read


Sent from my iPhone using Tapatalk
Title: Re: SDCC 2017 Hotel Lottery - Analysis and reflection
Post by: DRWHO42 on June 21, 2017, 02:53:37 PM
Fantastic article. Thanks to all involved. You
Title: Re: SDCC 2017 Hotel Lottery - Analysis and reflection
Post by: mark on June 21, 2017, 06:19:50 PM
Need to also mention [member=1]alyssa[/member] [member=1060]DRWHO42[/member] and [member=314]Transmute Jun[/member] for tons of help wrangling it into something more readable, also [member=3724]jristen[/member] and [member=3710]SteveD[/member] for suggestions and ideas that ended up in the article.

You are not allowed to view links. Register or Login
Mark, i noticed a trend that was not stated in the blog, it involves the hotel circle hotels and airport hotels, i and a friend did not have to do the hotel loto becaused we got SDCC hotels from hotel circle for under SDCC rates, T&C at $40 below rate per night, Hiton airport at $54 below SDCC rate per night, both july 18 -24 and paid, i found 2 how many other people found them, it might only be a small %

That's a good point, and you're right, we only looked at data from forms that were submitted, and we didn't ask about people that purchased outside the lottery. I wonder if more people will participate in the lottery if the process stabilizes and people trust it more. I know a number of people who avoided the old sale because it was so intense and stressful.
Title: Re: SDCC 2017 Hotel Lottery - Analysis and reflection
Post by: SteveD on June 21, 2017, 06:57:34 PM
Well done on the article.  Lots of great data collected.

Title: Re: SDCC 2017 Hotel Lottery - Analysis and reflection
Post by: Michael M on June 21, 2017, 08:00:22 PM
Great f'n read!  What will be interesting is, with this data, and us ideally collecting the same info next year, we might be able to start seeing trends and such.  Great article and great work!
Title: Re: SDCC 2017 Hotel Lottery - Analysis and reflection
Post by: Imogen16 on June 22, 2017, 06:47:39 PM
Thanks for taking the time to do this! It was a really interesting read  :)
Title: Re: SDCC 2017 Hotel Lottery - Analysis and reflection
Post by: mark on June 23, 2017, 05:49:17 PM
You are not allowed to view links. Register or Login
Great f'n read!  What will be interesting is, with this data, and us ideally collecting the same info next year, we might be able to start seeing trends and such.  Great article and great work!

It was tricky to compare with so many changes this year, also we asked more questions this year than the last. For example how many people got their #1 pick. Last year CCI put out a statement with a breakdown of results, which matched ours quite well, but they haven't released a similar breakdown this year.

There are 2 things that we can compare pretty well, one is the speed of the lottery, sessions were taken to the form faster than last year. Also, a smaller percentage of requests were waitlisted compared to last year, no doubt due to having the multiple notification groups. This makes me think that the number of requests that were offered a hotel but where they never booked it has been significant, and was a main reason for the new system. If I was involved I think I would make that a primary goal, get as many people into their final room as possible early on, and reduce the amount of bookings that happen outside of the lottery.

Thinking I'll post a little math stuff this weekend ;)
Title: Re: SDCC 2017 Hotel Lottery - Analysis and reflection
Post by: mark on June 25, 2017, 10:18:24 PM
I did promise some math so here we go, (don't worry, no equations just some talk and I'll skip a lot of details, especially since Preacher season 2 just started and I'm not going to be able to keep away from the tv for much longer.)

The biggest challenge with this data is what to do with times that are approximate. If you just use them as is then you would get all of these artificial spikes at 1 minute, 1:30, 2 minutes, etc., since that is what people will focus on. If you just look at browser timestamps, especially ones earlier in the lottery, then you see what you would expect to see if access times were truly random. It just looks like noise, and is consistent with the times being uniformly distributed. Now if we could somehow magically see the timestamps for the people who guessed, then those times should be distributed in a similar manner to the timestamps we did see since the underlying process that generates the times is the same. If we assume that everyone who estimated their time was off from the actual time by another random process then we can estimate the parameters of that process, and it seems that the average person tends to be somewhat overconfident and have an error range of about 30 seconds when they guess. (We actually went a bit further than this and made multiple categories based on how much confidence people expressed.)

Next we'll use a very powerful statistical technique called the bootstrap. Basically we make a new data set based on randomly sampling from the actual data, but with replacement so that one of the original records may show up multiple times. If that record had an estimated time then each time it is selected we will randomly dither it according to the parameters mentioned above. So if an estimated time was 1:00 the some of the time it would be included with a timestamp under a minute, sometimes over. If it had been estimated at 0:50 then more of those times would be under a minute, and so on.

So now we have this new data set, and we can ask it questions. It's fuzzier than the original data but that's because it incorporates the uncertainty due to the estimated times. We can still look for trends and as it turns out the trends for things like results and notification groups are still quite clear. To put it another way, combining the two types of data requires adding some noise, but the signal is strong enough to shine through it.

(Note, there are lots of other ways to approach this problem, one nice thing about the bootstrap is that is doesn't assume much and it can be adapted to try out lots of other things.)

There are some other things we looked at but which didn't end up in the write up. The timestamped data shows even clearer trends when looked at on its own. Situations where someone had a surprising result are much more likely when the times are approximated. There was also an asymmetry in the surprises, people were more likely to have a good result with a late time than to have a bad result with an early time, (taking into account what they requested and what options they used.) Even though we didn't publish these, they did lend some credence to the assumption that the process was fundamentally as it was described. If these had turned out differently or if the trends we looked for hadn't been clear then the analysis would have been much different.

All for now, it's Preacher time :)


Title: Re: SDCC 2017 Hotel Lottery - Analysis and reflection
Post by: TardisMom on June 26, 2017, 08:13:46 AM
I was horrible at Stats in college.  And I don't understand any of what you just said ???.  But thanks for trying!
Title: Re: SDCC 2017 Hotel Lottery - Analysis and reflection
Post by: SteveD on June 26, 2017, 08:19:17 AM
"It was my understanding that there would be no math." :o
Title: Re: SDCC 2017 Hotel Lottery - Analysis and reflection
Post by: Transmute Jun on June 26, 2017, 08:26:53 AM
Thanks for the explanation, Mark!

To paraphrase for those who got a bit lost, he essentially said that he tested the data in different ways, taking into account that many of the times were estimated and not exact, and no matter how he rearranged the data, the same overall results came through. So he's very confident about the general trend of results as discussed in the article.
Title: Re: SDCC 2017 Hotel Lottery - Analysis and reflection
Post by: mark on June 26, 2017, 01:17:36 PM
After I made the last post, I though I should go back and add a TL;DR version. But a few minutes into Preacher and it was clear I was getting nothing else done last night, the season premiere was insane.

But it's a new day, so here's a TL;DR haiku:

the soup was cloudy
but good, rice added for the
unexpected guest


(that might be too subtle, should've gone with a limerick.)

You are not allowed to view links. Register or Login
I was horrible at Stats in college.  And I don't understand any of what you just said ???.  But thanks for trying!

Here's an analogy, if you took a stats class one of the things they probably had you learn how to do was to compare the means from 2 samples, or the mean from a sample with some fixed value. You may have had to look up some number in the back of the book, and you would use a different table if you happened to know the standard deviation (they'd probably call this a z test) or if you did not know the standard deviation (t test.) The z test is just based on the normal distribution. The t distribution is a bit fatter, making the tests based on it more conservative, because it incorporates all of the possible values that the standard deviation could have had.

That's a general principle, when there are things you don't know you have to move that uncertainty into your model so that you'll be more cautious about making conclusions.

You are not allowed to view links. Register or Login
"It was my understanding that there would be no math." :o

Well then no discussion on conditional probabilities and maximum entropy optimizations for you young man! But +1 like for classic SNL.

You are not allowed to view links. Register or Login
Thanks for the explanation, Mark!

To paraphrase for those who got a bit lost, he essentially said that he tested the data in different ways, taking into account that many of the times were estimated and not exact, and no matter how he rearranged the data, the same overall results came through. So he's very confident about the general trend of results as discussed in the article.

That's right. Another, probably better, way I could have approached this would be to skip ahead to the model and then go into detail about parts of it. I'm assuming 3 parts.

(1) Your request is granted form access at a specific time on the onPeak server. This time (or something equivalent to it) is only known by that server.

(2) The next time your browser refreshes, it sees the access and goes to the form, possibly with some lag. At some point here the browser loads the elements that we can see in the browser cache.

(3) At some point you notice that your browser has gone to the form and you translate that to some estimated time, a very nervous brain is involved.

So we have:

Browser_timestamp = Actual_timestamp + Refresh_time + Lag_time
Estimated_time = Browser_timestamp + Brain_adjustment

and Refresh_time, Lag_time and Brain_adjustment are random factors that need to be estimated and described. My earlier post was some high level detail about trying to figure out the Brain_adjustment part. But when it's all done the questions now become:

Are the estimated factors in my model reasonable?

Are they sufficient to account for the results that we saw?

Are the trends we're looking for still present once I've included the uncertainties?

And in all cases the answer was a pretty comfortable yes.

In conclusion,
Mark made some posts, quite unclear
That left readers in need of a beer
But it all turned out okay
When later on in the day
They saw Conan notifications appear!
:)
Title: Re: SDCC 2017 Hotel Lottery - Analysis and reflection
Post by: Transmute Jun on June 26, 2017, 01:37:40 PM
You are not allowed to view links. Register or Login
But it all turned out okay
When later on in the day
They saw Conan notifications appear!
:)

From your keyboard to 1iota's ears!
Title: Re: SDCC 2017 Hotel Lottery - Analysis and reflection
Post by: TardisMom on June 26, 2017, 03:53:56 PM
I think [member=4270]mark[/member] needs a dedicated thread in which to post a haiku/limerick/poem each and every day.  Or more than one, if that's how his day is going.