On the 3rd of April (2 weeks before Easter) we released a new page at foody that showed all the fasting dishes in one single page.
This is a feature that was suggested to us many times since we first launched, and quite frankly we were very excited to launch it.
Until today, we were very happy with it and we intended to build a lot on it. Today… well we are just baffled – which is the perfect excuse to write a blog post.
So in this micro-analysis I will be evaluating the effectiveness of the page we created for Nistisima. The key metrics are:
What % of the users/sessions used it
What effect did it have on the sessions – especially on conversion rate, i.e. did it helped users place more orders which is always the goal
First thing I checked was what percentage of the people visited the page. Success Almost 8% of sessions visited the nistisima page – which I think is quite a high percentage.
So then I jumped into my analytics to see what was the conversion rate of Sessions with visit to the Nistisima page vs those without a visit to it and I was baffled to discover this
There was a drop, at the conversion rate of the people who visited the Nistisima page. What? Really? There must be an error in my data.
So I went back digging and trying to find an error in the data – but the data was solid.
Side Note: Time on page was increased by about 60% – WTF? – i ll get back to this later though. For now I have to test if this is statisftically singificant (though at first glance it seems like it is).
So I go to getdatadriven.com , put my data in and…
Fail There was a drop in the conversion rate. The drop seems to be statistically significant
The data have spoken
So the page was succesful in getting the interest of the people – lots of people visited and stayed there for very long (60% increase) but at the end fewer placed an order (Hypothesis: it seems that people gave up from the options they had. Oh how I wish I had hotjar installed at that point to review the sessions and understand why)
At this point I was baffled by the results (I know I already mentioned that many times but the results were really unexpected).
To further understand the difference I further broke down the results between new and returning users. The drop happened to returning users to foody which again is baffling(must really find a new word for that)
Nop. Nothing interesting here. CRAP
So as I start closing down my windows in disbelief (this blog was an internal micro analysis doc at that point) I see something very interesting.
The conversion rate of sessions that visited the nistisima page was much higher in the first week, and much lower after that. WTF
On the positive side, crazy observations call for crazy explanations. So it's Crappy Hypothesis time (at this point I am literally writing the blog as I progress):
1) People who fast order less in easter week cos they consider it more kosher/religious or something
2) Our Power users saw it immediately, and since they are our power users they have a much higher conversion rate it skewed results on the positive side, but when those saturated, the rest of the users had a lower conversion rate (so its an audience selection bias)
3) At the beggining we did not have all the fasting dishes from restaurants which were added day by day. Maybe we simply added too much and the page simply became to long and that was confusing/caused loading issues (couldnt find any)
4) I am doing some stupid and obvious mistake.
For what its worth, my money is on 4 – all the way 4. :)
So after a lot of WTFs I decided to switch my focus from Sessions to Users. The results really didn't dissapoint as at first glance they painted a completely different picture.
#notetoself: Ok I can sort of get why someone would hate analytics
So Users who visited the Nistisima page had a much higher conversion rate than those who didnt. WOOHOO? nahh it doesnt seem real.
The graph follows a similar trajectory and that definitely needs more thought. Maybe the assumption that this feature is most needed in the easter week (because more people fast then) was simply wrong and the earlier you are in the fasting period, the more useful it is #somethingtotestnexttime
But, when you look the data above a bit more closely you actually narrow down on the correct hypothesis. The magic number is the conversion rate for New Visitors which was lower for the the visitors who have visited the nistisima page – in stark contrast with all other data.
The way I see it this points to a skewed audience, i.e. the people who visited the nistisima page are our Power Users (especially in the first few days) which simply order more/have a higher conversion rate.
If only there was a way to test that hypothesis. Oh wait! I can see how these audiences did for the next 30 days.
Ughhh. Unfortunately thats not possible with segments as I did it (too much noise). I `ll have to think of another way to do it… but not today as I have already spent much more time than I planned for.
Till then I did the next best thing – wrote this blog entry and would really appreciate your thoughts and feedback :D