Solution was I had WP Maintenance Mode on! Of course it completely blocks Jetpack from accessing it publically. So expect it not to work when using some kind of maintenance mode. If there was some way to link my hosted WP and wordpress.com accounts so Jetpack would work even when in maintenance mode, that would be very useful
http://wordpress.org/extend/plugins/wp-maintenance-mode/
On a side note, it does work with WP Splash Image, which I thought may be the (or part of the) problem.
:)