There is a common misconception that chromecast can display the content that you have on your tablet or phone. Google made it look so seamless that people think so. It cannot display the media (photos, video) in your phone or tablet. But don't get disheartened yet, there are some hacks to work around the limitation.
The basic thing to understand is that when you choose to play a video (youtube, netflix), your phone/tablet will share the URL with the chromecast device. Chromecast will connect to your wifi router directly and get the content from the internet. There is more than this to make to look seamless. You can control the playback of the video from your device. To achieve this all your device has to do is to share the playback control information to the chromecast device and it will apply those controls on the content that it got from the web.
I think there is a good reason behind this. If it has to playback the media directly from your personal device, your personal device should have very good bandwidth capabilities to have a good video quality. Good video quality is important because the display will be on a large screen. Not all personal devices have this good bandwidth to stream local media.
I am guessing that google wants to give a good first-time-experience albeit with limited functionality. I think there will be a big demand from the users to be able to play the media on their phone/tablet. They may be willing to loose some video quality. Google may bring this feature down the line.
