Here’s How You Can Try Google’s Project Astra (Kind of)

Google announced a slew of updates yesterday including the Gemini 2.0 Flash model, Deep Research, and more. In addition, the search giant previewed its upcoming AI agents like Project Mariner, Project Astra, and Jules. Google said it’s improving Project Astra and its features will be integrated into the Gemini app.

Along with that, Google quietly released “Stream Realtime” on Google AI Studio which works very similar to Project Astra. The feature is meant for developers to test the API, but general users can also use it easily, and for free. You can use your camera to show your surroundings and talk to Gemini 2.0 in real time. The best part is that you can share your laptop screen and interact using real-time voice.

The latency has been reduced significantly, and it really feels like you are conversing with Gemini 2.0 in real time. You can use your smartphone camera or laptop’s webcam to show Gemini what you are looking at. You can also share your screen on your laptop. Here’s how you can try an early preview of something like Project Astra using Google AI Studio.

• Go ahead and launch aistudio.google.com/live in your mobile or laptop/desktop browser.
• Now, sign in with your Google account. Google AI Studio is free for everyone.
• After that, tap or click on “Show Gemini” and allow permission to use your live camera and microphone.

• Now, you can show your surroundings and talk to Gemini 2.0 in real time. I tested it using my smartphone, and it correctly identified my Raspberry Pi boards in real time.

• Next, on my laptop, I shared my screen with Gemini 2.0, and it worked pretty well. It analyzed my screen and gave me a few suggestions while looking for a laptop. That’s really cool.

For real-time coding assistance, the screen-sharing feature could come in handy. ChatGPT can already see your screen on macOS, but it’s limited to a select few coding apps. OpenAI previously announced that Live Camera is coming to ChatGPT, but there has been no official update so far. Perhaps, amid the ongoing “12 days of OpenAI” announcements, the company might launch Live Camera on ChatGPT.

By the way, Microsoft is also testing Copilot Vision which can see your screen as you browse the web, but it’s again limited to a few approved websites and available only to early access users.

With Stream Realtime, Google is showcasing something very close to Project Astra. Of course, the improved Project Astra can tap into real-time Google Maps, Google Lens, and Google Search data for an even more seamless experience, but it’s not available yet. I would say that even without live web access, Stream Realtime makes for a great Project Astra demo. Who knows Stream Realtime may be Project Astra behind the scenes.