While we have been waiting for Google, Anthropic, and DeepSeek to challenge OpenAI, Elon Musk’s xAI has swiftly emerged as its closest competitor this week. In such a short span, xAI has developed the Grok 3 model, showcasing impressive benchmark results. So, we did a deep dive and tested the Grok 3 base and reasoning models on a range of complex prompts, and what we discovered was truly surprising.
Reasoning Queries on Grok 3
I started testing the Grok 3 reasoning model with the popular Strawberry question, and it correctly answered that there are three r’s in the word Strawberry after thinking for 15 seconds. I threw another word “Lollapalooza” and asked it to count the number of l’s and it replied with 4, which is correct.
Next, I asked Grok 3 which number is larger — 9.11 or 9.9. Again, Grok 3 thought for 8 seconds and came up with the right answer. In fact, the Grok 3 model devised multiple mathematical methods to verify the final result, which was impressive.
After that, I posed this slightly tweaked puzzle to Grok 3 to misguide it. In my earlier comparison between ChatGPT o1 and DeepSeek R1, both models got the answer wrong and said the surgeon was the boy’s mother. Even the latest OpenAI o3-mini-high gets the answer wrong, completely ignoring the fact that it’s clearly stated in the prompt that the surgeon is the boy’s father.
The surgeon, who is the boy's father, says "I cannot operate on this boy, he's my son!" Who is the surgeon to the boy?
Finally, the Grok 3 reasoning model thought for 35 seconds and said the surgeon is the boy’s father which is correct. What I loved about Grok 3 is that it reasoned: “It’s possible this is a poorly phrased riddle lacking a clever twist, or perhaps it’s testing whether we overthink it. But based solely on the text, the relationship is explicit.” It also thought out loud, “the mention of not operating might be context or a red herring.“
Grok 3 is the only reasoning model to get the answer right, besides Gemini 2.0 Flash. It was not misguided and didn’t try endlessly to find the twist and establish a new relationship somehow.
In Greek mythology, who was Jason's maternal great-grandfather?
Lastly, I posed a question from Humanity’s Last Exam (HLE), and the Grok 3 Reasoning model aced it in just 47 seconds. Previously, only o3-mini-high has been able to get the answer right in 1 minute and 25 seconds. Even DeepSeek R1 failed to correctly find the answer. I would say, currently, Grok 3 has the best reasoning model, and it outranks OpenAI’s o3-mini-high, o1, and DeepSeek R1.
Grok 3’s Coding Performance
To test Grok 3’s coding capability, I asked the Reasoning model to write a Python program that shows a ball bouncing inside a hexagon. Basically, the ball should follow the principles of Physics and bounce off naturally.
Grok 3 thought for over a minute and generated the Python code. I ran the code on my PC, and the ball failed to bounce off. It simply jumped outside the hexagon. It was pretty surprising given that Grok 3’s Reasoning model achieved a great score on the LiveCodeBench benchmark.
Write a Python program that shows a ball bouncing inside a spinning hexagon. The ball should be affected by gravity and friction, and it must bounce off the rotating walls realistically
So I asked the non-reasoning base Grok 3 model to generate the same Python code. Surprisingly, it worked on the first try itself and the ball bounced off with great accuracy. The ball followed a natural path and simulated the ball’s movement perfectly. Perhaps, the Reasoning model overanalyzed the problem, leading to a glitch in the collision detection function.
I would say, the base Grok 3 non-reasoning model is solid for coding tasks. However, this is one of many tests, and you should use both reasoning and non-reasoning models on your codebase to check which one performs better.
Grok 3’s DeepSearch AI Agent
xAI has also launched a new AI agent called “DeepSearch” built on the Grok 3 model. It’s similar to OpenAI’s Deep Research agent, which is built on the full o3 model, which browses the web, does research, and generates a comprehensive report in 5 to 30 minutes. Grok 3’s DeepSearch, however, takes only a few minutes.
So I asked the Grok 3 DeepSearch AI agent to research “How is AI transforming the chip design process?” It started the thinking process and accessed multiple web pages, including scientific papers from IEEE, ACM, and more. In over a minute, the DeepSearch AI agent generated a 1300-word report including in-line citations, tables, and key points.
While the report explained Nvidia’s RL Circuits and Intel’s FloorSet dataset for an AI-powered chip designing process, it completely failed to mention Google’s AlphaChip framework for generating chip floorplans. The final report is similar to Perplexity’s new Deep Research tool. Both tools are quick but gloss over a lot of recent advancements.
Grok 3’s Political Bias
xAI’s owner Elon Musk has consistently criticized ChatGPT for being woke and having a left-leaning bias. In April 2023, Musk announced plans to create “TruthGPT” and to develop a “maximum truth-seeking AI”.
Just before the Grok 3 launch, Musk shared a response from the Grok 3 model calling a media outlet “garbage”. Many thought the Grok 3 model would be politically conservative and would lean right.
However, in my testing, Grok 3 is as politically neutral as possible. Even after pushing Grok 3 to take a stance on the subject matter, it explains the difference and leaves it to the user’s preference and discretion. Beyond the realm of politics, even on social issues such as transgender rights, DEI programs, immigration, and affirmative action — topics that Musk has openly criticized — Grok 3 maintains its neutral position.
What is interesting is that Grok 3 doesn’t shy away from joking about its owner, Elon Musk, and the current US President, Donald Trump.
Grok 3’s Safety Guardrails
When I tested Grok 2 last year, it was largely uncensored and didn’t have any safety guardrails. Grok 2 shockingly generated an email to scam people. However, Grok 3 has much better safety guardrails which is good news for AI safety. If you prompt Grok 3 with something harmful, it mentions, “I can’t assist with anything intended to harm or deceive others.”
As for AI image generation, the current Grok image generator on grok.com doesn’t generate images at all. However, on X, it still generates images of public figures and celebrities without any safety guardrails, which is concerning. It’s powered by xAI’s in-house Aurora image generation model.
Grok 3: Early Verdict
xAI has rolled out both larger Grok 3 base and reasoning models, and in my assessment, they both are frontier AI models that come close to the full OpenAI o3 model. OpenAI has so far only released o3-mini and o3-mini-high, besides the full o3 model which powers the Deep Research AI agent.
Based on my early testing, I can say the Grok 3 reasoning model surpasses (or at least matches) all available models, including OpenAI o3-mini and DeepSeek R1. Of course, this verdict is based on the standard “Thinking” effort. xAI has a “Big Brain” setting for the Grok 3 reasoning model, which uses more compute to think for a longer duration. It will be available to SuperGrok subscribers.
Its base Grok 3 non-reasoning model is also more capable than GPT-4o, Claude 3.5 Sonnet, and Gemini 2.0 Pro, becoming a solid alternative to ChatGPT. Perhaps for coding tasks, Claude 3.5 Sonnet may still win, but the gap is shrinking significantly.
Musk-led xAI has done a tremendous job at developing a powerful pre-trained Grok 3 model and an inference-scaled reasoning model. Now, we need to wait for OpenAI’s GPT-4.5 and GPT-5 models which are set to release in the coming weeks and months.
But at this moment, xAI has stood up to challenge OpenAI’s dominance in the AI space. Besides the legal battle, the fight between Elon Musk and Sam Altman rages on.
