OpenAI GPT-4: Multimodal, New Features, Image Input, How to Use & More

OpenAI made tremendous waves when it launched GPT-4 in 2023. For those unaware, GPT-4 kickstarted the intelligence age and finally showcased that language AI models are intelligent enough for many tasks. So in this article, we have scoured OpenAI’s blogs and the Internet to explain its multimodal capabilities including GPT-4 image input. If you are someone with little to no clue about it, get a cup of coffee and sit down as we tell you all about this AI model.

What is GPT-4?

Put simply, GPT-4 was OpenAI’s powerful large language model (LLM), which was launched in 2023. Language model systems, in general, are systems that try to predict the next word in a sentence and intelligently add their inputs to it. They do that by studying a large dataset that gives them the ability to identify patterns and act upon them.

GPT-4 brought huge improvements over previous-gen models like GPT 3 and 3.5. There were some specific things GPT-4 was better at, which we have discussed below. However, a simple point you should understand is that this model supercharged ChatGPT. The next model releases like GPT-4o, OpenAI o1, o3, and finally, GPT-5 completely changed the game.

Currently, the GPT-4 model is no longer available and it has been deprecated by OpenAI in both ChatGPT and API. In place, GPT-5 has become the default model on ChatGPT.

GPT-4 Explained

If you’ve used the previous GPT models, you might be aware of its limited ability to just interpret the text you input. However, one of the newest and biggest implementations in the new model is that it is multimodal. This means that GPT-4 is able to accept prompts of both text and images.

This translates to the AI not only receiving the image but actually interpreting and understanding it. This understanding will apply to prompts interspersed with both text and vision inputs.

Furthermore, GPT-4’s multimodal capability will spread across all sizes and types of images and text, including documents with text and photographs, diagrams (sketched or hand-drawn), or screenshots. GPT-4’s output will remain as capable as it would in just text-only inputs.

In a developer livestream organized by OpenAI, the company showcased GPT-4’s multimodal nature. GPT-4 was provided a screenshot of a Discord window in the stream and was asked to describe it in painstaking detail.

The model took a little over a minute and rendered an extremely descriptive and accurate response. The response captured almost every single element of the input screen. From the server name on the top left corner to the different voice channels and even naming all of the Discord members online in the right pane, GPT-4 captured everything.

GPT-4 was put through some more tests where people submitted several random artworks, including photos of a squirrel holding a camera. The model was then asked to identify “what was funny about this image.” It again churned out a response stating that the photo was funny because squirrels typically eat nuts and do not act like humans. As seen, it again provided a very specific answer like a human would.

However, as mentioned above, the model’s specificity goes beyond screenshots and onto text and image inputs of all types. OpenAI showcased the same when Greg captured a photo of a hand-drawn mockup of a joke website. He then uploaded the same to GPT-4’s API-connected Discord server. The model was then asked to ‘write brief HTML/JS code to turn the page into a website and replace the jokes with actual ones.’

Amazingly, GPT-4 compiled working code for the same. Upon testing it, it produced a fully working website wherein pressing the buttons revealed the jokes. The fact that the ChatGPT model could decipher human handwriting and create code out of a combination of text and image inputs is mindblowing. GPT-4’s multimodal capability is a huge step in the direction of AI fully understanding the prompts and delivering results with pitch-perfect accuracy.

While there weren’t any considerable hitches, OpenAI did claim that speed is something GPT-4 could use work with and it might take time. Furthermore, visual inputs for GPT-4 are still in research preview and are still not publicly available.

What Made GPT-4 Better than GPT 3.5/ GPT-3?

Besides its breathtaking multimodal approach, GPT-4 comes has other areas of improvement where the new model not only outperforms its older brothers. Some of these areas are:

1. Better Understanding Nuanced Prompts

OpenAI claims that it might be difficult to actually see the difference between GPT-4 and GPT-3.5 at first glance. However, the former’s capabilities come to light when you go into the nitty-gritty. To demonstrate the difference, the new model was pitted against GPT-3.5 in a variety of human-level exams. OpenAI used the most recent publicly available tests and gave the models no specific training for this.

The data itself paints a better picture than we could tell you. In all results, GPT-4 came out on top and scored above its former version. While the threshold was barely pushed in some exams (such as SAT EBRW), there was a tremendous leap in performance in other exams (Uniform bar exam, AP Chemistry, and more).

OpenAI stated, “GPT-4 is also more reliable, creative, and generally able to handle more nuanced instructions when compared to GPT-3.5.” This translates to the bot effectively understanding more complex prompts easily.

2. Exponentially Larger Word Limit

While everyone did love GPT 3 and GPT 3.5, people did wish it could recognize even longer inputs. The introduction of GPT-4 has solved that problem. The new GPT-4 AI language model comes with an astounding 25,000-word input limit, which is significantly large. For context, GPT 3.5 was limited to 8,000 words.

This means users will be able to feed the bot with much longer input prompts for it to read and then render outputs from. So when GPT-4 finally launches, you can expect to give a much more detailed response and take longer inputs without problems. What this means for the developers out there is that you will be able to feel new APIs and documentation to the chatbot and get help writing code or fixing bugs in existing code more easily.

3. Supports more languages

ChatGPT has predominantly been used by English speakers around the globe. However, GPT-4 takes other languages into consideration. The newest model has demonstrated support for over 26 different languages. This includes the likes of Ukranian, Korean, Germanic languages, and many more.

OpenAI tested the same by translating the MMLU benchmarks into a variety of languages. Out of 26 languages, GPT-4 outperforms the English-language performance of GPT-3.5 in 24 of them. However, there is more data training to be done before GPT-4 fully supports all languages fully.

4. Different Personalities

Steerability is a concept wherein you can instruct the AI to act a certain way with a fixed tone of speech. A good example of this is asking ChatGPT to act like a cowboy or a police officer (assigning it a role like we did while making our chatbot using ChatGPT API).

GPT-4 gets that steerability but OpenAI now makes it harder for the AI to break character. Developers can now fix their AI’s style from the get-go by describing those directions in the “system” message. Since these messages are easy to jailbreak, OpenAI is also working on making them more secure.

From the demos OpenAI showcased in its blog post, it was quite funny to see the user trying to get GPT-4 to stop being a Socratic tutor and just tell them the answer to their query. However, as it was programmed to be a tutor, GPT-4 refused to break character, which is the same as many developers can expect when they train their bots in the future.

Possible Applications of GPT-4?

While GPT-4’s multimodal application is not possible by users yet, OpenAI has already teamed with Be My Eyes, an app meant for the visually impaired. GPT-4’s inclusion in the app enables you to take a picture of what you’re looking at and the AI will describe exactly what’s on the screen, including dresses, plants, machines in a gym, reading maps, and much more.

GPT-4 has also partnered up with other apps like Duolingo, Khan Academy for intelligent learning, and even the government of Iceland for language preservation. While GPT-4’s API is currently available on a waitlist basis, we can expect developers to come out with amazing experiences once it is finally released. Even before that happens, the applications above are already live for people to use.

Does GPT-4 Have Any Limitations?

Even though GPT-4 is being heralded as the next step in artificial intelligence, it still has its speedbumps. For starters, GPT-4 lacks knowledge of any world knowledge of events that have occurred after September 2021. The model also doesn’t learn from its experience. This can lead to the GPT-4 model making reasoning errors and is even prone to accepting obvious false statements from a user.

GPT-4 can also fail at problems just like humans would. Like GPT 3.5, the new model can also hallucinate and be confidently wrong in its predictions. This might lead to GPT-4 not double-checking its work when sometimes making a mistake.

Despite that, however, OpenAI promises that GPT-4 has been better trained than the previous models to avoid this. In the company’s own internal adversarial factuality evaluations, the model scored 35% higher compared to GPT 3.5 in reducing hallucinations. While the model’s perceptions and predictions have improved, its results should still be taken in conjunction with human advice.

How to Get Access to GPT-4 Right Now

Unfortunately, GPT-4 is no longer available in both ChatGPT and API. Instead, you can use the GPT-5 model on ChatGPT for free and you can input images as well. OpenAI didn’t release the weights of the GPT-4 model so you can’t run it locally as well. Having said that, GPT-5 is much more intelligent and capable to perform various tasks, including image input and analysis.