It’s no secret, Apple is going to have a big artificial intelligence focus in 2024 starting with WWDC. Every operating system, including iOS 18, will have AI features throughout, and some information about these features has already been leaked.

Apple will focus on privacy and security with its on-device AI tools. Some rumors indicate Apple may be looking into server-side AI, but that may not be ready for iOS 18.

Features will focus on small models that run without connecting to the internet unless necessary. Tools in apps like Safari, Notes, and Messages will enhance basic interactions.

Other features will likely be announced and updated, like more widget options, customization tools, and changes to HomeKit.

The following information has been obtained from anonymous people familiar with Apple’s plans for iOS 18 or from other leaks and rumors that have been reported.

Privacy is paramount

Apple sees privacy as a fundamental human right. So, expect the company’s AI strategy for iOS 18 to have a central theme around privacy and data control.

Existing tools that use large language models (LLM) or generative AI tend to rely on sucking up a lot of information freely available on the web or consumer’s device. These data sets and the prompts provided by users are used to continue training the models, which leads to privacy concerns.

Apple is expected to focus on training the models separate from the user’s devices and data. When a model is used, it will use data available on the device to make decisions, but that user data will remain on the device and not become part of the larger model.

There has been some discussion of Apple offering paid access to LLMs made by other companies, but these would come with a warning about data privacy. An AI-focused App Store could be introduced.

Siri and Spotlight with generative AI

An Apple-made algorithm codenamed Ajax has been rumored to have generative AI capabilities in iOS 18. It could be used to provide improved responses from Siri, Spotlight, and other system-level search tools like those in Messages.

Ajax will be able to learn from user data found in Files, Contacts, Messages, and apps exposed to Spotlight to quickly find specific content buried deep across the system and provide summaries or direct links. The on-device model will be able to generate responses to queries about user data without an internet connection.

Apple isn’t looking to introduce chatbot capabilities, so don’t expect ChatGPT-like interactions with Siri. However, the search tools will be able to do more than the “result from Wikipedia” that frustrates many users today.

The generative AI assistant could help with texting by providing summaries of conversations. For example, if you’ve been away from a group chat and return to find hundreds of unread messages, Ajax could tell you what you missed in a quick summary.

Intelligent Browsing

Not much is known about Intelligent Browsing just yet other than the option exists. It appears to rely, in part, on Apple’s generative AI called Ajax.

The user will have to enable Intelligent Browsing in iOS 18 manually. It will then be able to provide information as the user is browsing, like webpage summaries or image search tools.

The image search functionality is in the works, but it may not make it for the iOS 18 release day. It apparently builds on the Visual Lookup feature in Photos and will enable users to perform actions on photos found on the web.

Another new tool is coming to Safari, but it isn’t clear if it will have AI-focused capabilities. It is a tool called Web Eraser, and it can remove elements of a page by tapping them.

Photos Clean Up

Another Apple AI feature in iOS 18 will be focused on photo editing. Clean Up will be able to remove objects from a photo using generative AI.

The feature will differ from conventional repair and removal tools, which rely on machine learning to determine how to remove an object. Generative AI will be able to better understand what is in a photo and what should be behind an object for more accurate object removal.

Apple could introduce other features like generative fill, but it isn’t clear if that will arrive when iOS 18 debuts. The Photos app on Mac already has a repair tool, and it appears that Clean Up replaces that entirely.

Voice transcription and summary

The Notes app will be getting Voice Memos integration, and both applications are getting AI capabilities. The on-device AI in iOS 18 will be able to take a Voice Memo, provide a searchable text transcription, and generate a summary.

Users will be able to open the Notes app and start an audio recording while taking notes simultaneously. The feature is available in third-party apps already, but Apple’s melding of Voice Memos and Notes may Sherlock those apps.

Once a recording has been completed, users will be able to ask questions about the content of the recording. For example, if you missed a specific part of a lecture, ask for when that topic occurred and a summary of it.

Apple Music Passthrough and Smart Transitions

A new feature called “Smart Transitions” is expected to replace or enhance the existing crossfade function. It would use an understanding of the current song and upcoming song to determine how to best transition between them.

The existing crossfade function works the same as it always has — it simply starts playing a song a few seconds before the previous song ends.

Another feature called Passthrough is coming, though nothing is known about the feature beyond appearing as a setting in Apple Music and QuickTime. It seems to be associated with Dolby Atmos.

Merging apps

A pattern has emerged from various insider details about iOS 18 — cross-app functionality is spreading. The Calculator is going to appear in Notes, Calendar will have a Reminders view, and Voice Memo recordings will also get added to Notes.

It isn’t clear if these are the only new integrations coming with the updated operating system, but it shows how Apple’s first-party apps continue to evolve. Being able to take notes with an intelligent calculator function could Sherlock some of the popular notes-meets-math apps like Soulver.

The Calendar and Reminders integration won’t be as in-depth as apps like Fantastical, but having a universal view of all tasks could be enough to bring more people into Apple’s free apps. Sherlocking doesn’t always have to do with being the best option, it can happen by being the good-enough default.

By integrating different functionality across the operating system, Apple may push more users to fall back to first-party apps. If a user can’t pick the default calculator, for example, they’re more likely to default to Apple’s apps. Apps like Apple Music and Safari have come under antitrust scrutiny for similar system-level Apple-only functionality.

iOS 18 release date

Apple will reveal iOS 18 during WWDC, which is being held June 10, 2024. It will be announced alongside iPadOS 18, tvOS 18, watchOS 11, macOS 15, and visionOS 2.

After a brief beta period over the summer, Apple will release iOS 18 to the public shortly after the iPhone 16 launch in September.