Someone Built an AI Agent for the iPhone Before Apple Could
This could be the future of how we use our phones.
Agentic AI is the next big step forward for artificial intelligence: bots that don't just respond with (often inaccurate) guidance and instruction, but can actually take actions on our behalf. Imagine ChatGPT, if it could research hotel prices and book your vacation, or Gemini being able to order the groceries you need on the web.
This technology is still in its nascent stage, but AI tools tend to progress rather rapidly these days, and we've already seen previews of browsers that can do your browsing for you, and bots that will find a new apartment for you and your roommates. These agents are going to be available very soon.
This all raises important questions: Do you want to give AI access to your calendar and credit card? Do you trust your favorite bot to pick the best restaurant in town? The promise is that you'll save time on these tasks, but AI makes mistakes and can't really be trusted for anything important right now.
Now we have an early AI agent for the iPhone, built by coder Rounak Jain (as reported by 9to5Google). It's not available as an app, but you can test it out if you're prepared to put in a bit of work and spend a few cents on OpenAI credits.
Our phone agent future?
The PhoneAgent assistant, which Jain originally put together as part of an OpenAI hackathon last year, is still rough—the programmer himself says this is experimental software that can "get things wrong sometimes." However, it does give us a glimpse of where Apple, Google, OpenAI, and the rest may soon take us.
The tool connects to OpenAI models, so you can tap into the same kind of resources you would get with ChatGPT, besides being able to take action on your iPhone. This can be particularly handy when you need to create text in an email, message, or document, which can be generated with a quick prompt.
In his demo, Jain gets the app to take a photo, share it with his friend Ron, and attach a generic haiku to the message. In another request, the agent messages flight details to Ron (that guy again!), before booking an Uber to the airport, figuring out from the flight number that the Delta terminal was needed. System-level operations are also possible, such as opening the Control Center and turning on the flashlight.
Commands can be issued via voice or typed text, and a notification gets displayed when each task is completed—you can actually reply to this notification with follow-up commands (to turn the flashlight off, for example).
In my testing, the PhoneAgent was occasionally buggy and always quite slow, but that's to be expected when you're testing early-stage software like this. On the whole, it carried out my commands as intended: I got it to send a message to my friend Beth expressing how much I was looking forward to seeing her, and managed to take a photo with the Camera and open it up in the Photos editing interface with a single command.
App support is (understandably) a bit patchy, and I wasn't able to use the agent to play the latest Self Esteem album on Spotify, or take a photo and automatically attach it to a note. Third-party integrations are going to be crucial, as devices like the Rabbit R1 have shown, but it's not clear exactly how much access app makers will want to give to AI bots.
If this is what Siri and Gemini eventually evolve into, I can see it making a significant difference: Imagine being able to check where your next meeting is and get directions there with a single command, or having AI book your car service while you're busy with something else. There are still a lot of hurdles to overcome, though—Jony Ive and Sam Altman, take note.
How to get it working
The process of getting PhoneAgent working is a little involved, but it's not too difficult—and it's free, apart from the OpenAI credits you need to spend to access its models. (I ran a few dozen commands through the software, which ended up costing me around $1 in credits.) First, you'll need to install Xcode (from the macOS App Store) and GitHub Desktop on your Mac, and a free GitHub account, if you don't already have one.
From GitHub Desktop, clone the PhoneAgent repository: Choose Clone a Repository from the Internet from the opening screen, or open the File menu and pick Clone Repository. Switch to the URL tab, paste "https://github.com/rounak/PhoneAgent.git" into the URL field, pick a destination folder on your Mac, then click Clone.
You've now got the PhoneAgent code on your Mac. Head to the folder you saved it to in Finder, double-click on the PhoneAgent.xcodeproj file, and it should open in Xcode. You're then ready to transfer it to your iPhone to test it out: Connect your phone to your Mac with a cable, unlock the phone, and choose to trust the computer if prompted. Then head to Privacy & Security from iOS Settings, and turn on Developer Mode.
After your iPhone restarts, open the drop-down menu at the top of Xcode on macOS (to the right of PhoneAgent), and pick your iPhone from the list. Then, select PhoneAgent at the top of the left-hand navigation pane, switch to the Signing & Capabilities tab, and use Add Account to enter your Apple credentials. With that done, choose your name from the Team drop-down under Signing & Capabilities. You also need to change the Bundle identifier to something unique.
Next, select PhoneAgentUITests.swift from the left-hand pane, and look for the "func testLoop()" entry in the code (just a little way down). If you hover over the code line label to the immediate left, it should turn into a play button: Click this to run PhoneAgent on your iPhone.
The first screen you see will prompt you for an OpenAI key. In a web browser, sign up for an OpenAI account, then head to this page: Add payment details and buy credits to use the service (the minimum spend is $5). After that, switch to the API keys page, and click Create new secret key to get a key you can give to the PhoneAgent (make sure you keep a copy of this key somewhere safe, too).
You can then start entering commands by tapping the mic button or tapping the text at the bottom of the screen and telling the agent what you want it to do.
