AI Agents: Where We Are Now

Last week, OpenAI launched their “Agent Mode” to ChatGPT’s paid tiers (Plus, Pro and Team), joining the group of startups and even Procore that have either announced or launched AI agents this year. As always, lots of promises are made and lots of inflated expectations.

What AI Agents can do:

We can separate things we’d ask an agent to do into a few buckets:

1. Fixed, repetitive, narrow tasks, like finding information from multiple sources and pulling it into an understood format. Agents like “Deep Research”, and “Agent Mode” are good at this, with a few caveats:
1. They will give you a good summary, but it’ll be based on a pattern they think is the right one, which might or might not include information or formatting you prefer. You can also upload an example of what you want the output to look like, or information that’s important. This can help a lot.
2. Open ended, or poorly defined tasks, like checking document quality or completeness. This type of task requires more judgment than we often realize and isn’t a good one for agents.
3. Decision making or judgment, like finding and choosing a product. Very often there are more than one factor that goes into these decisions, and exactly how to weigh these factors isn’t clear - it’s a judgment call.
   

Bottom line

AI, and AI agents in particular, is improving in power but not in judgment and definitely not in the experience and intuition that goes into making real decisions or catching errors or omissions from documents or information. Think of agents as powerful but limited helpers to support decisions and day-to-day work, not as workers who can do a job end to end.

← Back to AI Reality Checks