AI & Technology
I Asked ChatGPT the Same Business Question Four Ways. The Answers Changed.
One of the reasons AI has become so useful in business is its ability to take a complicated problem and produce a clear, well-reasoned recommendation.
You describe the situation, explain what you're trying to accomplish, and ask for advice.
A few seconds later, you have an answer.
Often, it's a pretty convincing one.
But I've been exploring a question that I think deserves more attention:
How much of an AI recommendation is based on an independent evaluation, and how much is influenced by what the AI already knows about your decisions?
I decided to test that question.
I asked ChatGPT to evaluate a business product strategy using four different conversational approaches.
The recommendations were noticeably different.
What made the experiment interesting wasn't that one answer was obviously right and the others were wrong.
It was that each recommendation came with a reasonable explanation.
And that creates a problem for anyone using AI to help make important business decisions.
The Business Decision
The scenario involved a service business evaluating how to structure its offerings.
There were several possibilities.
The business could offer multiple service options at different price points, giving customers flexibility based on their needs.
It could simplify its lineup around one primary service with an optional premium upgrade.
Or it could introduce a recurring membership intended to encourage customer retention and create more predictable revenue.
Each approach had potential advantages.
More options could serve different customer segments. Fewer options could simplify purchasing decisions and reduce operational complexity. A membership could support recurring relationships, but it could also introduce obligations that weren't justified by customer demand.
The question was which structure made the most sense.
I wasn't trying to get ChatGPT to recommend a particular answer.
I wanted to understand how its recommendations changed depending on whether it was evaluating an existing decision, a new proposal, or a hypothetical business.
Test One: Evaluate the Existing Product Lineup
In the first conversation, ChatGPT had context about an existing four-product structure.
The question was whether that structure made sense or whether it had become unnecessarily complicated.
ChatGPT largely endorsed the existing approach.
It explained how different offerings could address distinct customer needs and why maintaining multiple purchasing options might be reasonable.
The recommendation was essentially to preserve the structure rather than make substantial changes.
And the explanation made sense.
Customers don't all have the same requirements. Some prioritize price, others convenience, and others an ongoing relationship.
A business with multiple offerings can potentially accommodate those differences.
But there was something I wanted to test.
Would ChatGPT have recommended the same structure if it didn't already know that the business had chosen it?
That led to the second conversation.
Test Two: Start Fresh, Without the Existing Decision
For the second test, I opened a fresh ChatGPT conversation.
This time, I described the underlying business model and asked the AI to recommend a product structure without first revealing the existing lineup.
The answer changed considerably.
Instead of recommending four offerings, ChatGPT proposed a simpler structure.
Its recommendation centered on one primary service with an optional priority upgrade.
It also recommended against introducing a recurring membership at that stage.
The reasoning emphasized simplicity, customer clarity, and avoiding operational commitments before the business had sufficient evidence to justify them.
Again, the explanation was reasonable.
A simpler product lineup can reduce decision friction. It can make pricing easier to understand and limit the complexity associated with supporting multiple offerings.
But now I had two different recommendations for a broadly similar business problem.
In one conversation, the existing four-product structure was defensible and worth preserving.
In another, a simpler structure was preferable.
Both answers sounded confident.
Both presented logical arguments.
And neither, by itself, established which approach would produce better business results.
Test Three: Reveal the Existing Products
Next, I continued the second conversation.
After ChatGPT had recommended a simplified structure, I introduced the existing product lineup and explained that the business already offered multiple options.
I asked it to reconcile its earlier recommendation with that information.
Its position shifted.
Rather than recommending an immediate restructuring, it suggested a more incremental approach.
It emphasized retaining useful parts of the existing lineup, gathering additional evidence, and testing changes before making a larger commitment.
This was particularly interesting because the model wasn't simply repeating its first recommendation.
It was incorporating new context.
And there was a legitimate reason to do that.
A business considering a product launch is in a different position from one evaluating products that already exist.
An established offering may have customers, purchasing history, operating procedures, and expectations that should influence a decision.
Removing an existing product can carry costs that wouldn't exist if the product had never launched.
So a changed recommendation isn't necessarily a sign of AI bias.
Sometimes changing the recommendation is exactly what good analysis requires.
But that raises another question:
How do we distinguish a justified change in recommendation from a tendency to accommodate the decision that's already been made?
I wanted one more comparison.
Test Four: Present the Four Products as a Hypothetical Proposal
For the fourth evaluation, I started another fresh conversation.
This time, I described a proposed four-product lineup, but framed it as a hypothetical structure the business was considering.
The model could evaluate the options without treating them as established products.
Once again, ChatGPT favored simplification.
It recommended a primary service, an optional priority upgrade, and postponing the membership until the business had stronger evidence of demand.
That gave me an interesting pattern across the four evaluations.
| Evaluation | ChatGPT's general recommendation |
|---|---|
| Existing four-product lineup | Preserve the structure |
| Fresh evaluation without existing products | Simplify to a core service and priority upgrade |
| Existing products revealed afterward | Retain selectively and test changes |
| Four products presented as hypothetical | Simplify and defer membership |
The recommendations weren't identical, but there was a noticeable distinction between evaluating an existing decision and evaluating a proposed one.
When the product structure was presented as something already operating, ChatGPT was more inclined to preserve it.
When a similar structure was presented as something not yet implemented, it was more willing to recommend alternatives.
That's worth paying attention to.
But it also needs an important qualification.
What This Experiment Does—and Doesn't—Prove
This wasn't a controlled scientific experiment.
The prompts weren't identical. The conversations contained different amounts of historical information, and the business scenarios weren't framed in precisely the same way.
Those differences matter.
A model may reasonably recommend different strategies for a new business and an established one.
It may also change its recommendation because new information reveals implementation costs, customer expectations, or other constraints.
I can't conclude from these four responses that ChatGPT was definitively demonstrating anchoring bias or sycophancy.
And I can't conclude that the fresh conversation produced a better recommendation simply because it was more independent of the original decision.
What I can say is that the information and framing supplied to an AI assistant materially affect the analysis it produces.
That should concern anyone who uses AI recommendations as a form of decision validation.
Because a recommendation can sound objective even when it's built on assumptions introduced by the conversation itself.
The Bigger Problem: Every Answer Had a Convincing Explanation
This was the part of the experiment that stood out most.
ChatGPT didn't simply change its recommendation.
It provided a coherent business argument for each position.
Keeping multiple products made sense because customers have different needs.
Simplifying the lineup made sense because complexity can create friction.
Preserving existing offerings while testing alternatives made sense because operational history matters.
Deferring a membership made sense because recurring obligations should be supported by evidence.
None of those arguments was inherently unreasonable.
But they couldn't all be treated as definitive answers to the same strategic question.
This is where AI-assisted decision-making can become dangerous.
Not because AI necessarily produces bad advice.
But because it can produce persuasive advice from incomplete information.
A polished explanation can create the impression that a recommendation has been thoroughly validated when the model has actually identified one plausible interpretation of the problem.
The difference matters.
What Was Missing? Actual Business Evidence.
There was another weakness in the recommendations.
Some of the arguments depended on assumptions about how customers behave and how the business operates.
For example, a recommendation to simplify service options might assume that customers struggle to distinguish between products.
A recommendation to consolidate appointment lengths might assume that shorter appointments regularly require more time than allocated.
A recommendation to introduce recurring memberships might assume that customers want ongoing service relationships.
Those assumptions may be reasonable.
But reasonable isn't the same as verified.
When I compared some of the model's assumptions against actual operating information, certain concerns became less persuasive.
That didn't automatically make the existing product structure correct.
It demonstrated that a recommendation can change substantially when the analysis moves from hypothetical behavior to measured results.
And that leads to an important distinction.
AI can help develop a hypothesis.
It can help identify the evidence needed to test that hypothesis.
But the hypothesis itself shouldn't be mistaken for evidence.
For business decisions, that distinction can be the difference between an interesting recommendation and a reliable one.
A Better Approach: Independent First, History Second, Reconciliation Third
After comparing the four responses, I think there's a more effective way to use AI for strategic decisions.
It involves three separate stages.
1. Get an Independent Recommendation First
Begin by describing the business problem, objectives, and relevant constraints without unnecessarily revealing the solution you prefer.
Ask AI to recommend an approach from first principles.
For example:
A service business is evaluating how to structure its offerings. Its priorities are customer clarity, sustainable profitability, operational simplicity, and long-term retention. Recommend a product structure, identify your assumptions, and explain what evidence would be needed before implementing it.
The goal isn't to hide important business facts.
It's to avoid making the existing decision the default answer before alternatives have been considered.
2. Introduce the Existing Decision and Operating History
Next, provide the actual structure, relevant operating evidence, and constraints.
Ask the model to evaluate whether its original recommendation still makes sense.
For example:
The business currently offers several products with different service commitments. Here is the available information about customer demand, fulfillment costs, and product performance. Reevaluate your recommendation. Explain which assumptions were supported, which were contradicted, and what remains uncertain.
At this stage, historical context becomes valuable.
The AI can evaluate practical considerations that weren't available during the initial assessment.
3. Require a Reconciliation
This is the most important step.
Don't simply accept the revised answer.
Ask the model to explain why its recommendation changed.
For example:
Compare your original recommendation with your revised recommendation. Identify every material change, the new evidence responsible for that change, and whether the change reflects actual business constraints or an assumption that the existing structure should be preserved.
This forces the analysis to address the difference between the two positions.
It also creates an opportunity to identify unsupported reasoning.
If the AI cannot explain why its recommendation changed, that is useful information.
If it can explain the change, the explanation still needs to be evaluated against the evidence.
The goal isn't to make AI defend its consistency.
It's to make changes in reasoning visible.
Don't Assume the Fresh Answer Is Better
There's one final point I think is important.
It would be easy to look at this experiment and conclude that the fresh conversations were more objective and therefore more accurate.
I don't think that's a safe conclusion.
A fresh conversation may be less influenced by previous decisions, but it can also lack essential context.
It may overlook customer expectations, implementation costs, contractual obligations, or operational realities.
An established business can't always make decisions as though it were starting from zero.
The objective shouldn't be to eliminate historical context.
It should be to introduce that context deliberately and evaluate how it changes the recommendation.
Sometimes the existing decision will turn out to be the best one.
Sometimes the independent recommendation will reveal a better alternative.
And sometimes neither recommendation will be reliable until more evidence is available.
All three outcomes are useful.
The Real Lesson: AI Is a Decision-Support Tool, Not a Decision Authority
I've written previously about AI conversation bias and why AI agreement isn't the same as validation.
This experiment connects those ideas.
The context we provide influences the recommendations we receive.
Our existing preferences can influence the questions we ask.
And an AI assistant can produce convincing explanations for different strategies without having enough evidence to establish which one is best.
That doesn't make AI less useful.
It means we need to be more deliberate about how we use it.
For important business decisions, I would rather have two competing recommendations with clearly documented assumptions than one confident answer that never challenges the starting point.
The best AI-assisted decisions aren't necessarily the ones where the model agrees with us.
They're the ones where we understand why it recommends something, what evidence supports the recommendation, and what would cause it to change its mind.
Before asking AI whether your business decision makes sense, try asking what it would recommend if that decision hadn't already been made.
The difference between those two answers might be the most valuable part of the conversation.
Next in this series: When Real Business Data Contradicts AI Advice — why a convincing AI recommendation can fall apart when its assumptions are compared against actual operating results.