Say “AI voice agent” and most people picture either a smart speaker or the old phone menu that asks you to “briefly describe your problem” and then understands nothing. Today’s voice agents work differently from both: they hold a natural conversation while connecting to your store’s systems in the background and getting real work done.

Rather than draw an abstract architecture diagram, this article follows a single call: a customer asking where their order is. We’ll walk through what happens behind the scenes at each moment of that thirty-second conversation. The timings are approximate and only there to make the story easier to follow; real timings depend on the line, the network and the models used.

The short answer: four layers

A voice agent is built from four layers:

  1. Speech recognition (ASR/STT): turns the caller’s voice into text as they speak.
  2. Language model and flow: works out what the caller wants from that text, decides what to do and writes the reply.
  3. Tools: actions connected to your store that the model can call: look up an order, open a request, book a callback, transfer to a person.
  4. Speech synthesis (TTS): turns the reply into a natural voice.

On top of these sits turn management: knowing whether the caller has finished speaking, going quiet when they interrupt, knowing what to do in silence. Whether an agent feels robotic or human usually depends more on this layer than on how clever the model is.

Second 0: the phone rings

The customer dials the store’s number. The call travels from the store’s carrier over a SIP line to the system running the agent. The system first checks which number was called: if several stores or brands share the platform, that decides which agent answers.

The agent speaks its greeting. It usually includes the store’s name, the fact that the caller is talking to a voice agent and, if the call is recorded, a recording notice. It’s kept short: the customer called to ask their question, not to listen to a long welcome.

Meanwhile the first piece of work is already done: the caller’s number is matched against the phone numbers on orders. In a well-designed system this match doesn’t use the plain number; numbers are stored encrypted and matched through a keyed hash. So before the customer says a word, the agent already knows “there are two orders for this number”.

Second 3: the customer speaks

“Hi, I placed an order last week and it still hasn’t arrived.”

As the customer talks, audio streams to the speech recognition service and becomes text. Two questions matter here:

  • Is it hearing correctly? Accents, background noise and narrow-band phone audio all make recognition harder, and order numbers, product names and addresses are where accuracy counts most.
  • Has the sentence ended? People pause mid-sentence: “My order… one second… it was last week.” A system that only listens for silence will jump in while the caller is still thinking. Better systems use end-of-turn detection that looks at meaning as well as silence: it waits for “one second” and doesn’t wait after “still hasn’t arrived.”

Second 5: intent and decision

When the turn ends, the language model takes over. It knows:

  • The agent’s instructions and the store’s rules (return policy, working hours, which topic goes to whom).
  • The conversation so far.
  • The list of tools it can use.

The intent is clear: order status. Before answering, the model decides to make a tool call: “get the orders for the caller’s number”. The principle here is important: the model doesn’t guess the order status, it asks the system. An agent’s reliability depends less on how well it writes sentences than on calling the right tool at the right time and passing on what the tool returns without changing it.

Personal data protection is another principle. Phone numbers, addresses and emails can be masked before anything is sent to the language model; the model works with “the customer’s number” as a concept and never needs to see the number itself.

Second 6: filling the gap

Tool calls and generating a reply take time. Dead silence on the line makes the caller wonder whether the call dropped. A good agent fills the moment naturally: “Let me check that for you.”

These short phrases don’t have to be synthesized every time. A library of common acknowledgements and fillers can be generated in advance in the agent’s own voice and quality-checked, so a phrase plays instantly instead of waiting for synthesis each time.

Second 8: which order?

The tool returns two orders: one delivered, one in transit. Instead of guessing, the model clarifies:

“I can see two orders on your number. Is it the running shoes you ordered last Tuesday?”

That small question beats reading out the wrong order and confusing the customer. If the number matches no order, the agent asks for the order number and looks it up that way.

Second 12: reading the status

The customer confirms. The agent reads out the status, carrier and tracking number. Two details make a difference when speaking:

  • Long numbers are grouped. Nobody can follow “412788190” read in one breath; “four one two, seven eight eight, one nine zero” is easy to follow and repeat.
  • It says what the data says. The agent reads the status and tracking stored on the store’s order record. If the record is stale, so is the answer, which is why where the order data comes from and how often it updates is one of the most important setup questions.

Second 18: the customer interrupts

While the agent is giving the delivery estimate, the customer cuts in: “But I won’t be home this evening.”

This is barge-in. The agent notices the customer speaking, stops mid-sentence and listens. Without it, the call becomes a game of taking turns, and the customer has to wait for the agent to finish. Tuning it takes care: a cough or a quick “uh-huh” shouldn’t silence the agent, but a real objection should, immediately.

Second 22: work gets opened

What the customer wants now isn’t information but an action: rearranging the delivery. If the agent can’t arrange that with the carrier itself, its job is to hand the work to the right people, complete. The model calls an open request tool: the order number, what the customer wants and a summary of the call land in the team’s inbox.

“I’ve passed a note to our team; they’ll arrange the delivery with the carrier and get back to you.”

Sometimes the right step is a transfer instead: the customer is upset, the topic is outside the agent’s remit or the customer simply asks for a person. The transfer target depends on the topic and working hours. If nobody picks up, the call shouldn’t just drop; the agent should tell the customer and book a callback.

Second 30: wrap-up and records

The agent asks if there’s anything else and ends the call. Behind the scenes, one last round of work happens:

  • The transcript, recording and per-turn response times are stored.
  • The call is tagged by topic (for example “shipping status”, “delivery change”).
  • The opened request appears in the team’s inbox with the order attached.

These records aren’t just an archive. At month-end they answer “what share of calls were about shipping, and how many did the agent resolve on its own?”

When should it hand over to a person?

When evaluating a voice agent, pay most attention to what it does when it doesn’t know. A good agent:

  • Doesn’t attempt things it isn’t allowed to do (say, an exception to your return policy); it turns them into a request or a transfer.
  • Doesn’t invent missing data; it says it can’t see that information and states the next step.
  • Doesn’t resist when the customer asks for a person.
  • Warns ahead of time when a long call approaches its time limit instead of cutting off.

Questions to ask when evaluating an agent

  • How accurately does it catch order numbers and addresses in your language? Test with your own examples.
  • What happens when the caller trails off mid-sentence or interrupts?
  • Which tools does it have, and which actions need your approval?
  • Is personal data masked before it reaches the language model? Where is data processed?
  • Can you see the transcript, response times and tool calls for each call?
  • Can transfer and callback rules follow your working hours?

How this flow works in Zitena

Every step of the call above maps to a behaviour in Zitena’s inbound calls. Calls come in on your own SIP line and number; the caller is matched to order phones through a keyed hash and the number is shown masked in the console. If there are several orders the agent asks which one, it reads tracking numbers in groups, stops when the customer interrupts and waits out half-finished sentences. Acknowledgement and filler phrases are generated in advance in the agent’s own voice and quality-checked. During the live call the agent processes what the customer says so it can do the job; for post-call analysis (topics, quality, insights), phone numbers, emails, addresses and long numbers are masked before a transcript goes to a language model.

What it can’t resolve lands in your team’s inbox as a request, or the call is transferred by topic and working hours; if nobody answers, a callback is booked. Each call’s transcript, per-turn response times and requests sit on the call’s page in the console. The shipping side of the same flow is covered on the order status calls page. The agent is live in Turkish today; other languages are adapted on request.

To hear this flow with your own orders, book a demo.