How to judge an AI operating system

Every vendor in this category can show a polished demo. A demo is built to show the best day of the product, with clean data and a friendly example. Your real week has duplicate contacts, half-finished notes and a spreadsheet that only one person understands. This guide gives you the tests to run on that mess, the questions to put in writing, the red flags to watch for and a pilot you can finish in a month.

Judge it on four questions. Does it keep business context? Does it act inside real workflows? Does it record its decisions, with an owner for each approval? Does it get better with use? A product that answers yes to all four belongs in the category. A product that answers yes to only one or two is a tool with an AI feature.

These four questions come from the layer test in our guide to AI-native business systems. Put each one to the vendor in its own words, then ask for proof on a sample of your data.

The tests to run before you buy

Run four tests in a trial, each with your own data and one named person watching. Each takes an afternoon or less, and each exposes a gap that a demo hides. Write each result in a scorecard before you discuss price, so the choice rests on evidence.

  1. One task, start to finish. Pick a recurring request, such as a weekly summary of open deals, and follow it from the request to the record it leaves behind. If the trace stops at a chat answer, the system is not doing the work.
  2. The messy data test. Load a sample with duplicate contacts and missing fields. Count how often a person has to correct an output before it can be used.
  3. The approval test. Mark one sensitive step, such as sending a message to a customer. Check that the system waits for a yes, and that the log records who gave it.
  4. The exit test. Ask for an export of records, files and automation instructions, then read the export yourself.

Time the checking as well as the work, and note how long a person needs to review each output. A tool that saves time on drafting but adds the same time back in checking has not saved anything.

Questions to put to vendors

Put these questions in writing before the demo, and ask for written answers. A vendor that avoids a question has told you something useful. Ask about the context, the approvals, the log, the connections, the data, the cost and the limits, in that order, so each answer builds on the one before.

  • Where does the business context come from, how is it kept current, and who can change it?
  • Which actions run alone, which wait for approval, and who sets that rule?
  • Where is the log of each action, and can we export it?
  • Which tools connect natively, and how are the others linked, through an API, a webhook or an MCP server?
  • What happens to our data? Is it used to train any model, and where is it stored?
  • What does the monthly cost look like at our full workload, not during the trial?
  • What will you not do? Name the work you would decline.

Sistava is one of the options in this category, so these questions apply to us as well. Ask us in writing, and hold us to the same standard as any other vendor.

Red flags

A red flag is a sign that a label is covering a thin product. One flag is a reason to ask harder questions. Two or more are a reason to walk away, unless the vendor answers them in writing with proof.

Watch for a demo that ends at a chat answer, with no task moved through to a record. Watch for taglines that say AI is not bolted on, with no explanation of how the system works. A tagline is not evidence. Watch for copy that opens by saying what the product is not, such as not a trend, and never says what it is.

  • Pricing that is hard to predict once usage grows, or a spending cap you cannot set.
  • A promise that the system needs no oversight. Every real system needs a named owner.
  • No way to export your records, files and instructions.
  • Customer stories without names, results or dates.

Be careful with awards and user counts too. They do not show how a system is built, so set them aside and ask for the trace.

A one-workflow pilot plan

Run a pilot for four weeks on one workflow, with one owner and a written scorecard. Keep the scope small enough that the team can check every output. The steps below cover the whole pilot. Four weeks is long enough to see the normal case, and short enough to stop early if it fails.

Four-week pilot on one workflow

  1. Choose the workflow. Pick one recurring task that touches at least two people or tools, such as a weekly report or a follow-up after each demo.
  2. Write the success line. Agree on the time you expect to save, the number of fixes you will accept per output and the approvals required, before the trial starts.
  3. Load the context. Give the system the facts it needs, from sources you trust. Record every fact it gets wrong, because those errors show what the context is missing.
  4. Check every output in week one. In week two, check a sample. Only the risky steps should still need a yes once the scorecard holds.
  5. Score it and decide. Compare the results with the success line, turn each gap into a specific request to the vendor, and keep the option that handles the most of your work with the least checking.

Use the approval rules to decide which steps the pilot can run alone. Anything that sends a message to a customer, changes a record or moves money should wait for a yes until the results have held for several weeks.

Sistava is one place to try this. You describe the routine in plain language, hire an AI employee for it, and review the work before you add the next one.

Which to pick

Pick the option that handles your one workflow with the least checking, then widen the scope only if the results hold. A good buyer's guide ends with a decision, not a list of features. If no option passes the trace test, the honest answer is to fix your process first and run the test again in a month.