How to Evaluate an AI Consulting Engagement in New Zealand

The global AI services market is forecast to exceed USD 600 billion by 2028. This guide walks through the technical signals that distinguish genuine engineering capability from generic consultancy, the academic frameworks worth knowing, and the questions worth asking before any engagement is signed.

The AI services market in 2026

The market for AI-related professional services has grown faster than almost any other category of enterprise technology spend. According to IDC's Worldwide AI and Generative AI Spending Guide , global spending on AI-centric systems is projected to reach USD 632 billion by 2028, with a five-year compound annual growth rate of 29 percent. A significant share of that spend flows through consultancies, integrators, and specialist agencies — the upstream layer that turns models into deployed systems.

USD 632B

Projected worldwide AI systems spend by 2028

IDC 2024 ↗

78%

Of organisations now use AI in at least one business function

McKinsey 2024 ↗

29%

Five-year CAGR for global AI systems spending

IDC Spending Guide ↗

The challenge for any New Zealand or Australian business commissioning an AI engagement in this environment is not finding a vendor — it is distinguishing genuine engineering capability from rebranded generic consultancy. See our AI consulting service for how we structure engagements. This guide walks through the technical criteria that actually predict engagement quality, the structural questions worth asking, and the academic frameworks worth knowing before you sign anything.

The three layers of "AI consulting" — and why they get conflated

Most engagements marketed as "AI consulting" sit in one of three distinct layers, each with very different deliverables, talent requirements, and value:

The AI services stack LAYER 3 — Strategic advisory "Where should we apply AI?" Deliverable: prioritised opportunity map, governance model Skills: industry knowledge, change management, exec communication LAYER 2 — Solution architecture and integration "How should this be built and what should it integrate with?" Deliverable: working systems, monitoring, eval suites, runbooks Skills: software engineering, ML ops, systems integration LAYER 1 — Model and infrastructure "What model, hosted where, with what guardrails?" Deliverable: deployed models, fine-tuning, infra-as-code Skills: applied ML, cloud infrastructure, security engineering

The collapse of these layers under a single label is the single largest source of buyer confusion in the market. A Big-4 strategy team is excellent at Layer 3 and weak at Layers 1–2. A typical "no-code agency" is competent at parts of Layer 2 but rarely competent at Layer 1. An independent engineering shop may be strong at Layers 1–2 but lack the executive context of Layer 3. The Deloitte State of AI report consistently finds that engagements which fail tend to be ones where the layers were not made explicit.

What good engineering capability looks like

Rather than asking vendors about past projects in marketing terms, the technical signals that genuinely predict capability are observable and concrete:

  • They run evaluations. Ask to see an example eval suite — the set of test cases, expected outputs, and scoring logic that they use to detect regressions. Vendors who cannot produce one are working without instruments.
  • They use version control for prompts. Prompts are code. They should live in git, with diffs reviewed, and changes deployable through a CI pipeline. If "prompt engineering" means editing strings in a UI without history, the system will quietly degrade.
  • They have a defined fallback mode. What happens when the LLM API has an outage? When the model returns malformed output? When token limits are exceeded? Engineering-mature vendors have answers; the rest have hope.
  • They use the NIST AI Risk Management Framework or equivalent. Risk management is no longer optional. The framework provides a vocabulary for discussing context, measurement, and mitigation — and signals serious engineering maturity.
  • They monitor production systems. Real systems generate telemetry — latency, token usage, error rates, user feedback. Ask what dashboards exist for systems already shipped.
"The single largest predictor of AI project success in our sample was the presence of a measurement and evaluation discipline established before deployment — not after." — McKinsey QuantumBlack, State of AI 2024

The economics of AI engagement structures

Without naming dollar figures — which vary enormously by scope, region, and seniority — the three engagement structures that exist in this market each create different incentives:

StructureHow it worksWhat it incentivises
Time and materialsDay rate or hourly billing against actualsThroughput, flexibility, scope evolution — but no incentive to be efficient
Fixed-price projectDefined scope, agreed deliverables, fixed feeCost predictability — but creates strong incentive to minimise scope and refuse change requests
Retainer with outcome metricsMonthly fee tied to ongoing operation, with measurable KPIs in the contractLong-term system reliability, alignment between vendor and business success

The Australian Centre for the Public Awareness of Science and the OECD's policy database both document the increasing prevalence of outcome-aligned procurement in public-sector AI projects — a structure that's slowly migrating into private-sector best practice as well. For sector-specific context, see our pages on finance, medical, and professional services.

What to ask in an evaluation meeting

If you have 30 minutes with a prospective vendor, these are the questions that will tell you more than any case study:

  1. Show me a production system you maintain today. Our own case studies document live systems. Not a demo. A real system with real users, where you have access to logs.
  2. What's in your evaluation harness for that system? If they cannot describe the test cases, scoring, or regression process — they ship and pray.
  3. How do you handle model deprecation? Anthropic, OpenAI, and Google all sunset model versions. What happens when their primary model is deprecated?
  4. What's your incident response process? When the system breaks at 3am, what happens?
  5. What part of this would I need to learn to maintain it long-term? A good answer transfers knowledge; a bad answer creates lock-in.
  6. What's your view on the Model Context Protocol ? If they cannot articulate a view, they aren't tracking the infrastructure layer of the industry.

Free audits, discovery phases, and what they really mean

"Free audit" is a category that ranges from genuinely useful technical reviews to lightly-disguised sales calls. A genuinely useful audit produces three things: a written summary of the workflows that were assessed, a ranking by automation feasibility, and a specific list of next-step recommendations — whether or not you proceed with the vendor who conducted it. If what you receive looks like a slide deck pitching a six-figure project, the audit was the pitch.

The same logic applies to "discovery phases" billed at the front of large engagements. A discovery phase should produce artefacts — architecture diagrams, integration maps, eval criteria, risk assessments — that have standalone value, even if you choose to engage a different builder. Our free audit is structured this way deliberately.

The New Zealand and Australian context

According to Stats NZ , the New Zealand information and communications technology sector contributes around 7 percent of GDP, with the ratio of skilled-immigration visas for tech roles trailing demand by a significant margin. The New Zealand Productivity Commission has documented for over a decade that productivity growth lags the OECD average — a gap that AI-driven automation is uniquely positioned to close in the SME segment, where the marginal cost of senior engineering talent is otherwise prohibitive.

In Australia, the National AI Centre under CSIRO publishes regular landscape reports, and the federal government's Voluntary AI Safety Standard (2024) provides a useful procurement reference. Vendor responses to questions about how they comply with the Standard's ten guardrails are a useful filter.

When in-house, when outsourced, when hybrid

Building an internal AI engineering team is rational at certain scales and irrational at others. The variables that matter are: the volume of distinct workflows you need automated, the rate at which new use cases emerge, your existing engineering org maturity, and whether AI is a core competitive advantage for your business or a cost-reduction lever.

  • Pure in-house tends to make sense when AI capability is core to the product (a SaaS company building AI-native features) and the team is already over ~25 engineers.
  • Pure outsourced tends to make sense for businesses where AI is a cost-reduction lever applied to internal operations, and where the workflow volume is bounded.
  • Hybrid — an internal "AI product owner" working alongside an external engineering partner — is increasingly the dominant pattern for 50–500 person businesses. How we work documents this partnership model. The internal role owns the roadmap and metrics; the external partner owns delivery and operations.

Where to go from here

If you are early in the evaluation process, the most useful next step is to define what success looks like for the first workflow — measurable in hours saved, error reduction, or revenue impact — before contacting vendors. Vendor selection becomes vastly easier when the brief is concrete. Our free audit is built to produce exactly that brief, whether or not you proceed with us.

Talk to a senior team member.

30 minutes, no pitch. We map your specific opportunity and tell you what's worth doing first.

Book a free 30-min audit
← Claude implementation n8n vs Make vs Zapier →
SYS · ARKHAM
0%