Skip to main content

Private AI means the organization decides where prompts, data and models live. Microsoft offers three patterns: models sold by Azure inside your own tenant, with data at rest in your chosen geography and the deployment type deciding where prompts are processed, Foundry Local on Azure Local for open-weight models on on-premises Kubernetes (preview, can run disconnected), and Foundry Local for on-device inference with no Azure subscription. A multi-model architecture uses each where it fits.

Key Facts

  • For models sold by Azure, prompts and completions are not available to other customers or model providers and are not used to train foundation models without permission (Microsoft Learn, accessed 2026-10-27).
  • Foundry Local on Azure Local runs AI models on an Arc-enabled Kubernetes cluster on-premises and can operate in disconnected environments; it is in preview (Microsoft Learn, accessed 2026-10-27).
  • Foundry Local runs models on the device: data never leaves it, the app works offline, and no Azure subscription is required (Microsoft Learn, accessed 2026-10-27).
  • Microsoft sells models from providers including Cohere, DeepSeek, Meta, Mistral AI and xAI through Azure alongside Azure OpenAI models (Microsoft Learn, accessed 2026-10-27).

Last updated by Errin O'Connor, Founder & Chief AI Architect, EPC Group

The direct answer

A private AI solution keeps prompts, business data and, where required, the model itself inside boundaries the organization controls. With Microsoft technology there are three ways to do that, and most enterprises need more than one: regulated records may have to stay on servers in their own datacenter, while everyday drafting and search can run on models hosted in their own Azure tenant.

Three deployment patterns

PatternWhere the model runsFitsStatus (per Microsoft Learn)
Tenant-hostedModels sold by Azure, including Azure OpenAI and open-weight families, deployed in your Foundry resource in a geography you chooseMost enterprise workloads that need scale without running GPUsAvailable now; data at rest stays in the chosen Azure geography, but Global deployments may process prompts in any Azure region and Data Zone deployments only within the US, EU or APAC data zone; Standard and Regional Provisioned keep processing inside the geography (Microsoft Learn, Deployment types in Microsoft Foundry Models, accessed 2026-10-28)
On-premisesFoundry Local on Azure Local: an Arc-enabled Kubernetes cluster in your datacenter, CPU or GPU, models from the Foundry catalog or your own registrySovereignty or regulatory rules, low latency next to the data, disconnected sitesPreview
On-deviceFoundry Local inside the application on the user’s Windows, macOS or Linux deviceOffline work and sensitive data that should never leave the laptopSingle-user inference; not designed as a server stack

Requirements you can verify

“Private” is not one setting. Write each requirement down, then check it against the deployment instead of trusting a label. With models sold by Azure, data at rest stays in the Azure geography for every deployment type, but the deployment type decides where prompts are processed: a Global deployment may process them in any Azure region.

RequirementHow to meet itHow to check it
Prompts are processed only inside the EU (or the US)A Data Zone Standard or Data Zone Provisioned deployment in a region of that zone, or a Standard deployment in your geographyThe deployment’s SKU reads DataZoneStandard, DataZoneProvisionedManaged or Standard; an Azure Policy on the subscription denies GlobalStandard
Stored data stays in our geographyAny deployment type: files, Responses API history and other stored data stay in the resource’s Azure geographyThe resource’s region; check that the feature you rely on is generally available, because preview features may not support every data commitment
Microsoft staff do not review our promptsApproved modified abuse monitoringContentLogging = false in the resource’s JSON view or the Azure CLI output
No public network path to the modelsA private endpoint, with public network access set to DisabledThe private endpoint connection shows Approved, and nslookup inside the network returns its private IP
Inference keeps working with no internet connectionFoundry Local on Azure Local (preview) on your own clusterRun the workload with the site disconnected and record the result
Data never leaves the user’s deviceFoundry Local on the deviceRun inference with networking off and record the result

Record the result per workload. The private AI deployment record (CSV) has a row per workload for the data class, the pattern, the deployment type, the processing and storage boundaries, abuse monitoring, network access, key management and how each was verified, with sign-off columns. Its example rows are labeled SYNTHETIC-EXAMPLE, not client data.

Where multi-model AI comes in

A private deployment is rarely one model. EPC Group’s multi-model AI architecture puts every model behind one governed gateway, so each request goes to the model and location its data allows: open-weight models such as Llama or Mistral on infrastructure the client controls for regulated data, tenant-hosted models for general work, and Microsoft 365 Copilot where it already fits. Access control, logging and evaluation are shared across all of them, so a model can be swapped without rebuilding the application. The multi-model AI launch for Power BI shows the same pattern applied to analytics, and the engineering playbook covers the gateway layer.

How EPC Group delivers it

  1. Classify the data each AI workload touches and decide which of the three patterns each class allows.
  2. Pick the models per workload, including open-weight options for on-premises use.
  3. Stand up the gateway: identity with Microsoft Entra ID, logging, content safety and cost tracking in one place.
  4. Deploy the on-premises or on-device pieces and the tenant-hosted models, then evaluate quality on the client’s own test data.
  5. Hand over runbooks and the governance controls auditors will ask about.

Primary sources

Frequently Asked Questions

Can a private AI model run without sending data to the cloud?

Yes. EPC Group uses two Microsoft options for this. Foundry Local runs models on the user’s device, where data never leaves the device and no Azure subscription is required; Foundry Local on Azure Local runs models on an Arc-enabled Kubernetes cluster in your own datacenter and can operate in disconnected environments (Microsoft Learn, accessed 2026-10-27).

Are prompts sent to Azure-hosted models used to train them?

No, according to Microsoft. For models sold by Azure, including Azure OpenAI models, prompts, completions, embeddings and training data are not available to other customers or to the model providers, and are not used to train generative AI foundation models without your permission or instruction (Microsoft Learn, accessed 2026-10-27).

Which open-weight models can run on-premises?

Microsoft’s Foundry Local catalog covers chat models such as GPT OSS, Qwen, DeepSeek, Mistral and Phi, and Foundry Local on Azure Local can deploy models from the Foundry catalog or from your own registry (Microsoft Learn, accessed 2026-10-27). EPC Group chooses the model per workload rather than standardizing on one.

Is Foundry Local on Azure Local generally available?

Not yet. Microsoft lists Foundry Local on Azure Local as a preview, so features can change before general availability (Microsoft Learn, accessed 2026-10-27). EPC Group plans production designs around that status and keeps a tenant-hosted fallback for workloads that cannot wait.

How do we prove where prompts are processed?

Read the deployment type. GlobalStandard may process prompts in any Azure region; DataZoneStandard only inside the US, EU or Asia Pacific data zone; Standard inside the resource’s geography. Assign an Azure Policy that denies the deployment types your rules do not allow, so a new deployment cannot drift. Stored data, such as files and Responses API history, stays in the resource’s geography either way (Microsoft Learn, accessed 2026-11-02).

Why use more than one model for private AI?

Because no single model or location fits every workload. In EPC Group’s multi-model AI architecture, regulated data stays on models running on infrastructure the client controls, while general productivity work uses Microsoft 365 Copilot or tenant-hosted models, all behind one governed gateway with shared logging and access control.

Talk to a senior architect

A one-hour scoping call with a senior EPC Group architect covers which data must stay on your infrastructure, which models fit each workload and what the gateway needs. Email contact@epcgroup.net or call 888-381-9725.

Documented and anonymized engagements

Named engagements come first; anonymized summaries follow under the anonymized-engagement rule — the client is not named and every statement is EPC-reported. Documents behind named engagements are in the Evidence Center.

All case studies →

AI assistant — not a human