Skip to main content

Private AI means the organization decides where prompts, data and models live. Microsoft offers three patterns: models sold by Azure inside your own tenant and geography, Foundry Local on Azure Local for open-weight models on on-premises Kubernetes (preview, can run disconnected), and Foundry Local for on-device inference with no Azure subscription. A multi-model architecture uses each where it fits.

Key Facts

  • For models sold by Azure, prompts and completions are not available to other customers or model providers and are not used to train foundation models without permission (Microsoft Learn, accessed 2026-10-27).
  • Foundry Local on Azure Local runs AI models on an Arc-enabled Kubernetes cluster on-premises and can operate in disconnected environments; it is in preview (Microsoft Learn, accessed 2026-10-27).
  • Foundry Local runs models on the device: data never leaves it, the app works offline, and no Azure subscription is required (Microsoft Learn, accessed 2026-10-27).
  • Microsoft sells models from providers including Cohere, DeepSeek, Meta, Mistral AI and xAI through Azure alongside Azure OpenAI models (Microsoft Learn, accessed 2026-10-27).

Last updated by Errin O'Connor, Founder & Chief AI Architect, EPC Group

The direct answer

A private AI solution keeps prompts, business data and, where required, the model itself inside boundaries the organization controls. With Microsoft technology there are three ways to do that, and most enterprises need more than one: regulated records may have to stay on servers in their own datacenter, while everyday drafting and search can run on models hosted in their own Azure tenant.

Three deployment patterns

PatternWhere the model runsFitsStatus (per Microsoft Learn)
Tenant-hostedModels sold by Azure, including Azure OpenAI and open-weight families, deployed in your Foundry resource in a geography you chooseMost enterprise workloads that need scale without running GPUsAvailable now; data at rest stays in the chosen Azure geography, but Global deployments may process prompts in any Azure region and Data Zone deployments only within the US, EU or APAC data zone; Standard and Regional Provisioned keep processing inside the geography (Microsoft Learn, Deployment types in Microsoft Foundry Models, accessed 2026-10-28)
On-premisesFoundry Local on Azure Local: an Arc-enabled Kubernetes cluster in your datacenter, CPU or GPU, models from the Foundry catalog or your own registrySovereignty or regulatory rules, low latency next to the data, disconnected sitesPreview
On-deviceFoundry Local inside the application on the user’s Windows, macOS or Linux deviceOffline work and sensitive data that should never leave the laptopSingle-user inference; not designed as a server stack

Where multi-model AI comes in

A private deployment is rarely one model. EPC Group’s multi-model AI architecture puts every model behind one governed gateway, so each request goes to the model and location its data allows: open-weight models such as Llama or Mistral on infrastructure the client controls for regulated data, tenant-hosted models for general work, and Microsoft 365 Copilot where it already fits. Access control, logging and evaluation are shared across all of them, so a model can be swapped without rebuilding the application. The multi-model AI launch for Power BI shows the same pattern applied to analytics, and the engineering playbook covers the gateway layer.

How EPC Group delivers it

  1. Classify the data each AI workload touches and decide which of the three patterns each class allows.
  2. Pick the models per workload, including open-weight options for on-premises use.
  3. Stand up the gateway: identity with Microsoft Entra ID, logging, content safety and cost tracking in one place.
  4. Deploy the on-premises or on-device pieces and the tenant-hosted models, then evaluate quality on the client’s own test data.
  5. Hand over runbooks and the governance controls auditors will ask about.

Primary sources

Frequently Asked Questions

Can a private AI model run without sending data to the cloud?

Yes. EPC Group uses two Microsoft options for this. Foundry Local runs models on the user’s device, where data never leaves the device and no Azure subscription is required; Foundry Local on Azure Local runs models on an Arc-enabled Kubernetes cluster in your own datacenter and can operate in disconnected environments (Microsoft Learn, accessed 2026-10-27).

Are prompts sent to Azure-hosted models used to train them?

No, according to Microsoft. For models sold by Azure, including Azure OpenAI models, prompts, completions, embeddings and training data are not available to other customers or to the model providers, and are not used to train generative AI foundation models without your permission or instruction (Microsoft Learn, accessed 2026-10-27).

Which open-weight models can run on-premises?

Microsoft’s Foundry Local catalog covers chat models such as GPT OSS, Qwen, DeepSeek, Mistral and Phi, and Foundry Local on Azure Local can deploy models from the Foundry catalog or from your own registry (Microsoft Learn, accessed 2026-10-27). EPC Group chooses the model per workload rather than standardizing on one.

Is Foundry Local on Azure Local generally available?

Not yet. Microsoft lists Foundry Local on Azure Local as a preview, so features can change before general availability (Microsoft Learn, accessed 2026-10-27). EPC Group plans production designs around that status and keeps a tenant-hosted fallback for workloads that cannot wait.

Why use more than one model for private AI?

Because no single model or location fits every workload. In EPC Group’s multi-model AI architecture, regulated data stays on models running on infrastructure the client controls, while general productivity work uses Microsoft 365 Copilot or tenant-hosted models, all behind one governed gateway with shared logging and access control.

Talk to a senior architect

A one-hour scoping call with a senior EPC Group architect covers which data must stay on your infrastructure, which models fit each workload and what the gateway needs. Email contact@epcgroup.net or call 888-381-9725.

Documented and anonymized engagements

Named engagements come first; anonymized summaries follow under the anonymized-engagement rule — the client is not named and every statement is EPC-reported. Documents behind named engagements are in the Evidence Center.

All case studies →

AI assistant — not a human