Sanity check on a competing on-prem proposal vs. a cloud based solution
I am currently want to build my own automation business for German SMEs. I am talking to a a mid-sized manufacturer and he shared a proposal from a consulting / software consultancy firm with me.
Use cases: Standard SME processes:
Several document-processing workflows (incoming docs → OCR/VLM → ERP match → auto-process or route to human).
Plus a RAG layer over internal technical content: sales gets questions like "does article X meet specification Y," and the answer usually sits somewhere in old technical datasheets, internal wikis, or previous customer correspondence.
Proposed architecture: Fully on-prem: workstation GPU server, local open-source LLM, the consulting firm builds and operates their own custom RAG system on it, wrapped in their proprietary orchestration platform (user management, monitoring, prompt management). Mid-five-figures upfront, low-five-figures recurring annually for platform license, a per-user fees and a maintenance.
My instinct: cloud is the better fit here. Frontier model via EU-region cloud with DPA, n8n self-hosted for orchestration, Qdrant or pgvector for the vector store. Open-source RAG stack instead of proprietary. Fraction of the cost, frontier models instead of quantized local ones, no platform lock-in.
Genuinely want input on:
- Is on-prem actually warranted for a non-regulated SME? EU-region cloud with DPA covers GDPR. CLOUD Act risk is theoretical for ordinary business data. What am I missing?
- Custom proprietary RAG vs. open-source RAG. They build a bespoke system you can't see inside and pay for forever. Open-source equivalents exist for every component. Is there a real engineering reason to prefer the proprietary path, or is it pure lock-in?
- The 5-year question. Fixed on-prem hardware locks the company to today's capability. Cloud keeps improving in the background. Is this as big a deal as I think for a normal SME?
- Honest counter-argument. If you've shipped on-prem RAG in production at non-regulated SMEs, what's the case for it that I'm underweighting?
I am trying to be fair to both architectures and trying to understand what is the argument for a local hosted setup vs a cloud based setup? The proposal reads to me like it is optimized for the consulting firm recurring revenue...