Think Scale of 0 applies the fastest possible response time for all of the operations contained in the virtual service. (i.e., it is global)
However, virtual services will always be constrained by the amount of time it takes the VSM to execute the LISTEN step through RESPONDER step.
The OOTB VSM steps are oriented for high performance (for both functional or performance licenses), but I would never expect a virtual service to provide an absolute 0 ms response time. The types of additions to a VSM that slows the virtual service down include things such as DPHs processing large request payloads, Xpaths, Assertions, Filters, Custom scripts (JSR / Assertions), additional steps in the model such as DB I/O, Live Invocation response times, Match scripts in the VSI, etc. Each of these add to the time it takes the VSM to process the request and send a response.