In a .vmoptions file any filepath would be following java conventions, so I would try with either forward slashes or escaped backslashes.
Secondly, an agent is an argument to the JVM (not to the main class running in it) so it would be just -javaagent (not -Djavaagent)
And lastly the Oracle JVM docs specify that for -javaagent argument the filepath is added after a colon so it would be -javaagent: (not -javaagent=)
Have you tried?
-javaagent:C:/AppDynamics/agentDir/javaagent.jar
or
-javaagent:C:\\AppDynamics\\agentDir\\javaagent.jar
Hope this helps.
Cheers,
Danny