Is your question why the process is using roughly 11Gb of memory when you specified the max JVM memory to be 10GB or are you surprised that wasp used all the memory you allowed it to?
The first question is that the 10GB you indicated as the max is just the limit on the space that java does generic memory allocations out of. It is not a limit on the whole thing though all the rest of the memory that a Java VM is going to use is pretty static and so you'll see the -Xmx10240m exceeded but that will generally remain by a fixed amount. I wouldn't e surprised to see what you have here.
Regarding the second version of hte question, of the operations of taking memory, releasing memory, and determining what memory can be released, you get the most bang for the buck allocating free memory for a new demand. So, your typical JVM, if not otherwise affected by calls to the garbage collection routines, will fairly quickly allocate/use all the memory you allocate to it. It'll still be free memory within the JVM so it can be reused later but to the OS it will be allocated to the process. This is normal and it's done for performance reasons. The last thing you want to do is to run garbage collection to free up space for a new variable - better to ask the OS for a little bit more previously unused memory.
-Garin