thats just how java works if the java -xmx value is 1024 it should allocate 1gig of ram (for java to use), when all the ram gets used up it causes a out of memory error, usually un-checking vertex cache will fix the out of memory errors for this game. (but it does slow the game speed down somewhat), don't forget you have to restart the emulator each time you change these settings.
Try Out JPCSP Launcher v1.8.0.4 | How to post a log