are you seeing this problem immediately after a restart or is it possible that you have a memory problem with OutOfMemory errors in your log files? If the admin page (/index.jsp) shows 90% used you should increase the values as describe in the installation guide.
did you set Xms also to 256m? If you did please set it to 48m, that’‘s a much better initial value. So you shouldn’'t see 180 MB used memory, but maybe 70 MB if all users are connected.