Strange Wildfire server''s Linux OS behaviour

Openfire Openfire Support
21

So, i decided to install some plugins and it started to act slow again. Though it was ok, ssh at least. Say i press to install Registration or Search, it shows progress animation forever. At the same time i try to start mc via ssh. Shows nothing. If i break plugin installation, mc launches immediately. top is not working in any way, vmstat 1 is very slow and gives me that

$ vmstat 1

procs -

memory----

-swap -

io---- -system-- -

cpu----

r b swpd free buff cache si so bi bo in cs us sy id wa

0 1 0 3872 103964 27672 0 0 1 4 40 62 0 0 98 1

0 1 0 3872 103964 27672 0 0 0 0 1493 2078 1 0 0 99

0 1 0 3872 103964 27672 0 0 0 0 1233 796 0 0 0 100

0 1 0 3880 103964 27672 0 0 0 0 3108 6682 2 0 0 98

0 1 0 3880 103964 27672 0 0 0 0 1972 1976 0 0 0 99

0 1 0 3880 103964 27672 0 0 0 0 2536 7323 1 0 0 99

a lot of in, cs and wait. strange that swap is not in use after 4 days running. maybe some problem with Arch Linux indeed. I have setup the OS with default Auto-Partition option (swap, /home, /boot, / )

cfdisk 2.12r

Disk Drive: /dev/sda

Size: 40016019456 bytes, 40.0 GB

Heads: 255 Sectors per Track: 63 Cylinders: 4865

Name Flags Part Type FS Type Size (MB)

sda1 Boot, NC Primary Linux ext2 1052.84

sda2 Primary Linux swap / Solaris 1052.84

sda3 Primary Linux ext3 7871.60

sda4 Primary Linux ext3 30038.73

cant open Wildfire error.log

ok, will try to reboot the computer, doesnt respond to reboot command, so doing cold restart

interesting that Spark 2.5.0 is not reacting at first, only after a few seconds it shows reconnect window and when server is not loaded still it shows me the roster full of doubled offline users

error.log of Openfire:

2007.04.02 15:31:11 org.jivesoftware.util.log.util.CommonsLogFactory$1.error(CommonsLogFactory.java: 87) Line=19 The content of element type “dwr” must match “(init?,allow?,signatures?)”.

2007.04.05 18:04:56 org.jivesoftware.wildfire.stun.STUNService.startServer(STUNService.java:136) Disabling STUN server

java.net.BindException: Address already in use

at java.net.PlainDatagramSocketImpl.bind0(Native Method)

at java.net.PlainDatagramSocketImpl.bind(PlainDatagramSocketImpl.java:82)

at java.net.DatagramSocket.bind(DatagramSocket.java:368)

at java.net.DatagramSocket.(XMPPServer.java:148)

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessor Impl.java:39)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructor AccessorImpl.java:27)

at java.lang.reflect.Constructor.newInstance(Constructor.java:513)

at java.lang.Class.newInstance0(Class.java:355)

at java.lang.Class.newInstance(Class.java:308)

at org.jivesoftware.wildfire.starter.ServerStarter.start(ServerStarter.java:93)

at org.jivesoftware.wildfire.starter.ServerStarter.main(ServerStarter.java:49)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja va:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at com.exe4j.runtime.LauncherEngine.launch(Unknown Source)

at com.install4j.runtime.Launcher.main(Unknown Source

Have installed and uninstalled few plugins successfully

22

Hi,

a server which waits for IO can process nearly nothing, so I guess that it did write a lot of swap failures to /var/log/messages and did hang while trying to page out some unused pages.

r b swpd free buff cache si so bi bo in cs us sy id wa

0 1 0 3872 103964 27672 0 0 0 0 1233 796 0 0 0 100

Installing another kernel (probably an older one) should help a lot.

LG

23

it2000 wrote:

Installing another kernel (probably an older one) should help a lot.

or maybe newer? cause it’'s easier to get newer one with Arch Linux upgrade system. But it was updated a week ago. So this problem must be in a row of kernel upgrades in past. Thinking to post this on Arch Linux forums.

24

Hi,

a 2.4 kernel should work fine. If you update your kernel you should also update all your drivers (network, disc, …) as they may be causing the trouble.

LG

25

Well, i was using 2.6 from the begining, with my older PC, and the were no problems for a long time. I havent installed any specific drivers, nor compiled something in. I thought it should work fine without installing something additional. Only eth, ide, cdrom, video. Is it really could be caused by default drivers or so?

26

Hi Oleg,

as far as I can tell the “swapper: page allocation failure” was introduced with kernel 2.6.x, google returns some hits regarding this problem but solutions seem to be rare. I assume that you did never use all memory before you did install Wildfire so you did not hit this problem which should have existed for a long time on this server.

LG

27

Hm. Maybe there is something. My old server had only 64MB of RAM, so it was using swap always. And it was slow all the time, but it wasnt freezing like now. And i was using 2.6 kernel. But just before moving to a newer machine, there were so problems similar to these maybe (that’'s why i had to move). I thought that load on the server is just too big, so it was freezing more and more. But maybe it was the same problem with swap. Cant say now. That old server just refused to boot normally once (constant problems with main ext2 partition, so now i use ext3). So i had to move to other machine quickly, wipe the old one and put it away. Cant check logs, to find out whether there was same problems with swap.

I have posted a thread in Arch Kernel forum. Will see what arch devs and advanced user will suggest. If i find no solution, will try to move to 2.4 (or just reinstall it with 2.4).

28

What is in /proc/swaps ? Does it look correct?

Do you get any errors when you run “swapoff /dev/foo” (/dev/foo == your swap partition) then “swapon /dev/foo” ?

You might need to rerun “mkswap -c /dev/foo” is something went wrong in “formatting” the swap partition.

29

slushpupie wrote:

What is in /proc/swaps ? Does it look correct?

Filename Type Size Used Priority

/dev/sda2 partition 1028152 0 -1

dont know if this is correct, seems fine to me

Do you get any errors when you run “swapoff /dev/foo” (/dev/foo == your swap partition) then “swapon /dev/foo” ?

swapoff /dev/sda2

swapon /dev/sda2

no errors

You might need to rerun “mkswap -c /dev/foo” is something went wrong in “formatting” the swap partition.

mkswap -c /dev/sda2

Setting up swapspace version 1, size = 1052831 kB

no label, UUID=2b842cab-1c00-4d24-8ac8-16db0cc82d12

done, will see if that helps

/proc/swaps now is showing

Filename Type Size Used Priority

/dev/sda2 partition 1028152 0 -2

Used is still 0, but i had just rebooted the machine. Will see if that will change in the end of the day.

30

zgrep PREEMPT /proc/config.gz

  1. CONFIG_PREEMPT_NONE is not set

  2. CONFIG_PREEMPT_VOLUNTARY is not set

CONFIG_PREEMPT=y

CONFIG_PREEMPT_BKL=y

  1. CONFIG_DEBUG_PREEMPT is not set

31

Yeah, I would say the first thing to try is get a kernel without PREEMPT enabled. PREEMPT isnt really the best for servers anyway. If ARCH dosnt have a non-preempt kernel availible, you might want to compile your own. If you need help with that, I can help you.

32

So, i’‘m back again Well, i havent done much since then. I was restarting server every morning so it would run smoothly day long. But recently we’‘ve got back som old PC and i decided to test my installation woth it. It has only 256MB of RAM. This is the same installation of Arch linux (well, all soft parts and kernel is new ones). I have moved my Openfire to this machine and it works fine for 3 days without restarting. Swap is still not used, but there is no slowness. So i’'m thinking maybe there were some other problems with older machine (HDD maybe).

I’‘m going to swap these machines. But i’'m thinking, maybe i can stresstest this newer one. But i cant think of any test that would max RAM out and would need to use swap. Any ideas? I want to launch such test while Openfire running, so no Live testing distros.

33

Hi Oleg,

use -Xms200m or higher values when starting Openfire to check if swap gets used. As least a few bytes should be swapped out.

My server’'s top output looks like this:

Mem: 2043764k total, 1339780k used, 703984k free, 54992k buffers

Swap: 2097136k total, 132k used, 2097004k free, 372156k cached

so at least 132k swap space are allocated.

LG

34

mine is 0 k all the time.

maybe there is some other way to load RAM? some script? Dont want to mess with Openfire.

35

Hi Oleg,

within Openfire there’'s nothing which can be damaged if the server crashes. One could be afraid of the database but as this server is idle I guess that no one cares if you need to reinstall Openfire.

Writing a c program could be a solution to allocate a lot of memory but why should one make it more complicated than needed?

LG