Problem with cyrillic in messages

Openfire Openfire Support
1

Good afternoon!

While using Openfire we had some problems in messages with cyrillics. Some symbols (cyrillic) displayed incorrectly. It looks as one entered cyrillic symbol, after the sending, replaced with two rectangle symbols. It happens not more than for two symbols per message. Bad symbol each time different. And if you repeat your message - this symbol will display correct, but possible another symbol will play this joke again.

The same results for gentoo(Java Version: 1.6.0_03 Sun Microsystems Inc. – Java HotSpot™ Server VM) and for debian testing (1.6.0 Sun Microsystems Inc. – Java HotSpot™ Server VM)

2

We have tryed to use different client programs (psi, pidgin, miranda) on the different

platforms (Linux, Windows).

3

Hi,

this sounds a lot like JM-1175. Or did you already downgrade the MINA version?

LG

4

related:

5

this sounds a lot like

yes

Or did you already downgrade the MINA version?

no

svn co http://svn.igniterealtime.org/svn/repos/openfire/trunk openfire

copy mina files, rename:

mv mina-core-1.1.0.jar mina-core.jar; mv mina-filter-compression-1.1.0.jar mina-filter-compression.jar; mv mina-filter-ssl-1.1.0.jar mina-filter-ssl.jar

“Change dir to $ /build, and execute : ant openfire”

  1. ant openfire

Buildfile: build.xml

init:

compile:

Compiling 556 source files to /opt/svn/openfire/work/classes

/opt/svn/openfire/src/java/org/jivesoftware/openfire/sasl/SaslServerFactoryImpl .java:19: warning: com.sun.security.sasl.util.PolicyUtils is Sun proprietary API and may be removed in a future release

import com.sun.security.sasl.util.PolicyUtils;

^

/opt/svn/openfire/src/java/org/apache/mina/management/MINAStatCollector.java:30 9: cannot find symbol

symbol : method getEventQueueSize(org.apache.mina.common.IoSession)

location: class org.apache.mina.filter.executor.ExecutorFilter

tmpQueuevedEvents += executorFilter.getEventQueueSize(session);

^

/opt/svn/openfire/src/java/org/apache/mina/management/MINAStatCollector.java:30 9: inconvertible types

found : <nulltype>

required: long

tmpQueuevedEvents += executorFilter.getEventQueueSize(session);

^

/opt/svn/openfire/src/java/org/jivesoftware/openfire/sasl/SaslServerFactoryImpl .java:30: warning: com.sun.security.sasl.util.PolicyUtils is Sun proprietary API and may be removed in a future release

private static final int mechPolicies = { PolicyUtils.NOANONYMOUS };

^

/opt/svn/openfire/src/java/org/jivesoftware/openfire/sasl/SaslServerFactoryImpl .java:49: warning: com.sun.security.sasl.util.PolicyUtils is Sun proprietary API and may be removed in a future release

if (mechanism.equals(myMechs[PLAIN]) && PolicyUtils.checkPolicy(mechPolicies[PLAIN], props)) {

^

/opt/svn/openfire/src/java/org/jivesoftware/openfire/sasl/SaslServerFactoryImpl .java:65: warning: com.sun.security.sasl.util.PolicyUtils is Sun proprietary API and may be removed in a future release

return PolicyUtils.filterMechs(myMechs, mechPolicies, props);

^

Note: Some input files use or override a deprecated API.

Note: Recompile with -Xlint:deprecation for details.

Note: Some input files use unchecked or unsafe operations.

Note: Recompile with -Xlint:unchecked for details.

2 errors

4 warnings

BUILD FAILED

/opt/svn/openfire/build/build.xml:278: Compile failed; see the compiler error output for details.

Total time: 8 seconds

6

done

can not be compilled bacause files, placed in /org/apache/mina/filter/executor/* (mina-core-1.1.*)

=)

Can we hope for you fix this bug in mina?

7

Hey vovans,

I would like to confirm that this is a MINA problem so I can report it to MINA developers. If you no longer see this problem when using an old version of MINA then let us know and we will talk with MINA devs.

Thanks,

– Gato

8

If you no longer see this problem when using an old version of MINA then let us know and we will talk with MINA devs.

Yes, the problem has disappeared (fixed) (mina 1.1.0)

thanks

9

Confirm.

Built openfire with mina 1.1.0 and everything is clear (tested with 2 clients only sending each other large messages in russian).

Built with mina 1.1.1 and got this bug again.

10

Hello everyone!

I’ve enabled additional debugging in read method of XMLLightWeightParser:

/*

  • Main reading method

*/

public void read(ByteBuffer byteBuffer) throws Exception {

invalidateBuffer();

// Check that the buffer is not bigger than 1 Megabyte. For security reasons

// we will abort parsing when 1 Mega of queued chars was found.

if (buffer.length() > 1048576) {

throw new Exception(“Stopped parsing never ending stanza”);

}

Log.debug("XMLLightWeightParser: " + byteBuffer.toString());

CharBuffer charBuffer = encoder.decode(byteBuffer.buf());

Log.debug("XMLLightWeightParser: " + charBuffer.toString());

char[] buf = charBuffer.array();

int readByte = charBuffer.remaining();

to see what happens.

So, Openfire reads messages by parts. Cyrillic letters are double-byte. If first byte at the end of one part and second byte at the beginning of next one, each byte decodes in CharBuffer charBuffer = encoder.decode(byteBuffer.buf()) separetaly and replaces with Unicode replacement symbol 0xFFFD. Users see two 0xFFFD (square or question mark) symbols instead of one cyrillic (and other double-byte i think) letter.

It’s happens with mina 1.1.0 in my case too, but more rare. It’s uncommon for secure connections (i could reproduce this behaviour once or twice in all my tests).

I can send my debug logs, if you need them.

11

Hey Kinyapin,

Excellent analysis. I can now totally see why this problem is happening. I’m now trying to figure out the correct way to fix it but the problem has been identified now. Just as you mentioned this problem happens when each byte of a double-byte char arrives in 2 different iterations but what we are storing in XMLLightWeightParser is a corrupted/incomplete conversion of the byte into a character.

Thanks a lot for this information and the analysis. I will try to find a fix for this tonight or tomorrow.

Regards,

– Gato

12

Hey Kinyapin,

I’m no longer able to reproduce it locally so I was not able to test the fix. If you or anyone else is willing to test it I can send you a modified version of Openfire 3.4.4 or 3.4.5 with the patch so you can try it out.

Thanks,

– Gato

13

Hi Gato!

Yes, i can test it this weekend and send results on monday or tuesday.

Thank you for quick fix.

14

Hey Kinyapin,

Drop me an email with the Openfire version that you are using. Tomorrow morning (PST) I will send you back a modified version that you can try.

Thanks,

– Gato

15

Hey Gato,

I’m also interested in that patch. We’re still using 3.4.2 though. Could you send me the patch itself (instead of a completely modified server)? We already applied other (unrelated) changes that I’d rather not override. Thanks!

16

Guus so you will not upgrate your openfire? It’s not good…

if your changes are interesting for others send your patchs to developers of thr openfire

else make them as plugins

17

Thanks Gato, I’ve applied the patch to our source. I’ll try to deploy it to the one machine that gives us the best chances of reproducing the problem.

Vanaf, I’ve actually sent in patches for a lot of the non-plugin modifications that I have. I believe serveral of them are scheduled for the upcoming 3.5.0 release. We’re currently running four domains with Openfire, three of them in a clustered setup. We have been integrating a lot of custom systems (through 20+ plugins, client connections and external component implementations. Each time we update Openfire, we need to do regression tests on all of them, to check for compatibility issues. This entire process (which has to be repeated in several test environments) just eats up to much of our time to do it with ever minor Openfire release - that’s why I’m still running 3.4.2