Offline Spark Messages are Not being received

Hi All,
Realized yesterday that when a user who is online sends a message to a user that is offline, that message is not being sent to them. It actually doesn’t appear at all on that end. Perhaps another ‘bug’?

Thanks again for all you do!

Offline messages are being handled by the server. I have tested offline messages a few days ago with the latest Spark version and it was able to receive them. Openfire was being used as a server, with a setting to store offline messages turned on.

I can confirm with Kassandra we upgraded Openfire to 4.5.3 this past week and all clients have been upgraded to 2.9.0 and offline messages are not being stored despite the setting set to Always Store.

Puzzling. I have just tested this on a new server and a new install of Spark, and for me, things work as expected.

@bruce2400 / @kassandra.h are you both working on the same server? Could you check if there is anything in the log files?

Same here. Works for me on Openfire 4.5.3 and Spark 2.9.0.

What happens if you go back to Spark 2.8.3?

It would be good to check both Openfire and Spark logs. Openfire logs can be accessed in the Admin Console or in installation folder (all.log).

Spark error logs:
C:\Users\User\AppData\Roaming\Spark\logs
There is a bunch of files, look through all of them and select events that correlate with the time of your issue.

Hello,

Yes Kassandra and I are using the same Spark Server.

Testing with Debugging on here is what we get

2020.08.21 15:48:51 org.apache.mina.filter.codec.ProtocolCodecFilter - Processing a MESSAGE_RECEIVED for session 827
2020.08.21 15:48:51 org.jivesoftware.openfire.spi.RoutingTableImpl - Failed to route packet to JID: bruce.h@Hidden.com/DK4-PC packet: RqaXlS
2020.08.21 15:48:51 org.jivesoftware.openfire.MessageRouter - Message sent to unreachable address: RqaXlS
2020.08.21 15:48:51 org.apache.mina.filter.ssl.SslFilter - Session Server744: Writing Message : WriteRequest: HeapBuffer[pos=0 lim=222 cap=4096: 3C 6D 65 73 73 61 67 65 20 74 6F 3D 22 62 72 75…]
2020.08.21 15:48:51 org.apache.mina.filter.executor.OrderedThreadPoolExecutor - Adding event MESSAGE_SENT to session 744
Queue : [MESSAGE_SENT, ]

:thinking:

Maybe this happens when Spark does not cleanly log out. In such cases, the server will need various minutes to figure out that Spark is gone. But in that case, the sender of the message should still see the intended recipient as ‘online’. Is that the case?

The messages that are lost, are they sent me just after some went offline? Does the problem also occur when a message is sent to a user that had been offline for a longer period (15 minutes or more)?

I have tested this with one Spark client being in a VM connected to Openfire on a host machine. I then disconnect network in that VM and for a few minutes Openfire still sees it as online (because it wasn’t a clean logout/exit process). If messages are sent during that time, they won’t be stored as offline and the other side (in VM) won’t receive them. I think i sometimes saw such issue on my last job a few years ago when we were still using Spark. Users would miss some messages when going with their laptops from room to room and losing network connection or switching from WiFi to a wired connected ans so on. Maybe a proper message delivery receipts function is the only thing that could help in this situation. If you can see an indication that your message hasn’t reached the other side, even if they show up as online.

We have a ticket for this https://issues.igniterealtime.org/browse/SPARK-1238 and Smack library already supports this. But we don’t have many developers working on Spark. Especially those who can create a GUI for such a function. Although it seems somewhat simple. Just adding some checkmark icon for every message?

So I set the time to 30s before disconnecting and disable stream managment. This help us not to lose messages (as much as possible)

Thank you for helping us. I just tested and remained offline for about 30 minutes and had someone spark me. It did not come through when I came back online. To answer you question when a user logs out, the user shows up in the offline group.

What’s interesting is the current size of all offline messages remained 0.00. We use the Monitoring Service plugin and it captures the messages so I know the server is getting them just seems the offline messages part is not getting the message.

Is the same happening with Spark 2.8.3 or some other client, like Gajim? It seems that in your case this is some issue with Openfire rather. Have you tried restarting the server? Maybe changing offline messages settings back and forth.

We had issues with 2.8.3 and certificates so we moved all clients to 2.9.0. Do you know where we can get a 2.8.3 installer?

I just sent a ton of offline messages to a user and caused the data usage to go up to 0.1MB. Looking in the database I show the messages, see screenshot.

2020-08-24_9-20-40

Thank you for the link. Uninstalled Spark 2.9.0 and installed 2.8.3 and it works, I am getting offline messages now.

This is strange. Above i have posted where to find Spark logs. Exit Spark 2.9.0, wipe all log files, then send a few offline messages to that user, run Spark for that user and login. If offline messages don’t show up. Check the logs and post them here (either content or just zip them and attach). Also, make sure that in Openfire database those offline messages are not in offline storage anymore.

Hello,

I do see one warning in the logs you mentioned to review. The warning I see is this

Aug 25, 2020 9:01:39 AM org.jivesoftware.spark.util.log.Log warning
WARNING: ChatRoomDecorator: finished bruce.o@.com

I can send you the entire log, would it be possible to send you it securely?

I should also mention that Bruce.o sent me a message offline, It did reach the server and left the server upon me logging into Spark.

The ChatRoomDecorator warning is probably not related. At least that’s what i thought when creating https://issues.igniterealtime.org/browse/SPARK-2034, because i see this warning all the time when everything works fine. Removing this warning would help reducing noise in the logs. Unless this is somehow related to your issue.

You can send me the logs via a private message (click on my avatar, then Message, then can attach a file to it). You can also replace domain and user names in the logs with some random values.

I have checked the logs from Bruce. No clues still. In error log there are usual Jingle and SIP plugins errors, because these plugins can’t load (there are tickets for them). I get these errors too.

Warn log contains also the usual error that OTR plugin is not compatible. I think this happened after downgrading Spark to 2.8.3 and then upgrading back to 2.9.0, so OTR plugin is still present in plugins folder in user profile. You can manually remove this plugin (it is not working anyway). C:\Users\user\AppData\Roaming\Spark\plugins

Then there are a few certificates warnings, which i also get and i thought it is because of me using a self-signed cert. But i see that they use a proper cert, so wonder why it still logs such warning “Certificate doesn’t have subjectUniqueID”. I don’t get such warning for igniterealtime.org certificate. Still, seems unrelated to this problem so far.

And then the last one is the “ChatRoomDecorator” warning, which i also get all the time when opening and closing chat windows.

Bruce, have you tried a completely fresh Spark profile? I wonder if some setting maybe causing this or lef overs from older versions. Doing a clean reinstall of Spark (making sure Spark folder is gone in Program files) is also useful.

I suppose this is on Windows 10?

I have tried, you are correct Windows 10 Pro. I actually found a use for Windows Sandbox and tried both Spark online and offline installation packages.

I see Spark 2.9.1 is out. I will try that in Sandbox and send you the logs from the Sandbox environment this time around.