Smack-omemo rework #177 - Newly installed omemoDevice failed to setup omemo messaging chat with existing contacts

Smack Smack Dev
1

After testing with all implementation variants on my Note-8 with swordfish@atalk.org account, I am experiencing wiered device behavior:
a. omemo table is not in a proper state. e.g. the identity table contains the identityPrekey instead of identityPrekeyPairs
b. sending of omemo messaging is not working.
c At one instance, omemoManager throws “user not login” in actual fact the init was only call after callback from smack upon authentication. This was observed when account only successfully login on second attempt after first attempt failed; In impelementation, I block fetching new instance of omemoManger if not null. I suspect the problem arises due to aTalk smack and omemoManger is referring to a different instance of connection. I decided to revert the change to allow fetching new instance of omemoManger whenever a connection is connected (before authentication).

Sine unable to continue testing with a broken database, I decided to conduct the test with everything fresh by doing the followings.
a. Clear all the aTalk apk data on android Note-8
b. Clean up all the published items on my ejabberd server for user swordfish@atalk.org

Following are my observation with this new setup and confirmed with repeated tests.

  1. When I use the following method to get an instance of omemoManager upon connection.
    OmemoManager omemoManager = OmemoManager.getInstanceFor(mConnection);

I found that all my omemo database is populated with own OmemoDevice as swordfish@atalk.org:-1 and all other tables i.e.
prekeys, signedPrekey etc are filled with deviceId -1 in the respective deviceId column.
If I try to setup a omemo chat from my Note-3 (installed with aTalk v1.0.3 i.e. old omemo library), it was prompted to confirm the fingerPrint for swordfish@atalk.org:-1.

  1. When I use following method to get an instance of omemoManager upon connection i.e. with own local generated randomID e.g. 903218477
    defaultDeviceId = OmemoManager.randomDeviceId();
    ((SQLiteOmemoStore) mOmemoStore).setDefaultDeviceId(userJid, defaultDeviceId);
    OmemoManager omemoManager = OmemoManager.getInstanceFor(mConnection, defaultDeviceId);

As expected, the omemo database are now correctly filled with swordfish@atalk.org:903218477
and also Note-3 is prompted to confirm the fingerPrint for the correct omemoDevice swordfish@atalk.org:903218477.

However whenever I try to send omemo messages between Note-3 to Note-8, sent messages cannot received by either party.
And both throws error as follow: I restart both devices, one after the other repeated many times, but nothing help.

===== Note-3 =======

For the problem exhibit by Note-3, I believe it was the problem of PEP Manager failing to publish the deviceID on server as highlighted in my previous comments.

===== Note-8 =======

In my previous omemo implementation I actually have a private method i.e. private PEPListener buddyDeviceListUpdateListener(). You may refer to the source in my github repository. that listens for the PEP event deviceList broadcast from server, and re-create buddy identities when found missing. This is to resolve problem as exhibited by Note-8; it uses method OmemoService.getInstance().buildSessionFromOmemoBundle(mOmemoManager, omemoDevice, false) but has been removed from the latest OmemoService. I try the new method OmemoManager.getInstanceFor(mConnection).rebuildSessionWith(contactDevice) but does not seem to work. It actually corrupted the atalk omemo database for own IdentityKeyPairs and overwritten with identityKey containing only public key.

OmemoManger has a similar implementation i.e. private final PEPListener deviceListUpdateListener, but it takes care only for own DeviceIDs.

Any help is appreciated. Now I am stuck unable to carry out further test since Note-8 omemo data is not complete.

=========== On Note-3 on receive omemo message from Note-8 =============== 
01-20 16:19:30.371 D/SMACK: RECV (0): <message xml:lang='en' to='leopard@atalk.org/atalk' from='swordfish@atalk.org/atalk' type='chat' id='pCVp7-194'><encrypted xmlns='eu.siacs.conversations.axolotl'><header sid='903218477'><iv>iv5UvM3c1EkIKZuIp8jwlA==</iv></header><payload>1LaNeQI=</payload></encrypted><store xmlns='urn:xmpp:hints'/><encryption xmlns='urn:xmpp:eme:0' namespace='eu.siacs.conversations.axolotl' name='OMEMO'/><body>I sent you an OMEMO encrypted message but your client doesn&apos;t seem to support that. Find more information on https://conversations.im/omemo</body><thread>7745e59b-2a65-4c5e-8e08-fc38d47f42f6</thread></message>
01-20 16:19:30.371 I/aTalk: [18] org.jivesoftware.smackx.omemo.OmemoService.processReceivingMessage() There is no key with our deviceId. Silently discard the message.

=========== On Note-8 on receive omemo message from Note-3 ===============
01-20 16:18:10.081 D/SMACK: RECV (1): <message xml:lang='en' to='swordfish@atalk.org/atalk' from='leopard@atalk.org/atalk' type='chat' id='Tp0VL-947'><encrypted xmlns='eu.siacs.conversations.axolotl'><header sid='19359559'><key prekey='true' rid='903218477'>MwhiEiEFSyp2PitMJy0LXYIs+f0VHSfJC4ibPDm4pb4slxZXjF8aIQW2pyW5zLtfPT2nE2NqkvTRLDhuTjDEJwm9xcYf9wX2CiJiMwohBR1/taRvpJF0yVFd8BO2xWs2L1zKcVqKeWOEk3lLHnQyEAQYACIw9ymHWbdBW2qRK8w7gW00jrB3rPLgyHATx1VtUMtiXVGTGiCppn5UkRtSFjTeB5nGHjdRqPySxfMoADAB</key><key prekey='true' rid='10139552'>Mwg0EiEFbQ6dFIYDbf3+lxOZh1aVSHcc42epqcNrrEyTDM46/2EaIQW2pyW5zLtfPT2nE2NqkvTRLDhuTjDEJwm9xcYf9wX2CiJiMwohBRQoOLufXZZKFTdnmilBiCH+pCUkqf0rJQJ1r54Ar9cJECIYACIwIbV8dlzy9A7bIqPDdRGNmXRheukaMdcPO3ya30Wj2c7DIh7NBOByoR5AMSdJ6OdRiguEtn9VeDgoADAB</key><iv>G14sa3XUOYoqIuaW00mngQ==</iv></header><payload>QOee</payload></encrypted><store xmlns='urn:xmpp:hints'/><encryption xmlns='urn:xmpp:eme:0' namespace='eu.siacs.conversations.axolotl' name='OMEMO'/><body>I sent you an OMEMO encrypted message but your client doesn’t seem to support that. Find more information on https://conversations.im/omemo</body><thread>dd74fb02-6831-4e34-964e-43e3898bc7c9</thread></message>

01-20 16:18:10.159 E/aTalk: [15225] util.UtilActivator.uncaughtException().95 An uncaught exception occurred in thread=Thread[Thread-69,5,main] and message was: Cannot retrieve OmemoFingerprint of sender although decryption was successful: org.jivesoftware.smackx.omemo.exceptions.NoIdentityKeyException
                            java.lang.AssertionError: Cannot retrieve OmemoFingerprint of sender although decryption was successful: org.jivesoftware.smackx.omemo.exceptions.NoIdentityKeyException
                                at org.jivesoftware.smackx.omemo.OmemoService.decryptMessage(OmemoService.java:453)
                                at org.jivesoftware.smackx.omemo.OmemoService.onOmemoMessageStanzaReceived(OmemoService.java:1169)
                                at org.jivesoftware.smackx.omemo.OmemoManager$3$1.run(OmemoManager.java:984)
                                at java.lang.Thread.run(Thread.java:762)
2

The method rebuildSessionWith() is actually working. Corruption of the swordfish identityKeyPair is actually caused by aTalk method storeOmemoIdentityKey(); which mistakenly uses userDevice instead of contactDevice in storeOmemoIdentityKey() method.

After the correction in storeOmemoIdentityKey(), the last reported problems experienced by Note-3 and Note-8 is resolved, and Note-3 and Note-8 are now able to exchange omemo messages without problem.

The tests also demonstrated that the implementation of
PEPListener buddyDeviceListUpdateListener()
in aTalk is a must to take care and reconstruct the missing buddies identityKeys. Otherwise the previous reported problem will remain unresolved and omemo messaging will not work in aTalk.

I have included below aTalk implementation for your reference; for your consideration to include in your omemoManager
private final PEPListener deviceListUpdateListener
if you deem necessary:

    /**
     * PEPListener to add new buddy omemoDevice identityKey to identities table
     * This is necessary otherwise newly installed client will always fail to communicate with
     * buddies whom have been setup earlier.
     */

    private PEPListener buddyDeviceListUpdateListener = new PEPListener()
    {
        @Override
        public void eventReceived(EntityBareJid from, EventElement event,
                org.jivesoftware.smack.packet.Message message)
        {
            // Our deviceList, so nothing more to do as it is handled in OmemoManager
            if ((from == null) || from.equals(mOmemoManager.getOwnJid())) {
                return;
            }

            for (ExtensionElement items : event.getExtensions()) {
                if (!(items instanceof ItemsExtension)) {
                    continue;
                }

                final OmemoDevice userDevice = mOmemoManager.getOwnDevice();
                for (ExtensionElement item : ((ItemsExtension) items).getItems()) {
                    if (!(item instanceof PayloadItem<?>)) {
                        continue;
                    }

                    PayloadItem<?> payloadItem = (PayloadItem<?>) item;
                    if (!(payloadItem.getPayload() instanceof OmemoDeviceListElement)) {
                        continue;
                    }

                    // Get the published Device List
                    OmemoDeviceListElement receivedDeviceList = (OmemoDeviceListElement) payloadItem.getPayload();
                    Set<Integer> deviceList = receivedDeviceList.getDeviceIds();
                    for (int deviceID : deviceList) {
                        final OmemoDevice contactDevice = new OmemoDevice(from.asBareJid(), deviceID);
                        IdentityKey preKey = null;
                        try {
                            preKey = (IdentityKey) mOmemoStore.loadOmemoIdentityKey(userDevice, contactDevice);
                        } catch (CorruptedOmemoKeyException e) {
                            preKey = null;
                        }
                        // Get the preKeys for missing buddy deviceID
                        if (preKey == null) {
                            aTalkApp.showToastMessage("Rebuild new session for missing: " + contactDevice);
                            try {
                                OmemoManager.getInstanceFor(mConnection).rebuildSessionWith(contactDevice);
                            } catch (InterruptedException | SmackException.NotConnectedException
                                    | SmackException.NoResponseException | CorruptedOmemoKeyException
                                    | CannotEstablishOmemoSessionException
                                    | SmackException.NotLoggedInException e) {
                                logger.error("Could not rebuild session with: " + contactDevice + " Exception: ", e);
                            }
                        }
                    }
                }
            }
        }
    };
3

I think the above observed problems are actually due to the way aTalk is handling and storing of omemo identities data in SQL database.

In early aTalk implementation for the override method i.e
public void storeCachedDeviceList(OmemoDevice userDevice, BareJid contact, OmemoCachedDeviceList contactDeviceList)

It takes care only to update the active state for the omemoDevice if the device has a row entry in the table; Otherwise for those that are missing it performs
OmemoManager#rebuildSessionWith(contactDevice);

After I done a more careful review on the way FileBaseOmemoStore implementation, I change the aTalk public void storeCachedDeviceList() implementation to create a new but partially filled row entry for any missing omemoDevice. With this new changes, I also disabled the PEP listenerbuddyDeviceListUpdateListener().

With all the old data cleanup on both the devices (Note-3 and Note-8) and server.
I have done some initial testing again with this new implementation, I observe that omemo messaging is now working without any of the previous mentioned problems.

Please comment and confirm the new method is the correct way how it should be handled for identifies data.

4

I have tested multiple times with various scenarios, and confirmed the above new implementation for aTalk is working well without any of the earlier described problem.

The issue can be considered close. Sorry for any inconvenient cause.