Smack 4.4.0: Omemo does not update the identity info of a contact after fetching new data from server

During the testing the fix for identitykey index=0, I have observed the following on (Note-3 with omemoDevice user2@myserver.ltd:148608352)
Below is a description of the observation, I am not sure if the problem arise is due to aTalk DB design; or there is additional process aTalk needs to perform when below scenario happen.

While attempt to send an omemo message to contacts, omemoService will make call to the method below to get the fingerPrint->loadOmemoIdentityKey for each contact in the recipient list:

OmemoStore.getFingerprint(OmemoStore.java:609) →
loadOmemoIdentityKey(userDevice, contactsDevice)

(no sure why the contact identity table row has both the publickey and fingerPrint contain null)
a. Upon receive a null identityKey in reply from a specific contact, omemoStore throws an NoIdentityKeyException;
b. omemoService then add this contact into the undecidedDevices list and
c. return the undecidedDevices to the send message caller.

I see that omemoServie also proceed to fetch the prekeys bundle from the server for the contact with null identityKey. However this newly fetched contact publicKey info is not being updated to the database.

The user is unable to send the message; as above cycle repeats itself whenever user is trying to send another new omemo message.

Need your clarification on the followings:

  1. After fetching the new identityKey bundle, should omemoServer makes a call to
    OmemoStore.storeOmemoIdentityKey(OmemoDevice userDevice, OmemoDevice contactDevice, IdentityKey contactKey)
  2. Otherwise, is that something aTalk needs to do to get omemStore or aTalk to save the new contact prekey info into the database.
08-26 08:09:36.805 1870-1870/org.atalk.android W/aTalk: [1] org.jivesoftware.smackx.omemo.OmemoService.getUndecidedDevices() Could not load fingerprint of user1@myserver.ltd:1367773246
    org.jivesoftware.smackx.omemo.exceptions.NoIdentityKeyException
        at org.jivesoftware.smackx.omemo.OmemoStore.getFingerprint(OmemoStore.java:609)
        at org.jivesoftware.smackx.omemo.OmemoService.getUndecidedDevices(OmemoService.java:846)
        at org.jivesoftware.smackx.omemo.OmemoService.encrypt(OmemoService.java:361)
        at org.jivesoftware.smackx.omemo.OmemoService.createOmemoMessage(OmemoService.java:537)
        at org.jivesoftware.smackx.omemo.OmemoManager.encrypt(OmemoManager.java:342)
        at org.jivesoftware.smackx.omemo.OmemoManager.encrypt(OmemoManager.java:315)
        at net.java.sip.communicator.impl.protocol.jabber.OperationSetBasicInstantMessagingJabberImpl.sendInstantMessage(OperationSetBasicInstantMessagingJabberImpl.java:506)
        at org.atalk.android.gui.chat.MetaContactChatTransport.sendInstantMessage(MetaContactChatTransport.java:413)
        at org.atalk.android.gui.chat.ChatController.sendMessage(ChatController.java:294)
        at org.atalk.android.gui.chat.ChatController.onClick(ChatController.java:419)

08-26 08:09:36.945 1870-2103/org.atalk.android D/SMACK: SENT (0): 
    <iq to='user1@myserver.ltd' id='5PUFZ-119' type='get'>
      <pubsub xmlns='http://jabber.org/protocol/pubsub'>
        <items node='eu.siacs.conversations.axolotl.bundles:1367773246'/>
      </pubsub>
    </iq>
08-26 08:09:37.425 1870-2104/org.atalk.android D/SMACK: RECV (0): 
    <iq xml:lang='en' to='user2@myserver.ltd/atalk' from='user1@myserver.ltd' type='result' id='5PUFZ-119'>
      <pubsub xmlns='http://jabber.org/protocol/pubsub'>
        <items node='eu.siacs.conversations.axolotl.bundles:1367773246'>
          <item id='61EBCDC16D901'>
            <bundle xmlns='eu.siacs.conversations.axolotl'>
              <signedPreKeyPublic signedPreKeyId='1'>
                BcmSAz/M3BSBlz5crR4WYrHId1azx4GBxhiGbCM7OAg9
              </signedPreKeyPublic>
              <signedPreKeySignature>
                m890vfBB2be9ygtHPdmPOqi7de4zX4WOcHJaiSp7SPjxs78SZOs7XrK0qIyqwVmTxJe6qvK9l9927auWmQd4AQ==
              </signedPreKeySignature>
              <identityKey>
                Bb7i7uw0ZJwf7KHMcLt70R8TIvIoU/b+oTI+khZKh1hR
              </identityKey>

=========================
Further testing show that if the problem on Note-3 is not resolved, send a message from my Note-8 (user1@myserver.ltd:1367773246) is also not working. Although the message is sent successfully by Note-8, but Note-3 cannot repair the broken session, and hence no display of the received omemo message.

  1. Any proposal for the fix of all these observed problem?

Note: your proposed fix for index=0 is verified working.

<message xml:lang='en' to='user2@myserver.ltd/atalk' from='user1@myserver.ltd/atalk' type='chat' id='1566786562185188302834'>
      <archived by='user2@myserver.ltd' id='1566786563118350' xmlns='urn:xmpp:mam:tmp'/>
      <stanza-id by='user2@myserver.ltd' id='1566786563118350' xmlns='urn:xmpp:sid:0'/>
      <encrypted xmlns='eu.siacs.conversations.axolotl'>
        <header sid='1367773246'>
          <key prekey='true' rid='1481970607'>
            MwgyEiEF0KVrvBOEY8yw3Sv+FIRn9asw+yNufAbjL1NqdOS8i2EaIQW+4u7sNGScH+yhzHC7e9EfEyLyKFP2/qEyPpIWSodYUSJiMwohBRh37lvGXmGrno8QiwaeWpGSlkEnWtgtGF4JVw52xfxeEAEYACIwiaaqPsEGizcBVUhqL5B+woRP8hlzk1ENsEGsGE79QFEI8MxNO1WU8UK0AxErAXLJpylMyDwjzqMoADAB
          </key>
          <key prekey='true' rid='944885071'>
            MwgiEiEFRWf7vpABbGsE2hZd3/NqZTJ5SxFiUVNKOkujY2HtIjUaIQXtZA8Hhk9pABEUK+8UsUT/3wXNKAT7qsXHqt64u/ofQCJiMwohBfZkzhjVqy8GqjBOMw19bZTCiNKPL28NkgmPlSl9b9V3EAgYACIwjaDCl5QBwW1vnfCjBeB51MyU7w3tuzNFyWmwKeKPfgW44n0m0jmX8wiVx52yjlfZk9tRGt34fSooADAB
          </key>
          <key prekey='true' rid='1361'>
            MwhfEiEFKfnfAs92mpQHHlocyoYWkF912ecTo9W6Xnf8psQx5wUaIQW+4u7sNGScH+yhzHC7e9EfEyLyKFP2/qEyPpIWSodYUSJiMwohBeAXAakfcGY9d6HTdBS5VldTdZy9Gzr4ka1Frargs8tpEAEYACIwYUuxj17Ued/S1fYpTTfd85KSGB5B3ug6Ht2gauyeqgaJsk79CWYHLo10rprzUcSk0jZ9g2Ad3HwoADAA
          </key>
          <key rid='148608352'>
            MwohBSSY7Diw0Ep2x6VEbuJthbhG0H+iza
08-26 10:29:24.736 9133-14056/org.atalk.android D/SMACK: RECV (1): omk0xUGbUkzQdNEAMYACIweJGyQaPxTs0sDJ7CG2kB29jg5Wzas/Rono9u9QcOvJtlyv9xTvTADhy9fg8n8C1t47QiaZqGtxM=
          </key>
          <key prekey='true' rid='1389276080'>
            Mwg7EiEF3bnGwa6e160S4rv3k8ro8DOXP2+s30/GxpELQulF6E4aIQXtZA8Hhk9pABEUK+8UsUT/3wXNKAT7qsXHqt64u/ofQCJiMwohBYOr7MXV4tN5fuf6TubfbWKkMb38fL0H6CLC+KE+fCpkEAgYACIwPL29/OmGqOCI2PWROcLvm49Bu35/ZzskzTi1+b+xDJE/+MZgvNALYRT56A1Qels5LtN8Imb1hbIoADAB
          </key>
          <iv>
            KB1Stdi9fe6YhngRCb0rqg==
          </iv>
        </header>
        <payload>
          /qQ7
        </payload>
      </encrypted>
      <store xmlns='urn:xmpp:hints'/>
      <encryption xmlns='urn:xmpp:eme:0' namespace='eu.siacs.conversations.axolotl' name='OMEMO'/>
      <request xmlns='urn:xmpp:receipts'/>
      <body>
        I sent you an OMEMO encrypted message but your client doesn&apos;t seem to support that. Find more information on https://conversations.im/omemo
      </body>
    </message>
08-26 10:29:24.746 9133-14055/org.atalk.android D/SMACK: SENT (1): 
    <message to='user1@myserver.ltd/atalk' id='HCH53-341' type='chat'>
      <received xmlns='urn:xmpp:receipts' id='1566786562185188302834'/>
    </message>
    <r xmlns='urn:xmpp:sm:3'/>
08-26 10:29:24.746 9133-14056/org.atalk.android D/SMACK: RECV (1): 
    <r xmlns='urn:xmpp:sm:3'/>
08-26 10:29:24.746 9133-14055/org.atalk.android D/SMACK: SENT (1): 
    <a xmlns='urn:xmpp:sm:3' h='88'/>
08-26 10:29:24.756 9133-19383/org.atalk.android W/aTalk: [6] org.jivesoftware.smackx.omemo.OmemoService.onOmemoMessageStanzaReceived() No raw session found for contact user1@myserver.ltd:1367773246. 
    org.jivesoftware.smackx.omemo.exceptions.NoRawSessionException: org.whispersystems.libsignal.NoSessionException: No session for: user1@myserver.ltd:1367773246
        at org.jivesoftware.smackx.omemo.signal.SignalOmemoRatchet.doubleRatchetDecrypt(SignalOmemoRatchet.java:127)
        at org.jivesoftware.smackx.omemo.OmemoRatchet.retrieveMessageKeyAndAuthTag(OmemoRatchet.java:103)
        at org.jivesoftware.smackx.omemo.OmemoService.decryptMessage(OmemoService.java:456)
        at org.jivesoftware.smackx.omemo.OmemoService.onOmemoMessageStanzaReceived(OmemoService.java:1159)
        at org.jivesoftware.smackx.omemo.OmemoManager$3$1.run(OmemoManager.java:957)
        at java.lang.Thread.run(Thread.java:818)
     Caused by: org.whispersystems.libsignal.NoSessionException: No session for: user1@myserver.ltd:1367773246
        at org.whispersystems.libsignal.SessionCipher.decrypt(SessionCipher.java:239)
        at org.whispersystems.libsignal.SessionCipher.decrypt(SessionCipher.java:211)
        at org.jivesoftware.smackx.omemo.signal.SignalOmemoRatchet.doubleRatchetDecrypt(SignalOmemoRatchet.java:121)
        at org.jivesoftware.smackx.omemo.OmemoRatchet.retrieveMessageKeyAndAuthTag(OmemoRatchet.java:103) 
        at org.jivesoftware.smackx.omemo.OmemoService.decryptMessage(OmemoService.java:456) 
        at org.jivesoftware.smackx.omemo.OmemoService.onOmemoMessageStanzaReceived(OmemoService.java:1159) 
        at org.jivesoftware.smackx.omemo.OmemoManager$3$1.run(OmemoManager.java:957) 
        at java.lang.Thread.run(Thread.java:818) 

08-26 10:29:24.756 9133-19383/org.atalk.android W/aTalk: [6] org.jivesoftware.smackx.omemo.OmemoService.repairBrokenSessionWithPreKeyMessage() Attempt to repair the session by sending a fresh preKey message to user1@myserver.ltd:1367773246
08-26 10:29:24.766 9133-14055/org.atalk.android D/SMACK: SENT (1): 
    <iq to='user1@myserver.ltd' id='HCH53-342' type='get'>
      <pubsub xmlns='http://jabber.org/protocol/pubsub'>
        <items node='eu.siacs.conversations.axolotl.bundles:1367773246'/>
      </pubsub>
    </iq>
08-26 10:29:25.136 9133-14056/org.atalk.android D/SMACK: RECV (1): 
    <a h='57' xmlns='urn:xmpp:sm:3'/>
08-26 10:29:25.666 9133-14056/org.atalk.android D/SMACK: RECV (1): 
    <iq xml:lang='en' to='user2@myserver.ltd/atalk' from='user1@myserver.ltd' type='result' id='HCH53-342'>
      <pubsub xmlns='http://jabber.org/protocol/pubsub'>
        <items node='eu.siacs.conversations.axolotl.bundles:1367773246'>
          <item id='61EBFE649DC37'>
            <bundle xmlns='eu.siacs.conversations.axolotl'>
              <signedPreKeyPublic signedPreKeyId='1'>
                BcmSAz/M3BSBlz5crR4WYrHId1azx4GBxhiGbCM7OAg9
              </signedPreKeyPublic>
              <signedPreKeySignature>
                m890vfBB2be9ygtHPdmPOqi7de4zX4WOcHJaiSp7SPjxs78SZOs7XrK0qIyqwVmTxJe6qvK9l9927auWmQd4AQ==
              </signedPreKeySignature>
              <identityKey>
                Bb7i7uw0ZJwf7KHMcLt70R8TIvIoU/b+oTI+khZKh1hR
              </identityKey>
              <prekeys>
                <preKeyPublic preKeyId='1'>
                  BaM6jPM2U6f5ciIT+YoixsxYSC1Z7qdoq82k47QxCSlL
                </preKeyPublic>
                <preKeyPublic preKeyId='2'>
                  Bd9rLgwqeYbvSxr45+sZbU4TNIXyrqN2hT35ukMl6HFo
                </preKeyPublic>
                <preKeyPublic preKeyId='3'>
                  Bf15eqgMWDG4l/6l1Rv/WWfdZBTyMnlArpVcSlo70NUO
                </preKeyPublic>

08-26 10:29:25.736 9133-19383/org.atalk.android W/aTalk: [6] org.jivesoftware.smackx.omemo.OmemoService.repairBrokenSessionWithPreKeyMessage() Unable to repair session with user1@myserver.ltd:1367773246
    org.jivesoftware.smackx.omemo.exceptions.CorruptedOmemoKeyException: org.whispersystems.libsignal.InvalidKeyException: Invalid signature on device key!
        at org.jivesoftware.smackx.omemo.signal.SignalOmemoService.processBundle(SignalOmemoService.java:107)
        at org.jivesoftware.smackx.omemo.signal.SignalOmemoService.processBundle(SignalOmemoService.java:48)
        at org.jivesoftware.smackx.omemo.OmemoService.buildFreshSessionWithDevice(OmemoService.java:786)
        at org.jivesoftware.smackx.omemo.OmemoService.repairBrokenSessionWithPreKeyMessage(OmemoService.java:1294)
        at org.jivesoftware.smackx.omemo.OmemoService.onOmemoMessageStanzaReceived(OmemoService.java:1180)
        at org.jivesoftware.smackx.omemo.OmemoManager$3$1.run(OmemoManager.java:957)
        at java.lang.Thread.run(Thread.java:818)
     Caused by: org.whispersystems.libsignal.InvalidKeyException: Invalid signature on device key!
        at org.whispersystems.libsignal.SessionBuilder.process(SessionBuilder.java:176)
        at org.jivesoftware.smackx.omemo.signal.SignalOmemoService.processBundle(SignalOmemoService.java:104)
        at org.jivesoftware.smackx.omemo.signal.SignalOmemoService.processBundle(SignalOmemoService.java:48) 
        at org.jivesoftware.smackx.omemo.OmemoService.buildFreshSessionWithDevice(OmemoService.java:786) 
        at org.jivesoftware.smackx.omemo.OmemoService.repairBrokenSessionWithPreKeyMessage(OmemoService.java:1294) 
        at org.jivesoftware.smackx.omemo.OmemoService.onOmemoMessageStanzaReceived(OmemoService.java:1180) 
        at org.jivesoftware.smackx.omemo.OmemoManager$3$1.run(OmemoManager.java:957) 
        at java.lang.Thread.run(Thread.java:818)

=====================

With more testing and walking through smack omemo library source, I see that omemo does in fact trying to save the identity and established session into the database in the method below:

When omemo detects that there is no valid pre-established omemo session, it proceeds to perform fetchBundel(), follow by build session with the fetched bundle.

SessionBuilder.process(PreKeyBundle preKey)

However it throws InvalidKeyException while verifying the prekey from the fetch bundle

      if (preKey.getSignedPreKey() != null &&
          !Curve.verifySignature(preKey.getIdentityKey().getPublicKey(),
                                 preKey.getSignedPreKey().serialize(),
                                 preKey.getSignedPreKeySignature()))
      {
        throw new InvalidKeyException("Invalid signature on device key!");
      }

Previously I have already encountered this problem. This seems to be the fourth time while testing omemo chat; although the fetchBundle() returns seems normal as attached below.

Assume the following:
a. user1 omemoDeviceA prekey bundle is corrupted
b. user2 omemoDeviceB prekey is OK.

When this happen, the affected omemoDeviceA must “Regenerate OMEMO identities” in order to get out of this locked state.

The problem is that only the sender omemoDeviceB sees the problem as CorruptedOmemoKeyException while sending omemo message to user2, but the recipient omemoDeviceA is not aware of the problem.

The omemoDeviceA is able to send omemo chat without problem; however there is no omemo message being displayed on recipient omemoDeviceB chat window; and there is no error throws to app from omemo to inform user2:

Currently aTalk offers an option to purge the corrupted identity omomeDeviceA in database, so user2 can proceed to send omemo message to other contacts.

When I tested omemoDeviceA with conversations, conversations is able to communicate with user1 without any problem. How is conversatons handle the corrupted prekey bundle?

Any comment?

=================== debug log: user2 on receive message from user1 ============
08-28 11:49:43.517 6849-9115/org.atalk.android W/aTalk: [9] org.jivesoftware.smackx.omemo.OmemoService.repairBrokenSessionWithPreKeyMessage() Unable to repair session with user1@myserver.ltd:1367773246
    org.jivesoftware.smackx.omemo.exceptions.CorruptedOmemoKeyException: org.whispersystems.libsignal.InvalidKeyException: Invalid signature on device key!
        at org.jivesoftware.smackx.omemo.signal.SignalOmemoService.processBundle(SignalOmemoService.java:107)
        at org.jivesoftware.smackx.omemo.signal.SignalOmemoService.processBundle(SignalOmemoService.java:48)
        at org.jivesoftware.smackx.omemo.OmemoService.buildFreshSessionWithDevice(OmemoService.java:795)
        at org.jivesoftware.smackx.omemo.OmemoService.repairBrokenSessionWithPreKeyMessage(OmemoService.java:1303)
        at org.jivesoftware.smackx.omemo.OmemoService.onOmemoMessageStanzaReceived(OmemoService.java:1189)
        at org.jivesoftware.smackx.omemo.OmemoManager$3$1.run(OmemoManager.java:957)
        at java.lang.Thread.run(Thread.java:818)
     Caused by: org.whispersystems.libsignal.InvalidKeyException: Invalid signature on device key!
        at org.whispersystems.libsignal.SessionBuilder.process(SessionBuilder.java:176)
        at org.jivesoftware.smackx.omemo.signal.SignalOmemoService.processBundle(SignalOmemoService.java:104)
        at org.jivesoftware.smackx.omemo.signal.SignalOmemoService.processBundle(SignalOmemoService.java:48) 
        at org.jivesoftware.smackx.omemo.OmemoService.buildFreshSessionWithDevice(OmemoService.java:795) 
        at org.jivesoftware.smackx.omemo.OmemoService.repairBrokenSessionWithPreKeyMessage(OmemoService.java:1303) 
        at org.jivesoftware.smackx.omemo.OmemoService.onOmemoMessageStanzaReceived(OmemoService.java:1189) 
        at org.jivesoftware.smackx.omemo.OmemoManager$3$1.run(OmemoManager.java:957) 
        at java.lang.Thread.run(Thread.java:818)
=========== aTalk debug log = session build failed ============
08-28 10:30:31.597 23103-23103/org.atalk.android W/aTalk: [2] org.jivesoftware.smackx.omemo.OmemoService.encrypt() Could not build session with user1@myserver.ltd:1367773246.
    org.jivesoftware.smackx.omemo.exceptions.CorruptedOmemoKeyException: org.whispersystems.libsignal.InvalidKeyException: Invalid signature on device key!
        at org.jivesoftware.smackx.omemo.signal.SignalOmemoService.processBundle(SignalOmemoService.java:107)
        at org.jivesoftware.smackx.omemo.signal.SignalOmemoService.processBundle(SignalOmemoService.java:48)
        at org.jivesoftware.smackx.omemo.OmemoService.buildFreshSessionWithDevice(OmemoService.java:795)
        at org.jivesoftware.smackx.omemo.OmemoService.encrypt(OmemoService.java:386)
        at org.jivesoftware.smackx.omemo.OmemoService.createOmemoMessage(OmemoService.java:541)
        at org.jivesoftware.smackx.omemo.OmemoManager.encrypt(OmemoManager.java:342)
        at org.jivesoftware.smackx.omemo.OmemoManager.encrypt(OmemoManager.java:315)
        at net.java.sip.communicator.impl.protocol.jabber.OperationSetBasicInstantMessagingJabberImpl.sendInstantMessage(OperationSetBasicInstantMessagingJabberImpl.java:505)
        at org.atalk.android.gui.chat.MetaContactChatTransport.sendInstantMessage(MetaContactChatTransport.java:413)
        at org.atalk.android.gui.chat.ChatController.sendMessage(ChatController.java:294)
        at org.atalk.android.gui.chat.ChatController.onClick(ChatController.java:419)
        at android.view.View.performClick(View.java:5181)
        at android.view.View$PerformClick.run(View.java:20887)
        at android.os.Handler.handleCallback(Handler.java:739)
        at android.os.Handler.dispatchMessage(Handler.java:95)
        at android.os.Looper.loop(Looper.java:145)
        at android.app.ActivityThread.main(ActivityThread.java:5938)
        at java.lang.reflect.Method.invoke(Native Method)
        at java.lang.reflect.Method.invoke(Method.java:372)
        at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1389)
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1184)
     Caused by: org.whispersystems.libsignal.InvalidKeyException: Invalid signature on device key!
        at org.whispersystems.libsignal.SessionBuilder.process(SessionBuilder.java:176)
        at org.jivesoftware.smackx.omemo.signal.SignalOmemoService.processBundle(SignalOmemoService.java:104)
        at org.jivesoftware.smackx.omemo.signal.SignalOmemoService.processBundle(SignalOmemoService.java:48) 
        at org.jivesoftware.smackx.omemo.OmemoService.buildFreshSessionWithDevice(OmemoService.java:795) 
        at org.jivesoftware.smackx.omemo.OmemoService.encrypt(OmemoService.java:386) 
        at org.jivesoftware.smackx.omemo.OmemoService.createOmemoMessage(OmemoService.java:541) 
        at org.jivesoftware.smackx.omemo.OmemoManager.encrypt(OmemoManager.java:342) 
        at org.jivesoftware.smackx.omemo.OmemoManager.encrypt(OmemoManager.java:315) 
        at net.java.sip.communicator.impl.protocol.jabber.OperationSetBasicInstantMessagingJabberImpl.sendInstantMessage(OperationSetBasicInstantMessagingJabberImpl.java:505) 
        at org.atalk.android.gui.chat.MetaContactChatTransport.sendInstantMessage(MetaContactChatTransport.java:413) 
        at org.atalk.android.gui.chat.ChatController.sendMessage(ChatController.java:294) 
        at org.atalk.android.gui.chat.ChatController.onClick(ChatController.java:419) 
        at android.view.View.performClick(View.java:5181) 
        at android.view.View$PerformClick.run(View.java:20887) 
        at android.os.Handler.handleCallback(Handler.java:739) 
        at android.os.Handler.dispatchMessage(Handler.java:95) 
        at android.os.Looper.loop(Looper.java:145) 
        at android.app.ActivityThread.main(ActivityThread.java:5938) 
        at java.lang.reflect.Method.invoke(Native Method) 
        at java.lang.reflect.Method.invoke(Method.java:372) 
        at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1389) 
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1184)
============ Bundle fetching (InvalidKeyException) =========
08-28 10:30:27.357 23103-23773/org.atalk.android D/SMACK: SENT (0): 
    <iq to='user1@myserver.ltd' id='V5N2E-97' type='get'>
      <query xmlns='http://jabber.org/protocol/disco#info' node='eu.siacs.conversations.axolotl.bundles:1367773246'>
      </query>
    </iq>

08-28 10:30:30.377 23103-23774/org.atalk.android D/SMACK: RECV (0): 
    <iq xml:lang='en' to='user2@myserver.ltd' from='user1@myserver.ltd' type='result' id='V5N2E-101'>
      <pubsub xmlns='http://jabber.org/protocol/pubsub'>
        <items node='eu.siacs.conversations.axolotl.bundles:1367773246'>
          <item id='61ED3BFDBB514'>
            <bundle xmlns='eu.siacs.conversations.axolotl'>
              <signedPreKeyPublic signedPreKeyId='1'>
                BcmSAz/M3BSBlz5crR4WYrHId1azx4GBxhiGbCM7OAg9
              </signedPreKeyPublic>
              <signedPreKeySignature>
                m890vfBB2be9ygtHPdmPOqi7de4zX4WOcHJaiSp7SPjxs78SZOs7XrK0qIyqwVmTxJe6qvK9l9927auWmQd4AQ==
              </signedPreKeySignature>
              <identityKey>
                Bb7i7uw0ZJwf7KHMcLt70R8TIvIoU/b+oTI+khZKh1hR
              </identityKey>
              <prekeys>
                <preKeyPublic preKeyId='1'>
                  BaM6jPM2U6f5ciIT+YoixsxYSC1Z7qdoq82k47QxCSlL
                </preKeyPublic>

If I made the process(PreKeyBundle preKey) in aTalk ignores the return result of Curve.verifySignature() so as to allow process() to continue till the end i.e. to allow results to be stored to the database. After that everything is OK, user2 is now able to send omemo messages to user1 and vice versa.

I am not sure if conversations actually call up this part of code; the session is already established when the user2 account is added before I can set the break-point.

=================== New finding on conversations ============
I modified conversations source to create a condition so it always perform process(PreKeyBundle preKey) on user2. It too throws InvalidKeyException. So below explanations are possible:

a. The keyBundle is really corrupted either during the stage of creation, sending or retrieving from server.
b. But strange was that when the user2 contact is first added to conversations, it gives no error nor complain. The sending and receiving of omemo message is OK.
c. Curve.verifySignature passing parameters or the verification process has problem?

Have to rule out c, as checking on two other omemoDevices, both do not throw InvalidKeyException.

===========================

  identityKeyStore.saveIdentity(remoteAddress, preKey.getIdentityKey());
  sessionStore.storeSession(remoteAddress, sessionRecord);
============ ignore result returns from Curve.verifySignature() ==========
  public void process(PreKeyBundle preKey) throws InvalidKeyException, UntrustedIdentityException {
...
      if (preKey.getSignedPreKey() != null &&
          !Curve.verifySignature(preKey.getIdentityKey().getPublicKey(),
                                 preKey.getSignedPreKey().serialize(),
                                 preKey.getSignedPreKeySignature()))
      {
        // throw new InvalidKeyException("Invalid signature on device key!");
      }
...
}

As a side discussion, I see that OmemoService.buildFreshSessionWithDevice() converts
HashMap<Integer, T_Bundle> bundlesList to an array before get(randomIndex) for the PreKeyBundle.

T_Bundle randomPreKeyBundle2 = new ArrayList<>(bundlesList.values()).get(randomIndex);

The order of elements in HashMap are not guaranteed to be the same order as when the elements are added. The converted arrayList elements are ordered as per HashMap.

The below statement will return prekeyBundle where prekeyId == randomIndex; but no when it is first converted to an arrayList.

T_Bundle randomPreKeyBundle = bundlesList.get(randomIndex);

In the current process() context, the randomIndex is not used in PrekeyBundle for process later, hence it really does not give rise to any problem.

On further checking and has confirmed that 5 other omemoDevices omemokey bundles, the SessionBuilder.process(PreKeyBundle preKey) do not throw InvalidKeyException.
I have positively came to the conclusion that the problem is due to corruption of the omemo key bundles for user1:omemoDeviceA. Since it failed during Curve.verifySignature(),
One or more of the parameters in key bundle is/are corrupted i.e.

            <bundle xmlns='eu.siacs.conversations.axolotl'>
              <signedPreKeyPublic signedPreKeyId='1'>
                BcmSAz/M3BSBlz5crR4WYrHId1azx4GBxhiGbCM7OAg9
              </signedPreKeyPublic>
              <signedPreKeySignature>
                m890vfBB2be9ygtHPdmPOqi7de4zX4WOcHJaiSp7SPjxs78SZOs7XrK0qIyqwVmTxJe6qvK9l9927auWmQd4AQ==
              </signedPreKeySignature>
              <identityKey>
                Bb7i7uw0ZJwf7KHMcLt70R8TIvIoU/b+oTI+khZKh1hR
              </identityKey>
public void process(PreKeyBundle preKey) throws InvalidKeyException, UntrustedIdentityException {
...
      if (preKey.getSignedPreKey() != null &&
          !Curve.verifySignature(preKey.getIdentityKey().getPublicKey(),
                                 preKey.getSignedPreKey().serialize(),
                                 preKey.getSignedPreKeySignature()))
      {
        throw new InvalidKeyException("Invalid signature on device key!");
      }
....
}

However I am unable to determine in which stage is the cause of the problem:
i.e. either during the stage of prekey creation, uploading or retrieving from server.
Being unable to proceed further, just has to monitor the frequency of occurrences.

Currently the only way to get out for this tight loop, is for the user of the corrupted omemoDevice to perform the “Regenerate OMEMO identities”. However the user is unaware, until he is being told by his contact trying to send omemo messages to him.

Any recommendation?

Are the bundles that fail to verify generated by smack-omemo?

Yes, the one having problem is generated by aTalk (smack-omemo).

However all the others being verified Ok are generated by converse.js, conversations and one also by aTalk.

By the way, whenever I restart aTalk, I saw that aTalk always sent its key bundle to the server.
Are these key bundle generated new or retrieved from database. I have restart the problem device a couple of times, but the corrupted key still persist.

Just encountered another case of prekey bundle corruption. Following gives an account what happen that lead to this observations:

Just got a new Samsung Note-10. So I started using this new phone for testing:
Following are the various OmemoDevice installed on the various Samsung phones
a. Note-10 (swordfish@atalk.org:1175061843)
b. Note-8 (swordfish@atalk.org:1669278993)
c. Note-3 (leopard@atalk.org:339857885)

After installed aTalk on new Note-10 phone with (swordfish@atalk.org:1175061843), I start aTalk testing between Note-3 and Note-10.
a. Sending OMEMO message from Note-3 to Note-10.
b. Authenticated all the omemoDevices on Note-3 and resend message
c. Omemo message sent OK on Note-3, but not received on Note-10.
d. Send OMEMO message from Note-10 to Note-3.
e. Authentication pop up, showing (swordfish@atalk.org:1669278993) corrupted.
f. Authenticate leopard and purge corrupted (swordfish@atalk.org:1669278993) so message can be sent to Note-3.
g. Omemo message received on Note-3 OK
h. Send Omemo message from Note-3 to Note-10
i. Note-10 received the Omemo message OK.

Not sure why Note-8 omemo prekey is corrupted; as no other action was performed to the prekey.

2019-09-10 11:28:14.372 18127-19326/org.atalk.android D/SMACK: RECV (0): 
    <message xml:lang='en' to='swordfish@atalk.org/atalk' from='leopard@atalk.org/atalk' type='chat' id='1568086099153428086332'>
      <encrypted xmlns='eu.siacs.conversations.axolotl'>
        <header sid='339857885'>
          <key rid='1669278993'>
            MwohBauhAEb5xwDSJ+YC8WszA6e3KOv7f1dh/V1ZRCyFiOg7EAUYCSIwGT1tQVtMDn7kD90Rcfy7MVDFuv36Z3+Y5Pbs0AhLy3+mjsa84I47P65M8LyEZHxx/p3YkOg/aAk=
          </key>
          <key prekey='true' rid='482420002'>
            MwgbEiEFXK8uTlx/T7luKuk5NzDB9599Io0azXyO587dsrCESnoaIQUu97k4QxRbdYrc7ik4vSCTMPVPvoX2VHC5lrL+aymYLCJiMwohBf5qKYUkz1mkEMB7lxgXdX9E7qHqbJaxtUi5iBY0D7F/EAUYACIw8PI8oIwmNAOIbHMTkWx3H5DAXxJNEMu6skunYVOLNaO+2QvbgNovMI/hJaoxVxobdjhlOXH/zMQoADAB
          </key>
          <key rid='1175061843'>
            MwohBatRwzTKUlp1MRUlpPBA9BXEVS+LSKIF75xXdre6WAwqEAMYACIwvMidjCnopi0fJVURnz9xUAKqvArCuSn8s5rPNjs9qmcw4qSgddM8pEsC27zsxZem72UB6egTSFk=
          </key>
          <iv>
            V2eaWyvuHtRCg4HytLtOzA==
          </iv>
        </header>
        <payload>
          98a57vwf8+wteYh1pXRqtcPLBfJu8uiLY/mpBbTSHJFIDUUKnaPjDc01r5ddP/2Tq973Fw91zx+qDunyasRtVCHMBccJM8UHCPBD9S1atvcYKy5aAL/sRXo7fECRq2YRcClIBnaJFQ1SFmHbMy9Y8XDnC5AJZw2/rQ4OA0x25LCkgekEPJW9Km6oPp0dTeYlLuOZjSsbA95APuxFZStmtAdQtXG2MukxXqeJYvpx7pKwinIHxP5yodrLNCSzhJtnywdgvsHafgfF9QnVKid0PqrTyasfcbMoYZnel4eCePYlfgfpIxDhZhF7nfBBLDHYUGgRZl2QIqFUBRfNroM4AWfIuKxFMOy4ESs=
        </payload>
      </encrypted>
      <store xmlns='urn:xmpp:hints'/>
      <encryption xmlns='urn:xmpp:eme:0' namespace='eu.siacs.conversations.axolotl' name='OMEMO'/>
      <request xmlns='urn:xmpp:receipts'/>
      <body>
        I sent you an OMEMO encrypted message but your client doesn&apos;t seem to support that. Find more information on https://conversations.im/omemo
      </body>
    </message>

2019-09-10 11:28:14.471 18127-19325/org.atalk.android D/SMACK: SENT (0): 
    <iq id='HHJN9-421' type='get'>
      <pubsub xmlns='http://jabber.org/protocol/pubsub'>
        <items node='eu.siacs.conversations.axolotl.bundles:1669278993'/>
      </pubsub>
    </iq>
    <a xmlns='urn:xmpp:sm:3' h='87'/>
2019-09-10 11:28:14.505 18127-19326/org.atalk.android D/SMACK: RECV (0): 
    <iq xml:lang='en' to='swordfish@atalk.org/atalk' from='swordfish@atalk.org' type='result' id='HHJN9-421'>
      <pubsub xmlns='http://jabber.org/protocol/pubsub'>
        <items node='eu.siacs.conversations.axolotl.bundles:1669278993'>
          <item id='6201353762244'>
            <bundle xmlns='eu.siacs.conversations.axolotl'>
              <signedPreKeyPublic signedPreKeyId='3'>
                Bb2HRxPxwW7daU1+MIzzZrbQjpFTJr4btxWpjgPOT+Ek
              </signedPreKeyPublic>

2019-09-10 11:28:14.535 18127-18127/org.atalk.android W/aTalk: [2] org.jivesoftware.smackx.omemo.OmemoService.buildMissingSessionsWithDevices() swordfish@atalk.org:1175061843 could not establish session with swordfish@atalk.org:1669278993because their bundle seems to be corrupt.
    org.jivesoftware.smackx.omemo.exceptions.CorruptedOmemoKeyException: org.whispersystems.libsignal.InvalidKeyException: Invalid signature on device key!
        at org.jivesoftware.smackx.omemo.signal.SignalOmemoService.processBundle(SignalOmemoService.java:107)
        at org.jivesoftware.smackx.omemo.signal.SignalOmemoService.processBundle(SignalOmemoService.java:48)
        at org.jivesoftware.smackx.omemo.OmemoService.buildFreshSessionWithDevice(OmemoService.java:786)
        at org.jivesoftware.smackx.omemo.OmemoService.buildMissingSessionsWithDevices(OmemoService.java:815)
        at org.jivesoftware.smackx.omemo.OmemoService.encrypt(OmemoService.java:359)
        at org.jivesoftware.smackx.omemo.OmemoService.createOmemoMessage(OmemoService.java:537)
        at org.jivesoftware.smackx.omemo.OmemoManager.encrypt(OmemoManager.java:342)
        at org.jivesoftware.smackx.omemo.OmemoManager.encrypt(OmemoManager.java:315)
        at org.atalk.crypto.CryptoFragment.doHandleOmemoPressed(CryptoFragment.java:332)

2019-09-10 11:28:14.545 18127-18127/org.atalk.android W/(CryptoFragment.java:335)#doHandleOmemoPressed: There are undecided Omemo devices: [swordfish@atalk.org:1669278993]
2019-09-10 11:28:14.573 6066-20682/? D/GameManagerService: handleForegroundChange(). pkgName: org.atalk.android, clsName: org.atalk.crypto.omemo.OmemoAuthenticateDialog,FgActivityName:org.atalk.android/org.atalk.crypto.omemo.OmemoAuthenticateDialog,userID:0
2019-09-10 11:28:14.574 6066-21192/? D/MARsPolicyManager: onPackageResumedFG pkgName = org.atalk.android, userId = 0
2019-09-10 11:28:14.575 6066-6066/? D/Telecom:SamsungPrebindingServiceImpl: onTopActivityChanged : ComponentInfo{org.atalk.android/org.atalk.crypto.omemo.OmemoAuthenticateDialog}
2019-09-10 11:28:14.640 18127-19325/org.atalk.android D/SMACK: SENT (0): 
    <iq id='HHJN9-423' type='get'>
      <pubsub xmlns='http://jabber.org/protocol/pubsub'>
        <items node='eu.siacs.conversations.axolotl.bundles:1669278993'/>
      </pubsub>
    </iq>
2019-09-10 11:28:14.666 18127-19326/org.atalk.android D/SMACK: RECV (0): 
    <iq xml:lang='en' to='swordfish@atalk.org/atalk' from='swordfish@atalk.org' type='result' id='HHJN9-423'>
      <pubsub xmlns='http://jabber.org/protocol/pubsub'>
        <items node='eu.siacs.conversations.axolotl.bundles:1669278993'>
          <item id='6201353762244'>
            <bundle xmlns='eu.siacs.conversations.axolotl'>
              <signedPreKeyPublic signedPreKeyId='3'>
                Bb2HRxPxwW7daU1+MIzzZrbQjpFTJr4btxWpjgPOT+Ek
              </signedPreKeyPublic>
              <signedPreKeySignature>
                U2ZJQUNLXcJiYrbK836p6OpqZ/Jeuzibpz//7E0vMKpsnHaHDjedV0lSFjdlRP2MKEt+DBBlS2/aAFQ+1R32jQ==
              </signedPreKeySignature>

2019-09-10 11:28:33.193 18127-18127/org.atalk.android D/(DatabaseBackend.java:1333)#purgeOmemoDb: >>> Wiping OMEMO database for device : swordfish@atalk.org:1669278993
2019-09-10 11:28:33.202 6066-6066/? D/Telecom:SamsungPrebindingServiceImpl: onTopActivityChanged : ComponentInfo{org.atalk.android/org.atalk.android.gui.chat.ChatActivity}
2019-09-10 11:28:33.203 18127-19325/org.atalk.android D/SMACK: SENT (0): 
    <iq to='leopard@atalk.org' id='HHJN9-427' type='get'>
      <pubsub xmlns='http://jabber.org/protocol/pubsub'>
        <items node='eu.siacs.conversations.axolotl.devicelist'/>
      </pubsub>
    </iq>
2019-09-10 11:28:33.222 18127-19326/org.atalk.android D/SMACK: RECV (0): 
    <iq xml:lang='en' to='swordfish@atalk.org/atalk' from='leopard@atalk.org' type='result' id='HHJN9-427'>
      <pubsub xmlns='http://jabber.org/protocol/pubsub'>
        <items node='eu.siacs.conversations.axolotl.devicelist'>
          <item id='61CF081FB5DAD'>
            <list xmlns='eu.siacs.conversations.axolotl'>
              <device id='339857885'/>
            </list>
          </item>
        </items>
      </pubsub>
    </iq>

This topic was automatically closed 100 days after the last reply. New replies are no longer allowed.