Smack 4.4.0-beta2: OMEMO continues using old OmemoDevice Id even after the regeneration process with new DeviceID specified

Following up from the test observation described in:

Also with reference to:

I performed an Omemo regeneration on Device B. aTalk Omemo regeneration source is given below:
The latest aTalk regeneration routines have been updated to fix problem as described in
XEP-0060: Publish-Subscribe: Delete node and purge node operations in ejabberd #3418

Also it now includes action to purge all the old omemo device identities, in attempt to resolve the reported problem as given in the second reference herein (see above).

However as seen in the attached aTalk debug log, smack Omemo, when publishing prekeys to the server, it continues to use the old deviceID i.e. 889121650 instead of the new deviceID i.e.

Omemo device added for: parrot@atalk.sytes.net; 116536437.

It also pushes all the old purged parrot Omemo device ID’s to the server, the new parrot device ID is not being updated to the server.

When I restart Device A (Note-10), it retrieves all the old parrot OmemoDevice Id’s from the server. The list contain no new parrot device Id.

2020-11-06 10:10:15.219 30495-26191/org.atalk.android D/(DatabaseBackend.java:1012)#storeCachedDeviceList: Identities table - updating for activeDevice: parrot@atalk.sytes.net:[1566361092, 522622186, 889121650]

I have verified that aTalk SQL database the omemo device B has already with this new ID 116536437 and all the old 889121650 data has already been purged from the database.

It seems smack OMEMO still contains a cached of the old Omemo Device ID; and aTalk regeneration source still has error.

Appreciate if smack team can advice, what else aTalk needs to do, in order to get smack Omemo to use the newly specified Omemo DeviceId.

=============== aTalk OMEMO Regeration Routines ===============
    /**
     * Clean up omemo database and omemo bundle data on server for user account when deleted.
     *
     * @param accountId the omemo local database/server of the accountID to be purged.
     */

    public void purgeUserOmemoData(AccountID accountId)
    {
        // Purge server omemo bundle nodes for the deleted account (only if online and registered)
        ProtocolProviderService pps = accountId.getProtocolProvider();
        if (pps != null) {
            XMPPConnection connection = pps.getConnection();
            if ((connection != null) && connection.isAuthenticated()) {
                BareJid userJid = accountId.getBareJid();
                PubSubManager pubsubManager = PubSubManager.getInstanceFor(connection, userJid);
                OmemoCachedDeviceList deviceList = mDB.loadCachedDeviceList(userJid);
                try {
                    for (int deviceId : deviceList.getAllDevices()) {
                        String nodeId = new OmemoDevice(userJid, deviceId).getBundleNodeName();
                        LeafNode leafNode = pubsubManager.getLeafNode(nodeId);
                        leafNode.deleteAllItems();
                        pubsubManager.deleteNode(nodeId);
                    }
                } catch (SmackException | InterruptedException | XMPPException.XMPPErrorException e) {
                    aTalkApp.showToastMessage(R.string.omemo_purge_inactive_device_error, userJid);
                }
            }
        }
        // Purge local omemo database for the deleted account
        mDB.purgeOmemoDb(accountId);
        trustCache.evictAll();
    }

    /**
     * Regenerate new omemo identity for the device
     * 1. Purge account Omemo info in database
     * 2. generate fresh user identityKeyPairs, bundle
     * 3. publish it to the server.
     */
    public void regenerate(AccountID accountId)
    {
        ProtocolProviderService pps = accountId.getProtocolProvider();
        if (pps != null) {
            XMPPConnection connection = pps.getConnection();
            if ((connection != null) && connection.isAuthenticated()) {
                // Purge all omemo devices info in database and server for the specific account
                purgeUserOmemoData(accountId);

                // Generate new omemoDevice
                BareJid userJid = accountId.getBareJid();
                int defaultDeviceId = OmemoManager.randomDeviceId();
                // Update the default OmemoDevice in database omemo_devices table
                setDefaultDeviceId(userJid, defaultDeviceId);

                /*
                 * (201908024) must perform the omemo initialization with the current active omemoManager.
                 * Do not create new, otherwise subsequent omemo message will still use the old omemoDevice
                 * and result to many omemo error messages
                 */
                mOmemoManager = OmemoManager.getInstanceFor(connection);
                // Init and publish to server
                mOmemoManager.initializeAsync(this);
            }
        }
    }

    /**
     * Call by OMEMO regeneration to perform clean up for:
     * 1. purge own Omemo deviceId
     * 2. All the preKey records for the deviceId
     * 3. Singed preKey data
     * 4. All the identities and sessions that are associated with the accountUuid
     *
     * @param accountId the specified AccountID to regenerate
     */
    public void purgeOmemoDb(AccountID accountId)
    {
        String account = accountId.getAccountJid();
        Timber.d(">>> Wiping OMEMO database for account : %s", account);

        SQLiteDatabase db = this.getWritableDatabase();
        String[] args = {account};

        db.delete(SQLiteOmemoStore.OMEMO_DEVICES_TABLE_NAME, SQLiteOmemoStore.OMEMO_JID + "=?", args);
        db.delete(SQLiteOmemoStore.PREKEY_TABLE_NAME, SQLiteOmemoStore.BARE_JID + "=?", args);
        db.delete(SQLiteOmemoStore.SIGNED_PREKEY_TABLE_NAME, SQLiteOmemoStore.BARE_JID + "=?", args);

        // Cleanup all the session and identities records for own resources and the contacts
        List<String> identityJids = getContactsForAccount(accountId.getAccountUuid());
        identityJids.add(account);
        for (String identityJid : identityJids) {
            args = new String[]{identityJid};
            db.delete(SQLiteOmemoStore.IDENTITIES_TABLE_NAME, SQLiteOmemoStore.BARE_JID + "=?", args);
            db.delete(SQLiteOmemoStore.SESSION_TABLE_NAME, SQLiteOmemoStore.BARE_JID + "=?", args);
        }
    }
====== Device B Debug Log for OMEMO regeneration =========
2020-11-06 09:11:28.345 13978-14157/org.atalk.android D/SMACK: SENT (0): 
    <iq id='MSHDU-25' type='get'>
      <query xmlns='http://jabber.org/protocol/disco#info' node='eu.siacs.conversations.axolotl.bundles:1566361092'>
      </query>
    </iq>
2020-11-06 09:11:28.384 13978-14158/org.atalk.android D/SMACK: RECV (0): 
    <iq xml:lang='en-US' to='parrot@atalk.sytes.net/atalk' from='parrot@atalk.sytes.net' type='result' id='MSHDU-25'>
      <query node='eu.siacs.conversations.axolotl.bundles:1566361092' xmlns='http://jabber.org/protocol/disco#info'>
        <identity type='pep' category='pubsub'/>
        <identity type='leaf' category='pubsub'/>
        <identity type='registered' category='account'/>
        <feature var='http://jabber.org/protocol/pubsub'/>
        <feature var='http://jabber.org/protocol/rsm'/>
        <feature var='http://jabber.org/protocol/pubsub#create-nodes'/>
        <feature var='http://jabber.org/protocol/pubsub#auto-create'/>
        <feature var='http://jabber.org/protocol/pubsub#auto-subscribe'/>
        <feature var='http://jabber.org/protocol/pubsub#delete-nodes'/>
        <feature var='http://jabber.org/protocol/pubsub#delete-items'/>
        <feature var='http://jabber.org/protocol/pubsub#filtered-notifications'/>
        <feature var='http://jabber.org/protocol/pubsub#modify-affiliations'/>
        <feature var='http://jabber.org/protocol/pubsub#outcast-affiliation'/>
        <feature var='http://jabber.org/protocol/pubsub#persistent-items'/>
        <feature var='http://jabber.org/protocol/pubsub#publish'/>
        <feature var='http://jabber.org/protocol/pubsub#publish-options'/>
        <feature var='http://jabber.org/protocol/pubsub#purge-nodes'/>
        <feature var='http://jabber.org/protocol/pubsub#retract-items'/>
        <feature var='http://jabber.org/protocol/pubsub#retrieve-affiliations'/>
        <feature var='http://jabber.org/protocol/pubsub#retrieve-items'/>
        <feature var='http://jabber.org/protocol/pubsub#retrieve-subscriptions'/>
        <feature var='http://jabber.org/protocol/pubsub#subscribe'/>
      </query>
    </iq>
2020-11-06 09:11:28.450 13978-14157/org.atalk.android D/SMACK: SENT (0): 
    <iq id='E2WWW-48' type='set'>
      <pubsub xmlns='http://jabber.org/protocol/pubsub#owner'>
        <purge node='eu.siacs.conversations.axolotl.bundles:1566361092'/>
      </pubsub>
    </iq>
2020-11-06 09:11:28.480 13978-14158/org.atalk.android D/SMACK: RECV (0): 
    <iq xml:lang='en-US' to='parrot@atalk.sytes.net/atalk' from='parrot@atalk.sytes.net' type='result' id='E2WWW-48'/>
2020-11-06 09:11:28.493 13978-14157/org.atalk.android D/SMACK: SENT (0): 
    <iq id='E2WWW-50' type='set'>
      <pubsub xmlns='http://jabber.org/protocol/pubsub#owner'>
        <delete node='eu.siacs.conversations.axolotl.bundles:1566361092'/>
      </pubsub>
    </iq>
2020-11-06 09:11:28.533 13978-14158/org.atalk.android D/SMACK: RECV (0): 
    <iq xml:lang='en-US' to='parrot@atalk.sytes.net/atalk' from='parrot@atalk.sytes.net' type='result' id='E2WWW-50'/>
2020-11-06 09:11:28.541 13978-14157/org.atalk.android D/SMACK: SENT (0): 
    <iq id='MSHDU-26' type='get'>
      <query xmlns='http://jabber.org/protocol/disco#info' node='eu.siacs.conversations.axolotl.bundles:522622186'>
      </query>
    </iq>

2020-11-06 09:11:28.627 13978-14158/org.atalk.android D/SMACK: RECV (0): 
    <iq xml:lang='en-US' to='parrot@atalk.sytes.net/atalk' from='parrot@atalk.sytes.net' type='result' id='MSHDU-26'>
      <query node='eu.siacs.conversations.axolotl.bundles:522622186' xmlns='http://jabber.org/protocol/disco#info'>
        <identity type='pep' category='pubsub'/>
        <identity type='leaf' category='pubsub'/>
        <identity type='registered' category='account'/>
        <feature var='http://jabber.org/protocol/pubsub'/>
        <feature var='http://jabber.org/protocol/rsm'/>
        <feature var='http://jabber.org/protocol/pubsub#create-nodes'/>
        <feature var='http://jabber.org/protocol/pubsub#auto-create'/>
        <feature var='http://jabber.org/protocol/pubsub#auto-subscribe'/>
        <feature var='http://jabber.org/protocol/pubsub#delete-nodes'/>
        <feature var='http://jabber.org/protocol/pubsub#delete-items'/>
        <feature var='http://jabber.org/protocol/pubsub#filtered-notifications'/>
        <feature var='http://jabber.org/protocol/pubsub#modify-affiliations'/>
        <feature var='http://jabber.org/protocol/pubsub#outcast-affiliation'/>
        <feature var='http://jabber.org/protocol/pubsub#persistent-items'/>
        <feature var='http://jabber.org/protocol/pubsub#publish'/>
        <feature var='http://jabber.org/protocol/pubsub#publish-options'/>
        <feature var='http://jabber.org/protocol/pubsub#purge-nodes'/>
        <feature var='http://jabber.org/protocol/pubsub#retract-items'/>
        <feature var='http://jabber.org/protocol/pubsub#retrieve-affiliations'/>
        <feature var='http://jabber.org/protocol/pubsub#retrieve-items'/>
        <feature var='http://jabber.org/protocol/pubsub#retrieve-subscriptions'/>
        <feature var='http://jabber.org/protocol/pubsub#subscribe'/>
      </query>
    </iq>
2020-11-06 09:11:28.766 13978-14157/org.atalk.android D/SMACK: SENT (0): 
    <iq id='E2WWW-52' type='set'>
      <pubsub xmlns='http://jabber.org/protocol/pubsub#owner'>
        <purge node='eu.siacs.conversations.axolotl.bundles:522622186'/>
      </pubsub>
    </iq>
    <r xmlns='urn:xmpp:sm:3'/>
2020-11-06 09:11:28.813 13978-14158/org.atalk.android D/SMACK: RECV (0): 
    <iq xml:lang='en-US' to='parrot@atalk.sytes.net/atalk' from='parrot@atalk.sytes.net' type='result' id='E2WWW-52'/>
2020-11-06 09:11:28.827 13978-14157/org.atalk.android D/SMACK: SENT (0): 
    <iq id='E2WWW-54' type='set'>
      <pubsub xmlns='http://jabber.org/protocol/pubsub#owner'>
        <delete node='eu.siacs.conversations.axolotl.bundles:522622186'/>
      </pubsub>
    </iq>
2020-11-06 09:11:28.861 13978-14158/org.atalk.android D/SMACK: RECV (0): 
    <iq xml:lang='en-US' to='parrot@atalk.sytes.net/atalk' from='parrot@atalk.sytes.net' type='result' id='E2WWW-54'/>

2020-11-06 09:11:28.883 13978-14157/org.atalk.android D/SMACK: SENT (0): 
    <iq id='E2WWW-56' type='set'>
      <pubsub xmlns='http://jabber.org/protocol/pubsub#owner'>
        <purge node='eu.siacs.conversations.axolotl.bundles:889121650'/>
      </pubsub>
    </iq>
2020-11-06 09:11:28.932 13978-13996/org.atalk.android I/g.atalk.androi: Background concurrent copying GC freed 152753(7MB) AllocSpace objects, 0(0B) LOS objects, 34% free, 11MB/17MB, paused 27.674ms total 403.645ms
2020-11-06 09:11:28.934 13978-14158/org.atalk.android D/SMACK: RECV (0): 
    <iq xml:lang='en-US' to='parrot@atalk.sytes.net/atalk' from='parrot@atalk.sytes.net' type='result' id='E2WWW-56'/>
2020-11-06 09:11:28.942 13978-14157/org.atalk.android D/SMACK: SENT (0): 
    <iq id='E2WWW-58' type='set'>
      <pubsub xmlns='http://jabber.org/protocol/pubsub#owner'>
        <delete node='eu.siacs.conversations.axolotl.bundles:889121650'/>
      </pubsub>
    </iq>
2020-11-06 09:11:29.005 13978-14158/org.atalk.android D/SMACK: RECV (0): 
    <iq xml:lang='en-US' to='parrot@atalk.sytes.net/atalk' from='parrot@atalk.sytes.net' type='result' id='E2WWW-58'/>
2020-11-06 09:11:29.008 13978-16155/org.atalk.android D/(DatabaseBackend.java:1337)#purgeOmemoDb: >>> Wiping OMEMO database for account : parrot@atalk.sytes.net
2020-11-06 09:12:09.144 13978-16155/org.atalk.android I/(DatabaseBackend.java:499)#storeOmemoRegId: ### Omemo device added for: parrot@atalk.sytes.net; 116536437

2020-11-06 09:12:09.939 13978-16162/org.atalk.android I/(SQLiteOmemoStore.java:375)#storeOmemoIdentityKeyPair: Store omemo identityKeyPair for :parrot@atalk.sytes.net:889121650
2020-11-06 09:12:39.247 13978-14157/org.atalk.android D/SMACK: SENT (0): 
    <iq id='E2WWW-60' type='set'>
      <pubsub xmlns='http://jabber.org/protocol/pubsub'>
        <publish node='eu.siacs.conversations.axolotl.bundles:889121650'>
          <item>
            <bundle xmlns='eu.siacs.conversations.axolotl'>
              <signedPreKeyPublic signedPreKeyId='1'>
                BciL2zw7Bs+4fnsgnCCOucgoR2RHbooMPZATaTXXJ60q
              </signedPreKeyPublic>
              <signedPreKeySignature>
                Bi0/qyCxbugQQVJeKmJasZqwssz/5wFMbEIc4gtFUGsRE6jI5mimr6nD2551G/vWZTHY1gpvZGsdx/12++04iw==
              </signedPreKeySignature>
              <identityKey>
                BVhKF82sA97fijJiMICn5FhbVwcnGvQgi4SImu0K3PZ3
              </identityKey>
              <prekeys>
                <preKeyPublic preKeyId='1'>
                  Bd87WaHaC9sThhKok78IKIpHeczZzDybs+sf4FZ7dOdm
                </preKeyPublic>
                <preKeyPublic preKeyId='2'>
                  BaGSIvX9+77MXv25RvRbuQ+RzhYm/jntloZ6G6glHcwB
                </preKeyPublic>
.....
                <preKeyPublic preKeyId='98'>
                  BSVgVtafr6qeSHKCCYLP44fE3bz/YXgIeTRhD2UGBgpD
                </preKeyPublic>
                <preKeyPublic preKeyId='99'>
                  BRO2A205NEkN2CRZCcvDZlTgSgwu25ALL43KeRUxu8pW
                </preKeyPublic>
                <preKeyPublic preKeyId='100'>
                  BVCyTNhHr2AvvLDmS4mByyAEcUBor9vCtY3JHwTtgKp/
                </preKeyPublic>
              </prekeys>
            </bundle>
          </item>
        </publish>
      </pubsub>
    </iq>
2020-11-06 09:12:40.929 13978-14158/org.atalk.android D/SMACK: RECV (0): 
    <message to='parrot@atalk.sytes.net/atalk' from='parrot@atalk.sytes.net' type='headline'>
      <event xmlns='http://jabber.org/protocol/pubsub#event'>
        <items node='eu.siacs.conversations.axolotl.bundles:889121650'>
          <item id='64498A09CB7F3'>
            <bundle xmlns='eu.siacs.conversations.axolotl'>
              <signedPreKeyPublic signedPreKeyId='1'>
                BciL2zw7Bs+4fnsgnCCOucgoR2RHbooMPZATaTXXJ60q
              </signedPreKeyPublic>
              <signedPreKeySignature>
                Bi0/qyCxbugQQVJeKmJasZqwssz/5wFMbEIc4gtFUGsRE6jI5mimr6nD2551G/vWZTHY1gpvZGsdx/12++04iw==
              </signedPreKeySignature>
              <identityKey>
                BVhKF82sA97fijJiMICn5FhbVwcnGvQgi4SImu0K3PZ3
              </identityKey>
              <prekeys>
                <preKeyPublic preKeyId='1'>
                  Bd87WaHaC9sThhKok78IKIpHeczZzDybs+sf4FZ7dOdm
                </preKeyPublic>
                <preKeyPublic preKeyId='2'>
                  BaGSIvX9+77MXv25RvRbuQ+RzhYm/jntloZ6G6glHcwB
                </preKeyPublic>
.....                
                <preKeyPublic preKeyId='98'>
                  BSVgVtafr6qeSHKCCYLP44fE3bz/YXgIeTRhD2UGBgpD
                </preKeyPublic>
                <preKeyPublic preKeyId='99'>
                  BRO2A205NEkN2CRZCcvDZlTgSgwu25ALL43KeRUxu8pW
                </preKeyPublic>
                <preKeyPublic preKeyId='100'>
                  BVCyTNhHr2AvvLDmS4mByyAEcUBor9vCtY3JHwTtgKp/
                </preKeyPublic>
              </prekeys>
            </bundle>
          </item>
        </items>
      </event>
      <addresses xmlns='http://jabber.org/protocol/address'>
        <address jid='parrot@atalk.sytes.net/atalk' type='replyto'/>
      </addresses>
    </message>
2020-11-06 09:12:41.460 13978-14158/org.atalk.android D/SMACK: RECV (0): 
    <iq xml:lang='en-US' to='parrot@atalk.sytes.net/atalk' from='parrot@atalk.sytes.net' type='result' id='E2WWW-60'>
      <pubsub xmlns='http://jabber.org/protocol/pubsub'>
        <publish node='eu.siacs.conversations.axolotl.bundles:889121650'>
          <item id='64498A09CB7F3'/>
        </publish>
      </pubsub>
    </iq>
2020-11-06 09:12:41.525 13978-14157/org.atalk.android D/SMACK: SENT (0): 
    <iq id='E2WWW-62' type='get'>
      <pubsub xmlns='http://jabber.org/protocol/pubsub'>
        <items node='eu.siacs.conversations.axolotl.devicelist'/>
      </pubsub>
    </iq>
2020-11-06 09:12:41.735 13978-14158/org.atalk.android D/SMACK: RECV (0): 
    <iq xml:lang='en-US' to='parrot@atalk.sytes.net/atalk' from='parrot@atalk.sytes.net' type='result' id='E2WWW-62'>
      <pubsub xmlns='http://jabber.org/protocol/pubsub'>
        <items node='eu.siacs.conversations.axolotl.devicelist'>
          <item id='current'>
            <list xmlns='eu.siacs.conversations.axolotl'>
              <device id='1566361092'/>
              <device id='522622186'/>
            </list>
          </item>
          <item id='64480FA448DD'>
            <list xmlns='eu.siacs.conversations.axolotl'>
              <device id='1566361092'/>
              <device id='294375861'/>
            </list>
          </item>
          <item id='64480FA9F38D'>
            <list xmlns='eu.siacs.conversations.axolotl'>
              <device id='1566361092'/>
              <device id='294375861'/>
            </list>
          </item>
          <item id='64482798558F'>
            <list xmlns='eu.siacs.conversations.axolotl'>
              <device id='1566361092'/>
            </list>
          </item>
          <item id='6448412C11D9'>
            <list xmlns='eu.siacs.conversations.axolotl'>
              <device id='1566361092'/>
              <device id='889121650'/>
            </list>
          </item>
          <item id='644841317511'>
            <list xmlns='eu.siacs.conversations.axolotl'>
              <device id='1566361092'/>
              <device id='889121650'/>
            </list>
          </item>
          <item id='644841329F86'>
            <list xmlns='eu.siacs.conversations.axolotl'>
              <device id='1566361092'/>
              <device id='889121650'/>
            </list>
          </item>
          <item id='64487704C2788'>
            <list xmlns='eu.siacs.conversations.axolotl'>
              <device id='1566361092'/>
              <device id='522622186'/>
              <device id='889121650'/>
            </list>
          </item>
          <item id='64487704EBA89'>
            <list xmlns='eu.siacs.conversations.axolotl'>
              <device id='1566361092'/>
              <device id='522622186'/>
              <device id='889121650'/>
            </list>
          </item>
          <item id='6448770550FB0'>
            <list xmlns='eu.siacs.conversations.axolotl'>
              <device id='1566361092'/>
              <device id='522622186'/>
              <device id='889121650'/>
            </list>
          </item>
        </items>
      </pubsub>
    </iq>

I believe I found the solution for the problem. By reusing the previous aTalk class, to init the Omemo i.e.
The class is in the aTalk official release gitbub i.e. org.atalk.crypto.omemo.AndroidOmemoService

new AndroidOmemoService(pps).initOmemoDevice();

The new regenerate() routine now changes to below. After I performed an OMEMO regeneration with this new routine, Device A and B are now back to normal operation for OMEMO message exchanges.
Sorry for any inconvenience causes to the smack team.

    /**
     * Regenerate new omemo identity for the device
     * 1. Purge account Omemo info in database
     * 2. generate fresh user identityKeyPairs, bundle
     * 3. publish it to the server.
     */
    public void regenerate(AccountID accountId)
    {
        ProtocolProviderService pps = accountId.getProtocolProvider();
        if (pps != null) {
            XMPPConnection connection = pps.getConnection();
            if ((connection != null) && connection.isAuthenticated()) {
                // Purge all omemo devices info in database and server for the specific account
                purgeUserOmemoData(accountId);
                new AndroidOmemoService(pps).initOmemoDevice();

//                // Generate new omemoDevice
//                BareJid userJid = accountId.getBareJid();
//                int defaultDeviceId = OmemoManager.randomDeviceId();
//                // Update the default OmemoDevice in database omemo_devices table
//                setDefaultDeviceId(userJid, defaultDeviceId);
//
//                /*
//                 * (201908024) must perform the omemo initialization with the current active omemoManager.
//                 * Do not create new, otherwise subsequent omemo message will still use the old omemoDevice
//                 * and result to many omemo error messages
//                 */
//                mOmemoManager = OmemoManager.getInstanceFor(connection);
//                // Init and publish to server
//                mOmemoManager.initializeAsync(this);
            }
        }
    }

This topic was automatically closed 100 days after the last reply. New replies are no longer allowed.