Cannot make server to server connection-help

tonyg · August 6, 2019, 12:53pm

Hi, i have just installed a second openfire server (4.4.0) on our network in a different domain. I cannot get server to server communications to work.
I tried the server to server test tool and these are the results: can someone help me understand the output?

thanks

xmpp:

<iq type="get" id="42-89264" from="ultra-fei.com" to="uedi-gbs.com"><ping xmlns="urn:xmpp:ping"/></iq>
<iq type="error" id="42-89264" to="ultra-fei.com" from="uedi-gbs.com"><ping xmlns="urn:xmpp:ping"/><error code="404" type="cancel"><remote-server-not-found xmlns="urn:ietf:params:xml:ns:xmpp-stanzas"/></error></iq>

Certs:
blank

logs:

Sending server to server ping request to uedi-gbs.com
Successful server to server response received.
Primary packet routing failed
org.jivesoftware.openfire.PacketException: Cannot route packet of type IQ or Presence to bare JID: <iq type="error" id="42-89264" to="ultra-fei.com" from="uedi-gbs.com"><ping xmlns="urn:xmpp:ping"/><error code="404" type="cancel"><remote-server-not-found xmlns="urn:ietf:params:xml:ns:xmpp-stanzas"/></error></iq>
	at org.jivesoftware.openfire.spi.RoutingTableImpl.routeToLocalDomain(RoutingTableImpl.java:306) ~[xmppserver-4.4.0.jar:4.4.0]
	at org.jivesoftware.openfire.spi.RoutingTableImpl.routePacket(RoutingTableImpl.java:239) [xmppserver-4.4.0.jar:4.4.0]
	at org.jivesoftware.openfire.server.OutgoingSessionPromise$PacketsProcessor.returnErrorToSender(OutgoingSessionPromise.java:343) [xmppserver-4.4.0.jar:4.4.0]
	at org.jivesoftware.openfire.server.OutgoingSessionPromise$PacketsProcessor.addPacket(OutgoingSessionPromise.java:361) [xmppserver-4.4.0.jar:4.4.0]
	at org.jivesoftware.openfire.server.OutgoingSessionPromise$1.run(OutgoingSessionPromise.java:134) [xmppserver-4.4.0.jar:4.4.0]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
Failed to establish server to server session.

tonyg · August 8, 2019, 12:21pm

bump, can anyone point me in the right direction?

thanks in advance!

totzkotz · August 8, 2019, 2:58pm

Do you have the neccecary DNS SRV records ?
Open port 5269 on both servers (forward port if needed)

tonyg · August 8, 2019, 4:45pm

hi, thanks for the reply.

yes, i have the dns svr records in and i have confirmed that tcp/5269 is open

totzkotz · August 8, 2019, 7:33pm

Log into a shell on the servers and try to telnet to the other servers port 5269 (take hostname not ip)… if there are no errors then it is not a network problem so maybe wrong configuration
Check your ssl certs…

tonyg · August 13, 2019, 6:03pm

hi, sorry for the delay. its not network, i tried telnet to the fqdn and the right IP resolved and it did connect to the port.

also, under config, starttls is optional and mutual auth is disabled.

any idea where i should look next?

tonyg · August 13, 2019, 6:27pm

i see this in the debug logs, it seems to indicate the local server cannot find the foreign server.

2019.08.13 14:21:56 org.jivesoftware.openfire.spi.RoutingTableImpl - Failed to route packet to JID: ultra-fei.com packet: <iq type="error" id="278-219" to="ultra-fei.com" from="uedi-gbs.com"><ping xmlns="urn:xmpp:ping"/><error code="404" type="cancel"><remote-server-not-found xmlns="urn:ietf:params: xml:ns:xmpp-stanzas"/></error></iq>
2019.08.13 14:21:56 org.jivesoftware.openfire.IQRouter - IQ sent to unreachable address: <iq type="error" id="278-219" to="ultra-fei.com" from="uedi-gbs.com"><ping xmlns="urn:xmpp:ping"/><error code="404" type="cancel"><remote-server-not-found xmlns="urn:ietf:params:xml:ns:xmpp-stanzas"/></error></iq>
2019.08.13 14:21:56 org.jivesoftware.openfire.server.OutgoingSessionPromise$PacketsProcessor - Error sending packet to domain 'uedi-gbs.com' (outbound queue full):
<iq type="get" id="278-219" from="ultra-fei.com" to="uedi-gbs.com">
  <ping xmlns="urn:xmpp:ping"/>
</iq>

however, the dns is there:

dig _xmpp-server._tcp.uedi-gbs.com. SRV

; <<>> DiG 9.9.4-RedHat-9.9.4-74.el7_6.2 <<>> _xmpp-server._tcp.uedi-gbs.com. SRV
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 54534
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 2

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;_xmpp-server._tcp.uedi-gbs.com.        IN      SRV

;; ANSWER SECTION:
_xmpp-server._tcp.uedi-gbs.com. 3600 IN SRV     0 0 5269 gbs-vic-chat-1.uedi-gbs.com.

;; ADDITIONAL SECTION:
gbs-vic-chat-1.uedi-gbs.com. 3600 IN    A       172.30.0.15

;; Query time: 1 msec
;; SERVER: 172.30.0.38#53(172.30.0.38)
;; WHEN: Tue Aug 13 14:26:21 EDT 2019
;; MSG SIZE  rcvd: 122

totzkotz · August 15, 2019, 8:03am

you send the ping to domain: ultra-fei.com so this domain must also have its service records

tonyg · August 15, 2019, 11:42am

hi, no, the ping was sent from ultra-fei.com TO uedi-gbs.com

totzkotz · August 15, 2019, 2:27pm

do you sit behind a router/firewall/nat?
i tested “telnet uedi-gbs.com 5269” and i could not establish a connection…
so i tested dns with “nslookup uedi-gbs.com” and did not get an ip. so it should be a dns problem…

tonyg · August 15, 2019, 2:40pm

hi, these are internal systems. to be clear, clients cannot connect to them from the Internet but they have internet access out. also, the 2 systems are NOT separated by a firewall, and in fact, they are on the same subnet

totzkotz · August 15, 2019, 6:32pm

Try to add the servers to
each others whitelist in the admin console

tonyg · August 15, 2019, 7:03pm

yes, i tried that

btw, i was confused by the interface, do you add the domain you want to connect to…or the server in that domain? i added both.

tonyg · August 15, 2019, 7:03pm

also, i upgraded to 4.41…did not help

totzkotz · August 16, 2019, 4:07am

you have to add the xmpp domain of the server…

totzkotz · August 16, 2019, 4:10am

check:
STARTTLS policy > optional
Mutual Authentication > disabled
Certificate chain checking > Allow peer certificates to be self-signed.
Encryption Protocols > TLSv1, TLSv1.1, TLSv1.2
Encryption Cipher Suites > all to left (for testing)…

tonyg · August 16, 2019, 11:28am

totzkotz, thanks for sticking with this, I appreciate your time.

i made the changes you suggested, most of the settings were that way already. it had no affect. a test results in the following:
xmpp:

<iq type="get" id="220-7048" from="gbs-vic-chat-1.uedi-gbs.com" to="ultra-fei.com"><ping xmlns="urn:xmpp:ping"/></iq>
<iq type="error" id="220-7048" to="gbs-vic-chat-1.uedi-gbs.com" from="ultra-fei.com"><ping xmlns="urn:xmpp:ping"/><error code="404" type="cancel"><remote-server-not-found xmlns="urn:ietf:params:xml:ns:xmpp-stanzas"/></error></iq>

certs:
empty

logs:

Sending server to server ping request to ultra-fei.com
Successful server to server response received.
Failed to establish server to server session.

tonyg · August 30, 2019, 12:35pm

can anyone point me in the right direction?

guus · September 3, 2019, 12:21pm

Enabling debug logging will give you a wealth of data. I’d start with that.

I’ve also found issues with s2s in the 4.4.x branch. Things are improving this week - the first fix should already be in the nightly build.

tonyg · September 18, 2019, 5:53pm

sorry for the delay. I enabled debug logging and then attempted to send a chat to someone on the foreign server (uedi-gbs.com)…this is what was in the debug log

at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49) [mina-core-2.1.3.jar:?]
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128) [mina-core-2.1.3.jar:?]
at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:122) [mina-core-2.1.3.jar:?]
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650) [mina-core-2.1.3.jar:?]
at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49) [mina-core-2.1.3.jar:?]
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128) [mina-core-2.1.3.jar:?]
at org.apache.mina.filter.codec.ProtocolCodecFilter$ProtocolDecoderOutputImpl.flush(ProtocolCodecFilter.java:413) [mina-core-2.1.3.jar:?]
at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:257) [mina-core-2.1.3.jar:?]
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650) [mina-core-2.1.3.jar:?]
at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49) [mina-core-2.1.3.jar:?]
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128) [mina-core-2.1.3.jar:?]
at org.apache.mina.core.filterchain.IoFilterEvent.fire(IoFilterEvent.java:106) [mina-core-2.1.3.jar:?]
at org.apache.mina.core.session.IoEvent.run(IoEvent.java:89) [mina-core-2.1.3.jar:?]
at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTask(OrderedThreadPoolExecutor.java:766) [mina-core-2.1.3.jar:?]
at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTasks(OrderedThreadPoolExecutor.java:758) [mina-core-2.1.3.jar:?]
at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.run(OrderedThreadPoolExecutor.java:697) [mina-core-2.1.3.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]
2019.09.18 13:50:59 org.jivesoftware.openfire.net.SASLAuthentication - SASL negotiation failed for session: org.jivesoftware.openfire.session.LocalClientSession@1b555fe3 status: 1 address: gbs-vic-chat-1.uedi-gbs.com/33kk1q0oxj id: 33kk1q0oxj presence:
<presence type="unavailable"/>
javax.security.sasl.SaslException: PLAIN authentication failed for: stan.braithwaite
at org.jivesoftware.openfire.sasl.SaslServerPlainImpl.evaluateResponse(SaslServerPlainImpl.java:144) ~[xmppserver-4.4.1.jar:4.4.1]
at org.jivesoftware.openfire.net.SASLAuthentication.handle(SASLAuthentication.java:357) [xmppserver-4.4.1.jar:4.4.1]
at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:185) [xmppserver-4.4.1.jar:4.4.1]
at org.jivesoftware.openfire.nio.ConnectionHandler.messageReceived(ConnectionHandler.java:183) [xmppserver-4.4.1.jar:4.4.1]
at org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:1015) [mina-core-2.1.3.jar:?]
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650) [mina-core-2.1.3.jar:?]
at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49) [mina-core-2.1.3.jar:?]
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128) [mina-core-2.1.3.jar:?]
at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:122) [mina-core-2.1.3.jar:?]
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650) [mina-core-2.1.3.jar:?]
at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49) [mina-core-2.1.3.jar:?]
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128) [mina-core-2.1.3.jar:?]
at org.apache.mina.filter.codec.ProtocolCodecFilter$ProtocolDecoderOutputImpl.flush(ProtocolCodecFilter.java:413) [mina-core-2.1.3.jar:?]
at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:257) [mina-core-2.1.3.jar:?]
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650) [mina-core-2.1.3.jar:?]
at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49) [mina-core-2.1.3.jar:?]
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128) [mina-core-2.1.3.jar:?]
at org.apache.mina.core.filterchain.IoFilterEvent.fire(IoFilterEvent.java:106) [mina-core-2.1.3.jar:?]
at org.apache.mina.core.session.IoEvent.run(IoEvent.java:89) [mina-core-2.1.3.jar:?]
at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTask(OrderedThreadPoolExecutor.java:766) [mina-core-2.1.3.jar:?]
at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTasks(OrderedThreadPoolExecutor.java:758) [mina-core-2.1.3.jar:?]
at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.run(OrderedThreadPoolExecutor.java:697) [mina-core-2.1.3.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]
Caused by: javax.security.sasl.SaslException: PLAIN: user not authorized: stan.braithwaite
at org.jivesoftware.openfire.sasl.SaslServerPlainImpl.evaluateResponse(SaslServerPlainImpl.java:133) ~[xmppserver-4.4.1.jar:4.4.1]
... 22 more
2019.09.18 13:50:59 org.apache.mina.filter.ssl.SslFilter - Session Server[18](SSL): Writing Message : WriteRequest: HeapBuffer[pos=0 lim=77 cap=128: 3C 66 61 69 6C 75 72 65 20 78 6D 6C 6E 73 3D 22...]
2019.09.18 13:50:59 org.apache.mina.filter.executor.OrderedThreadPoolExecutor - Adding event MESSAGE_SENT to session 18
Queue : [MESSAGE_SENT, ]