Java-Bells: A Jingle implementation for Java based on LibJitsi, Ice4J and Smack

I’ve noticed there were some efforts to put jingle back into smack, since smackx-jingle is discontinued, but none of them encorporate ice and none of them really show developers how to encorporate jingle into their own apps. So I worked on some code to make it easier for Java developers to use libjitsi and Ice4j to add AV chat features to smack applications. You can find the code here:

Please star/follow/report bugs/send fixes as you can.

It’s not a lot of code, but I think it’s very helpful in navigating libjitsi and ice4j.

2 Likes

Since Jingle is a often requestes feature for smack this deserves a star on github from me and a mark as featured content.

I really like to see some more Jingle work with Smack and maybe your project will be a new kickstart for it

1 Like

Thanks Flow!

Hi Bjorn,

Great sample! Thanks for the example! Do you know how it would be extended to allow multi-user conferencing?

Jee

Assuming you don’t want to try to support many simultaneous connections on the client, you would need support for this on the server side. I think the Jitsi project has some extensions for this.

Hmmm, I was thinking that a multi-user conference could be done by using the standard Jingle protocol, and having an audio mixer. I’m new to this, so I’m just trying to get validation for my thinking. I did see this post, which this guy seems to have been trying a similar thing:

http://lists.jitsi.org/pipermail/dev/2012-November/001032.html

Does this seem like it could work, in your opinion?

Jee

That is what Jitsi Video Bridge does. It is an Openfire plugin and handles all audio mixing on the server (Openfire) and forwards all video streams to each client to render. You should be able to develop a client with Java-Bells that connects to Jitsi Video Bridge. You only need to send an IQ message in order to connect. There is no documentation for it, but if you use Jitsi client and Openfire xmldebugger plugin you can see what is being sent and recieved.

Awesome! Thanks for the info! We are doing audio only. Does the VideoBridge work for audio only? We are doing point-to-point Jingle communication using Ice4j, so how does the server receive the audio stream, if we are connecting point-to-point?

Thanks again for your help!

Jee

Dele’s solution sounds correct. The lack of documentation is a problem with all Jitsi projects, (and the reason I created java-bells in the first place), but it shouldn’t be too terribly difficult to get an openfire plugin working once you know the basics.

Incidentally, the reason you need this plugin is that Jingle uses XMPP only for signalling. Actual media data is sent over another channel. For more details (which you may need in order to understand Java Bells and openfire) here are the appropriate specs:

http://xmpp.org/extensions/xep-0166.html

http://xmpp.org/extensions/xep-0176.html

http://tools.ietf.org/html/rfc5245

Less technical info also abounds.

Does the VideoBridge work for audio only?

Despite the name, “Video Bridge”, it can do audio only. Just test it with two Jitsi clients

Also, the audio mixer is part of libJitsi. This means that each client can host a conference and mix the audio of all participants. You could modify Java-Bells to do this and handle conferences peer to peer instead of using Jitsi Video Bridge. Again, the Jitsi client does this.

This gives you choice and flexibility

Dele,

Thanks for the taking the time to respond! I have been going down this Audio Mixer route, but not having too much luck. Do you think the VideoBridge route is an easier way to go?

Isn’t the VideoBridge route just connecting to one peer, and XMPP plugin would be that peer? So, instead of a peer to peer connection with every peer, all participants connect to the XMPP plugin as their peer? Or am I not thinking correctly?

Jeeman

Isn’t the VideoBridge route just connecting to one peer, and XMPP plugin would be that peer? So, instead of a peer to peer connection with every peer, all participants connect to the XMPP plugin as their peer? Or am I not thinking correctly?

Absolutely correct in your thinking

Using Jitsi VideoBridge would be the easiest and simplest way to go considering the lack of documentation and the fact that you would have to reverse-engineer a solution looking at the IQ messages between the client and Video Bridge component on your Openfire server.

Last time I tested, it was two or three messages exchanged and they were self-documenting.

Dele,

I’d be happy to create an initial VideoBridge IQ document to contribute to the community, once I get this thing figured out!

So, I went through the xmldebugger output. And I seem to only see one additional message that is VideoBridge action related. Is the other message(s) of the 2-3 that you said, related to discovery of the VB plugin?

Here is the one in-conference related IQ that I found:

C2S - SENT (256745658):

<?xml version="1.0" encoding="UTF-8"?>
      <conference-info xmlns="urn:ietf:params:xml:ns:conference-info" entity="jm2@scrubbed.com/jitsi-7q1vfv" sid="bpcq25g5bvl34" state="full" version="7">

2

                          <user entity="xmpp:jm2@scrubbed.com/jitsi-7q1vfv">

                                    <endpoint entity="xmpp:jm2@scrubbed.com/jitsi-7q1vfv">

connected

audio

-1657501349

sendonly

                          <user entity="jm@scrubbed.com/jitsi-1g7haco">

jm dev

                                    <endpoint entity="xmpp:jm@scrubbed.com/jitsi-1g7haco">

on-hold

audio

inactive

Thanks for all your help!

Jee

This is my trace

Jitsi sends audio conference request for a two person (channels) audio conference

VideoBridge responds by creating an audio bridge and allocates a pair of RTP/RTCP ports for the two channels. The fact that both channels have the same IP address which is the IP address of the VideoBridge confirms it is an audio mixer.

<iq type="result" id="4NoH5-30" from="jitsi-videobridge.btg199251" to="dele.olajide@btg199251/jitsi-15ep2bo">
   <conference xmlns="http://jitsi.org/protocol/colibri" id="1c540bd97828792f">
             <content name="audio">
                       <channel id="415a50e062c8517d" host="192.168.1.98" rtpport="10000" rtcpport="10001" expire="60"/>
                       <channel id="57584d9fc5a5f435" host="192.168.1.98" rtpport="10002" rtcpport="10003" expire="60"/>
             </content>
   </conference>
</iq>

Later on, it allocate SSRC codes and sends that information back to client

<iq type="set" id="dlEFJ-0" from="jitsi-videobridge.btg199251" to="dele.olajide@btg199251/jitsi-15ep2bo">
    <conference xmlns="http://jitsi.org/protocol/colibri" id="1c540bd97828792f">
          <content name="audio">
                    <channel id="415a50e062c8517d" host="192.168.1.98" rtpport="10000" rtcpport="10001" expire="60">
                              <ssrc>-1308648441</ssrc>
                    </channel>
          </content>
    </conference>
</iq> <iq type="set" id="dlEFJ-1" from="jitsi-videobridge.btg199251" to="dele.olajide@btg199251/jitsi-15ep2bo">
    <conference xmlns="http://jitsi.org/protocol/colibri" id="1c540bd97828792f">
              <content name="audio">
                        <channel id="57584d9fc5a5f435" host="192.168.1.98" rtpport="10002" rtcpport="10003" expire="60">
                                  <ssrc>-1308656147</ssrc>
                        </channel>
              </content>
    </conference>
</iq>

If you are familiar or used JingleNodes, this will all make sense. Otherwise, first understand how to work with a media relay when working with peer connections.

Ok, I found out what I did wrong, and now have the correct messages showing up. I don’t have the VideoBridge “set” message with the ssrc, but I only ran the conference for about 15 seconds before ending it.

Now, in the remote peer’s “session-accept”, it sends back its own IP/port. I, as the VideoBridge initiator, ignore this, right? I connect only to the IP provided by the VideoBridge “result” message, correct?

Pseudo-code (I hope this isn’t too confusing):

LocalPeer = LP

VideoBridgePlugin = VBP

RemotePeer = RP

IP and Ports = IPP

LP “get” -> VBP

VBP “result” with VBP-IPP -> LP

LP “session-initiate” with VBP-IPP -> RP

RP “session-accept” with RP-IPP -> LP

LP ignores RP-IPP

LP connects to VBP-IPP

RP connects to VBP-IPP

Bridge occurs

Someone sends "session-terminate"

LP “set” with channel expire set to “0” -> VBP

VBP closes conference bridge.

As always, thanks for your help.

Jeeman

Hi,

Another question…My VideoBridge and Jabber server are behind a firewall. Do you use ICE to traverse the VideoBridge call? Thanks,

Jeeman

Provided your firewall does not block UDP packets, you should be ok

Yeah, I am using a raw udp protocol right now (I believe that is all that VB supports right now).

But now, I am encountering another problem…my client starts a VB, audio only, and when I connect to the RTP/RTCP channels, I get the following…any ideas off the top of your head what I could be doing wrong?:

Aug 5, 2013 5:26:06 PM net.sf.fmj.media.Log error

SEVERE: Failed to build a graph for the given custom options.

Aug 5, 2013 5:26:06 PM net.sf.fmj.media.Log error

SEVERE: Failed to realize: net.sf.fmj.media.ProcessEngine@4446a5c4

Aug 5, 2013 5:26:06 PM net.sf.fmj.media.Log error

SEVERE: Cannot build a flow graph with the customized options:

Aug 5, 2013 5:26:06 PM net.sf.fmj.media.Log error

SEVERE: Unable to transcode format: LINEAR, 48000.0 Hz, 16-bit, Mono, LittleEndian, Signed

Aug 5, 2013 5:26:06 PM net.sf.fmj.media.Log error

SEVERE: to: opus/rtp, 48000.0 Hz, Mono

Aug 5, 2013 5:26:06 PM net.sf.fmj.media.Log error

SEVERE: outputting to: raw.rtp

Aug 5, 2013 5:26:06 PM net.sf.fmj.media.Log error

SEVERE: Error: Unable to realize net.sf.fmj.media.ProcessEngine@4446a5c4

Aug 5, 2013 5:26:06 PM org.jitsi.util.Logger warn

WARNING: javax.media.ResourceUnavailableEvent[source=com.sun.media.processor.unknown.Han dler@7a2d09e0,message=Failed to realize: cannot handle the customized options set on the Processor.

Check the logs for full details.]

Aug 5, 2013 5:26:06 PM org.jitsi.util.Logger warn

WARNING: ControllerErrorEvent: javax.media.ResourceUnavailableEvent[source=com.sun.media.processor.unknown.Han dler@7a2d09e0,message=Failed to realize: cannot handle the customized options set on the Processor.

Check the logs for full details.]

Aug 5, 2013 5:26:06 PM org.jitsi.util.Logger warn

WARNING: javax.media.ControllerClosedEvent[source=com.sun.media.processor.unknown.Handle r@7a2d09e0]

Aug 5, 2013 5:26:06 PM org.jitsi.util.Logger info

INFO: audio codec/freq: opus/48000 Hz

Aug 5, 2013 5:26:06 PM org.jitsi.util.Logger info

INFO: audio remote IP/port: 74.62.16.98/5002

Aug 5, 2013 5:26:06 PM net.sf.fmj.media.Log info

INFO: Resetting queue, last seq added: 9223372036854775806, current seq: 57601

Aug 5, 2013 5:26:07 PM org.jitsi.util.Logger info

INFO: rtpstat:Received a sender report for audio stream SSRC:2924214753 [packet count:51, bytes:3718 ]

Aug 5, 2013 5:26:12 PM net.sf.fmj.media.Log info

INFO: Growing packet queue to 8

By the way, it works fine Jitsi->Jitsi, or Jitsi->My custom client. It doesn’t work My Custom Client->Jitsi, and VideoBridge spits out the errors in the previous post.

Jeeman