A bit over a week ago, I fin­ished host­ing the vir­tual X.​Org De­vel­oper Con­fer­ence (XDC) 2020 with my friend Arka­diusz Hiler (AKA Arek / ivyl). This con­fer­ence has been livestreamed every sin­gle year since 2012, but this was the first time that we went fully-vir­tual and needed to have hosts / speak­ers pre­sent from their homes.

Of course, the XDC was not the only con­fer­ence this year that needed to be­come fully vir­tual, so we have been lucky-enough to learn from them (thanks to LWN for its ar­ti­cle on LPC2020 and Guy Lu­nardi), and this blog post is my at­tempt at shar­ing some of the knowl­edge we ac­quired by run­ning XDC 2020.

Over this blog post, I will an­swer the ques­tions how we se­lected jitsi over other video-con­fer­enc­ing so­lu­tions, how to de­ploy it, then pre­sent 3 dif­fer­ent ways to use it for live-stream­ing your con­fer­ence. Let’s get to it, shall we?

Why we se­lected Jitsi? Scal­ing con­cerns!

Given that the XDC is about open source graph­ics, it felt wrong to use pro­pri­etary tools for the con­fer­ence. This pretty much lim­ited our choices to two op­tions: Big Blue But­ton(BBB), and Jitsi. Luck­ily, both are ex­cel­lent!

BBB is a fan­tas­tic all-in-one con­fer­ence tool that is meant for on­line learn­ing and col­lab­o­ra­tion. Rooms are con­trolled by one-or-more mod­er­a­tors who split par­tic­i­pants into mul­ti­ple sub-room which is use­ful for group work in an eLearn­ing en­vi­ron­ment, or for break-out rooms in a con­fer­ence. It ex­cells at keep­ing band­width/cpu us­age low as im­ages are shared rather than a full video stream. This also makes it less likely to hit we­bRTC-re­lated is­sues. Sounds great for schools where stu­dents could be run­ning any­thing, from some cheap chrome­books to Win­dows lap­tops. The rec­om­mended server spec­i­fi­ca­tions are 8GB or RAM and 8 cores.

Jitsi on the other hand has a sim­pler UI cen­tered around we­bRTC. Users join a room, share their we­b­cam or screen, and talk us­ing their mi­cro­phones. It is also pos­si­ble to col­lab­o­ra­tively work on a shared doc­u­ment with a lo­cal ether­pad (an open-source web-based UI for col­lab­o­ra­tive text edi­tion). Most of the work done by Jitsi is hap­pen­ing client-side, which means that band­width and CPU re­quire­ments are higher than BBB’s, and that browser com­pat­i­bil­ity is also more prob­lem­atic (Chromium is fine, Fire­fox is prob­lem­atic, Sa­fari is bust), al­though I am sure the sit­u­a­tion will con­tin­u­ously im­prove. On the other hand, the server re­quire­ments are much lower than BBB. The only real re­source-heavy fea­ture is the built-in livestream­ing which spawns an X-Server, Chromium, screen grabs the out­put and streams it to youtube us­ing ffm­peg for the en­cod­ing.

XDC had an ex­pected at­ten­dance of up to 200 par­tic­i­pants, which meant that we needed to make sure that what­ever so­lu­tion we would go for had to work nicely with that many at­ten­dees. Ad­di­tion­ally, we needed to test this so­lu­tion ahead of time. Us­ing an in­te­grated so­lu­tion such as BBB pre­sented the risk that too many at­ten­dees would try to join and over­load the server which would have pre­vented the con­fer­ence from hap­pen­ing. In­stead, by lim­it­ing the au­di­ence to just the speak­ers, test­ing was rel­a­tively easy (find 10 friends to have a chat with on a test in­stance), and view­ers could just watch the stream live on Youtube (AKA, scal­ing is not my prob­lem). The in­con­ve­nient of such method is of course that we need to pro­vide a way for at­ten­dees to in­ter­act with the speaker and dis­cuss with each other. For this, we just use IRC since this is what the com­mu­nity is used to. See Arek’s blog post to learn more about max­i­miz­ing the sat­is­fac­tion of livestream view­ers.

So, in the end, the choice to use Jitsi for XD­C2020 came down to… be­ing a chicken and not hav­ing the time to or­ga­nize the needed test­ing, like the Linux Plumber Con­fer­ence did! How­ever, in hind­sight, I would still not have changed any­thing (more on that later).

How to de­ploy Jitsi?

If you are go­ing to de­ploy Jitsi for a con­fer­ence, I highly sug­gest us­ing the “cloud” for this! It brings a lot of con­ve­nience for test­ing, du­pli­cat­ing in­stances, back­ups, and it keeps the in­fra cost ex­tremely low! FYI, for XDC 2020, we used 4 in­stances of Jitsi (Yes, I am a para­noid chicken!), and the over­all cost was less than $39. This in­cludes the test in­stances I set up, up to a month ahead of the con­fer­ence.

I per­son­nally se­lected Dig­i­tal Ocean be­cause I was al­ready a cus­tomer of theirs, and it was easy for me to cre­ate a $5/month in­stance that could be scaled to a $80/month in­stance with 4 ded­i­cated CPUs and 16GB of RAM just for the con­fer­ence.

When se­lect­ing a lo­ca­tion for the VM, choose one that is some­what at the cen­ter of the ex­pected par­tic­i­pants. For ex­am­ple, if the con­fer­ence has a ma­jor­ity of Eu­ro­peans and Amer­i­cans (North and South), se­lect­ing New York City (NYC) or Lon­don (LON) will bring the low­est av­er­age la­tency be­tween par­tic­i­pants. In case of doubt, set up mul­ti­ple in­stances of Jitsi in dif­fer­ent data cen­ters so as you can move peo­ple to an­other server in case of an is­sue. Check out Dig­i­tal Ocean’s speed test to ex­pe­ri­ence the ef­fect of world-wide rout­ing on your ping and band­width, and ask your at­ten­dees to ver­ify ahead of time that they will meet the band­width re­quire­ments.

When in­stalling Jitsi, I strongly sug­gest you fol­low the Docker Self-Host­ing Guide if you want a pain­less de­ploy­ment. I tried the other two rec­om­mended meth­ods and al­ways had is­sues with the bridge which I never man­aged to fix. Just go with docker, don’t waste your time!

If you want to cus­tomize some style/code, I sug­gest you mod­ify docker-compose.​yml to mount-bind some re­sources such as a re­place­ment wa­ter­mark for the jitsi in­stance, or a new css file.

Fi­nally, I sug­gest you cre­ate a sys­temd ser­vice to auto-start Jitsi on boot. Cre­ate the fol­low­ing file at /etc/sys­temd/sys­tem/jitsi.​service:

[Unit]
Description=Jitsi
Requires=docker.service
After=docker.service

[Service]
Type=simple
WorkingDirectory=/root/docker-jitsi-meet
ExecStart=/usr/bin/docker-compose -f docker-compose.yml -f etherpad.yml -f jibri.yml up

[Install]
WantedBy=multi-user.target

Don’t for­get to en­able/start the ser­vice af­ter that:

# systemctl daemon-reload
# systemctl enable jitsi
# systemctl start jitsi

Jitsi-based live-streamed con­fer­ence: 3 lev­els of pro­fes­sion­al­ism

Okay, let’s now get into the gist of this ar­ti­cle! Propos­ing three dif­fer­ent ways of host­ing a Jitsi-based con­fer­ence.

Pro­vid­ing a good ex­pe­ri­ence to livestream view­ers is im­por­tant, es­pe­cially since it also ends up be­ing the record­ings for the con­fer­ence. As sug­gested by Arek on his XD­C2020 blog post, here are im­por­tant as­pects that should drive the qual­ity up:

• No dead air (the stream ap­pears dead);
• Min­i­mize the room for (se­ri­ous) er­rors;
• Fo­cus on the au­dio qual­ity;
• Em­brace ex­ist­ing means of com­mu­ni­ca­tion.

We will use these ideas when propos­ing so­lu­tions.

Level 1: All par­tic­i­pants in a Jitsi room

This is the sim­plest so­lu­tion by far. All it re­quires is to de­ploy a beefy Jitsi in­stance, cre­ate a room, and in­vite the par­tic­i­pants there.

To start live-stream­ing, sim­ply click on the set­tings but­ton, press “Start livestream­ing”, then copy the livestream­ing key you got from Youtube live!

At this point, you will have two pools of at­ten­dees: the live ones on Jitsi, and the view­ers-only on Youtube. It is up to you to de­cide how many peo­ple will be on one or the other, but in the event of poor per­for­mance of the server, it is pos­si­ble to just kick peo­ple out of the room to keep only the most im­por­tant peo­ple there.

Rather than us­ing Jitsi’s and Youtube’s chat, Arek and I sug­gest hav­ing the same chat sys­tem for both groups of at­ten­dees. Use your com­mu­nity’s favourite mode of text com­mu­ni­ca­tion (IRC?) there!

To re­duce the amount of dead-air which can hap­pen in-be­tween two talks, and to im­prove the ex­pe­ri­ence of live-view­ers that would just tune in the con­fer­ence dur­ing a break, the or­ga­nizer should share their desk­top show­ing the name of the next talk and when it will start. It could be as sim­ple as a full-screen text ed­i­tor con­tain­ing the data, but if you want to do it very well, hav­ing a count­down helps re­duce con­fu­sion with time­zones:

(Click to go to the video)

Pros:

• Sim­ple to de­ploy
• Mul­ti­ple hosts can take care of ask­ing ques­tions or prepar­ing/show­ing the next-talk slide

Cons:

• Im­pos­si­ble to im­prove the sound qual­ity of par­tic­i­pants for the view­ers on Youtube as Jitsi streams di­rectly
• No pre-test­ing pos­si­ble for pre­sen­ters on the live in­stance, so the switch from one pre­sen­ter to an­other can be shaky
• Can’t change rooms with­out los­ing the livestream: if a pre­sen­ter can’t join, tough luck!

Level 2: Only speak­ers in a room + stream­ing us­ing OBS

Level 2 is fo­cus­ing on im­prov­ing the sound qual­ity of pre­sen­ters, and the flex­i­bil­ity of pre­sen­ta­tion. This can be achieved by mov­ing the stream­ing from the Jitsi in­stance to a host’s com­puter. This en­ables the stream­ing of al­most any­thing: video, other jitsi in­stances, …

For any kind of stream­ing setup, I would rec­om­mend us­ing OBS Stu­dio which has pretty much every­thing you need built-in. Please fol­low Arek’s blog post to get tips on how to max­i­mize the qual­ity of your setup!

The live-stream­ing host should fo­cus on video and sound qual­ity only, leav­ing an­other host to deal with col­lect­ing ques­tions dur­ing the talk, then ask­ing them live on Jitsi.

The ma­jor draw­back of this so­lu­tion is that the host’s in­ter­net con­nec­tion is be­com­ing crit­i­cal to stream­ing the con­fer­ence. This means that any power or in­ter­net cut will lead to a loss of livestream­ing. To re­duce the power risk, I sug­gest buy­ing a UPS to power both the mo­dem, the com­puter, and all the equi­ments needed for stream­ing. To re­duce the in­ter­net cut risk, I sug­gest set­ting a router with a failover to a 4G net­work so as OBS would just re-es­tab­lish the con­nec­tion af­ter los­ing it.

Pros:

• Flex­i­bil­ity to ad­just sound, video, and show any wait­ing screen with a nice tran­si­tion
• Set­ting up of the next pre­sen­ter can hap­pen off-cam­era dur­ing the breaks
• Pos­si­bil­ity to tell speak­ers how much time is left with­out the stream hear­ing it

Cons:

• The host needs to take steps to re­duce the risk of in­ter­net or power is­sues dur­ing the con­fer­ence
• Pre­sen­ters might not be able to set up their sys­tem dur­ing the break, which in­creases stress level of every­one

Level 3: Only speak­ers in a room, trip­ple OBS setup

For our last level, we will fo­cus on re­duc­ing the risk that pre­sen­ters might not be able to set up their sys­tem dur­ing the break, and al­low­ing the host to take breaks by hav­ing 2 Jitsi and 3 OBS in­stances.

Why cre­ate so many in­stances? Hav­ing 3 OBS in­stances en­ables two hosts to share the load. When one is live, the other one can take a break and can pre­pare for the next talk. The 3rd OBS in­stance is sim­ply used to switch be­tween the two hosts.

Now, you may won­der why would we need 2 Jitsi in­stances? Re­mem­ber what I said about be­ing para­noid and a chicken? Well, I re­ally don’t like touch­ing any­thing sys­tem that is live, so hav­ing one Jitsi in­stance per host en­able each host to have full con­trol of their Jitsi and OBS in­stance, with noth­ing else hap­pen­ing in the back­ground. This thus re­duces the chances of crashes or that set­ting up the next talk would starve the live room \o/!

Again, see Arek’s blog post for the full rea­son­ing be­hind this, and how to set it up!

Pros:

• Flex­i­bil­ity to ad­just sound, video, and show any wait­ing screen with a nice tran­si­tion
• Set­ting up of the next pre­sen­ter can hap­pen off-cam­era dur­ing the breaks
• Pos­si­bil­ity to speak to speak­ers with­out the stream hear­ing any­thing (good for time-check­ing pre­sen­ters)
• En­ables hosts to take breaks every other talk

Cons:

• The hosts needs to take steps to re­duce the risk of in­ter­net or power is­sues dur­ing the con­fer­ence

Fi­nal thoughts?

Host­ing a con­fer­ence us­ing Jitsi is rel­a­tively sim­ple! Now that I have done it, I could host a new one in less than an hour if needed, and that is pretty im­pres­sive!

Of course, there are a lot of as­pects that will el­e­vate the level of pro­fes­sion­al­ism of your con­fer­ence, but many of these things can be done as time per­mits. How­ever, the ma­jor jump in qual­ity re­ally comes from im­prov­ing the au­dio, and go­ing with at least the level 2 ar­chi­tec­ture for any­thing but a break­out room is rec­om­mended as it en­ables a lot more con­trol over what goes in the livestream.

No mat­ter what so­lu­tion you end up with, I highly sug­gest you doc­u­ment the way at­ten­dees and speak­ers should in­ter­act with the con­fer­ence. For ref­er­ence, here are the pre­sen­ter’s guide and the at­tendee’s guide.

I guess that’s it! I am sure I for­got many things which com­menters might re­mind me of, so ex­pect up­dates!