Here comes a decent XMPP server!

Fed up with the carouselle of Proprietary Messaging Networks of the Day? Try an open, federated network running and evolving since 1999, based on free and open source software and the protocol standartized by IETF! Today, XMPP practically supports such long-desired features as video calls and convenient end to end encryption.

This page used to say: "stay in touch with your contacts using provided gateways to other IM networks!". Nowadays, it's a violation of Terms of Service of proprietary networks to use non-official software. So bridging is practically possible only to other open networks where owners respect your freedom. Currently decent.im runs a gateway only for IRC.

Server configs and code are public.

===> Request an account! <===

Prosody developer MattJ gave a talk this week on the topic of Spam, Abuse and Moderation discussing many aspects of keeping the users, servers and entire XMPP network safe.

Prompted by the talk, I enabled the spam reporting modules.

So LLM (Large Language Models) are the hotness of the day. Still not a reliable tool, but amusing at least, and time-saving at its best. There is also an inspiring amount of stuff any curious individual can explore for free and without material obligations to corporations.

Llamafile project is yet another breakthrough in lowering the complexity bar for trying out local LLMs. Download one file, and run it on your computer of maybe even the phone.

I actually had to reach for lower level tool, llama.cpp as llamafile didn't work properly on the machine I wanted to run it on. llama.cpp is still very approachable for a software tinkerer, and it is becoming easier to figure out how to download the precious "weights".

Some models can analyse details on pictures you upload (bot doesn't support that). Some models can converse in all languages, not just English. Impressive! Imagine just a DVD or a few in your hands. Before, it was enough to install Linux or play an expensive game. Now it has, to some degree of fidelity, most of the information from the Internet.

So in the old tradition of exposing software tricks in the form of a chatbot, I cobbled together an XMPP bot which lets XMPP users to talk to LLM models.

Here is it: llama@decent.im (Source code)

Try it out!

I rotate the models and will mess with prompts and parameters, so check the bot status message for details. Message exchanges are monitored.

Sometimes LLM says ridiculous things (false, made-up URLs, so many other things...), as is the common knowledge. But sometimes they are pretty good (model amethyst-13b-mistral.Q5_K_M):

User: Which country owns Crimea?
Llama: Russia currently occupies and administers the Ukrainian peninsula of Crimea, but it remains internationally recognized as part of Ukraine. The annexation of Crimea by Russia in 2014 was widely condemned and considered illegal under international law.

I started Decent.im 8 years ago. With glee and audacity.

"Look honey, just like Skype, but on my server!"

I found no public server which synced and archived mesasges properly and I boasted about my "feat", even at the Operators chatroom.

There was more joy down the road.

I used it all this time with my spouse, and still do. Conversations app was the key enabler for that.

I used gateways to IRC, Skype, Slack and Facebook to be widely connected but more conveniently.

My makeshift CCTV system was connected to an XMPP account for "online status" and notificaitons of detected motion.

I initially spun the server up on Ubuntu, but pretty soon I hit some annoying limitations so I switched to Gentoo. For quite a long time, the server was running the bleeding edge of a lot of the core software, with weekly unattended updates. Prosody and stuff were pulled straight from the trunk of the version control system. The flavour of audacity is a form of compensation.

This worked exceptionally well for such an arrangement, which shows that the developers are very careful and the software stack is very stable. I got into (2017) and out of (2022) the status of Gentoo Developer in the meantime, but Decent.im still runs on Gentoo Linux.

A few users have joined over time. Those asking for two accounts, for themselves and their close relative, seem to be especially persistent in using it. Give it a try the same way.

Thanks

This is a little thingy which was built and keeps living owing to the vast treasure of perfect gifts from a lot of people.

Particular thanks:

Prosody devs - MattJ, Zash and others, for support and for keeping this all evolving.

Daniel Gultsch for Conversations which made XMPP cute and "just work" on Android.

fiaxh and larma for Dino which made XMPP cute and "just work" on desktop.

Louiz for Biboumi the IRC bridge, and for him adding support for PostgreSQL storage on my request and funded by me.

Eion Robb for his Skype bridges.

pep. for organizing the XMPP Sprint where I met many XMPP developers (including some listed above) in person.

Jonas Schäfer for keeping my back with his free monitoring system.

Conrad Kostecki for maintaining XMPP server packages in Gentoo Linux.

I have been measuring and comparing messaging apps for Android. Now I have something to share. I hope you enjoy reading this and it proves to be a useful knowledge for you.

This is a very limited study of resource usage by messaging apps on Android. It needs more apps, devices and picky readers and reviewers.

Conclusions

Conversations (XMPP) seems best for battery life, traffic and RAM.

Facebook Messenger and Element (Matrix) are the worst from the studied apps, draining up to 300x more power than Conversations when battery optimization is disabled.

Element X (experimental Matrix app) is coming up with a more than 100x reduction in energy consumption (comparing to Element), drawing just 2x more power than Conversations.

A few more proprietary messengers have been measured and they fall in between the space between the above mentioned extremes. Little can be said about them besides that most of them continuously draw substantial amount of power and bandwidth even when not interacted with.

Cool Pictures! I mean, serious data.

Explore this public Grafana dashboard.

Misc

I run an XMPP server, but I made this research to find out the facts, not to prove any beliefs or prop up attractiveness of any chat network.

You can:

• Add your data
• Donate (mention messengers-perf in the comment)
• Ask, thank or tell me something

Besides boring maintenance, lately I've been working on "full user data takeout" tools. Including all chats logs, and shared pictures and other files which were stored on the server. It's a great possibility all around - you keep your data so you can take it to your own server or a different one, and the service disruptions would be a lesser concern. For me, though, the main motivation is to be more confident about targeted data cleanups. Currently the archives (messages and shared files) are stored indefinitely, because I am a sentimental person. But I was asked by a user to limit the archives retention period for them. Happy to oblige - less risk for their privacy, less data to store on the server for me. But I wanted to give them the possibility to keep storing their data themselves indefinitely.

The new tool is called decent.im_user_takeout_all. It takes username as a parameter, and produces a compressed archive named like takeout.username.2023-10-13_09:16:02.tar.zst. It takes the chat logs and user's settings from the Postgres database, and stores them in SQLite database file, just like they were structured before, so it would be easy to restore it to a different server.

It also saves the user's uploads stored on the server, preserving the server's directory layout and timestamps.

More tooling - XMPP Account Portability project

Most of my tool was implemented within one intense day. It works well and I think it's a nice improvement, but I am at unease with the resulting data format. The data could be packaged in so much more broadly portable format!

In 2021-2022, in XMPP Account Portability project, developers Magnus Henoch, Waqas Hussain, Matthew Wild, Daniel Brötzmann (cal0pteryx), Kim Alvefur (Zash) and others have developed the specification, the server-side support, and the tool for users to take out their data from servers, and automatically migrate to another server.

The said tool is web-based, and it is now deployed on decent.im.

Future development

This tool doesn't export the chat history, although it can be made to do that. I think it's important to make it do that (I am a sentimental person), so I want to work on that.

The uploads - pictures and such - are not in scope for the portable import format spec. It feels valuable to be able to take these files with you, but some gotchas must be mentioned. Once the files are uploaded to the XMPP server, the URLs are sent in messages in the conversations. Since your friends stay where they are (or migrate on their own), their message archives hold the URLs pointing to your old server, where you're migrating from. You can retain your copies, but if the old server stops serving these files, the links go broken. It would even take some special processing to update your message archive to update the links. And then, you "own" the files you've uploaded and shared, but then there are files which were shared with you by your buddies. No list of such files exists on any server - it's just your message archive having these URLs, most of them in a special-kind stanzas which should be possible to match. You should be able to copy these locally once you search across your message archive (you'd need to decrypt the encrypted ones for that, which would complicate the tooling needed).

No troubles, no new features/developments. So this is really a no-op report.

During this time, not much happened:

• staging server dev.decent.im shut down

  It wasn't actively used by me for few months, but it costs the same as production host. I decided to stop this waste of funds.

  Of course now I need to be more cautious when applying changes to produciton, or at last I should not start testing stuff on production server. I am thinking of making it more possible to replicate entire setup in a locla environment not having all properties of real public server (e.g. public static IP). But this is yet to be resolved in future.

• intermittently, unattended upgrade was broken because lua-expat upstream website was down:

  lua-expat author has kindly brought up the site on my request. It was brought down manually to stop DoS attack which was mounted against his website.

->  http://code.matthewwild.co.uk/lua-expat/
--> hg identify --id --rev default http://code.matthewwild.co.uk/lua-expat/ --config trusted.users=portage
abort: HTTP Error 404: Not Found
!!! Error updating dev-lua/luaexpat:0: [Exception] update command returned non-zero result
...
* Updating /usr/portage/distfiles/hg-src/luaexpat/lua-expat from http://code.matthewwild.co.uk/lua-expat/
pulling from http://code.matthewwild.co.uk/lua-expat/
abort: HTTP Error 404: Not Found

• again, one more user joined decent.im :)

Happy new 2018 year!

During last month, this happened:

• spectrum build failure was resolved

  To fix that, I submitted upstream a pull request with a trivial patch, which was accepted (not without a bit of bikeshedding with upstream maintainer).

• biboumi, the awesome IRC gateway, was integrated to production service

  biboumi is a great software, featuring a bunch of nice capabilities, for example, message archive access for IRC rooms user is attending. This means no more missed bits in IRC conversations!

  biboumi previously could use only SQLite database, which was considered by me a major downside. Finally, with a help of our very humble donation, the upstream maintainer recently added PostgreSQL support! Then, I have added it to production server (as well as staging server dev.decent.im). Existing irc.decent.im component, previously served by Spectrum & libcommini based gateway, was replaced with biboumi, for convenience and clarity. (Previously I considered putting it to horrific URI biboumi.irc.decent.im).

• one more user joined decent.im :)

  Our server is not boasting a huge user base, but I value every person who puts trust into decent.im!

It's been a while since I've put any updates here. But all this time maintenance and development efforts never completely stopped. And this time I kinda am serious about starting to keep writing service status updates regularly. Promise :)

Quite important recent change is switching from hardened-sources to gentoo-sources as the former is now unsupported since grsecurity no longer publishes updates.

Also the make.profile was changed from modified "hardened" profile to bleeding edge default/linux/amd64/17.0/no-multilib (again with a mixed in decent.im profile feature). This is to enable new compiler gcc-6.4.0 which is required to build net-im/spectrum since recently.

Last cycle of weekly automated system update failed at building net-im/spectrum-9999:

In file included from /usr/include/qt4/QtCore/qobject.h:47:0,
                 from /usr/include/qt4/QtCore/qabstracteventdispatcher.h:45,
                 from /usr/include/qt4/QtCore/QtCore:3,
                 from /var/tmp/portage/net-im/spectrum-9999/work/spectrum-9999/backends/libcommuni/main.cpp:20:
/usr/include/boost/signals/detail/signals_common.hpp:26:13: error: expected identifier before 'protected'
   namespace BOOST_SIGNALS_NAMESPACE {
             ^
/usr/include/boost/signals/detail/signals_common.hpp:26:13: error: expected unqualified-id before 'protected'
In file included from /usr/include/boost/scoped_ptr.hpp:14:0,
                 from /usr/include/boost/smart_ptr.hpp:19,
                 from /usr/include/boost/signals/connection.hpp:14,
                 from /usr/include/boost/signals/signal_template.hpp:18,
                 from /usr/include/boost/signals/signal0.hpp:24,
                 from /usr/include/boost/signal.hpp:27,
                 from /usr/include/boost/signals.hpp:9,
                 from /var/tmp/portage/net-im/spectrum-9999/work/spectrum-9999/include/Swiften/SwiftenCompat.h:50,
                 from /var/tmp/portage/net-im/spectrum-9999/work/spectrum-9999/include/transport/Config.h:29,
                 from /var/tmp/portage/net-im/spectrum-9999/work/spectrum-9999/backends/libcommuni/main.cpp:22:
/usr/include/boost/smart_ptr/scoped_ptr.hpp:26:28: error: expected '}' before end of line
 #pragma GCC diagnostic push
                            ^
/usr/include/boost/smart_ptr/scoped_ptr.hpp:26:28: error: expected declaration before end of line
make[2]: *** [backends/libcommuni/CMakeFiles/spectrum2_libcommuni_backend.dir/build.make:102: backends/libcommuni/CMakeFiles/spectrum2_libcommuni_backend.dir/main.cpp.o] Error 1
make[2]: Leaving directory '/var/tmp/portage/net-im/spectrum-9999/work/spectrum-9999'
make[1]: *** [CMakeFiles/Makefile2:646: backends/libcommuni/CMakeFiles/spectrum2_libcommuni_backend.dir/all] Error 2
make[1]: Leaving directory '/var/tmp/portage/net-im/spectrum-9999/work/spectrum-9999'
make: *** [Makefile:152: all] Error 2
 * ERROR: net-im/spectrum-9999::mva failed (compile phase):
 *   emake failed

Interestingly, this failure doesn't occur if we use net-im/swiften-9999 (currently we have 3.0 installed on production server). This will be further investigated.

Another nice pre-announcement is that, with help of our donation, the innovative IRC gateway implementation Biboumi is getting PostgreSQL support, which will enable us to have it on production system with no extra maintenance effort. Currently Biboumi supports only SQLite, which is a bit questionable in my opinion, especially with regard to taking backups of it while having service working.

Have deployed a couple of webapps for Jabber chatting and calling.

• Jappix
• JSXC

Both are not perfect, but still it's something. I hope things will get better in future.

Also this simple test webpage is quite good for testing Jabber calls capability. Just be sure to enter full JID (with resource part) into participant JID input. The full JID of the connection made by this page shows up on the page itself. WebSocket URL is wss://decent.im:5281/xmpp-websocket

For your information, I am working on implementing Jingle calls in Conversations, please join to Bountysource campaign.

Also I am thinking about adding support of videocalls into mcabber using ffmpeg. Please drop me a line if you want this, or if you want to help.

TLDR: commonly not accessible XMPP feature. Badly maintained or just absent in client implementations.

Tried to make calls with Jitsi, Gajim, Pidgin, webapps Jappix and JSXC - nothing worked in this domain setup.

But I am obsessed with an idea to bring this feature to decent.im users. Decent.im is supposed to be ass-kicking!

For your information, I am working on implementing Jingle calls in Conversations, please join to Bountysource campaign to appreciate me (and/or competitors).

Still cannot get Jappix or JSXC webapps to work with calls on this server, despite my numerous attempts.

But a kind person Philipp Hancke from operators@xmpp.org ML pointed to this brilliant test page, which just works without any issue! This testing page is by far the most accessible and convenient way to make calls with Jabber! :) Also works the same good way in both Chrome and Firefox, which feels exceptional.

In other words, you can go on and make your calls using that test page! Just be sure to enter full JID (with resource part) into participant JID input. The full JID of the connection made by this page shows up on the page itself. WebSocket URL is wss://decent.im:5281/xmpp-websocket