• chevron_right

      Keycloak Identity and Access Management on FreeBSD

      Slixfeed · Sunday, 10 March - 10:35 · 8 minutes

    Many times I wrote about FreeIPA/IDM – but I have one problem with it – its not currently possible to run FreeIPA on FreeBSD … so I searched for other open source alternatives and found Keycloak. What surprised me even more is that its even available in the FreeBSD Ports as net/keycloak port. So I wanted to check how it works/runs on FreeBSD … and this is exactly how this article happened.

    keycloak.logo

    My earlier FreeIPA/IDM attempts are below.

    First – we will create new VM for our server. I will use sysutils/vm-bhyve-devel for Bhyve but feel free to use any other hypervisor (or even w/o one). To not waste time installing I will also use provided by FreeBSD project VM-IMAGE with ZFS enabled – FreeBSD-14.0-RELEASE-amd64-zfs.raw disk0.img

    host # cat /vm/.templates/freebsd.conf
    loader="bhyveload"
    cpu=1
    memory=256M
    network0_type="virtio-net"
    network0_switch="public"
    disk0_type="nvme"
    disk0_name="disk0.img"
    
    host # vm create -t freebsd -c 2 -m 4G -s 10G keycloak
    
    host # ls -lh /vm/keycloak
    total 3402399
    -rw-------  1 root wheel   10G Mar 10 10:47 disk0.img
    -rw-r--r--  1 root wheel  209B Mar 10 07:20 keycloak.conf
    -rw-r--r--  1 root wheel   96B Mar 10 07:22 vm-bhyve.log
    
    host # cd /vm/keycloak
    
    host # rm -f disk0.img
    
    host # cp /vm/TEMPLATE/FreeBSD-14.0-RELEASE-amd64-zfs.raw disk0.img
    
    host # truncate -s 10G disk0.img
    
    host # vm start keycloak
    Starting keycloak
      * found guest in /vm/keycloak
      * booting...
    
    host # vm console keycloak
    

    Type root as user and hit [ENTER] for empty password. Now the FreeBSD setup and needed packages.

    root@freebsd:~ # :> ~/.hushlogin
    
    root@freebsd:~ # cat << EOF > /etc/rc.conf
    hostname="keycloak.lab.org"
    ifconfig_DEFAULT="inet 10.1.1.211/24"
    defaultrouter="10.1.1.1"
    growfs_enable="YES"
    zfs_enable="YES"
    sshd_enable="YES"
    postgresql_enable="YES"
    keycloak_enable="YES"
    keycloak_env="KEYCLOAK_ADMIN=admin KEYCLOAK_ADMIN_PASSWORD=password"
    EOF
    
    root@freebsd:~ # echo 10.1.1.211 keycloak.lab.org keycloak >> /etc/hosts
    
    root@freebsd:~ # mkdir -p /usr/local/etc/pkg/repos
    
    root@freebsd:~ # sed -e s/quarterly/latest/g /etc/pkg/FreeBSD.conf \
                       > /usr/local/etc/pkg/repos/FreeBSD.conf
    
    root@freebsd:~ # echo nameserver 1.1.1.1 > /etc/resolv.conf
    
    root@freebsd:~ # drill freebsd.org | grep '^[^;]'
    freebsd.org.        799     IN      A       96.47.72.84
    
    root@freebsd:~ # service netif restart
    
    root@freebsd:~ # service routing restart
    
    root@freebsd:~ # service hostname restart
    Setting hostname: keycloak.lab.org.
    
    root@keycloak:~ # passwd
    Changing local password for root
    New Password:
    Retype New Password:
    
    root@keycloak:~ # cat << EOF >> /etc/ssh/sshd_config
    PermitRootLogin yes
    UseDNS no
    EOF
    
    root@keycloak:~ # service sshd enable
    
    root@keycloak:~ # service sshd start
    
    root@keycloak:~ # exit
    

    Now switch to ssh(1) for better experience – needed to paste larger blocks of configs/text.

    host % ssh root@10.1.1.211
    
    root@keycloak:~ # pkg install -y keycloak postgresql16-server postgresql16-client
    
    root@keycloak:~ # service postgresql enable
    
    root@keycloak:~ # service postgresql initdb
    
    root@keycloak:~ # service postgresql start
    
    root@keycloak:~ # sockstat -l4
    USER     COMMAND    PID   FD  PROTO  LOCAL ADDRESS         FOREIGN ADDRESS      
    postgres postgres    2265 7   tcp4   127.0.0.1:5432        *:*
    root     syslogd      656 7   udp4   *:514                 *:*
    
    root@keycloak:~ # su - postgres -c psql
    psql (16.2)
    Type "help" for help.
    
    postgres=# ALTER USER postgres WITH PASSWORD 'password';
    
    postgres=# CREATE DATABASE keycloak with encoding 'UTF8';
    CREATE DATABASE
    
    postgres=# GRANT ALL ON DATABASE keycloak TO postgres;
    GRANT
    
    postgres=# \q
    
    root@keycloak:~ # cd /usr/local/share/java/keycloak/conf
    
    root@keycloak:~ # openssl req -x509 -newkey rsa:2048 -keyout server.key.pem -out server.crt.pem -days 36500 -nodes -subj "/C=PL/ST=lodzkie/L=Lodz/O=Vermaden/OU=HR/CN=keycloak.lab.org"
    
    root@keycloak:~ # chmod 600 server.crt.pem server.key.pem
    
    root@keycloak:~ # chown keycloak:keycloak server.crt.pem server.key.pem
    
    root@keycloak:~ # cat << EOF > /usr/local/share/java/keycloak/conf/keycloak.conf               
    db=postgres
    db-username=postgres
    db-password=password
    db-url=jdbc:postgresql://localhost:5432/keycloak
    hostname-strict-https=true
    hostname-url=https://keycloak.lab.org:8443/
    hostname-admin-url=https://keycloak.lab.org:8443/
    https-certificate-file=/usr/local/share/java/keycloak/conf/server.crt.pem
    https-certificate-key-file=/usr/local/share/java/keycloak/conf/server.key.pem
    proxy=edge
    EOF
    
    root@keycloak:~ # echo quarkus.transaction-manager.enable-recovery=true \
                        > /usr/local/share/java/keycloak/conf/quarkus.properties
    
    root@keycloak:~ # chown keycloak:keycloak /usr/local/share/java/keycloak/conf/quarkus.properties
    
    root@keycloak:~ # service keycloak enable
    
    root@keycloak:~ # service keycloak build
    The following run time non-cli properties were found, but will be ignored during build time: kc.db-url, kc.db-username, kc.db-password, kc.hostname-url, kc.hostname-admin-url, kc.hostname-strict-https, kc.https-certificate-file, kc.https-certificate-key-file, kc.proxy
    Updating the configuration and installing your custom providers, if any. Please wait.
    2024-03-10 09:01:17,701 INFO  [io.quarkus.deployment.QuarkusAugmentor] (main) Quarkus augmentation completed in 29796ms
    Server configuration updated and persisted. Run the following command to review the configuration:
    
            kc.sh show-config
    
    root@keycloak:~ # /usr/local/share/java/keycloak/bin/kc.sh show-config
    Current Mode: production
    Current Configuration:
            kc.config.built =  true (SysPropConfigSource)
            kc.db =  postgres (PropertiesConfigSource)
            kc.db-password =  ******* (PropertiesConfigSource)
            kc.db-url =  jdbc:postgresql://localhost:5432/keycloak (PropertiesConfigSource)
            kc.db-username =  postgres (PropertiesConfigSource)
            kc.hostname-admin-url =  https://keycloak.lab.org:8443/ (PropertiesConfigSource)
            kc.hostname-strict-https =  true (PropertiesConfigSource)
            kc.hostname-url =  https://keycloak.lab.org:8443/ (PropertiesConfigSource)
            kc.https-certificate-file =  /usr/local/share/java/keycloak/conf/server.crt.pem (PropertiesConfigSource)
            kc.https-certificate-key-file =  /usr/local/share/java/keycloak/conf/server.key.pem (PropertiesConfigSource)
            kc.log-console-output =  default (PropertiesConfigSource)
            kc.log-file =  ${kc.home.dir:default}${file.separator}data${file.separator}log${file.separator}keycloak.log (PropertiesConfigSource)
            kc.optimized =  true (PersistedConfigSource)
            kc.proxy =  edge (PropertiesConfigSource)
            kc.spi-hostname-default-admin-url =  https://keycloak.lab.org:8443/ (PropertiesConfigSource)
            kc.spi-hostname-default-hostname-url =  https://keycloak.lab.org:8443/ (PropertiesConfigSource)
            kc.spi-hostname-default-strict-https =  true (PropertiesConfigSource)
            kc.version =  23.0.6 (SysPropConfigSource)
    
    

    We now have needed packages installed. Self signed certificate for HTTPS generated. PostgreSQL database and Keycloak configured. We will need small patch to enable passing env(1) variables at the Keycloak daemon start. It will allow to use keycloak_env at the /etc/rc.conf main FreeBSD config file. This is needed to configure the initial admin user as sated in the Keycloak documentation.

    keycloak-0-initial-admin-user

    Now back to the patch.

    root@keycloak:~ # cat /root/keycloak.patch
    --- /root/keycloak      2024-03-08 11:46:21.847315000 +0000
    +++ /usr/local/etc/rc.d/keycloak        2024-03-08 11:47:22.027102000 +0000
    @@ -28,6 +28,7 @@
     : ${keycloak_enable:=NO}
     : ${keycloak_user:=keycloak}
     : ${keycloak_group:=keycloak}
    +: ${keycloak_env:=""}
     : ${keycloak_flags="start"}
     : ${keycloak_java_home="/usr/local/openjdk17"}
     
    @@ -54,6 +55,7 @@
     
            echo "Starting keycloak."
             ${command} ${command_args} \
    +                env ${keycloak_env} \
                     /usr/local/share/java/keycloak/bin/kc.sh \
                     ${keycloak_flags}
     }
    
    root@keycloak:~ # cd /usr/local/etc/rc.d
    
    root@keycloak:/usr/local/etc/rc.d # patch < /root/keycloak.patch
    Hmm...  Looks like a unified diff to me...
    The text leading up to this was:
    --------------------------
    |--- /root/keycloak      2024-03-08 11:46:21.847315000 +0000
    |+++ /usr/local/etc/rc.d/keycloak        2024-03-08 11:47:22.027102000 +0000
    --------------------------
    Patching file keycloak using Plan A...
    Hunk #1 succeeded at 28.
    Hunk #2 succeeded at 55 with fuzz 2.
    Hmm...  Ignoring the trailing garbage.
    done
    
    

    Now we will start Keycloak. Its possible to track its startup process in the /var/log/keycloak/keycloak.out file. Below You will find last 4 lines that you want to see – with Keycloak 23.0.6 on JVM (powered by Quarkus 3.2.10.Final) started in 19.251s. message 🙂

    root@keycloak:~ # service keycloak start
    
    root@keycloak:~ # tail -f /var/log/keycloak/keycloak.out
    (...)
    2024-03-10 09:12:15,550 INFO  [io.quarkus] (main) Keycloak 23.0.6 on JVM (powered by Quarkus 3.2.10.Final) started in 19.251s. Listening on: http://0.0.0.0:8080 and https://0.0.0.0:8443
    2024-03-10 09:12:15,551 INFO  [io.quarkus] (main) Profile prod activated. 
    2024-03-10 09:12:15,552 INFO  [io.quarkus] (main) Installed features: [agroal, cdi, hibernate-orm, jdbc-h2, jdbc-mariadb, jdbc-mssql, jdbc-mysql, jdbc-oracle, jdbc-postgresql, keycloak, logging-gelf, micrometer, narayana-jta, reactive-routes, resteasy-reactive, resteasy-reactive-jackson, smallrye-context-propagation, smallrye-health, vertx]
    2024-03-10 09:12:16,303 INFO  [org.keycloak.services] (main) KC-SERVICES0009: Added user 'admin' to realm 'master'
    [CTRL]-[C]
    
    root@keycloak:~ # top -ab -o res 10
    last pid:  3067;  load averages:  0.50,  0.47,  0.42  up 0+02:56:35    09:19:04
    18 processes:  1 running, 17 sleeping
    CPU:  1.4% user,  0.0% nice,  0.4% system,  0.2% interrupt, 98.0% idle
    Mem: 299M Active, 176M Inact, 3247M Wired, 264K Buf, 202M Free
    ARC: 2965M Total, 902M MFU, 1982M MRU, 4096B Anon, 12M Header, 50M Other
         2766M Compressed, 2934M Uncompressed, 1.06:1 Ratio
    Swap: 1024M Total, 1024M Free
    
      PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
     2981 keycloak     41  68    0  1425M   299M uwait    1   0:37   0.00% /usr/local/openjdk17/bin/java -Dkc.config.built=true -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Dfile.encoding=UTF-8 -Dsun.stdout.encoding=UTF-8 -Dsun.err.encoding=UTF-8 -Dstdout.encoding=UTF-8 -Dstderr.encoding=UTF-8 -XX:+ExitOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -XX:+UseParallelGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:FlightRecorderOptions=stackdepth=512 --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.security=ALL-UNNAMED -Dkc.home.dir=/usr/local/share/java/keycloak/bin/.. -Djboss.server.config.dir=/usr/local/share/java/keycloak/bin/../conf -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Dquarkus-log-max-startup-records=10000 -cp /usr/local/share/java/keycloak/bin/../lib/quarkus-run.jar io.quarkus.bootstrap.runner.QuarkusEntryPoint start
     3063 postgres      1  24    0   181M    49M kqread   1   0:00   0.00% postgres: postgres keycloak 127.0.0.1(21936) idle (postgres)
     2266 postgres      1  20    0   178M    48M kqread   1   0:00   0.00% postgres: checkpointer  (postgres)
     3062 postgres      1  20    0   181M    47M kqread   1   0:00   0.00% postgres: postgres keycloak 127.0.0.1(22820) idle (postgres)
     2270 postgres      1  20    0   179M    31M kqread   0   0:00   0.00% postgres: autovacuum launcher  (postgres)
     2271 postgres      1  20    0   179M    31M kqread   0   0:00   0.00% postgres: logical replication launcher  (postgres)
     2269 postgres      1  20    0   178M    31M kqread   0   0:00   0.00% postgres: walwriter  (postgres)
     2267 postgres      1  20    0   178M    31M kqread   0   0:00   0.00% postgres: background writer  (postgres)
     2265 postgres      1  20    0   178M    30M kqread   0   0:00   0.00% /usr/local/bin/postgres -D /var/db/postgres/data16
     2420 root          1  20    0    22M    11M select   1   0:01   0.00% sshd: root@pts/0 (sshd)
    
    

    Add also on the host system the IP information to the /etc/hosts file and check https://keycloak.lab.org:8443 in your browser.

    host # echo 10.1.1.211 keycloak.lab.org keycloak >> /etc/hosts
    
    host % firefox 'https://keycloak.lab.org:8443'
    

    As we use self signed certificate You will be warned by potential security risk. Hit ‘Advanced’ and then ‘Accept the Risk and Continue’ buttons.

    keycloak-1-self-cert

    Next click the Administration Console link.

    keycloak-2-main-page

    Login with admin and password (or your password if You used other one).

    keycloak-3-admin-login

    … and You can now create your new realm, add users, create groups etc. You have fully working Keycloak in production mode.

    keycloak-4-admin-console

    Now … like with FreeIPA/IDM – it would be nice to attach FreeBSD to it so one could login to FreeBSD system with Keycloak user … not so fast unfortunately. To make such things be possible You need a PAM module for Keycloak … and I was not able to find one that will work on FreeBSD … and the Keycloak package also comes without one.

    root@keycloak:~ # pkg info -l keycloak | grep -i pam
    root@keycloak:~ # 
    

    After grepping the Internet I found two solutions … but only for Linux.

    One of them was a step by step Keycloak PAM Module Development Tutorial guide which showed you how to write such PAM module.

    pam-dev

    The other one was Keycloak SSH PAM project on GitHub which provided more or less ready solution for Linux systems.

    pam-kc

    So while with FreeIPA/IDM we had server on Linux that allowed to connect FreeBSD systems to it – we now hat Keycloak server hosted on FreeBSD that allows connecting Linux systems 🙂

    Not much of an improvement – but maybe someone will find that guide useful.

    EOF
    • chevron_right

      Connect FreeBSD 14.0-STABLE to FreeIPA/IDM

      Slixfeed · Wednesday, 6 March - 15:42 · 21 minutes

    In the open source world everything lives/evolves/changes. This is why the new version of connecting latest FreeBSD 14.0-STABLE system to the FreeIPA/IDM is needed. One of the things that changed is that security/sssd is now deprecated and security/sssd2 is its successor. Also new version of ports-mgmt/poudriere-devel is available – with needed fixes already merged – and also with new restyled web interface.

    FreeIPA-logo

    I already messed with that topic several times in the past:

    This article will try to address and contain all steps needed – including setting up the FreeIPA/IDM server and including the Poudriere setup. Below You will find Table of Contents for this article. All of these systems will be Bhyve virtual machines.

    • FreeIPA/IDM Server – Installation
    • FreeIPA/IDM Server – Configuration
    • Poudriere Server – Setup
    • Poudriere Server – Build FreeIPA/IDM Client Packages
    • Poudriere Server – Update Repo/Packages
    • FreeBSD 14.0-STABLE Client – Setup
    • FreeBSD 14.0-STABLE Client – Debug Commands
    • Summary

    The FreeBSD project recently started to provide ZFS based VM images … but unfortunately only for 14.0-RELEASE and they are not created for 14.0-STABLE or 15-CURRENT versions – so we will use the UFS based ones for both Poudriere server and FreeBSD FreeIPA/IDM client. For the record – https://download.freebsd.org/snapshots/VM-IMAGES/14.0-STABLE/amd64/Latest/ – they are available here.

    Some note about commands run in this article – different colors for various hosts.

    host # top -ba -o res 3                                  // executed on the host system
    [root@idm ~]# yum update -y                              // executed on IDM server
    root@freebsd:~ # geom disk list                          // executed on Poudriere server
    root@poudriere-devel-14-stable:~ # poudriere ports -l    // executed on Poudriere server
    root@idm-client:~ # hostname idm-client.lab.org          // executed on IDM client (FreeBSD)
      important information                                  // marked as GREEN color

    For the FreeIPA/IDM server I have used Alma Linux RHEL clone – but we know that Rocky Linux or Oracle Linux would also work well. We will use three systems in this article.

    FreeIPA/IDM server – with idm.lab.org hostname.

          OS: Alma Linux
          IP: 10.0.0.200/24
          GW: 10.0.0.1
      domain: lab.org
       realm: LAB.ORG
    hostname: idm.lab.org
    

    Poudriere builder system – with poudriere-devel-14-stable.lab.org hostname.

          OS: FreeBSD 14.0-STABLE
          IP: 10.0.0.124/24
          GW: 10.0.0.1
         DNS: 1.1.1.1
      domain: -
       realm: -
    hostname: poudriere-devel-14-stable.lab.org
    

    FreeBSD client for FreeIPA/IDM system – with idm-client.lab.org hostname.

          OS: FreeBSD 14.0-STABLE
          IP: 10.0.0.233/24
          GW: 10.0.0.1
         DNS: 10.0.0.200
      domain: lab.org
       realm: LAB.ORG
    hostname: idm-client.lab.org
    

    I really like the FreeBSD Bhyve memory ballooning – which means the guest VMs only take as much RAM as guest OS allocated and not 12 GB RAM as is configured.

    host # vm list | grep -e STATE -e Running
    NAME                       DATASTORE  LOADER     CPU  MEMORY  VNC           AUTO     STATE
    idm                        default    uefi       2    4g      0.0.0.0:5900  No       Running (25284)
    idm-client-14-stable       default    bhyveload  2    1g      -             No       Running (29517)
    poudriere-devel-14-stable  default    bhyveload  8    12g     -             Yes [1]  Running (23419)
    
    host # top -ba -o res 3
    last pid:  1290;  load averages:  0.09,  0.09,  0.08  up 0+00:47:08    07:05:20
    32 processes:  1 running, 31 sleeping
    CPU:  0.0% user,  0.0% nice,  0.6% system,  0.0% interrupt, 99.4% idle
    Mem: 2983M Active, 463M Inact, 1060M Wired, 56K Buf, 27G Free
    ARC: 619M Total, 115M MFU, 497M MRU, 32K Anon, 2346K Header, 4231K Other
         551M Compressed, 1080M Uncompressed, 1.96:1 Ratio
    Swap: 4096M Total, 4096M Free
    
      PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
    25284 root         13  20    0  4159M  1168M kqread  13   3:12   1.27% bhyve: idm (bhyve)
    23419 root         19  20    0    12G   109M kqread  15   0:27   0.00% bhyve: poudriere-devel-14-stable (bhyve)
    29517 root         13  20    0  1075M    77M kqread   5   0:20   0.00% bhyve: idm-client-14-stable (bhyve)
    
    

    As you can see I am using sysutils/vm-bhyve-devel for the Bhyve management – but You may as well use bare /usr/share/examples/bhyve/vmrun.sh instead … or even entirely different hypervisor like KVM on Linux or VirtualBox on Windows – it does not matter as long as machines have access to the Internet and they see each other in the same LAN network.

    FreeIPA/IDM Server – Installation

    I installed the Alma Linux some time ago – so the screenshot shows older 8.7 version.

    LAB.IDM.Server.ROOT

    After reboot its network is configured as shown below.

    [root@idm ~]# cat /etc/sysconfig/network-scripts/ifcfg-enp0s3
    TYPE=Ethernet
    PROXY_METHOD=none
    BROWSER_ONLY=no
    BOOTPROTO=none
    DEFROUTE=yes
    IPV4_FAILURE_FATAL=no
    IPV6INIT=no
    IPV6_DEFROUTE=yes
    IPV6_FAILURE_FATAL=no
    IPV6_ADDR_GEN_MODE=eui64
    NAME=enp0s3
    UUID=120efe1f-3cb6-40cf-8aad-b17066c08543
    DEVICE=enp0s3
    ONBOOT=yes
    IPADDR=10.0.0.200
    PREFIX=24
    GATEWAY=10.0.0.1
    DNS1=1.1.1.1
    IPV6_DISABLED=yes
    

    Some more basic setup commands below.

    
    [root@idm ~]# echo 10.0.0.200 idm.lab.org idm >> /etc/hosts
    
    [root@idm ~]# cat << EOF >> /etc/sysctl.conf
    # DISABLE IPv6 FOR MAIN enp0s3 INTERFACE
    net.ipv6.conf.enp0s3.disable_ipv6=1
    EOF
    
    [root@idm ~]# hostnamectl set-hostname idm.lab.org
    
    [root@idm ~]# timedatectl set-timezone Europe/Warsaw
    
    [root@idm ~]# timedatectl set-local-rtc 0
    
    [root@idm ~]# yum update -y
    
    [root@idm ~]# reboot
    

    Continuation after reboot.

    [root@idm ~]# yum module enable idm:DL1 -y
    
    [root@idm ~]# yum distro-sync -y
    
    [root@idm ~]# yum install -y bind-utils chrony nc
    
    [root@idm ~]# ipa-server-install                        \
                        --domain lab.org                    \
                        --realm LAB.ORG                     \
                        --reverse-zone=0.0.10.in-addr.arpa. \
                        --allow-zone-overlap                \
                        --no-forwarders                     \
                        --ntp-pool pool.ntp.org             \
                        --setup-dns                         \
                        --ds-password    password           \
                        --admin-password password           \
                        --unattended
    
    [root@idm ~]# ipactl status
    Directory Service: RUNNING
    krb5kdc Service: RUNNING
    kadmin Service: RUNNING
    named Service: RUNNING
    httpd Service: RUNNING
    ipa-custodia Service: RUNNING
    pki-tomcatd Service: RUNNING
    ipa-otpd Service: RUNNING
    ipa-dnskeysyncd Service: RUNNING
    ipa: INFO: The ipactl command was successful
    
    [root@idm ~]# systemctl list-unit-files | grep ipa | grep service
    ipa-ccache-sweep.service                   static   
    ipa-custodia.service                       disabled 
    ipa-dnskeysyncd.service                    disabled 
    ipa-healthcheck.service                    disabled 
    ipa-ods-exporter.service                   disabled 
    ipa-otpd@.service                          static   
    ipa.service                                enabled
    
    [root@idm ~]# systemctl enable --now httpd
    
    [root@idm ~]# systemctl list-unit-files | grep httpd.service
    httpd.service                              enabled  
    
    [root@idm ~]# systemctl disable firewalld
    
    [root@idm ~]# systemctl stop    firewalld
    
    [root@idm ~]# cat /etc/sssd/sssd.conf
    [domain/lab.org]
      ipa_server_mode                = True
      ipa_server                     = idm.lab.org
      ipa_hostname                   = idm.lab.org
      ipa_domain                     = lab.org
      id_provider                    = ipa
      auth_provider                  = ipa
      chpass_provider                = ipa
      access_provider                = ipa
      cache_credentials              = True
      ldap_tls_cacert                = /etc/ipa/ca.crt
      krb5_store_password_if_offline = True
    
    [sssd]
      services = nss, pam, ifp, ssh, sudo
      domains  = lab.org
    
    [nss]
      homedir_substring = /home
      memcache_timeout  = 600
    
    [pam]
    
    [sudo]
    
    [autofs]
    
    [ssh]
    
    [pac]
    
    [ifp]
      allowed_uids = ipaapi, root
    
    [session_recording]
    

    If you would like to see what a successful ipa-server-install(8) looks like – you can take a look HERE.

    We have our FreeIPA/IDM server installed.

    You will need to add 10.0.0.200 as idm.lab.org to your /etc/hosts on the system where you will be using the browser (or to your local DNS).

    host # grep idm /etc/hosts
    10.0.0.200  idm.lab.org  idm
    

    You can login to it typing https://10.0.0.200 at your local browser – you will be redirected to https://idm.lab.org/ipa/ui/ immediately and you will see the login page as shown below.

    FreeIPA-login-1

    You may login with admin username and the password you specified for the ipa-server-install(8) command (or password if you just copy pasted that command 🙂

    FreeIPA/IDM Server – Configuration

    … and after logging in I created a regular vermaden user as shown below.

    FreeIPA-login-2

    Keep in mind to reset your password by connecting to FreeIPA/IDM server.

    host # ssh -l vermaden 10.0.0.200
    (vermaden@10.0.0.200) Password:
    (vermaden@10.0.0.200) Password expired. Change your password now.
    Current Password:
    (vermaden@10.0.0.200) New password:
    (vermaden@10.0.0.200) Retype new password:
    Last failed login: Wed Oct 19 00:47:57 CEST 2022 from 10.0.0.33 on ssh:notty
    There was 1 failed login attempt since the last successful login.
    
    [vermaden@idm /]$ w
     12:58:50 up  6:39,  1 user,  load average: 0.02, 0.05, 0.00
    USER     TTY      FROM             LOGIN@   IDLE   JCPU   PCPU WHAT
    vermaden pts/0    10.0.0.4         12:58    1.00s  0.04s  0.01s w
    

    The more important configuration is in HBAC and Sudo rules.

    Here are HBAC related settings.

    idm-1-hbac-rules-menu

    idm-2-hbac-rules-menu

    idm-3-hbac-rules-freebsd-details

    … and the Sudo part.

    idm-4-sudo-rules-menu

    idm-5-sudo-rules-freebsd

    idm-6-sudo-rules-freebsd-details

    Poudriere Server – Setup

    One note for the FreeBSD setups below – please use /bin/sh shell (default for root since 14.0-RELEASE) for the commands … or zsh(1) for example … or other POSIX compatible shell. Some of these commands may not work properly on ‘C’ based shells or in fish(1) shell.

    This is the template I used for Bhyve VMs.

    host # cat /vm/.templates/freebsd.conf 
    loader="bhyveload"
    cpu=1
    memory=256M
    network0_type="virtio-net"
    network0_switch="public"
    disk0_type="nvme"
    disk0_name="disk0.img"
    

    We will now create poudriere-devel-14-stable VM for Poudriere server.

    host # vm create -t freebsd -s 20g -m 12g -c 8 poudriere-devel-14-stable
    
    host # du -sgA /vm/poudriere-devel-14-stable/*
    20      /vm/poudriere-devel-14-stable/disk0.img
    1       /vm/poudriere-devel-14-stable/poudriere-devel-14-stable.conf
    1       /vm/poudriere-devel-14-stable/vm-bhyve.log
    

    Now we will replace disk0.img with Latest FreeBSD 14.0-STABLE snapshot.

    host # fetch -o - 'https://download.freebsd.org/snapshots/VM-IMAGES/14.0-STABLE/amd64/Latest/FreeBSD-14.0-STABLE-amd64.raw.xz' \
             | xz -d > /vm/poudriere-devel-14-stable/disk0.img
    
    host # file -b /vm/poudriere-devel-14-stable/disk0.img 
    DOS/MBR boot sector; partition 1 : ID=0xee, start-CHS (0x0,0,2), end-CHS (0x3ff,255,63), startsector 1, 12649684 sectors
    
    host # du -sgA /vm/poudriere-devel-14-stable/*
    7       /vm/poudriere-devel-14-stable/disk0.img
    1       /vm/poudriere-devel-14-stable/poudriere-devel-14-stable.conf
    1       /vm/poudriere-devel-14-stable/vm-bhyve.log
    

    We now need to add additional disk1.img disk to for ZFS pool.

    host # truncate -s 10G /vm/poudriere-devel-14-stable/disk0.img
    
    host # vm add -d disk -t file -s 100g poudriere-devel-14-stable
    
    host # vm info poudriere-devel-14-stable | grep -A 16 virtual-disk
      virtual-disk
        number: 0
        device-type: file
        emulation: nvme
        options: -
        system-path: /vm/poudriere-devel-14-stable/disk0.img
        bytes-size: 10737418240 (10.000G)
        bytes-used: 1720046592 (1.601G)
    
      virtual-disk
        number: 1
        device-type: file
        emulation: nvme
        options: -
        system-path: /vm/poudriere-devel-14-stable/disk1.img
        bytes-size: 107374182400 (100.000G)
        bytes-used: 1024 (1.000K)
    
    host # du -sgA /vm/poudriere-devel-14-stable/*
    10      /vm/poudriere-devel-14-stable/disk0.img
    100     /vm/poudriere-devel-14-stable/disk1.img
    1       /vm/poudriere-devel-14-stable/poudriere-devel-14-stable.conf
    1       /vm/poudriere-devel-14-stable/vm-bhyve.log
    

    Now internally inside VM.

    host # vm start poudriere-devel-14-stable                              
    Starting poudriere-devel-14-stable
      * found guest in /vm/poudriere-devel-14-stable
      * booting...
    
    host # vm console poudriere-devel-14-stable
    (...)
    Starting devd.
    Starting dhclient.
    DHCPDISCOVER on vtnet0 to 255.255.255.255 port 67 interval 4
    DHCPOFFER from 10.0.0.1
    DHCPREQUEST on vtnet0 to 255.255.255.255 port 67
    DHCPACK from 10.0.0.1
    bound to 10.0.0.23 -- renewal in 43200 seconds.
    add host 127.0.0.1: gateway lo0 fib 0: route already in table
    add host ::1: gateway lo0 fib 0: route already in table
    add net fe80::: gateway ::1
    add net ff02::: gateway ::1
    add net ::ffff:0.0.0.0: gateway ::1
    add net ::0.0.0.0: gateway ::1
    Updating motd:.
    Updating /var/run/os-release done.
    Clearing /tmp (X related).
    Creating and/or trimming log files.
    Starting syslogd.
    Mounting late filesystems:.
    Starting cron.
    Starting background file system checks in 60 seconds.
    
    Wed Mar  6 08:23:03 UTC 2024
    
    FreeBSD/amd64 (freebsd) (ttyu0)
    
    login: 
    
    
    

    Use the root user with ’empty’ password – just hit [ENTER] key on password prompt.

    root@freebsd:~ # :> ~/.hushlogin
    
    root@freebsd:~ # passwd root
    Changing local password for root
    New Password:
    Retype New Password:
    
    root@freebsd:~ # geom disk list
    Geom name: nda0
    Providers:
    1. Name: nda0
       Mediasize: 10737418240 (10G)
       Sectorsize: 512
       Mode: r3w3e8
       descr: bhyve-NVMe
       lunid: 589cfc2012350001
       ident: NVME-4-0
       rotationrate: 0
       fwsectors: 0
       fwheads: 0
    
    Geom name: nda1
    Providers:
    1. Name: nda1
       Mediasize: 107374182400 (100G)
       Sectorsize: 512
       Mode: r0w0e0
       descr: bhyve-NVMe
       lunid: 589cfc20d2f40001
       ident: NVME-4-1
       rotationrate: 0
       fwsectors: 0
       fwheads: 0
    
    root@freebsd:~ # zpool create zroot nda1
    ZFS filesystem version: 5
    ZFS storage pool version: features support (5000)
    
    root@freebsd:~ # zfs set mountpoint=none zroot
    
    root@freebsd:~ # zfs list
    NAME    USED  AVAIL  REFER  MOUNTPOINT
    zroot   100K  96.4G    24K  none
    

    Now some basic configuration.

    root@freebsd:~ # cat /etc/rc.conf
    hostname="poudriere-devel-14-stable.lab.org"
    ifconfig_DEFAULT="inet 10.0.0.124/24 up"
    defaultrouter="10.0.0.1"
    zfs_enable="YES"
    sshd_enable="YES"
    nginx_enable="YES"
    
    root@freebsd:~ # cat /etc/hosts
    ::1         localhost  localhost.my.domain
    127.0.0.1   localhost  localhost.my.domain
    10.0.0.124  poudriere-devel-14-stable.lab.org  poudriere-devel-14-stable
    
    root@freebsd:~ # service sshd start
    
    root@freebsd:~ # mkdir -p /usr/local/etc/pkg/repos
    
    root@freebsd:~ # sed -e s/quarterly/latest/g /etc/pkg/FreeBSD.conf \
                       > /usr/local/etc/pkg/repos/FreeBSD.conf
    
    root@freebsd:~ # pkg install -y    \
                       beadm           \
                       lsblk           \
                       poudriere-devel \
                       nginx           \
                       git-lite        \
                       ccache4         \
                       tree
    
    root@freebsd:~ # reboot
    

    Fortunately we do not need to patch ports-mgmt/poudriere-devel anymore as the -u flag for sort(1) is already there.

    root@poudriere-devel-14-stable:~ # grep remote_all_ /usr/local/share/poudriere/common.sh | grep sort
                "${remote_all_options}" | sort -k1.2 -u | paste -s -d ' ' -)
                "${remote_all_dept}" | sort -u | paste -s -d ' ' -)
    

    We will now setup actual Poudriere server.

    root@poudriere-devel-14-stable:~ # export SSL=/usr/local/etc/ssl
    
    root@poudriere-devel-14-stable:~ # mkdir -p \
                                         /usr/ports/distfiles \
                                         ${SSL}/keys \
                                         ${SSL}/certs
    
    root@poudriere-devel-14-stable:~ # chmod 0600 ${SSL}/keys
    
    root@poudriere-devel-14-stable:~ # openssl genrsa -out ${SSL}/keys/poudriere.key 4096
    
    root@poudriere-devel-14-stable:~ # openssl rsa \
                                         -in  ${SSL}/keys/poudriere.key -pubout \
                                         -out ${SSL}/certs/poudriere.cert
    
    root@poudriere-devel-14-stable:~ # zfs create -p -o mountpoint=/var/ccache zroot/var/ccache
    
    root@poudriere-devel-14-stable:~ # zfs list
    NAME               USED  AVAIL  REFER  MOUNTPOINT
    zroot              213K  96.4G    24K  none
    zroot/var           48K  96.4G    24K  none
    zroot/var/ccache    24K  96.4G    24K  /var/ccache
    
    root@poudriere-devel-14-stable:~ # export IP=10.0.0.124
    
    root@poudriere-devel-14-stable:~ # cat << EOF > /usr/local/etc/poudriere.conf
    ZPOOL=zroot
    BASEFS=/usr/local/poudriere
    ZROOTFS=/usr/local/poudriere
    FREEBSD_HOST=ftp://ftp.freebsd.org
    POUDRIERE_DATA=/usr/local/poudriere/data
    CHECK_CHANGED_OPTIONS=verbose
    CHECK_CHANGED_DEPS=yes
    PKG_REPO_SIGNING_KEY=/usr/local/etc/ssl/keys/poudriere.key
    URL_BASE=http://${IP}/
    USE_TMPFS=no
    TMPFS_LIMIT=12
    MAX_MEMORY=12
    PARALLEL_JOBS=8
    PREPARE_PARALLEL_JOBS=8
    MAX_FILES=4096
    DISTFILES_CACHE=/usr/ports/distfiles
    KEEP_OLD_PACKAGES=yes
    KEEP_OLD_PACKAGES_COUNT=3
    CHECK_CHANGED_OPTIONS=verbose
    CHECK_CHANGED_DEPS=yes
    CCACHE_DIR=/var/ccache
    RESTRICT_NETWORKING=no
    EOF
    
    root@poudriere-devel-14-stable:~ # mkdir -p /usr/local/poudriere/data/logs/bulk
    
    root@poudriere-devel-14-stable:~ # ln -s \
                                         /usr/local/etc/ssl/certs/poudriere.cert \
                                         /usr/local/poudriere/data/logs/bulk/poudriere.cert
    
    root@poudriere-devel-14-stable:~ # service nginx enable
    
    root@poudriere-devel-14-stable:~ # sed -i '' -E 's|text/plain[\t\ ]*txt|text/plain txt log|g' /usr/local/etc/nginx/mime.types
    
    root@poudriere-devel-14-stable:~ # export IP=10.0.0.124
    
    root@poudriere-devel-14-stable:~ # cat << EOF > /usr/local/etc/nginx/nginx.conf
    events {
      worker_connections 1024;
    }
    
    http {
      include      mime.types;
      default_type application/octet-stream;
    
      server {
        listen 80 default;
        server_name ${IP};
        root /usr/local/share/poudriere/html;
    
        location /data {
          alias /usr/local/poudriere/data/logs/bulk;
          autoindex on;
        }
    
        location /packages {
          root /usr/local/poudriere/data;
          autoindex on;
        }
      }
    }
    EOF
    
    root@poudriere-devel-14-stable:~ # service nginx restart
    
    root@poudriere-devel-14-stable:~ # mkdir -p /root/.cache/ccache                                  
    
    root@poudriere-devel-14-stable:~ # ln -sf /var/ccache /root/.cache/ccache
    
    root@poudriere-devel-14-stable:~ # cat << EOF > /usr/local/etc/poudriere.d/make.conf
    ALLOW_UNSUPPORTED_SYSTEM=yes
    DISABLE_LICENSES=yes
    EOF
    
    root@poudriere-devel-14-stable:~ # cat << EOF > /var/ccache/ccache.conf
    max_size = 0
    cache_dir = /var/ccache
    base_dir = /var/ccache
    hash_dir = false
    EOF
    
    root@poudriere-devel-14-stable:~ # poudriere jail -c -j 14-0-S-amd64 -v 14.0-STABLE
    (...)
    [00:20:45] Jail 14-0-S-amd64 14.0-STABLE amd64 is ready to be used
    
    root@poudriere-devel-14-stable:~ # poudriere jail -l
    JAILNAME     VERSION     ARCH  METHOD TIMESTAMP           PATH
    14-0-S-amd64 14.0-STABLE amd64 http   2024-03-06 09:44:27 /usr/local/poudriere/jails/14-0-S-amd64
    
    root@poudriere-devel-14-stable:~ # poudriere ports -c -p idm
    [00:00:00] Creating idm fs at /usr/local/poudriere/ports/idm... done
    [00:00:00] Cloning the ports tree... done
    
    root@poudriere-devel-14-stable:~ # poudriere ports -l
    PORTSTREE METHOD    TIMESTAMP           PATH
    idm       git+https 2024-03-06 10:10:53 /usr/local/poudriere/ports/idm
    
    
    

    Poudriere Server – Build FreeIPA/IDM Client Packages

    Now we will choose needed options for our FreeBSD Ports and then start the bulk process of fetching and building them.

    root@poudriere-devel-14-stable:~ # poudriere options -c -n -p idm security/cyrus-sasl2-gssapi
    //   SELECT: (*) GSSAPI_MIT
    
    root@poudriere-devel-14-stable:~ # poudriere options -c -n -p idm net/openldap26-client
    //   SELECT: [x] GSSAPI
    
    root@poudriere-devel-14-stable:~ # poudriere options -c -n -p idm security/sudo
    // DESELECT: [ ] PAM
    //   SELECT: (*) GSSAPI_MIT
    //   SELECT: (*) SSSD2
    
    root@poudriere-devel-14-stable:~ # cat << EOF > /usr/local/etc/poudriere.d/idm
    security/krb5
    security/sudo
    security/sssd2
    security/cyrus-sasl2
    security/cyrus-sasl2-gssapi
    security/pam_mkhomedir
    net/openldap26-client
    net/samba416
    EOF
    
    root@poudriere-devel-14-stable:~ # poudriere bulk -j 14-0-S-amd64 -b latest -p idm -f /usr/local/etc/poudriere.d/idm
    
    root@poudriere-devel-14-stable:~ # zfs list
    NAME                                           USED  AVAIL  REFER  MOUNTPOINT
    zroot                                         1.51G  94.9G    24K  none
    zroot/usr                                     1.39G  94.9G    24K  none
    zroot/usr/local                               1.39G  94.9G    24K  none
    zroot/usr/local/poudriere                     1.39G  94.9G    24K  none
    zroot/usr/local/poudriere/jails               1.07G  94.9G    24K  none
    zroot/usr/local/poudriere/jails/14-0-S-amd64  1.07G  94.9G  1.07G  /usr/local/poudriere/jails/14-0-S-amd64
    zroot/usr/local/poudriere/ports                328M  94.9G    24K  none
    zroot/usr/local/poudriere/ports/idm            328M  94.9G   328M  /usr/local/poudriere/ports/idm
    zroot/var                                      117M  94.9G    24K  none
    zroot/var/ccache                               117M  94.9G   117M  /var/ccache
    
    
    

    This is how the Poudriere build process looks like from the terminal … and a view for its new ZFS datasets that Poudriere created.

    xterm-poudriere

    It was 2nd or 3rd run so when You first will run the bulk there will be more information about fetching packages etc.

    Below You can see what processes are running in htop(1) during the build.

    xterm-htop

    You can also follow the status of the build process in the browser at https://10.0.0.124 page.

    poudriere-devel-100-latest-builds

    Generally the new Poudriere interface is quite ‘large’ I would say – so I use it at 70% scale/zoom on Firefox and IMHO its more usable like that.

    poudriere-devel-70-latest-builds

    And below are the details about our build job.

    poudriere-devel-70-build-complete

    Poudriere Server – Update Repo/Packages

    Everytime you will need to update the packages in that FreeIPA/IDM repo You will need to run these commands.

    root@poudriere-devel-14-stable:~ # poudriere ports -u -p idm
    
    root@poudriere-devel-14-stable:~ # poudriere bulk -j 14-0-S-amd64 -b latest -p idm -f /usr/local/etc/poudriere.d/idm
    

    You may as well update the FreeBSD Jail when needed.

    root@poudriere-devel-14-stable:~ # poudriere jail -u -j 14-0-S-amd64
    

    FreeBSD 14.0-STABLE Client – Setup

    I will not repeat the process – but the same as with Poudriere server – you need to create FreeBSD client – for example as Bhyve VM.

    Now – the needed configuration on FreeBSD 14.0-STABLE system to connect it to FreeIPA/IDM server.

    root@idm-client:~ # :> ~/.hushlogin
    
    root@idm-client:~ # mkdir -p              \
                         /usr/local/etc/ipa   \
                         /var/log/sssd        \
                         /var/run/sss/private \
                         /var/db/sss
    
    root@idm-client:~ # echo '10.0.0.233  idm-client.lab.org  idm-client' >> /etc/hosts
    
    root@idm-client:~ # echo '10.0.0.200  idm.lab.org         idm'        >> /etc/hosts
    
    root@idm-client:~ # hostname idm-client.lab.org
    
    root@idm-client:~ # sysrc hostname=idm-client.lab.org
    
    root@idm-client:~ # fetch -o /usr/local/etc/ipa/ca.crt http://idm.lab.org/ipa/config/ca.crt
    
    

    Now we will need to add or FreeBSD client to FreeIPA/IDM. Instructions below.

    [root@idm ~]# kinit admin
    
    [root@idm ~]# ipa dnsrecord-add lab.org idm-client --a-rec=10.0.0.233 --a-create-reverse
      Record name: idm-client
      A record: 10.0.0.233
    
    [root@idm ~]# ipa host-add idm-client.lab.org
    -------------------------------
    Added host "idm-client.lab.org"
    -------------------------------
      Host name: idm-client.lab.org
      Principal name: host/idm-client.lab.org@LAB.ORG
      Principal alias: host/idm-client.lab.org@LAB.ORG
      Password: False
      Keytab: False
      Managed by: idm-client.lab.org
    
    [root@idm ~]# ipa-getkeytab -s idm.lab.org -p host/idm-client.lab.org@LAB.ORG -k /root/idm-client.lab.org.keytab
    Keytab successfully retrieved and stored in: /root/idm-client.lab.org.keytab
    
    [root@idm ~]# cp /root/idm-client.lab.org.keytab /usr/share/ipa/html/
    
    [root@idm ~]# chmod 644 /usr/share/ipa/html/idm-client.lab.org.keytab
    
    

    Now lets get back to our FreeBSD client.

    root@idm-client:~ # fetch -o /usr/local/etc/ipa/krb5.keytab \
                          http://idm.lab.org/ipa/config/idm-client.lab.org.keytab
    
    root@idm-client:~ # chmod 600 /usr/local/etc/ipa/krb5.keytab
    
    root@idm-client:~ # mkdir -p /usr/local/etc/ssl/certs
    
    root@idm-client:~ # mkdir -p /usr/local/etc/pkg/repos
    
    root@idm-client:~ # sed -e 's|quarterly|latest|g' /etc/pkg/FreeBSD.conf \
                          > /usr/local/etc/pkg/repos/FreeBSD.conf
    
    root@idm-client:~ # pkg install -y beadm
    
    root@idm-client:~ # fetch -o /usr/local/etc/ssl/certs/poudriere.cert \
                          http://idm.lab.org/data/poudriere.cert
    
    root@idm-client:~ # export IP=10.0.0.124
    
    root@idm-client:~ # cat << EOF > /usr/local/etc/pkg/repos/14-0-S-amd64.conf
    14-0-S-amd64-idm: {
      url: "http://${IP}/packages/14-0-S-amd64-idm/",
      mirror_type: "http",
      signature_type: "pubkey",
      pubkey: "/usr/local/etc/ssl/certs/poudriere.cert",
      enabled: yes,
      priority: 100
    }
    EOF
    
    root@idm-client:~ # pkg update -f
    
    root@idm-client:~ # pkg install -y      \
                          krb5              \
                          sudo              \
                          sssd2             \
                          cyrus-sasl        \
                          cyrus-sasl-gssapi \
                          openldap26-client \
                          pam_mkhomedir
    
    root@idm-client:~ # cat << EOF >> /etc/ssh/ssh_config
    GSSAPIAuthentication yes
    EOF
    
    root@idm-client:~ # cat << EOF >> /etc/ssh/sshd_config
    GSSAPIAuthentication yes
    UsePAM yes
    EOF
    
    root@idm-client:~ # cat << EOF > /usr/local/etc/sssd/sssd.conf
    [sssd]
      config_file_version      = 2
      services                 = pam, ssh, sudo, ifp, pac, nss
      domains                  = lab.org
      timeout                  = 20
    
    [domain/lab.org]
      ipa_server               = idm.lab.org
      ipa_domain               = lab.org
      pam_gssapi_services      = sudo, sudo-i
      enumerate                = True
      cache_credentials        = True
      override_shell           = /usr/local/bin/bash
      override_homedir         = /home/%u
      default_shell            = /bin/sh
      ldap_group_nesting_level = 10
      default_ccache_template  = FILE:/tmp/krb5cc_:%U
    
      krb5_ccache_template     = FILE:/tmp/krb5cc_:%U
      krb5_server              = idm.lab.org:88
      krb5_realm               = LAB.ORG
      krb5_keytab              = /usr/local/etc/ipa/krb5.keytab
      krb5_auth_timeout        = 20
    
      id_provider              = ipa
      sudo_provider            = ipa
      access_provider          = ipa
      subdomains_provider      = ipa
      auth_provider            = ipa
      chpass_provider          = ipa
      selinux_provider         = none
    EOF
    
    root@idm-client:~ # chmod 600 /usr/local/etc/sssd/sssd.conf
    
    root@idm-client:~ # cat << EOF > /etc/nsswitch.conf
    #
    # nsswitch.conf(5) - name service switch configuration file
    # $FreeBSD$
    #
    group: files sss
    group_compat: nis
    hosts: files dns
    networks: files
    passwd: files sss
    passwd_compat: nis
    shells: files
    services: compat
    services_compat: nis
    protocols: files
    rpc: files
    sudoers: sss files
    netgroup: files
    EOF
    
    root@idm-client:~ # cat /etc/rc.conf
    hostname="idm-client.lab.org"
    ifconfig_vtnet0="inet 10.0.0.233/24"
    defaultrouter="10.0.0.1"
    syslogd_flags="-ss"
    clear_tmp_enable="YES"
    sshd_enable="YES"
    zfs_enable="YES"
    sssd_enable="YES"
    
    root@idm-client:~ # cat << EOF > /usr/local/etc/openldap/ldap.conf
    BASE        dc=org,dc=lab
    URI         ldap://idm.lab.org/
    SASL_MECH   GSSAPI
    SASL_REALM  LAB.ORG
    ssl         start_tls
    TLS_CACERT  /usr/local/etc/ipa/ca.crt
    EOF
    
    root@idm-client:~ # cat << EOF > /etc/krb5.conf
    [libdefaults]
      default_realm        = LAB.ORG
      default_keytab_name  = FILE:/usr/local/etc/ipa/krb5.keytab
      default_tkt_enctypes = aes256-cts des-cbc-crc aes128-cts arcfour-hmac
      default_tgs_enctypes = aes256-cts des-cbc-crc aes128-cts arcfour-hmac
      dns_lookup_realm     = false
      dns_lookup_kdc       = false
      rdns                 = false
      ticket_lifetime      = 24h
      forwardable          = yes
    
    [realms]
      LAB.ORG = {
        kdc            = idm.lab.org:88
        master_kdc     = idm.lab.org:88
        admin_server   = idm.lab.org:749
        default_domain = lab.org
        pkinit_anchors = FILE:/usr/local/etc/ipa/ca.crt
      }
    
    [domain_realm]
      .lab.org = LAB.ORG
       lab.org = LAB.ORG
    
    [logging]
      kdc          = FILE:/var/log/krb5/krb5kdc.log
      admin_server = FILE:/var/log/krb5/kadmin.log
      kadmin_local = FILE:/var/log/krb5/kadmin_local.log
      default      = FILE:/var/log/krb5/krb5lib.log
    EOF
    
    root@idm-client:~ # cat << EOF > /etc/pam.d/system
    #
    #
    # System-wide defaults
    #
    
    # AUTH
      auth      sufficient  pam_krb5.so                      no_warn try_first_pass
    # auth      sufficient  pam_ssh.so                       no_warn try_first_pass
      auth      sufficient  /usr/local/lib/pam_sss.so        no_warn use_first_pass
      auth      required    pam_unix.so                      no_warn try_first_pass nullok
    
    # ACCOUNT
    # account   required    pam_krb5.so
      account   required    pam_login_access.so
      account   required    /usr/local/lib/pam_sss.so        ignore_unknown_user ignore_authinfo_unavail
      account   required    pam_unix.so
    
    # SESSION
    # session   optional    pam_ssh.so                       want_agent
      session   required    pam_lastlog.so                   no_fail
      session   required    /usr/local/lib/pam_mkhomedir.so  mode=0700
    
    # PASSWORD
    # password  sufficient  pam_krb5.so                      no_warn try_first_pass
      password  sufficient  /usr/local/lib/pam_sss.so        no_warn use_authtok
      password  required    pam_unix.so                      no_warn try_first_pass
    EOF
    
    root@idm-client:~ # cat << EOF > /etc/pam.d/sshd
    #
    #
    # PAM configuration for the "sshd" service
    #
    
    # AUTH
      auth      sufficient  pam_krb5.so                      no_warn try_first_pass
    # auth      sufficient  pam_ssh.so                       no_warn try_first_pass
      auth      sufficient  /usr/local/lib/pam_sss.so        no_warn use_first_pass
      auth      required    pam_unix.so                      no_warn try_first_pass
    
    # ACCOUNT
      account   required    pam_nologin.so
    # account   required    pam_krb5.so
      account   required    pam_login_access.so
      account   required    pam_unix.so
      account   required    /usr/local/lib/pam_sss.so        ignore_unknown_user ignore_authinfo_unavail
    
    # SESSION
    # session   optional    pam_ssh.so                       want_agent
      session   required    pam_permit.so
      session   required    /usr/local/lib/pam_mkhomedir.so  mode=0700
      session   optional    /usr/local/lib/pam_sss.so
    
    # PASSWORD
    # password  sufficient  pam_krb5.so                      no_warn try_first_pass
      password  sufficient  /usr/local/lib/pam_sss.so        no_warn use_authtok
      password  required    pam_unix.so                      no_warn try_first_pass
    EOF
    
    

    Our idm-client.lab.org in the FreeIPA/IDM below.

    idm-hosts

    Now reboot your idm-client.lab.org and You should be able to login to it with FreeIPA/IDM account.

    host # ssh vermaden@10.0.0.233
    (vermaden@10.0.0.233) Password:
    Last login: Wed Mar  6 07:04:42 2024
    
    vermaden@idm-client:~ $ id
    uid=1374600003(vermaden) gid=1374600000(admins) groups=1374600000(admins)
    
    vermaden@idm-client:~ $ klist
    Credentials cache: FILE:/tmp/krb5cc_1374600003
            Principal: vermaden@LAB.ORG
    
      Issued                Expires               Principal
    Mar  6 07:04:34 2024  Mar  7 06:19:19 2024  krbtgt/LAB.ORG@LAB.ORG
    
    vermaden@idm-client:~ $ sudo -i
    Password for vermaden@LAB.ORG:
    
    root@idm-client:~ # id
    uid=0(root) gid=0(wheel) groups=0(wheel),5(operator)
    

    FreeBSD 14.0-STABLE Client – Debug Commands

    Below are some commands that you may (or may not) find useful.

    root@idm-client:~ # sssctl user-checks vermaden
    user: vermaden
    action: acct 
    service: system-auth
    
    SSSD nss user lookup result:
     - user name: vermaden
     - user id: 1374600003
     - group id: 1374600000
     - gecos: Vermaden Nedamrev
     - home directory: /home/vermaden
     - shell: /bin/sh
      
    Unable to connect to system bus!
    InfoPipe User lookup with [vermaden] failed.
    testing pam_acct_mgmt   
      
    pam_acct_mgmt: Success  
      
    PAM Environment:
     - no env -
    
      
      
    root@idm-client:~ # ldapsearch -H ldap://idm.lab.org -x -b "" -s base -LLL supportedSASLMechanisms
    dn:
    supportedSASLMechanisms: EXTERNAL
    supportedSASLMechanisms: GSS-SPNEGO                      
    supportedSASLMechanisms: GSSAPI
    supportedSASLMechanisms: DIGEST-MD5
    supportedSASLMechanisms: CRAM-MD5
    supportedSASLMechanisms: LOGIN
    supportedSASLMechanisms: PLAIN
    supportedSASLMechanisms: ANONYMOUS
    
    
    
    
    
    
    root@idm-client:~ # ldapsearch -x -v -W -D 'cn=Directory Manager' uid=vermaden
    ldap_initialize(  )
    Enter LDAP Password: 
    filter: uid=vermaden
    requesting: All userApplication attributes
    # extended LDIF
    #
    # LDAPv3
    # base  (default) with scope subtree
    # filter: uid=vermaden
    # requesting: ALL
    #
    
    # search result
    search: 2
    result: 32 No such object
    
    # numResponses: 1
    
    
    
    
    
    root@idm-client:~ # ldapsearch -Y GSSAPI -Omaxssf=0 -H ldaps://idm.lab.org -b dc=lab,dc=org CN=vermaden
    SASL/GSSAPI authentication started
    SASL username: vermaden@LAB.ORG
    SASL SSF: 0
    # extended LDIF
    #
    # LDAPv3
    # base  with scope subtree
    # filter: CN=vermaden
    # requesting: ALL
    #
    
    # vermaden, groups, compat, lab.org
    dn: cn=vermaden,cn=groups,cn=compat,dc=lab,dc=org
    objectClass: posixGroup
    objectClass: ipaOverrideTarget
    objectClass: ipaexternalgroup
    objectClass: top
    gidNumber: 1374600003
    ipaAnchorUUID:: OklQQTpsYWIub3JnOjcyN2FlMjM2LTMyMTktMTFlZS04OGMyLTU4OWNmYzA4MW
     QzNQ==
    cn: vermaden
    
    # vermaden, groups, accounts, lab.org
    dn: cn=vermaden,cn=groups,cn=accounts,dc=lab,dc=org
    objectClass: posixgroup
    objectClass: ipaobject
    objectClass: mepManagedEntry
    objectClass: top
    cn: vermaden
    gidNumber: 1374600003
    description: User private group for vermaden
    mepManagedBy: uid=vermaden,cn=users,cn=accounts,dc=lab,dc=org
    ipaUniqueID: 727ae236-3219-11ee-88c2-589cfc081d35
    
    # search result
    search: 4
    result: 0 Success
    
    # numResponses: 3
    # numEntries: 2
    
    
    

    Thats it – you have FreeBSD 14.0-STABLE connected to FreeIPA/IDM server.

    Summary

    Let me know in comments how it went.

    EOF
    • chevron_right

      Writing a Chat Client from Scratch

      Stephen Paul Weber · Wednesday, 30 November, 2022 - 03:30 edit · 27 minutes

    There are a lot of things that go into building a chat system, such as client, server, and protocol.  Even for only making a client there are lots of areas of focus, such as user experience, features, and performance.  To keep this post a manageable size, we will just be building a client and will use an existing server and protocol (accessing Jabber network services using the XMPP protocol).  We’ll make a practical GUI so we can test things, but not spend too much time on polish, and look at getting to a useful baseline of features.

    You can find all the code for this post in git.  All code licensed AGPL3+.

    Use a Library

    As with most large programming tasks, if we wanted to do every single thing ourselves we would spend a lot more time, so we should find some good libraries.  There is another reason to use a library: any improvements we make to the library benefits others.  While releasing our code might help someone else if they choose to read it, a library improvement can be picked up by users of that library right away.

    We need to speak the XMPP protocol so let’s choose Blather.  We need a GUI so we can see this working, but don’t really want to futz with it much so let’s choose Glimmer.  The code here will use these libraries and be written in the Ruby programming language, but these ideas are general purpose to the task and hopefully we won’t get too bogged down in syntax specifics.

    One little language-specific thing you will need to create is a description of which ruby packages are being used, so let’s make that file (named Gemfile):

    Gemfile

    source "https://rubygems.org"
    
    gem "blather", git: "https://github.com/adhearsion/blather", branch: "develop"
    gem "glimmer-dsl-libui", "~> 0.5.24"

    Run this to get the packages installed:

    bundle install --path=.gems

    Let’s get the bare minimum: a connection to a Jabber service and a window.

    client.rb

    require "glimmer-dsl-libui"
    require "blather/client"
    
    BLATHER = self
    include Glimmer
    
    Thread.new do
    	window("Contacts") {
    		on_destroy {
    			BLATHER.shutdown
    		}
    	}
    end

    When required in this way, Blather will automatically set up a connection with event processing on the main thread, and will process command line arguments to get connection details.  So we put the GUI on a second thread to not have them block each other.  When the window is closed (on_destroy), be sure to disconnect from the server too.  You can run this barely-a-client like this:

    bundle exec ruby client.rb user@example.com password

    The arguments are a Jabber ID (which you can get from manyexistingservices), and the associated password.

    You should get a blank window and no errors in your terminal.  If you wanted to you could even look in another client and confirm that it is connected to the account by seeing it come online.

    Show a Contact List

    Let’s fetch the user’s contacts from the server and show them in the window (if you use this with a new, blank test account there won’t be any contacts yet of course, but still).

    $roster = [["", ""]]
    
    Thread.new do
    	window("Contacts") {
    		vertical_box {
    			table {
    				button_column("Contact") {
    				}
    				editable false
    				cell_rows $roster
    			}
    		}
    
    		on_destroy {
    			BLATHER.shutdown
    		}
        }.show
    end
    
    after(:roster) do
    	LibUI.queue_main do
    		$roster.clear
    		my_roster.each do |item|
    			$roster << [item.name || item.jid, item.jid]
    		end
    	end
    end

    In a real app you would probably want some kind of singleton object to represent the contacts window and the contact list (“roster”) etc.  For simplicity here we just use a global variable for the roster, starting with some dummy data so that the GUI framework knows what it will look like, etc.

    We fill out the window from before a little bit to have a table with a column of buttons, one for each contact.  The button_column is the first (and in this case, only) column definition so it will source data from the first element of each item in cell_rows.  It’s not an editable table, and it gets its data from the global variable.

    We then add an event handler to our XMPP connection to say that once the roster has been loaded from the server, we hand control over to the GUI thread and there we clear out the global variable and fill it up with the roster as we now see it.  The first item in each row is the name that will be shown on the button (either item.name or item.jid if there is no name set), the second item is the Jabber ID which won’t be shown because we didn’t define that column when we made the window.  Any updates to the global variable will be automatically painted into the GUI so we’re done.

    One Window Per Conversation

    For simplicity, let’s say we want to show one window per conversation, like so:

    $conversations = {}
    
    class Conversation
    	include Glimmer
    
    	def self.open(jid, m=nil)
    		return if $conversations[jid]
    
    		($conversations[jid] = new(jid, m)).launch
    	end
    
    	def initialize(jid, m=nil)
    		@jid = jid
    		@messages = [["", ""]]
    		new_message(m) if m
    	end
    
    	def launch
    		window("Conversation With #{@jid}") {
    			vertical_box {
    				table {
    					text_column("Sender")
    					text_column("Message")
    					editable false
    					cell_rows @messages
    					@messages.clear
    				}
    
    				horizontal_box {
    					stretchy false
    
    					@message_entry = entry
    					button("Send") {
    						stretchy false
    
    						on_clicked do
    							BLATHER.say(@jid, @message_entry.text)
    							@messages << [ARGV[0], @message_entry.text]
    							@message_entry.text = ""
    						end
    					}
    				}
    			}
    
    			on_closing do
    				$conversations.delete(@jid)
    			end
    		}.show
    	end
    
    	def format_sender(jid)
    		BLATHER.my_roster[jid]&.name || jid
    	end
    
    	def message_row(m)
    		[
    			format_sender(m.from&.stripped || BLATHER.jid.stripped),
    			m.body
    		]
    	end
    
    	def new_message(m)
    		@messages << message_row(m)
    	end
    end
    
    message :body do |m|
    	LibUI.queue_main do
    		conversation = $conversations[m.from.stripped.to_s]
    		if conversation
    			conversation.new_message(m)
    		else
    			Conversation.open(m.from.stripped.to_s, m)
    		end
    	end
    end

    Most of this is the window definition again, with a table of the messages in this conversation sourced from an instance variable @messages.  At the bottom of the window is an entry box to type in text and a button to trigger sending it as a message.  When the button is clicked, send that message to the contact this conversation is with, add it to the list of messages so that it shows up in the GUI, and make the entry box empty again.  When the window closes (on_closing this time because it’s not the “main” window) delete the object from the global set of open conversations.

    This object also has a helper to open a conversation window if there isn’t already one with a given Jabber ID (jid), some helpers to format message objects into table rows by extracting the sender and body (including format_sender which gets the roster item if there is one, uses &.name to get the name if there was a roster item or else nil, and if there was no roster item or no name just show jid) and a helper that adds new messages into the GUI.

    Finally we add a new XMPP event handler for incoming messages that have a body.  Any such incoming message we look up in the global if there is a conversation open already, if so we pass the new message there to have it appended to the GUI table, otherwise we open the conversation with this message as the first thing it will show.

    Getting from the Contact List to a Conversation

    Now we wire up the contact list to the conversation view:

    button_column("Contact") {
    	on_clicked do |row|
    		Conversation.open($roster[row][1].to_s)
    	end
    }

    When a contact button is clicked, grab the Jabber ID from the hidden end of the table row that we had stashed there, and open the conversation.

    horizontal_box {
    	stretchy false
    
    	jid_entry = entry {
    		label("Jabber ID")
    	}
    
    	button("Start Conversation") {
    		stretchy false
    
    		on_clicked do
    			Conversation.open(jid_entry.text)
    		end
    	}
    }

    And let’s provide a way to start a new conversation with an address that isn’t a contact too.  An entry to type in a Jabber ID and a button that opens the conversation.

    Adding a Contact

    Might as well add a button to the main window that re-uses that entry box to allow adding a contact as well:

    button("Add Contact") {
    	stretchy false
    
    	on_clicked do
    		BLATHER.my_roster << jid_entry.text
    	end
    }

    Handling Multiple Devices

    In many chat protocols, it is common to have multiple devices or apps connected simultaneously. It is often desirable to show messages sent to or from one device on all the others as well.  So let’s implement that.  First, a helper for creating XML structures we may need:

    def xml_child(parent, name, namespace)
    	child = Niceogiri::XML::Node.new(name, parent.document, namespace)
    	parent << child
    	child
    end

    We need to tell the server that we support this feature:

    when_ready do
    	self << Blather::Stanza::Iq.new(:set).tap { |iq|
    		xml_child(iq, :enable, "urn:xmpp:carbons:2")
    	}
    end

    We will be handling live messages from multiple event handlers so let’s pull the live message handling out into a helper:

    def handle_live_message(m, counterpart: m.from.stripped.to_s)
    	LibUI.queue_main do
    		conversation = $conversations[counterpart]
    		if conversation
    			conversation.new_message(m)
    		else
    			Conversation.open(counterpart, m)
    		end
    	end
    end

    And the helper that will handle messages from other devices of ours:

    def handle_carbons(fwd, counterpart:)
    	fwd = fwd.first if fwd.is_a?(Nokogiri::XML::NodeSet)
    	return unless fwd
    
    	m = Blather::XMPPNode.import(fwd)
    	return unless m.is_a?(Blather::Stanza::Message) && m.body.present?
    
    	handle_live_message(m, counterpart: counterpart.call(m))
    end

    This takes in the forwarded XML object (allowing for it to be a set of which we take the first one) and imports it with Blather’s logic to become hopefully a Message object.  If it’s not a Message or has no body, we don’t really care so we stop there. Otherwise we can handle this extracted message as though we had received it ourselves.

    And then wire up the event handlers:

    message(
    	"./carbon:received/fwd:forwarded/*[1]",
    	carbon: "urn:xmpp:carbons:2",
    	fwd: "urn:xmpp:forward:0"
    ) do |_, fwd|
    	handle_carbons(fwd, counterpart: ->(m) { m.from.stripped.to_s })
    end

    Because XMPP is just XML, we can use regular XPath stuff to extract from incoming messages.  Here we say that if the message contains a forwarded element inside a carbons received element, then we should handle this with the carbons handler instead of just the live messages handler.  The XML that matches our XPath comes in as the second argument and that is what we pass to the handler to get converted into a Message object.

    message(
    	"./carbon:sent/fwd:forwarded/*[1]",
    	carbon: "urn:xmpp:carbons:2",
    	fwd: "urn:xmpp:forward:0"
    ) do |_, fwd|
    	handle_carbons(fwd, counterpart: ->(m) { m.to.stripped.to_s })
    end

    This handler is for messages sent by other devices instead of received by other devices.  It is pretty much the same, except that we know the “other side of the conversation” (here called counterpart) is in the to not the from.

    message :body do |m|
    	handle_live_message(m)
    end

    And our old message-with-body handler now just needs to call the helper.

    History

    So far our client only processes and displays live messages.  If you close the app, or even close a conversation window, the history is gone.  If you chat with another client or device, you can’t see that when you re-open this one.  To fix that we’ll need to store messages persistently, and also fetch any history from while we were disconnected from the server.  We will need a few more lines in our Gemfile first:

    gem "sqlite3"
    gem "xdg"

    And then to set up a basic database schema:

    require "securerandom"
    require "sqlite3"
    require "xdg"
    
    DATA_DIR = XDG::Data.new.home + "jabber-client-demo"
    DATA_DIR.mkpath
    DB = SQLite3::Database.new(DATA_DIR + "db.sqlite3")
    
    if DB.user_version < 1
    	DB.execute(<<~SQL)
    		CREATE TABLE messages (
    			mam_id TEXT PRIMARY KEY,
    			stanza_id TEXT NOT NULL,
    			conversation TEXT NOT NULL,
    			created_at INTEGER NOT NULL,
    			stanza TEXT NOT NULL
    		)
    	SQL
    	DB.execute("CREATE TABLE data (key TEXT PRIMARY KEY, value TEXT)")
    	DB.user_version = 1
    end

    user_version is a SQLite feature that allows storing a simple integer alongside the database.  It starts at 0 if never set, and so here we use it to check if our schema has been created or not.  We store the database in a new directory created according to the XDG Base Directory specification.  There are two relevant IDs for most XMPP operations: the MAM ID (the ID in the server’s archive) and the Stanza ID (which was usually selected by the original sender).  We also create a data table for storing basic key-value stuff, which we’ll use in a minute to remember where we have sync’d up to so far.  Let’s edit the Conversation object to store messages as we send them, updating the send button on_clicked handler:

    def message
    	Blather::Stanza::Message.new(@jid, @message_entry.text, :chat).tap { |m|
    		m.id = SecureRandom.uuid
    	}
    end

    on_clicked do
    	m = message
    	EM.defer do
    		BLATHER << m
    		DB.execute(<<~SQL, [nil, m.id, @jid, m.to_s])
    			INSERT INTO messages
    			(mam_id, stanza_id, conversation, created_at, stanza)
    			VALUES (?,?,?,unixepoch(),?)
    		SQL
    	end
    	@messages << message_row(m)
    	@message_entry.text = ""
    end

    When we send a message we don’t yet know the server’s archive ID, so we set that to nil for now.  We set mam_id to be the primary key, but SQLite allows multiple rows to have NULL in there so this will work.  We don’t want to block the GUI thread while doing database work so we use EM.defer to move this to a worker pool.  We also want to store messages when we receive them live, so add this to the start of handle_live_message:

    mam_id = m.xpath("./ns:stanza-id", ns: "urn:xmpp:sid:0").find { |el|
    	el["by"] == jid.stripped.to_s
    }&.[]("id")
    delay = m.delay&.stamp&.to_i || Time.now.to_i
    DB.execute(<<~SQL, [mam_id, m.id, counterpart, delay, m.to_s])
    	INSERT INTO messages (mam_id, stanza_id, conversation, created_at, stanza)
        VALUES (?,?,?,?,?)
    SQL

    Here we extract the server archive’s ID for the message (added by the server in a stanza-id with by="Your Jabber ID") and figure out what time the message was originally sent (usually this is just right now for a live message, but if it is coming from offline storage because every client was offline or similar, then there can be a “delay” set on it which we can use).  Now that we have stored the history of message we received we need to load them into the GUI when we start up a Conversation so add this at the end of initialize:

    EM.defer do
    	mam_messages = []
    	query = <<~SQL
    		SELECT stanza
    		FROM messages
    		WHERE conversation=?
    		ORDER BY created_at
    	SQL
    	DB.execute(query, [@jid]) do |row|
    		m = Blather::XMPPNode.import(
    			Nokogiri::XML.parse(row[0]).root
    		)
    		mam_messages << m
    	end
    
    	LibUI.queue_main do
    		mam_messages.map! { |m| message_row(m) }
    		@messages.replace(mam_messages + @messages)
    	end
    end

    In the worker pool we load up all the stored messages for the current conversation in order, then we take the XML stored as a string in the database and parse it into a Blather Message object.  Once we’ve done as much of the work as we can in we worker pool we use queue_main to switch back to the GUI thread and actually build the rows for the table and replace them into the GUI.

    With these changes, we are now storing all messages we see while connected and displaying them in the conversation.  But what about messages sent or received by other devices or clients while we were not connected?  For that we need to sync with the server’s archive, fetching messages at a reasonable page size from whatever we already have until the end.

    def sync_mam(last_id)
    	start_mam = Blather::Stanza::Iq.new(:set).tap { |iq|
    		xml_child(iq, :query, "urn:xmpp:mam:2").tap do |query|
    			xml_child(query, :set, "http://jabber.org/protocol/rsm").tap do |rsm|
    				xml_child(rsm, :max, "http://jabber.org/protocol/rsm").tap do |max|
    					max.content = (EM.threadpool_size * 5).to_s
    				end
    				next unless last_id
    
    				xml_child(rsm, :after, "http://jabber.org/protocol/rsm").tap do |after|
    					after.content = last_id
    				end
    			end
    		end
    	}
    
    	client.write_with_handler(start_mam) do |reply|
    		next if reply.error?
    
    		fin = reply.find_first("./ns:fin", ns: "urn:xmpp:mam:2")
    		next unless fin
    
    		handle_rsm_reply_when_idle(fin)
    	end
    end

    The first half of this creates the XML stanza to request a page from the server’s archive. We create a query with a max page size based on the size of our worker threadpool, and ask for messages only after the last known id (if we have one, which we won’t on first run). Then we use write_with_handler to send this request to the server and wait for a reply. The reply is sent after all messages have been sent down (sent seperately, not returned in this reply, see below), but we may still be processing some of them in the worker pool so we next create a helper to wait for the worker pool to be done:

    def handle_rsm_reply_when_idle(fin)
    	unless EM.defers_finished?
    		EM.add_timer(0.1) { handle_rsm_reply_when_idle(fin) }
    		return
    	end
    
    	last = fin.find_first(
    		"./ns:set/ns:last",
    		ns: "http://jabber.org/protocol/rsm"
    	)&.content
    
    	if last
    		DB.execute(<<~SQL, [last, last])
    			INSERT INTO data VALUES ('last_mam_id', ?)
    			ON CONFLICT(key) DO UPDATE SET value=? WHERE key='last_mam_id'
    		SQL
    	end
    	return if fin["complete"].to_s == "true"
    
    	sync_mam(last)
    end

    Poll with a timer until the worker pool is all done so that we aren’t fetching new pages before we have handled the last one.  Get the value of the last archive ID that was part of the page just processed and store it in the database for next time we start up.  If this was the last page (that is, complete="true") then we’re all done, otherwise get the next page.  We need to make sure we actually start this sync process inside the when_ready handler:

    last_mam_id = DB.execute(<<~SQL)[0]&.first
    	SELECT value FROM data WHERE key='last_mam_id' LIMIT 1
    SQL
    sync_mam(last_mam_id)

    And also, we need to actually handle the messages as they come down from the server archive:

    message "./ns:result", ns: "urn:xmpp:mam:2" do |_, result|
    	fwd = result.xpath("./ns:forwarded", ns: "urn:xmpp:forward:0").first
    	fwd = fwd.find_first("./ns:message", ns: "jabber:client")
    	m = Blather::XMPPNode.import(fwd)
    	next unless m.is_a?(Blather::Stanza::Message) && m.body.present?
    
    	mam_id = result.first["id"]&.to_s
    	# Can't really race because we're checking for something from the past
    	# Any new message inserted isn't the one we're looking for here anyway
    	sent = DB.execute(<<~SQL, [m.id])[0][0]
    		SELECT count(*) FROM messages WHERE stanza_id=? AND mam_id IS NULL
    	SQL
    	if sent < 1
    		counterpart = if m.from.stripped.to_s == jid.stripped.to_s
    			m.to.stripped.to_s
    		else
    			m.from.stripped.to_s
    		end
    		delay =
    			fwd.find_first("./ns:delay", ns: "urn:xmpp:delay")
    			&.[]("stamp")&.then(Time.method(:parse))
    		delay = delay&.to_i || m.delay&.stamp&.to_i || Time.now.to_i
    		DB.execute(<<~SQL, [mam_id, m.id, counterpart, delay, m.to_s])
    			INSERT OR IGNORE INTO messages
    			(mam_id, stanza_id, conversation, created_at, stanza)
    			VALUES (?,?,?,?,?)
    		SQL
    	else
    		DB.execute(<<~SQL, [mam_id, m.id])
    			UPDATE messages SET mam_id=? WHERE stanza_id=?
    		SQL
    	end
    end

    Any message which contains a MAM (server archive) result will get handled here.  Just like with carbons we extract the forwarded message and import, making sure it ends up as a Blather Message object with a body.

    Remember how when we stored a sent message we didn’t know the archive ID yet?  Here we check if there is anything in our database already with this stanza ID and no archive ID, if no we will insert it as a new message, but otherwise we can update the row we already have to store the server archive ID on it, which we now know.

    And with that, our client now stores and syncs all history with the server, to give the user a full view of their conversation no matter where or when it happened.

    Display Names

    If a user is added to the contact list with a name, we already show that name instead of their address in conversations.  What if a user is not a contact yet, or we haven’t set a name for them?  It might be useful to be able to fetch any display name they advertise for themselves and show that.  First we add a simple helper to expose write_with_handler outside of the main object:

    public def write_with_handler(stanza, &block)
    	client.write_with_handler(stanza, &block)
    end

    We need an attribute on the Conversation to hold the nickname:

    attr_accessor :nickname

    And then we can use this in Conversation#initialize to fetch the other side’s nickname if they advertise one and we don’t have one for them yet:

    self.nickname = BLATHER.my_roster[jid]&.name || jid
    return unless nickname.to_s == jid.to_s
    
    BLATHER.write_with_handler(
    	Blather::Stanza::PubSub::Items.new(:get).tap { |iq|
    		iq.node = "http://jabber.org/protocol/nick"
    		iq.to = jid
    	}
    ) do |reply|
    	self.nickname = reply.items.first.payload_node.text rescue self.nickname
    end

    Inside the window declaration we can use this as the window title:

    title <=> [self, :nickname]

    and in format_sender we can use this as well:

    return nickname if jid.to_s == @jid.to_s

    Avatars

    Names are nice, but what about pictures?  Can we have nice avatar images that go with each user?  What should we display if they don’t have an avatar set?  Well not only is there a protocol to get an avatar, but a specification that allows all clients to use the same colours to represent things, so we can use a block of that if there is no avatar set.  Let’s generate the colour blocks first.  Add this to Gemfile:

    gem "hsluv"

    Require the library at the top:

    require "hsluv"
    
    $avatars = {}

    And a method on Conversation to use this:

    def default_avatar(string)
    	hue = (Digest::SHA1.digest(string).unpack1("v").to_f / 65536) * 360
    	rgb = Hsluv.rgb_prepare(Hsluv.hsluv_to_rgb(hue, 100, 50))
    	rgba = rgb.pack("CCC") + "xff".b
    	image { image_part(rgba * 32 * 32, 32, 32, 4) }
    end

    This takes the SHA-1 of a string, unpacks the first two bytes as a 16-bit little-endian integer, converts the range from 0 to MAX_SHORT into the range from 0 to 360 for hue degrees, then passes to the library we added to convert from HSV to RGB colour formats.  The GUI library expects images as a byte string where every 4 bytes are 0 to 255 for red, then green, then blue, then transparency.  Because we want a square of all one colour, we can create the byte string for one pixel and then multiply the string by the width and height (multiplying a string by a number in Ruby make a new string with that many copies repeated) to get the whole image.

    In Conversation#initialize we can use this to make a default avatar on the dummy message row then the window first opens:

    @messages = [[default_avatar(""), "", ""]]

    And we will need to add a new column definition to be beginning of the table { block:

    image_column("Avatar")

    And actually add the image to message_row:

    def message_row(m)
    	from = m.from&.stripped || BLATHER.jid.stripped
    	[
    		$avatars[from.to_s] || default_avatar(from.to_s),
    		format_sender(from),
    		m.body
    	]
    end

    If you run this you should now see a coloured square next to each message.  We would now like to get actual avatars, so add this somewhere at the top level to advertise support for this:

    set_caps(
    	"https://git.singpolyma.net/jabber-client-demo",
    	[],
    	["urn:xmpp:avatar:metadata+notify"]
    )

    Then in the when_ready block make sure to send it to the server:

    send_caps

    And handle the avatars as they come in:

    pubsub_event(
    	"//ns:items[@node='urn:xmpp:avatar:metadata']",
    	ns: "http://jabber.org/protocol/pubsub#event"
    ) do |m|
    	id = m.items.first&.payload_node&.children&.first&.[]("id")
    	next $avatars.delete(m.from.stripped.to_s) unless id
    
    	path = DATA_DIR + id.to_s
    	key = m.from.stripped.to_s
    	if path.exist?
    		LibUI.queue_main { $avatars[key] = image(path.to_s, 32, 32) rescue nil }
    	else
    		write_with_handler(
    			Blather::Stanza::PubSub::Items.new(:get).tap { |iq|
    				iq.node = "urn:xmpp:avatar:data"
    				iq.to = m.from
    			}
    		) do |reply|
    			next if reply.error?
    
    			data = Base64.decode64(reply.items.first&.payload_node&.text.to_s)
    			path.write(data)
    			LibUI.queue_main { $avatars[key] = image(path.to_s, 32, 32) rescue nil }
    		end
    	end
    end

    When an avatar metadata event comes in, we check what it is advertising as the ID of the avatar for this user.  If there is none, that means they don’t have an avatar anymore so delete anything we may have in the global cache for them, otherwise create a file path in the same folder as the database based on this ID.  If that file exists already, then no need to fetch it again, create the image from that path on the GUI thread and set it into our global in-memory cache.  If the file does not exist, then use write_with_handler to request their avatar data.  It comes back Base64 encoded, so decode it and then write it to the file.

    If you run this you should now see avatars next to messages for anyone who has one set.

    Delivery Receipts

    The Internet is a wild place, and sometimes things don’t work out how you’d hope.  Sometimes something goes wrong, or perhaps just all of a user’s devices are turned off.  Whatever the reason, it can be useful to see if a message has been delivered to at least one of the intended user’s devices yet or not.  We’ll need a new database column to store that status, add after the end of the DB.user_version < 1 if block:

    if DB.user_version < 2
    	DB.execute(<<~SQL)
    		ALTER TABLE messages ADD COLUMN delivered INTEGER NOT NULL DEFAULT 0
    	SQL
    	DB.user_version = 2
    end

    Let’s advertise support for the feature:

    set_caps(
    	"https://git.singpolyma.net/jabber-client-demo",
    	[],
    	["urn:xmpp:avatar:metadata+notify", "urn:xmpp:receipts"]
    )

    We need to add delivery status and stanza id to the dummy row for the messages table:

    @messages = [[default_avatar(""), "", "", false, nil]]

    And make sure we select the status out of the database when loading up messages:

    SELECT stanza,delivered FROM messages WHERE conversation=? ORDER BY created_at

    And pass that through when building the message rows

    mam_messages << [m, row[1]]

    mam_messages.map! { |args| message_row(*args) }

    Update the messages table to expect the new data model:

    table {
    	image_column("Avatar")
    	text_column("Sender")
    	text_column("Message")
    	checkbox_column("Delivered")
    	editable false
    	cell_rows @messages
    	@messages.clear if @messages.length == 1 && @messages.first.last.nil?
    }

    And update the row builder to include this new data:

    def message_row(m, delivered=false)
    	from = m.from&.stripped || BLATHER.jid.stripped
    	[
    		$avatars[from.to_s] || default_avatar(from.to_s),
    		format_sender(from),
    		m.body,
    		delivered,
    		m.id
    	]
    end

    Inbound messages are always considered delivered, since we have them:

    def new_message(m)
    	@messages << message_row(m, true)
    end

    And a method to allow signalling that a delivery receipt should be displayed, using the fact that we now hide the stanza id off the end of the rows in the table to find the relevant message to update:

    def delivered_message(id)
    	row = @messages.find_index { |r| r.last == id }
    	return unless row
    
    	@messages[row] = @messages[row][0..-3] + [true, id]
    end

    In the Send button’s on_clicked handler we need to actually request that others send us receipts:

    m = message
    xml_child(m, :request, "urn:xmpp:receipts")

    And we need to handle the receipts when they arrive:

    message "./ns:received", ns: "urn:xmpp:receipts" do |m, received|
    	DB.execute(<<~SQL, [received.first["id"].to_s])
    		UPDATE messages SET delivered=1 WHERE stanza_id=?
    	SQL
    
    	conversation = $conversations[m.from.stripped.to_s]
    	return unless conversation
    
    	LibUI.queue_main do
    		conversation.delivered_message(received.first["id"].to_s)
    	end
    end

    When we get a received receipt, we get the id attribute off of it, which represents a stanza ID that this receipt is for.  We update the database, and inform any open conversation window so the GUI can be updated.

    Finally, if someone requests a receipt from us we should send it to them:

    message :body do |m|
    	handle_live_message(m)
    
    	if m.id && m.at("./ns:request", ns: "urn:xmpp:receipts")
    		self << m.reply(remove_children: true).tap { |receipt|
    			xml_child(receipt, :received, "urn:xmpp:receipts").tap { |received|
    				received["id"] = m.id
    			}
    		}
    	end
    end

    If the stanza has an id and a receipt request, we construct a reply that contains just the received receipt and send it.

    Message Correction

    Sometimes people send a message with a mistake in it and want to send another to fix it.  It is convenvient for the GUI to support this and render only the new version of the message.  So let’s implement that.  First we add it to the list of things we advertise support for:

    set_caps(
    	"https://git.singpolyma.net/jabber-client-demo",
    	[],
    	[
    		"urn:xmpp:avatar:metadata+notify",
    		"urn:xmpp:receipts",
    		"urn:xmpp:message-correct:0"
    	]
    )

    Then we need a method on Conversation to process incoming corrections and update the GUI:

    def new_correction(replace_id, m)
    	row = @messages.find_index { |r| r.last == replace_id }
    	return unless row
    
    	@messages[row] = message_row(m, true)
    end

    We look up the message row on the stanza id, just as we did for delivery receipts, and just completely replace it with a row based on the new incoming message.  That’s it for the GUI.  Corrections may come from live messages, from carbons, or even from the server archive if they happened while we were disconnected, so we create a new insert_message helper to handle any case we previously did the SQL INSERT for an incoming message:

    def insert_message(
    	m,
    	mam_id:,
    	counterpart: m.from.stripped.to_s,
    	delay: m.delay&.stamp&.to_i
    )
    	if (replace = m.at("./ns:replace", ns: "urn:xmpp:message-correct:0"))
    		DB.execute(<<~SQL, [m.to_s, counterpart, replace["id"].to_s])
    			UPDATE messages SET stanza=? WHERE conversation=? AND stanza_id=?
    		SQL
    	else
    		delay ||= Time.now.to_i
    		DB.execute(<<~SQL, [mam_id, m.id, counterpart, delay, m.to_s])
    			INSERT OR IGNORE INTO messages
    			(mam_id, stanza_id, conversation, created_at, stanza, delivered)
    			VALUES (?,?,?,?,?,1)
    		SQL
    	end
    end

    The else case here is the same as the INSERTs we’ve been using up to this point, but we also check first for an element that signals this as a replacement and if that is the case we issue an UPDATE instead to correct our internal archive to the new version.

    Then in handle_live_message we also signal the possibly-open GUI:

    if (replace = m.at("./ns:replace", ns: "urn:xmpp:message-correct:0"))
    	conversation.new_correction(replace["id"].to_s, m)
    else
    	conversation.new_message(m)
    end

    We can now display incoming corrections, but it would also be nice to be able to send them.  Add a second button after the Send button in Conversation that can re-use the @message_entry box to correct the most recently sent message:

    button("Correct") {
    	stretchy false
    
    	on_clicked do
    		replace_row = @messages.rindex { |message|
    			message[1] == format_sender(BLATHER.jid.stripped)
    		}
    		next unless replace_row
    
    		m = message
    		m << xml_child(m, :replace, "urn:xmpp:message-correct:0").tap { |replace|
    			replace["id"] = @messages[replace_row].last
    		}
    		EM.defer do
    			BLATHER << m
    			DB.execute(<<~SQL, [m.to_s, @jid, @messages[replace_row].last])
    				UPDATE messages SET stanza=? WHERE conversation=? AND stanza_id=?
    			SQL
    		end
    		@messages[replace_row] = message_row(m, @messages[replace_row][-2])
    		@message_entry.text = ""
    	end
    }

    When the button is clicked we find the row for the most recently sent message, construct a message to send just as in the Send case but add the message correction replace child with the id matching the stanza id of the most recently sent message.  We send that message and also update our own local copy of the stanza both in the database and in the memory model rendered in the GUI.

    Conclusion

    There are a lot more features that a chat system can implement, but hopefully this gives you a useful taste of how each one can be incrementally layered in, and what the considerations might be for a wide variety of different kinds of features.  All the code for the working application developed in this article is available in git under AGPLv3+, with commits that corrospond to the path we took here.

    • wifi_tethering open_in_new

      This post is public

      blog.jmp.chat /b/2022-chat-client-from-scratch

    • Pictures 1 image

    • visibility
    • favorite

      4 Like

      𝖈𝖍𝖚𝖓𝖐, Bigou, le VRAI!, Kris, chipmnk

    • chevron_right

      Writing a Chat Client from Scratch

      Stephen Paul Weber · Wednesday, 30 November, 2022 - 03:30 edit · 27 minutes

    There are a lot of things that go into building a chat system, such as client, server, and protocol.  Even for only making a client there are lots of areas of focus, such as user experience, features, and performance.  To keep this post a manageable size, we will just be building a client and will use an existing server and protocol (accessing Jabber network services using the XMPP protocol).  We’ll make a practical GUI so we can test things, but not spend too much time on polish, and look at getting to a useful baseline of features.

    You can find all the code for this post in git.  All code licensed AGPL3+.

    Use a Library

    As with most large programming tasks, if we wanted to do every single thing ourselves we would spend a lot more time, so we should find some good libraries.  There is another reason to use a library: any improvements we make to the library benefits others.  While releasing our code might help someone else if they choose to read it, a library improvement can be picked up by users of that library right away.

    We need to speak the XMPP protocol so let’s choose Blather.  We need a GUI so we can see this working, but don’t really want to futz with it much so let’s choose Glimmer.  The code here will use these libraries and be written in the Ruby programming language, but these ideas are general purpose to the task and hopefully we won’t get too bogged down in syntax specifics.

    One little language-specific thing you will need to create is a description of which ruby packages are being used, so let’s make that file (named Gemfile):

    Gemfile

    source "https://rubygems.org"
    
    gem "blather", git: "https://github.com/adhearsion/blather", branch: "develop"
    gem "glimmer-dsl-libui", "~> 0.5.24"

    Run this to get the packages installed:

    bundle install --path=.gems

    Let’s get the bare minimum: a connection to a Jabber service and a window.

    client.rb

    require "glimmer-dsl-libui"
    require "blather/client"
    
    BLATHER = self
    include Glimmer
    
    Thread.new do
    	window("Contacts") {
    		on_destroy {
    			BLATHER.shutdown
    		}
    	}
    end

    When required in this way, Blather will automatically set up a connection with event processing on the main thread, and will process command line arguments to get connection details.  So we put the GUI on a second thread to not have them block each other.  When the window is closed (on_destroy), be sure to disconnect from the server too.  You can run this barely-a-client like this:

    bundle exec ruby client.rb user@example.com password

    The arguments are a Jabber ID (which you can get from manyexistingservices), and the associated password.

    You should get a blank window and no errors in your terminal.  If you wanted to you could even look in another client and confirm that it is connected to the account by seeing it come online.

    Show a Contact List

    Let’s fetch the user’s contacts from the server and show them in the window (if you use this with a new, blank test account there won’t be any contacts yet of course, but still).

    $roster = [["", ""]]
    
    Thread.new do
    	window("Contacts") {
    		vertical_box {
    			table {
    				button_column("Contact") {
    				}
    				editable false
    				cell_rows $roster
    			}
    		}
    
    		on_destroy {
    			BLATHER.shutdown
    		}
        }.show
    end
    
    after(:roster) do
    	LibUI.queue_main do
    		$roster.clear
    		my_roster.each do |item|
    			$roster << [item.name || item.jid, item.jid]
    		end
    	end
    end

    In a real app you would probably want some kind of singleton object to represent the contacts window and the contact list (“roster”) etc.  For simplicity here we just use a global variable for the roster, starting with some dummy data so that the GUI framework knows what it will look like, etc.

    We fill out the window from before a little bit to have a table with a column of buttons, one for each contact.  The button_column is the first (and in this case, only) column definition so it will source data from the first element of each item in cell_rows.  It’s not an editable table, and it gets its data from the global variable.

    We then add an event handler to our XMPP connection to say that once the roster has been loaded from the server, we hand control over to the GUI thread and there we clear out the global variable and fill it up with the roster as we now see it.  The first item in each row is the name that will be shown on the button (either item.name or item.jid if there is no name set), the second item is the Jabber ID which won’t be shown because we didn’t define that column when we made the window.  Any updates to the global variable will be automatically painted into the GUI so we’re done.

    One Window Per Conversation

    For simplicity, let’s say we want to show one window per conversation, like so:

    $conversations = {}
    
    class Conversation
    	include Glimmer
    
    	def self.open(jid, m=nil)
    		return if $conversations[jid]
    
    		($conversations[jid] = new(jid, m)).launch
    	end
    
    	def initialize(jid, m=nil)
    		@jid = jid
    		@messages = [["", ""]]
    		new_message(m) if m
    	end
    
    	def launch
    		window("Conversation With #{@jid}") {
    			vertical_box {
    				table {
    					text_column("Sender")
    					text_column("Message")
    					editable false
    					cell_rows @messages
    					@messages.clear
    				}
    
    				horizontal_box {
    					stretchy false
    
    					@message_entry = entry
    					button("Send") {
    						stretchy false
    
    						on_clicked do
    							BLATHER.say(@jid, @message_entry.text)
    							@messages << [ARGV[0], @message_entry.text]
    							@message_entry.text = ""
    						end
    					}
    				}
    			}
    
    			on_closing do
    				$conversations.delete(@jid)
    			end
    		}.show
    	end
    
    	def format_sender(jid)
    		BLATHER.my_roster[jid]&.name || jid
    	end
    
    	def message_row(m)
    		[
    			format_sender(m.from&.stripped || BLATHER.jid.stripped),
    			m.body
    		]
    	end
    
    	def new_message(m)
    		@messages << message_row(m)
    	end
    end
    
    message :body do |m|
    	LibUI.queue_main do
    		conversation = $conversations[m.from.stripped.to_s]
    		if conversation
    			conversation.new_message(m)
    		else
    			Conversation.open(m.from.stripped.to_s, m)
    		end
    	end
    end

    Most of this is the window definition again, with a table of the messages in this conversation sourced from an instance variable @messages.  At the bottom of the window is an entry box to type in text and a button to trigger sending it as a message.  When the button is clicked, send that message to the contact this conversation is with, add it to the list of messages so that it shows up in the GUI, and make the entry box empty again.  When the window closes (on_closing this time because it’s not the “main” window) delete the object from the global set of open conversations.

    This object also has a helper to open a conversation window if there isn’t already one with a given Jabber ID (jid), some helpers to format message objects into table rows by extracting the sender and body (including format_sender which gets the roster item if there is one, uses &.name to get the name if there was a roster item or else nil, and if there was no roster item or no name just show jid) and a helper that adds new messages into the GUI.

    Finally we add a new XMPP event handler for incoming messages that have a body.  Any such incoming message we look up in the global if there is a conversation open already, if so we pass the new message there to have it appended to the GUI table, otherwise we open the conversation with this message as the first thing it will show.

    Getting from the Contact List to a Conversation

    Now we wire up the contact list to the conversation view:

    button_column("Contact") {
    	on_clicked do |row|
    		Conversation.open($roster[row][1].to_s)
    	end
    }

    When a contact button is clicked, grab the Jabber ID from the hidden end of the table row that we had stashed there, and open the conversation.

    horizontal_box {
    	stretchy false
    
    	jid_entry = entry {
    		label("Jabber ID")
    	}
    
    	button("Start Conversation") {
    		stretchy false
    
    		on_clicked do
    			Conversation.open(jid_entry.text)
    		end
    	}
    }

    And let’s provide a way to start a new conversation with an address that isn’t a contact too.  An entry to type in a Jabber ID and a button that opens the conversation.

    Adding a Contact

    Might as well add a button to the main window that re-uses that entry box to allow adding a contact as well:

    button("Add Contact") {
    	stretchy false
    
    	on_clicked do
    		BLATHER.my_roster << jid_entry.text
    	end
    }

    Handling Multiple Devices

    In many chat protocols, it is common to have multiple devices or apps connected simultaneously. It is often desirable to show messages sent to or from one device on all the others as well.  So let’s implement that.  First, a helper for creating XML structures we may need:

    def xml_child(parent, name, namespace)
    	child = Niceogiri::XML::Node.new(name, parent.document, namespace)
    	parent << child
    	child
    end

    We need to tell the server that we support this feature:

    when_ready do
    	self << Blather::Stanza::Iq.new(:set).tap { |iq|
    		xml_child(iq, :enable, "urn:xmpp:carbons:2")
    	}
    end

    We will be handling live messages from multiple event handlers so let’s pull the live message handling out into a helper:

    def handle_live_message(m, counterpart: m.from.stripped.to_s)
    	LibUI.queue_main do
    		conversation = $conversations[counterpart]
    		if conversation
    			conversation.new_message(m)
    		else
    			Conversation.open(counterpart, m)
    		end
    	end
    end

    And the helper that will handle messages from other devices of ours:

    def handle_carbons(fwd, counterpart:)
    	fwd = fwd.first if fwd.is_a?(Nokogiri::XML::NodeSet)
    	return unless fwd
    
    	m = Blather::XMPPNode.import(fwd)
    	return unless m.is_a?(Blather::Stanza::Message) && m.body.present?
    
    	handle_live_message(m, counterpart: counterpart.call(m))
    end

    This takes in the forwarded XML object (allowing for it to be a set of which we take the first one) and imports it with Blather’s logic to become hopefully a Message object.  If it’s not a Message or has no body, we don’t really care so we stop there. Otherwise we can handle this extracted message as though we had received it ourselves.

    And then wire up the event handlers:

    message(
    	"./carbon:received/fwd:forwarded/*[1]",
    	carbon: "urn:xmpp:carbons:2",
    	fwd: "urn:xmpp:forward:0"
    ) do |_, fwd|
    	handle_carbons(fwd, counterpart: ->(m) { m.from.stripped.to_s })
    end

    Because XMPP is just XML, we can use regular XPath stuff to extract from incoming messages.  Here we say that if the message contains a forwarded element inside a carbons received element, then we should handle this with the carbons handler instead of just the live messages handler.  The XML that matches our XPath comes in as the second argument and that is what we pass to the handler to get converted into a Message object.

    message(
    	"./carbon:sent/fwd:forwarded/*[1]",
    	carbon: "urn:xmpp:carbons:2",
    	fwd: "urn:xmpp:forward:0"
    ) do |_, fwd|
    	handle_carbons(fwd, counterpart: ->(m) { m.to.stripped.to_s })
    end

    This handler is for messages sent by other devices instead of received by other devices.  It is pretty much the same, except that we know the “other side of the conversation” (here called counterpart) is in the to not the from.

    message :body do |m|
    	handle_live_message(m)
    end

    And our old message-with-body handler now just needs to call the helper.

    History

    So far our client only processes and displays live messages.  If you close the app, or even close a conversation window, the history is gone.  If you chat with another client or device, you can’t see that when you re-open this one.  To fix that we’ll need to store messages persistently, and also fetch any history from while we were disconnected from the server.  We will need a few more lines in our Gemfile first:

    gem "sqlite3"
    gem "xdg"

    And then to set up a basic database schema:

    require "securerandom"
    require "sqlite3"
    require "xdg"
    
    DATA_DIR = XDG::Data.new.home + "jabber-client-demo"
    DATA_DIR.mkpath
    DB = SQLite3::Database.new(DATA_DIR + "db.sqlite3")
    
    if DB.user_version < 1
    	DB.execute(<<~SQL)
    		CREATE TABLE messages (
    			mam_id TEXT PRIMARY KEY,
    			stanza_id TEXT NOT NULL,
    			conversation TEXT NOT NULL,
    			created_at INTEGER NOT NULL,
    			stanza TEXT NOT NULL
    		)
    	SQL
    	DB.execute("CREATE TABLE data (key TEXT PRIMARY KEY, value TEXT)")
    	DB.user_version = 1
    end

    user_version is a SQLite feature that allows storing a simple integer alongside the database.  It starts at 0 if never set, and so here we use it to check if our schema has been created or not.  We store the database in a new directory created according to the XDG Base Directory specification.  There are two relevant IDs for most XMPP operations: the MAM ID (the ID in the server’s archive) and the Stanza ID (which was usually selected by the original sender).  We also create a data table for storing basic key-value stuff, which we’ll use in a minute to remember where we have sync’d up to so far.  Let’s edit the Conversation object to store messages as we send them, updating the send button on_clicked handler:

    def message
    	Blather::Stanza::Message.new(@jid, @message_entry.text, :chat).tap { |m|
    		m.id = SecureRandom.uuid
    	}
    end

    on_clicked do
    	m = message
    	EM.defer do
    		BLATHER << m
    		DB.execute(<<~SQL, [nil, m.id, @jid, m.to_s])
    			INSERT INTO messages
    			(mam_id, stanza_id, conversation, created_at, stanza)
    			VALUES (?,?,?,unixepoch(),?)
    		SQL
    	end
    	@messages << message_row(m)
    	@message_entry.text = ""
    end

    When we send a message we don’t yet know the server’s archive ID, so we set that to nil for now.  We set mam_id to be the primary key, but SQLite allows multiple rows to have NULL in there so this will work.  We don’t want to block the GUI thread while doing database work so we use EM.defer to move this to a worker pool.  We also want to store messages when we receive them live, so add this to the start of handle_live_message:

    mam_id = m.xpath("./ns:stanza-id", ns: "urn:xmpp:sid:0").find { |el|
    	el["by"] == jid.stripped.to_s
    }&.[]("id")
    delay = m.delay&.stamp&.to_i || Time.now.to_i
    DB.execute(<<~SQL, [mam_id, m.id, counterpart, delay, m.to_s])
    	INSERT INTO messages (mam_id, stanza_id, conversation, created_at, stanza)
        VALUES (?,?,?,?,?)
    SQL

    Here we extract the server archive’s ID for the message (added by the server in a stanza-id with by="Your Jabber ID") and figure out what time the message was originally sent (usually this is just right now for a live message, but if it is coming from offline storage because every client was offline or similar, then there can be a “delay” set on it which we can use).  Now that we have stored the history of message we received we need to load them into the GUI when we start up a Conversation so add this at the end of initialize:

    EM.defer do
    	mam_messages = []
    	query = <<~SQL
    		SELECT stanza
    		FROM messages
    		WHERE conversation=?
    		ORDER BY created_at
    	SQL
    	DB.execute(query, [@jid]) do |row|
    		m = Blather::XMPPNode.import(
    			Nokogiri::XML.parse(row[0]).root
    		)
    		mam_messages << m
    	end
    
    	LibUI.queue_main do
    		mam_messages.map! { |m| message_row(m) }
    		@messages.replace(mam_messages + @messages)
    	end
    end

    In the worker pool we load up all the stored messages for the current conversation in order, then we take the XML stored as a string in the database and parse it into a Blather Message object.  Once we’ve done as much of the work as we can in we worker pool we use queue_main to switch back to the GUI thread and actually build the rows for the table and replace them into the GUI.

    With these changes, we are now storing all messages we see while connected and displaying them in the conversation.  But what about messages sent or received by other devices or clients while we were not connected?  For that we need to sync with the server’s archive, fetching messages at a reasonable page size from whatever we already have until the end.

    def sync_mam(last_id)
    	start_mam = Blather::Stanza::Iq.new(:set).tap { |iq|
    		xml_child(iq, :query, "urn:xmpp:mam:2").tap do |query|
    			xml_child(query, :set, "http://jabber.org/protocol/rsm").tap do |rsm|
    				xml_child(rsm, :max, "http://jabber.org/protocol/rsm").tap do |max|
    					max.content = (EM.threadpool_size * 5).to_s
    				end
    				next unless last_id
    
    				xml_child(rsm, :after, "http://jabber.org/protocol/rsm").tap do |after|
    					after.content = last_id
    				end
    			end
    		end
    	}
    
    	client.write_with_handler(start_mam) do |reply|
    		next if reply.error?
    
    		fin = reply.find_first("./ns:fin", ns: "urn:xmpp:mam:2")
    		next unless fin
    
    		handle_rsm_reply_when_idle(fin)
    	end
    end

    The first half of this creates the XML stanza to request a page from the server’s archive. We create a query with a max page size based on the size of our worker threadpool, and ask for messages only after the last known id (if we have one, which we won’t on first run). Then we use write_with_handler to send this request to the server and wait for a reply. The reply is sent after all messages have been sent down (sent seperately, not returned in this reply, see below), but we may still be processing some of them in the worker pool so we next create a helper to wait for the worker pool to be done:

    def handle_rsm_reply_when_idle(fin)
    	unless EM.defers_finished?
    		EM.add_timer(0.1) { handle_rsm_reply_when_idle(fin) }
    		return
    	end
    
    	last = fin.find_first(
    		"./ns:set/ns:last",
    		ns: "http://jabber.org/protocol/rsm"
    	)&.content
    
    	if last
    		DB.execute(<<~SQL, [last, last])
    			INSERT INTO data VALUES ('last_mam_id', ?)
    			ON CONFLICT(key) DO UPDATE SET value=? WHERE key='last_mam_id'
    		SQL
    	end
    	return if fin["complete"].to_s == "true"
    
    	sync_mam(last)
    end

    Poll with a timer until the worker pool is all done so that we aren’t fetching new pages before we have handled the last one.  Get the value of the last archive ID that was part of the page just processed and store it in the database for next time we start up.  If this was the last page (that is, complete="true") then we’re all done, otherwise get the next page.  We need to make sure we actually start this sync process inside the when_ready handler:

    last_mam_id = DB.execute(<<~SQL)[0]&.first
    	SELECT value FROM data WHERE key='last_mam_id' LIMIT 1
    SQL
    sync_mam(last_mam_id)

    And also, we need to actually handle the messages as they come down from the server archive:

    message "./ns:result", ns: "urn:xmpp:mam:2" do |_, result|
    	fwd = result.xpath("./ns:forwarded", ns: "urn:xmpp:forward:0").first
    	fwd = fwd.find_first("./ns:message", ns: "jabber:client")
    	m = Blather::XMPPNode.import(fwd)
    	next unless m.is_a?(Blather::Stanza::Message) && m.body.present?
    
    	mam_id = result.first["id"]&.to_s
    	# Can't really race because we're checking for something from the past
    	# Any new message inserted isn't the one we're looking for here anyway
    	sent = DB.execute(<<~SQL, [m.id])[0][0]
    		SELECT count(*) FROM messages WHERE stanza_id=? AND mam_id IS NULL
    	SQL
    	if sent < 1
    		counterpart = if m.from.stripped.to_s == jid.stripped.to_s
    			m.to.stripped.to_s
    		else
    			m.from.stripped.to_s
    		end
    		delay =
    			fwd.find_first("./ns:delay", ns: "urn:xmpp:delay")
    			&.[]("stamp")&.then(Time.method(:parse))
    		delay = delay&.to_i || m.delay&.stamp&.to_i || Time.now.to_i
    		DB.execute(<<~SQL, [mam_id, m.id, counterpart, delay, m.to_s])
    			INSERT OR IGNORE INTO messages
    			(mam_id, stanza_id, conversation, created_at, stanza)
    			VALUES (?,?,?,?,?)
    		SQL
    	else
    		DB.execute(<<~SQL, [mam_id, m.id])
    			UPDATE messages SET mam_id=? WHERE stanza_id=?
    		SQL
    	end
    end

    Any message which contains a MAM (server archive) result will get handled here.  Just like with carbons we extract the forwarded message and import, making sure it ends up as a Blather Message object with a body.

    Remember how when we stored a sent message we didn’t know the archive ID yet?  Here we check if there is anything in our database already with this stanza ID and no archive ID, if no we will insert it as a new message, but otherwise we can update the row we already have to store the server archive ID on it, which we now know.

    And with that, our client now stores and syncs all history with the server, to give the user a full view of their conversation no matter where or when it happened.

    Display Names

    If a user is added to the contact list with a name, we already show that name instead of their address in conversations.  What if a user is not a contact yet, or we haven’t set a name for them?  It might be useful to be able to fetch any display name they advertise for themselves and show that.  First we add a simple helper to expose write_with_handler outside of the main object:

    public def write_with_handler(stanza, &block)
    	client.write_with_handler(stanza, &block)
    end

    We need an attribute on the Conversation to hold the nickname:

    attr_accessor :nickname

    And then we can use this in Conversation#initialize to fetch the other side’s nickname if they advertise one and we don’t have one for them yet:

    self.nickname = BLATHER.my_roster[jid]&.name || jid
    return unless nickname.to_s == jid.to_s
    
    BLATHER.write_with_handler(
    	Blather::Stanza::PubSub::Items.new(:get).tap { |iq|
    		iq.node = "http://jabber.org/protocol/nick"
    		iq.to = jid
    	}
    ) do |reply|
    	self.nickname = reply.items.first.payload_node.text rescue self.nickname
    end

    Inside the window declaration we can use this as the window title:

    title <=> [self, :nickname]

    and in format_sender we can use this as well:

    return nickname if jid.to_s == @jid.to_s

    Avatars

    Names are nice, but what about pictures?  Can we have nice avatar images that go with each user?  What should we display if they don’t have an avatar set?  Well not only is there a protocol to get an avatar, but a specification that allows all clients to use the same colours to represent things, so we can use a block of that if there is no avatar set.  Let’s generate the colour blocks first.  Add this to Gemfile:

    gem "hsluv"

    Require the library at the top:

    require "hsluv"
    
    $avatars = {}

    And a method on Conversation to use this:

    def default_avatar(string)
    	hue = (Digest::SHA1.digest(string).unpack1("v").to_f / 65536) * 360
    	rgb = Hsluv.rgb_prepare(Hsluv.hsluv_to_rgb(hue, 100, 50))
    	rgba = rgb.pack("CCC") + "xff".b
    	image { image_part(rgba * 32 * 32, 32, 32, 4) }
    end

    This takes the SHA-1 of a string, unpacks the first two bytes as a 16-bit little-endian integer, converts the range from 0 to MAX_SHORT into the range from 0 to 360 for hue degrees, then passes to the library we added to convert from HSV to RGB colour formats.  The GUI library expects images as a byte string where every 4 bytes are 0 to 255 for red, then green, then blue, then transparency.  Because we want a square of all one colour, we can create the byte string for one pixel and then multiply the string by the width and height (multiplying a string by a number in Ruby make a new string with that many copies repeated) to get the whole image.

    In Conversation#initialize we can use this to make a default avatar on the dummy message row then the window first opens:

    @messages = [[default_avatar(""), "", ""]]

    And we will need to add a new column definition to be beginning of the table { block:

    image_column("Avatar")

    And actually add the image to message_row:

    def message_row(m)
    	from = m.from&.stripped || BLATHER.jid.stripped
    	[
    		$avatars[from.to_s] || default_avatar(from.to_s),
    		format_sender(from),
    		m.body
    	]
    end

    If you run this you should now see a coloured square next to each message.  We would now like to get actual avatars, so add this somewhere at the top level to advertise support for this:

    set_caps(
    	"https://git.singpolyma.net/jabber-client-demo",
    	[],
    	["urn:xmpp:avatar:metadata+notify"]
    )

    Then in the when_ready block make sure to send it to the server:

    send_caps

    And handle the avatars as they come in:

    pubsub_event(
    	"//ns:items[@node='urn:xmpp:avatar:metadata']",
    	ns: "http://jabber.org/protocol/pubsub#event"
    ) do |m|
    	id = m.items.first&.payload_node&.children&.first&.[]("id")
    	next $avatars.delete(m.from.stripped.to_s) unless id
    
    	path = DATA_DIR + id.to_s
    	key = m.from.stripped.to_s
    	if path.exist?
    		LibUI.queue_main { $avatars[key] = image(path.to_s, 32, 32) rescue nil }
    	else
    		write_with_handler(
    			Blather::Stanza::PubSub::Items.new(:get).tap { |iq|
    				iq.node = "urn:xmpp:avatar:data"
    				iq.to = m.from
    			}
    		) do |reply|
    			next if reply.error?
    
    			data = Base64.decode64(reply.items.first&.payload_node&.text.to_s)
    			path.write(data)
    			LibUI.queue_main { $avatars[key] = image(path.to_s, 32, 32) rescue nil }
    		end
    	end
    end

    When an avatar metadata event comes in, we check what it is advertising as the ID of the avatar for this user.  If there is none, that means they don’t have an avatar anymore so delete anything we may have in the global cache for them, otherwise create a file path in the same folder as the database based on this ID.  If that file exists already, then no need to fetch it again, create the image from that path on the GUI thread and set it into our global in-memory cache.  If the file does not exist, then use write_with_handler to request their avatar data.  It comes back Base64 encoded, so decode it and then write it to the file.

    If you run this you should now see avatars next to messages for anyone who has one set.

    Delivery Receipts

    The Internet is a wild place, and sometimes things don’t work out how you’d hope.  Sometimes something goes wrong, or perhaps just all of a user’s devices are turned off.  Whatever the reason, it can be useful to see if a message has been delivered to at least one of the intended user’s devices yet or not.  We’ll need a new database column to store that status, add after the end of the DB.user_version < 1 if block:

    if DB.user_version < 2
    	DB.execute(<<~SQL)
    		ALTER TABLE messages ADD COLUMN delivered INTEGER NOT NULL DEFAULT 0
    	SQL
    	DB.user_version = 2
    end

    Let’s advertise support for the feature:

    set_caps(
    	"https://git.singpolyma.net/jabber-client-demo",
    	[],
    	["urn:xmpp:avatar:metadata+notify", "urn:xmpp:receipts"]
    )

    We need to add delivery status and stanza id to the dummy row for the messages table:

    @messages = [[default_avatar(""), "", "", false, nil]]

    And make sure we select the status out of the database when loading up messages:

    SELECT stanza,delivered FROM messages WHERE conversation=? ORDER BY created_at

    And pass that through when building the message rows

    mam_messages << [m, row[1]]

    mam_messages.map! { |args| message_row(*args) }

    Update the messages table to expect the new data model:

    table {
    	image_column("Avatar")
    	text_column("Sender")
    	text_column("Message")
    	checkbox_column("Delivered")
    	editable false
    	cell_rows @messages
    	@messages.clear if @messages.length == 1 && @messages.first.last.nil?
    }

    And update the row builder to include this new data:

    def message_row(m, delivered=false)
    	from = m.from&.stripped || BLATHER.jid.stripped
    	[
    		$avatars[from.to_s] || default_avatar(from.to_s),
    		format_sender(from),
    		m.body,
    		delivered,
    		m.id
    	]
    end

    Inbound messages are always considered delivered, since we have them:

    def new_message(m)
    	@messages << message_row(m, true)
    end

    And a method to allow signalling that a delivery receipt should be displayed, using the fact that we now hide the stanza id off the end of the rows in the table to find the relevant message to update:

    def delivered_message(id)
    	row = @messages.find_index { |r| r.last == id }
    	return unless row
    
    	@messages[row] = @messages[row][0..-3] + [true, id]
    end

    In the Send button’s on_clicked handler we need to actually request that others send us receipts:

    m = message
    xml_child(m, :request, "urn:xmpp:receipts")

    And we need to handle the receipts when they arrive:

    message "./ns:received", ns: "urn:xmpp:receipts" do |m, received|
    	DB.execute(<<~SQL, [received.first["id"].to_s])
    		UPDATE messages SET delivered=1 WHERE stanza_id=?
    	SQL
    
    	conversation = $conversations[m.from.stripped.to_s]
    	return unless conversation
    
    	LibUI.queue_main do
    		conversation.delivered_message(received.first["id"].to_s)
    	end
    end

    When we get a received receipt, we get the id attribute off of it, which represents a stanza ID that this receipt is for.  We update the database, and inform any open conversation window so the GUI can be updated.

    Finally, if someone requests a receipt from us we should send it to them:

    message :body do |m|
    	handle_live_message(m)
    
    	if m.id && m.at("./ns:request", ns: "urn:xmpp:receipts")
    		self << m.reply(remove_children: true).tap { |receipt|
    			xml_child(receipt, :received, "urn:xmpp:receipts").tap { |received|
    				received["id"] = m.id
    			}
    		}
    	end
    end

    If the stanza has an id and a receipt request, we construct a reply that contains just the received receipt and send it.

    Message Correction

    Sometimes people send a message with a mistake in it and want to send another to fix it.  It is convenvient for the GUI to support this and render only the new version of the message.  So let’s implement that.  First we add it to the list of things we advertise support for:

    set_caps(
    	"https://git.singpolyma.net/jabber-client-demo",
    	[],
    	[
    		"urn:xmpp:avatar:metadata+notify",
    		"urn:xmpp:receipts",
    		"urn:xmpp:message-correct:0"
    	]
    )

    Then we need a method on Conversation to process incoming corrections and update the GUI:

    def new_correction(replace_id, m)
    	row = @messages.find_index { |r| r.last == replace_id }
    	return unless row
    
    	@messages[row] = message_row(m, true)
    end

    We look up the message row on the stanza id, just as we did for delivery receipts, and just completely replace it with a row based on the new incoming message.  That’s it for the GUI.  Corrections may come from live messages, from carbons, or even from the server archive if they happened while we were disconnected, so we create a new insert_message helper to handle any case we previously did the SQL INSERT for an incoming message:

    def insert_message(
    	m,
    	mam_id:,
    	counterpart: m.from.stripped.to_s,
    	delay: m.delay&.stamp&.to_i
    )
    	if (replace = m.at("./ns:replace", ns: "urn:xmpp:message-correct:0"))
    		DB.execute(<<~SQL, [m.to_s, counterpart, replace["id"].to_s])
    			UPDATE messages SET stanza=? WHERE conversation=? AND stanza_id=?
    		SQL
    	else
    		delay ||= Time.now.to_i
    		DB.execute(<<~SQL, [mam_id, m.id, counterpart, delay, m.to_s])
    			INSERT OR IGNORE INTO messages
    			(mam_id, stanza_id, conversation, created_at, stanza, delivered)
    			VALUES (?,?,?,?,?,1)
    		SQL
    	end
    end

    The else case here is the same as the INSERTs we’ve been using up to this point, but we also check first for an element that signals this as a replacement and if that is the case we issue an UPDATE instead to correct our internal archive to the new version.

    Then in handle_live_message we also signal the possibly-open GUI:

    if (replace = m.at("./ns:replace", ns: "urn:xmpp:message-correct:0"))
    	conversation.new_correction(replace["id"].to_s, m)
    else
    	conversation.new_message(m)
    end

    We can now display incoming corrections, but it would also be nice to be able to send them.  Add a second button after the Send button in Conversation that can re-use the @message_entry box to correct the most recently sent message:

    button("Correct") {
    	stretchy false
    
    	on_clicked do
    		replace_row = @messages.rindex { |message|
    			message[1] == format_sender(BLATHER.jid.stripped)
    		}
    		next unless replace_row
    
    		m = message
    		m << xml_child(m, :replace, "urn:xmpp:message-correct:0").tap { |replace|
    			replace["id"] = @messages[replace_row].last
    		}
    		EM.defer do
    			BLATHER << m
    			DB.execute(<<~SQL, [m.to_s, @jid, @messages[replace_row].last])
    				UPDATE messages SET stanza=? WHERE conversation=? AND stanza_id=?
    			SQL
    		end
    		@messages[replace_row] = message_row(m, @messages[replace_row][-2])
    		@message_entry.text = ""
    	end
    }

    When the button is clicked we find the row for the most recently sent message, construct a message to send just as in the Send case but add the message correction replace child with the id matching the stanza id of the most recently sent message.  We send that message and also update our own local copy of the stanza both in the database and in the memory model rendered in the GUI.

    Conclusion

    There are a lot more features that a chat system can implement, but hopefully this gives you a useful taste of how each one can be incrementally layered in, and what the considerations might be for a wide variety of different kinds of features.  All the code for the working application developed in this article is available in git under AGPLv3+, with commits that corrospond to the path we took here.

    • chevron_right

      Kaidan - A user-friendly XMPP client, and ATT - Automatic Trust Transfer

      debacle · pubsub.movim.eu / berlin-xmpp-meetup · Saturday, 2 March, 2019 - 12:48 edit

    Kaidan - A user-friendly XMPP client, and ATT - Automatic Trust Transfer

    At this months Berlin XMPP meetup, we will probably

    When? Wednesday, 2019-03-13 18:00 CET

    Where?JWD: Takustraße 3, 14195 Berlin

    #xmpp #meeting #meetup #berlin #jabber #kaidan #client #sprint #att #omemo #jwd

    • chevron_right

      Chinwag Messaging moves to Movim

      Timothée Jaussoin · pubsub.movim.eu / Movim · Wednesday, 22 June, 2016 - 19:18 edit

    Moving to Movim

    Chinwag Messaging an Australian IM service based on XMPP just moved to Movim for their web client solution.

    • Moving to Movim

      Since I started this little project, one of the main things I've been looking for is a decent web interface for chatting. I've tested and rejected a lot of things. Some options came really close to being what I wanted but just didn't seem to have any momentum behind them and stagnated. I wanted t

    • wifi_tethering open_in_new

      This post is public

      nl.movim.eu

    • chevron_right

      Cryptocat, ou le piège du client magique

      Timothée Jaussoin · Wednesday, 6 April, 2016 - 19:35 edit · 6 minutes

    Il y a quelques jours je suis tombé sur un article annonçant la sortie de la nouvelle mouture du client de messagerie sécurisé Cryptocat. Je ne m'étais jamais réellement penché sur ce client car il faisait, selon moi, partie des dizaines d'autres clients surfant sur la vague des messageries instantanées chiffrées sorties suite aux révélations d'E.Snowden. J'ai donc voulu creuser un peu et voir de quoi il était composé.

    Comme vous le savez peut-être, je travaille moi-même sur un client de messagerie "sociale" appelée Movim exploitant le protocole XMPP. J'essaye, au travers de ce projet, de montrer aux gens qu'il est parfaitement possible de créer une solution à la fois sécurisée, standard et décentralisée tout en offrant une interface agréable et simple à comprendre. Petite précision toutefois: Movim ne possède pas (encore, mais j'y travaille) de fonctionnalité de chiffrement de bout en bout. Néanmoins il est possible de faire ce genre de chiffrement sur XMPP de façon standard et compatible avec des technologies standardisées comme OTR et le récent OMEMO (c'est d'ailleurs cette norme que j'ai décidé d'implémenter dans Movim).

    La chose que je trouve dommage avec tous ces services et clients c'est qu'ils ne sont pas compatibles entre eux. Pourtant il existe aujourd'hui plusieurs clients XMPP proposant la norme OTR et OMEMO mais qui ne sont malheureusement pas si connus et médiatisés que les autres. Je mentionnerais en particulier l'excellent Conversations (sur Android) dont l'équipe est à l'origine de la norme OMEMO (qui est elle-même une implémentation de la norme Axolotl dans XMPP, notamment utilisée sur le client Signal).

    Étant un peu curieux par nature je me suis plongé dans le code source de la nouvelle version de ce cher Cryptocat pour regarder un peu ce qu'il a dans le ventre.

    Ni une, ni deux, je clone le dépôt sur ma machine et je commence à explorer le code.

    Première surprise !

    Avant même d'ouvrir le moindre fichier j'avais déjà un petit sourire en coin en voyant le nom de certains d'entre eux dans le répertoire src/js/.

    axolotl.js
    omemo.js
    xmpp.js
    

    Tiens, tiens, tiens. Un petit tour sur la page Security du site me confirme bien ça. Cryptocat utilise XMPP comme transport et OMEMO comme méthode de chiffrement. Le reste du blabla n'est rien d'autre qu'une réexplication de la norme Axolotl sans pour autant la mentionner.

    Donc Cryptocat est un client XMPP, en Javascript, avec la norme OMEMO par dessus.

    XMPP…

    Je ne me suis pas particulièrement penché sur la partie chiffrement du projet, j'avoue ne pas avoir assez de compétences en cryptographie pour offrir une critique constructive et je pense que l'auteur du projet est bien plus expérimenté que moi là dessus. Je salue par ailleurs sa volonté de transparence à ce propos. Néanmoins je commence à bien connaitre le protocole XMPP et je sais que son point fort est la possibilité d'interconnecter les serveurs entre eux et de laisser le choix du serveur à utiliser aux utilisateurs.

    L'interconnexion est une fonctionnalité que j'estime nécessaire car d'une part, elle permet aux utilisateurs d'avoir le choix du serveur sur lequel ils iront déposer leurs données et d'autre part, elle décentralise les points d'échange du réseau (et rend ainsi plus difficile la censure et l'écoute du réseau).

    Après avoir regardé plus en détails le fichier xmpp.js j'ai enfin la confirmation de ce dont je me doutais jusque là.

    jid: username + '@crypto.cat',
    server: 'crypto.cat',
    

    Oui, il y a bien un serveur XMPP écrit en dur dans le code source du projet.

    Explorons un petit peu le serveur crypto.cat

    Le site XMPP IM Observatory nous retourne une note de A pour la sécurité du serveur, ici rien à dire. Par contre je découvre que l'interconnexion avec les autres serveurs du réseau n'est pas autorisée. La connexion entre le client Javascript et le serveur se fait au moyen d'un Websocket et le serveur XMPP est un serveur Prosody tout bête.

    Cryptocat n'est donc pas qu'une application de chat offerte par Nadim Kobeissi, mais aussi un service centralisé auquel les utilisateurs sont contraints de se connecter (sauf s'ils changent le code source du client, ce qui est possible, mais combien le feront ?).

    Précisions sur le chiffrement de bout en bout dans XMPP

    Le chiffrement de bout en bout est une très grande avancée dans l'anonymisation des données échangées. Néanmoins il ne permet pas tout. Sur XMPP par exemple il faut savoir que OTR et OMEMO ne chiffrent que le contenu des messages et pas la signalisation qui les accompagne (certains diront les métadonnées). Mais que pouvons nous trouver dans ces métadonnées ? Et bien l'identifiant de l'émetteur et du destinataire du message mais aussi beaucoup d'autres choses pouvant êtres sensibles. Une page sur le Wiki de Reporters Sans Frontieres résume plutôt bien tout ça. Ces informations peuvent être conservées dans des historiques et journaux sur les serveurs XMPP.

    En choisissant de centraliser son service sur un seul et unique serveur (qui semble être hébergé dans le datacenter d'OVH à Roubaix) l'auteur de Cryptocat a aussi fait le choix de concentrer l'intégralité du trafic de sa plateforme sur un seul et unique point du réseau Internet.

    Cela me serait égal si toutes ces informations étaient précisées sur le site du projet, au moins de façon temporaire si l'objectif à terme était de permettre la connexion à d'autres serveurs XMPP. Mais je trouve vraiment dommage qu'il ait tu ces informations.

    Petit bonus, qu'en est-il de la v1 de Cryptocat ?

    Même si je salue le fait d'utiliser des normes comme XMPP et OMEMO dans Cryptocat, j'ai quand même jeté un œil à la première version du projet pour voir sur quelle architecture il s'était basé.

    Sans grande surprise la première version se base également sur XMPP et OTR. Néanmoins elle permettait de changer le serveur XMPP sur lequel le client se connectait. J'espère donc qu'il sera également possible de faire la même chose sur la seconde version.

    Pas de quoi s'inquiéter donc ?

    Oui, pour le moment pas de quoi crier au scandale mais je souhaiterais que Nadim soit aussi transparent sur l'architecture de son projet que sur les méthodes de chiffrement qu'il utilise au sein de son application.

    Je pense qu'il est aussi important d'informer les utilisateurs de comment sont protégées leurs messages mais aussi par où et comment ceux-ci sont acheminés.

    Dernière petite question, pourquoi vouloir à tout prix fermer la plateforme alors que XMPP permet nativement l'interconnexion avec les autres réseaux ?

    En bref, si vous voulez avoir les mêmes fonctionnalités que Cryptocat, plus tous les avantages de XMPP (interconnexion, gestion de l'historique, profiles, notifications, accusés de réception…) tournez vous vers des clients comme Conversations ou Gajim (et bientôt Movim concernant le chiffrement je vous promet !).

    That's all folks !