For Sonos products that only support Airplay v2 (ERA100s, ERA300s, Sonos Arc Ultram etc.), there is currently a bug when placing your Sonos devices in a separate VLAN and trying to Airplay to them.

See this community thread for more information: https://en.community.sonos.com/advanced-setups-229000/unable-to-airplay-to-era-100-in-separate-vlan-ubiquity-6920955

Update 16/05/2024 notice: version 80.xx.xx.xx

After the mobile app update to version 80.xx.xx.xx, the method that the app use to discover speakers has completely changed. It is now required to allow destination UDP port 1900 (UPNP) traffic and source TCP port 1443 to flow to br0 instead:

iptables -I YazFiFORWARD -i wl0.1 -o br0 -s 172.27.15.3 -p udp --dport 1900 -j ACCEPT
iptables -I YazFiFORWARD -i wl0.1 -o br0 -s 172.27.15.3 -p tcp --sport 1443 -j ACCEPT

Where 172.27.15.3 is the IP address of the Sonos system.

For Airplay to work it is also required to allow destination TCP ephemeral ports, source TCP 7000 (BBS), along with destination UDP 319:320 (PTP) ports to flow to br0:

iptables -I YazFiFORWARD -i wl0.1 -o br0 -s 172.27.15.3 -p tcp --dport 32768:65535 -j ACCEPT
iptables -I YazFiFORWARD -i wl0.1 -o br0 -s 172.27.15.3 -p tcp --sport 7000 -j ACCEPT
iptables -I YazFiFORWARD -i wl0.1 -o br0 -s 172.27.15.3 -p udp --dport 319:320 -j ACCEPT

Where 172.27.15.3 is the IP address of the Sonos system.

I've also noticed that Sonos also need source TCP 4444 (krb524) to br0 while running updates:

iptables -I YazFiFORWARD -i wl0.1 -o br0 -s 172.27.15.3 -p tcp --sport 4444 -j ACCEPT

Where 172.27.15.3 is the IP address of the Sonos system.

Relevant new tcpdumps:

# SONOS APP TESTING
jarylc@RT-AX88U:/jffs/addons/YazFi.d/userscripts.d# tcpdump -i any src 172.27.15.3 and dst 172.27.14.98
08:23:59.721602 wl0.1 In  IP 172.27.15.3.34242 > 172.27.14.98.upnp: UDP, length 741
08:23:59.721669 br0   Out IP 172.27.15.3.34242 > 172.27.14.98.upnp: UDP, length 741
08:23:59.721678 eth7  Out IP 172.27.15.3.34242 > 172.27.14.98.upnp: UDP, length 741
08:23:59.870068 wl0.1 In  IP 172.27.15.3.1443 > 172.27.14.98.50146: Flags [S.], seq 165454314, ack 134039494, win 28960, options [mss 1460,sackOK,TS val 355714560 ecr 4169303348,nop,wscale 7], length 0
08:23:59.870090 br0   Out IP 172.27.15.3.1443 > 172.27.14.98.50146: Flags [S.], seq 165454314, ack 134039494, win 28960, options [mss 1460,sackOK,TS val 355714560 ecr 4169303348,nop,wscale 7], length 0
08:23:59.870093 eth7  Out IP 172.27.15.3.1443 > 172.27.14.98.50146: Flags [S.], seq 165454314, ack 134039494, win 28960, options [mss 1460,sackOK,TS val 355714560 ecr 4169303348,nop,wscale 7], length 0
08:23:59.926271 wl0.1 In  IP 172.27.15.3.1443 > 172.27.14.98.50146: Flags [.], ack 157, win 235, options [nop,nop,TS val 355714574 ecr 4169303497], length 0
08:23:59.926303 br0   Out IP 172.27.15.3.1443 > 172.27.14.98.50146: Flags [.], ack 157, win 235, options [nop,nop,TS val 355714574 ecr 4169303497], length 0
08:23:59.926307 eth7  Out IP 172.27.15.3.1443 > 172.27.14.98.50146: Flags [.], ack 157, win 235, options [nop,nop,TS val 355714574 ecr 4169303497], length 0
08:23:59.926275 wl0.1 In  IP 172.27.15.3.1443 > 172.27.14.98.50146: Flags [P.], seq 1:63, ack 157, win 235, options [nop,nop,TS val 355714574 ecr 4169303497], length 62
08:23:59.926330 br0   Out IP 172.27.15.3.1443 > 172.27.14.98.50146: Flags [P.], seq 1:63, ack 157, win 235, options [nop,nop,TS val 355714574 ecr 4169303497], length 62
08:23:59.926333 eth7  Out IP 172.27.15.3.1443 > 172.27.14.98.50146: Flags [P.], seq 1:63, ack 157, win 235, options [nop,nop,TS val 355714574 ecr 4169303497], length 62
08:23:59.926280 wl0.1 In  IP 172.27.15.3.1443 > 172.27.14.98.50146: Flags [.], seq 63:1511, ack 157, win 235, options [nop,nop,TS val 355714574 ecr 4169303497], length 1448
08:23:59.926357 br0   Out IP 172.27.15.3.1443 > 172.27.14.98.50146: Flags [.], seq 63:1511, ack 157, win 235, options [nop,nop,TS val 355714574 ecr 4169303497], length 1448
08:23:59.926361 eth7  Out IP 172.27.15.3.1443 > 172.27.14.98.50146: Flags [.], seq 63:1511, ack 157, win 235, options [nop,nop,TS val 355714574 ecr 4169303497], length 1448
08:24:01.100909 wl0.1 In  IP 172.27.15.3.1443 > 172.27.14.98.50146: Flags [P.], seq 2093:2124, ack 476, win 243, options [nop,nop,TS val 355714868 ecr 4169304590], length 31
08:24:01.100937 br0   Out IP 172.27.15.3.1443 > 172.27.14.98.50146: Flags [P.], seq 2093:2124, ack 476, win 243, options [nop,nop,TS val 355714868 ecr 4169304590], length 31
08:24:01.100942 eth7  Out IP 172.27.15.3.1443 > 172.27.14.98.50146: Flags [P.], seq 2093:2124, ack 476, win 243, options [nop,nop,TS val 355714868 ecr 4169304590], length 31
08:24:01.100913 wl0.1 In  IP 172.27.15.3.1443 > 172.27.14.98.50146: Flags [F.], seq 2124, ack 476, win 243, options [nop,nop,TS val 355714868 ecr 4169304590], length 0
08:24:01.100960 br0   Out IP 172.27.15.3.1443 > 172.27.14.98.50146: Flags [F.], seq 2124, ack 476, win 243, options [nop,nop,TS val 355714868 ecr 4169304590], length 0
08:24:01.100963 eth7  Out IP 172.27.15.3.1443 > 172.27.14.98.50146: Flags [F.], seq 2124, ack 476, win 243, options [nop,nop,TS val 355714868 ecr 4169304590], length 0
08:24:01.190110 wl0.1 In  IP 172.27.15.3.1443 > 172.27.14.98.50146: Flags [F.], seq 2124, ack 476, win 243, options [nop,nop,TS val 355714891 ecr 4169304590], length 0
08:24:01.190127 br0   Out IP 172.27.15.3.1443 > 172.27.14.98.50146: Flags [F.], seq 2124, ack 476, win 243, options [nop,nop,TS val 355714891 ecr 4169304590], length 0
08:24:01.190130 eth7  Out IP 172.27.15.3.1443 > 172.27.14.98.50146: Flags [F.], seq 2124, ack 476, win 243, options [nop,nop,TS val 355714891 ecr 4169304590], length 0
08:25:01.196936 wl0.1 In  IP 172.27.15.3.33005 > 172.27.14.98.57454: UDP, length 741
08:25:03.196969 br0   Out IP 172.27.15.3.33005 > 172.27.14.98.57454: UDP, length 741
08:25:05.196976 eth7  Out IP 172.27.15.3.33005 > 172.27.14.98.57454: UDP, length 741

# AIRPLAY TESTING
jarylc@RT-AX88U-0BC0:/tmp/home/root# tcpdump -i any src 172.27.15.3 and dst 172.27.14.110
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
08:59:34.272732 wl0.1 In  IP 172.27.15.3.bbs > 172.27.14.110.53646: Flags [S.], seq 361871852, ack 1905326041, win 28960, options [mss 1460,sackOK,TS val 356248160 ecr 2914930491,nop,wscale 7], length 0
08:59:34.272761 br0   Out IP 172.27.15.3.bbs > 172.27.14.110.53646: Flags [S.], seq 361871852, ack 1905326041, win 28960, options [mss 1460,sackOK,TS val 356248160 ecr 2914930491,nop,wscale 7], length 0
08:59:34.272769 eth7  Out IP 172.27.15.3.bbs > 172.27.14.110.53646: Flags [S.], seq 361871852, ack 1905326041, win 28960, options [mss 1460,sackOK,TS val 356248160 ecr 2914930491,nop,wscale 7], length 0
08:59:34.284023 wl0.1 In  IP 172.27.15.3.bbs > 172.27.14.110.53646: Flags [.], ack 281, win 235, options [nop,nop,TS val 356248164 ecr 2914930539], length 0
08:59:34.284038 br0   Out IP 172.27.15.3.bbs > 172.27.14.110.53646: Flags [.], ack 281, win 235, options [nop,nop,TS val 356248164 ecr 2914930539], length 0
08:59:34.284041 eth7  Out IP 172.27.15.3.bbs > 172.27.14.110.53646: Flags [.], ack 281, win 235, options [nop,nop,TS val 356248164 ecr 2914930539], length 0
08:59:34.288263 wl0.1 In  IP 172.27.15.3.bbs > 172.27.14.110.53646: Flags [P.], seq 1:977, ack 281, win 235, options [nop,nop,TS val 356248164 ecr 2914930539], length 976
08:59:34.288281 br0   Out IP 172.27.15.3.bbs > 172.27.14.110.53646: Flags [P.], seq 1:977, ack 281, win 235, options [nop,nop,TS val 356248164 ecr 2914930539], length 976
08:59:34.288285 eth7  Out IP 172.27.15.3.bbs > 172.27.14.110.53646: Flags [P.], seq 1:977, ack 281, win 235, options [nop,nop,TS val 356248164 ecr 2914930539], length 976
08:59:35.333401 wl0.1 In  IP 172.27.15.3 > 172.27.14.110: ICMP 172.27.15.3 udp port 320 unreachable, length 112
08:59:35.333430 br0   Out IP 172.27.15.3 > 172.27.14.110: ICMP 172.27.15.3 udp port 320 unreachable, length 112
08:59:35.333435 eth7  Out IP 172.27.15.3 > 172.27.14.110: ICMP 172.27.15.3 udp port 320 unreachable, length 112
08:59:35.333405 wl0.1 In  IP 172.27.15.3 > 172.27.14.110: ICMP 172.27.15.3 udp port 320 unreachable, length 142
08:59:35.333452 br0   Out IP 172.27.15.3 > 172.27.14.110: ICMP 172.27.15.3 udp port 320 unreachable, length 142
08:59:35.333455 eth7  Out IP 172.27.15.3 > 172.27.14.110: ICMP 172.27.15.3 udp port 320 unreachable, length 142
08:59:35.333408 wl0.1 In  IP 172.27.15.3.46785 > 172.27.14.110.53647: Flags [S.], seq 4199358917, ack 972859376, win 28960, options [mss 1460,sackOK,TS val 356248426 ecr 2879974157,nop,wscale 7], length 0
08:59:35.333474 br0   Out IP 172.27.15.3.46785 > 172.27.14.110.53647: Flags [S.], seq 4199358917, ack 972859376, win 28960, options [mss 1460,sackOK,TS val 356248426 ecr 2879974157,nop,wscale 7], length 0
08:59:35.333477 eth7  Out IP 172.27.15.3.46785 > 172.27.14.110.53647: Flags [S.], seq 4199358917, ack 972859376, win 28960, options [mss 1460,sackOK,TS val 356248426 ecr 2879974157,nop,wscale 7], length 0

# UPDATE TESTING
jarylc@RT-AX88U-0BC0:/tmp/home/root# tcpdump -i any src 172.27.15.3 or src 172.27.15.1 or src 172.27.15.3
08:22:01.901621 wl0.1 In  IP 172.27.15.3.krb524 > 172.27.14.98.35390: Flags [R.], seq 0, ack 3400968069, win 0, length 0
08:22:01.901646 br0   Out IP 172.27.15.3.krb524 > 172.27.14.98.35390: Flags [R.], seq 0, ack 1, win 0, length 0
08:22:01.901653 eth7  Out IP 172.27.15.3.krb524 > 172.27.14.98.35390: Flags [R.], seq 0, ack 1, win 0, length 0

Overall, my latest script looks like this now:

HOME_ASSISTANT='172.27.14.254'
MEDIA_RANGE='172.27.15.2-172.27.15.20'
iptables -I YazFiFORWARD -i wl0.1 -o br0 -d ${HOME_ASSISTANT} -p tcp -m multiport --dports 1400,8123 -j ACCEPT
iptables -I YazFiFORWARD -i wl0.1 -o br0 -m iprange --src-range "${MEDIA_RANGE}" -p tcp --dport 32768:65535 -j ACCEPT
iptables -I YazFiFORWARD -i wl0.1 -o br0 -m iprange --src-range "${MEDIA_RANGE}" -p udp -m multiport --dports 319,320,1900 -j ACCEPT
iptables -I YazFiFORWARD -i wl0.1 -o br0 -m iprange --src-range "${MEDIA_RANGE}" -p tcp -m multiport --sports 1400,1443,4444,7000 -j ACCEPT

End of update here, original post follows

Most recently I've discovered that Asuswrt-Merlin (community firmware for Asus routers) has a terminal UI interface `amtm`.

Within it there is a project called Yazfi by @jackyaz which provides a "feature expansion of guest Wi-Fi networks on AsusWRT-Merlin".

I've always wanted to move all my smart devices to another VLAN with only pinhole access to my main LAN. Thanks to Yazfi, the effort is reduced tremendously trying to configure these while leveraging on the Guest Network functionality of Asus Routers. With that, I started my project to reformat my Asus Router, rotating my Wi-Fi passwords and start moving everything over.

I did however encounter some issues with discovering Sonos devices on the mobile app even with `One way to guest` enabled, it was either totally not discovering or very slow, and decided to start digging into it.

Investigation

This section explains how I investigated how the Sonos works in my network under the hood and you can skip to the Solution section below if you only care about that!

Temporarily allowing two-way traffic between LAN and Guest Network

Since we are investigating how Sonos operates, I temporarily allowed two-way communication between the 2 networks in Yazfi configuration.

Enabling multicast routing

Since various online sources says that Sonos uses Simple Service Discovery Protocol (SSDP) and multicast DNS (mDNS) to support discoverability, for good measure, I enabled multicast routing under LAN > IPTV > Enable multicast routing

However I'm not 100% sure there is a need for this.

Installing entware and tcpdump on my router

You can follow these instructions to install entware on your router.

After installationof entware is complete, it's as simple as

jarylc@RT-AX88U:/tmp/home/root# opkg install tcpdump

Running tcpdump

After snooping around the UI and the Sonos App, the private IP addresses of the relevant devices are the following:

• My Android phone: 172.27.14.98 (172.27.14.x is my main LAN)
• One of my Sonos devices: 172.27.15.3 (172.27.15.x is my smart devices Guest network)

I limited the snooping to UDP packets only as that is what SSDP and mDNS primarily uses

jarylc@RT-AX88U:/jffs/addons/YazFi.d/userscripts.d# tcpdump -i any 'udp and (src 172.27.15.3 or src 172.27.14.98)'
09:26:30.044460 wl0.4 Out IP 172.27.14.98.37972 > 239.255.255.250.upnp: UDP, length 200
09:26:30.044469 wl0.1 Out IP 172.27.14.98.37972 > 239.255.255.250.upnp: UDP, length 200
09:26:30.044433 br0   M   IP 172.27.14.98.37972 > 239.255.255.250.upnp: UDP, length 200
09:26:30.044582 wl0.4 Out IP 172.27.14.98.37972 > 255.255.255.255.upnp: UDP, length 200
09:26:30.044589 wl0.1 Out IP 172.27.14.98.37972 > 255.255.255.255.upnp: UDP, length 200
09:26:30.044443 br0   B   IP 172.27.14.98.37972 > 255.255.255.255.upnp: UDP, length 200
09:26:30.045267 wl0.4 Out IP 172.27.14.98.37972 > 239.255.255.250.upnp: UDP, length 254
09:26:30.045274 wl0.1 Out IP 172.27.14.98.37972 > 239.255.255.250.upnp: UDP, length 254
09:26:30.045259 br0   M   IP 172.27.14.98.37972 > 239.255.255.250.upnp: UDP, length 254
09:26:30.045539 wl0.4 Out IP 172.27.14.98.37972 > 255.255.255.255.upnp: UDP, length 254
09:26:30.045544 wl0.1 Out IP 172.27.14.98.37972 > 255.255.255.255.upnp: UDP, length 254
09:26:30.045530 br0   B   IP 172.27.14.98.37972 > 255.255.255.255.upnp: UDP, length 254
09:26:30.048134 wl0.4 Out IP 172.27.14.98.5353 > mdns.mcast.net.5353: 1 PTR (QU)? _sonos._tcp.local. (35)
09:26:30.048139 wl0.1 Out IP 172.27.14.98.5353 > mdns.mcast.net.5353: 1 PTR (QU)? _sonos._tcp.local. (35)
09:26:30.048125 br0   M   IP 172.27.14.98.5353 > mdns.mcast.net.5353: 1 PTR (QU)? _sonos._tcp.local. (35)
09:26:30.227307 wl0.1 M   IP 172.27.15.3.5353 > mdns.mcast.net.5353: 0*- [0q] 1/0/6 PTR Sonos-XXXXXXXXXXXX@Media Room._sonos._tcp.local. (676)
09:26:30.266863 wl0.4 Out IP 172.27.14.98.37972 > 239.255.255.250.upnp: UDP, length 254
09:26:30.266871 wl0.1 Out IP 172.27.14.98.37972 > 239.255.255.250.upnp: UDP, length 254
09:26:30.266828 br0   M   IP 172.27.14.98.37972 > 239.255.255.250.upnp: UDP, length 254
09:26:30.266945 wl0.4 Out IP 172.27.14.98.37972 > 255.255.255.255.upnp: UDP, length 254
09:26:30.266950 wl0.1 Out IP 172.27.14.98.37972 > 255.255.255.255.upnp: UDP, length 254
09:26:30.266840 br0   B   IP 172.27.14.98.37972 > 255.255.255.255.upnp: UDP, length 254
09:26:30.580416 wl0.1 In  IP 172.27.15.3.37360 > 172.27.14.98.37972: UDP, length 741
09:26:30.580456 br0   Out IP 172.27.15.3.37360 > 172.27.14.98.37972: UDP, length 741
09:26:31.072123 wl0.4 Out IP 172.27.14.98.5353 > mdns.mcast.net.5353: 2 PTR (QM)? _sonos._tcp.local. (35)
09:26:31.072128 wl0.1 Out IP 172.27.14.98.5353 > mdns.mcast.net.5353: 2 PTR (QM)? _sonos._tcp.local. (35)
09:26:31.071869 br0   M   IP 172.27.14.98.5353 > mdns.mcast.net.5353: 2 PTR (QM)? _sonos._tcp.local. (35)
09:26:31.072203 wl0.4 Out IP 172.27.14.98.37972 > 239.255.255.250.upnp: UDP, length 200
09:26:31.072208 wl0.1 Out IP 172.27.14.98.37972 > 239.255.255.250.upnp: UDP, length 200
09:26:31.071885 br0   M   IP 172.27.14.98.37972 > 239.255.255.250.upnp: UDP, length 200
09:26:31.072269 wl0.4 Out IP 172.27.14.98.37972 > 255.255.255.255.upnp: UDP, length 200
09:26:31.072274 wl0.1 Out IP 172.27.14.98.37972 > 255.255.255.255.upnp: UDP, length 200
09:26:31.071892 br0   B   IP 172.27.14.98.37972 > 255.255.255.255.upnp: UDP, length 200
09:26:31.309102 wl0.1 In  IP 172.27.15.3.38139 > 172.27.14.98.37972: UDP, length 741
09:26:31.309125 br0   Out IP 172.27.15.3.38139 > 172.27.14.98.37972: UDP, length 741
09:26:31.343765 wl0.4 Out IP 172.27.14.98.37972 > 239.255.255.250.upnp: UDP, length 254
09:26:31.343773 wl0.1 Out IP 172.27.14.98.37972 > 239.255.255.250.upnp: UDP, length 254
09:26:31.343737 br0   M   IP 172.27.14.98.37972 > 239.255.255.250.upnp: UDP, length 254
09:26:31.343853 wl0.4 Out IP 172.27.14.98.37972 > 255.255.255.255.upnp: UDP, length 254
09:26:31.343858 wl0.1 Out IP 172.27.14.98.37972 > 255.255.255.255.upnp: UDP, length 254
09:26:31.343748 br0   B   IP 172.27.14.98.37972 > 255.255.255.255.upnp: UDP, length 254
09:26:31.692594 wl0.1 In  IP 172.27.15.3.38570 > 172.27.14.98.37972: UDP, length 741
09:26:31.692617 br0   Out IP 172.27.15.3.38570 > 172.27.14.98.37972: UDP, length 741
09:26:32.271746 wl0.4 Out IP 172.27.14.98.5353 > mdns.mcast.net.5353: 1 PTR (QU)? _sonos._tcp.local. (35)
09:26:32.271753 wl0.1 Out IP 172.27.14.98.5353 > mdns.mcast.net.5353: 1 PTR (QU)? _sonos._tcp.local. (35)
09:26:32.271598 br0   M   IP 172.27.14.98.5353 > mdns.mcast.net.5353: 1 PTR (QU)? _sonos._tcp.local. (35)
09:26:32.278203 eth7  M   IP 172.27.14.98.37972 > 239.255.255.250.upnp: UDP, length 254
09:26:32.278226 wl0.4 Out IP 172.27.14.98.37972 > 239.255.255.250.upnp: UDP, length 254
09:26:32.278232 wl0.1 Out IP 172.27.14.98.37972 > 239.255.255.250.upnp: UDP, length 254
09:26:32.278203 br0   M   IP 172.27.14.98.37972 > 239.255.255.250.upnp: UDP, length 254
09:26:32.278296 wl0.4 Out IP 172.27.14.98.37972 > 255.255.255.255.upnp: UDP, length 254
09:26:32.278301 wl0.1 Out IP 172.27.14.98.37972 > 255.255.255.255.upnp: UDP, length 254
09:26:32.278214 br0   B   IP 172.27.14.98.37972 > 255.255.255.255.upnp: UDP, length 254
09:26:32.465141 wl0.1 M   IP 172.27.15.3.5353 > mdns.mcast.net.5353: 0*- [0q] 1/0/6 PTR Sonos-XXXXXXXXXXXX@Media Room._sonos._tcp.local. (676)
09:26:32.701778 wl0.1 In  IP 172.27.15.3.41457 > 172.27.14.98.37972: UDP, length 741
09:26:32.701801 br0   Out IP 172.27.15.3.41457 > 172.27.14.98.37972: UDP, length 741
09:26:33.215347 wl0.4 Out IP 172.27.14.98.5353 > mdns.mcast.net.5353: 2 PTR (QM)? _sonos._tcp.local. (35)
09:26:33.215355 wl0.1 Out IP 172.27.14.98.5353 > mdns.mcast.net.5353: 2 PTR (QM)? _sonos._tcp.local. (35)
09:26:33.215331 br0   M   IP 172.27.14.98.5353 > mdns.mcast.net.5353: 2 PTR (QM)? _sonos._tcp.local. (35)
09:26:33.905757 wl0.4 Out IP 172.27.14.98.37972 > 239.255.255.250.upnp: UDP, length 254
09:26:33.905763 wl0.1 Out IP 172.27.14.98.37972 > 239.255.255.250.upnp: UDP, length 254
09:26:33.905736 br0   M   IP 172.27.14.98.37972 > 239.255.255.250.upnp: UDP, length 254
09:26:33.905846 wl0.4 Out IP 172.27.14.98.37972 > 255.255.255.255.upnp: UDP, length 254
09:26:33.905851 wl0.1 Out IP 172.27.14.98.37972 > 255.255.255.255.upnp: UDP, length 254
09:26:33.905744 br0   B   IP 172.27.14.98.37972 > 255.255.255.255.upnp: UDP, length 254
09:26:34.034082 wl0.1 In  IP 172.27.15.3.59807 > 172.27.14.98.37972: UDP, length 741
09:26:34.034122 br0   Out IP 172.27.15.3.59807 > 172.27.14.98.37972: UDP, length 741
09:26:34.220437 wl0.4 Out IP 172.27.14.98.5353 > mdns.mcast.net.5353: 3 PTR (QM)? _sonos._tcp.local. (35)
09:26:34.220444 wl0.1 Out IP 172.27.14.98.5353 > mdns.mcast.net.5353: 3 PTR (QM)? _sonos._tcp.local. (35)
09:26:34.220422 br0   M   IP 172.27.14.98.5353 > mdns.mcast.net.5353: 3 PTR (QM)? _sonos._tcp.local. (35)
09:26:34.387343 wl0.1 M   IP 172.27.15.3.5353 > mdns.mcast.net.5353: 0*- [0q] 1/0/6 PTR Sonos-XXXXXXXXXXXX@Media Room._sonos._tcp.local. (676)

The problem

09:26:30.580416 wl0.1 In  IP 172.27.15.3.37360 > 172.27.14.98.37972: UDP, length 741
09:26:30.580456 br0   Out IP 172.27.15.3.37360 > 172.27.14.98.37972: UDP, length 741
09:26:31.309102 wl0.1 In  IP 172.27.15.3.38139 > 172.27.14.98.37972: UDP, length 741
09:26:31.309125 br0   Out IP 172.27.15.3.38139 > 172.27.14.98.37972: UDP, length 741
09:26:32.701778 wl0.1 In  IP 172.27.15.3.41457 > 172.27.14.98.37972: UDP, length 741
09:26:32.701801 br0   Out IP 172.27.15.3.41457 > 172.27.14.98.37972: UDP, length 741
09:26:34.034082 wl0.1 In  IP 172.27.15.3.59807 > 172.27.14.98.37972: UDP, length 741
09:26:34.034122 br0   Out IP 172.27.15.3.59807 > 172.27.14.98.37972: UDP, length 741

Filtering out all the noise, I noticed that the Sonos device was trying to send a UDP packet from the ephemeral UDP ports (range 32768-65535) to my mobile device.

This is actually not surprising as that is how SSDP works as the device would switch to unicast after being discovered. Prior to this, I was not very familiar with SSDP and did not know that.

I noticed in a previous tcpdump before enabling two-way communication that this was being blocked:

00:13:65.371058 wl0.1 In  IP 172.27.15.3.46814 > 172.27.14.98.49972: UDP, length 737
00:13:54.373059 wl0.1 Out IP 172.27.15.1 > 172.27.15.3: ICMP 172.27.14.98 udp port 49972 unreachable, length 556

Solution

Adding custom firewall allow rules to Yazfi to allow this UDP traffic from Sonos devices

Yazfi allows additional iptables rules by adding a shell script with iptables commands to any file in `/jffs/addons/YazFi.d/userscripts.d/`

With that I created a file called `myscript.sh` within the folder and placed this in the file to allow UDP traffic on the ephemeral ports (range 32768-65535) originating from the Sonos device to flow back to the main LAN

#!/bin/sh
iptables -I YazFiFORWARD -i wl0.1 -o br0 -s 172.27.15.3 -p udp --dport 32768:65535 -j ACCEPT

• `-I YazFiForward` - adds the rule to the FORWARD chain in the YazFi table
• `-i wl0.1` - make sure to change this to your actual Guest Wi-Fi interface
• `-o br0` - packets moving to the bridge interface
• `-s 172.27.15.3` - make sure to change this to your actual Sonos device's IP address
• `-p udp` - UDP packets only
• `--dport 32768:65536` - ephemeral UDP port ranges only
• `-j ACCEPT` - allow packets filtered by this rule

You can duplicate that iptables line for as many Sonos device you have and their respective IPs, or you can also use `-srcrange` (i.e. `-m iprange --src-range 172.27.15.2-172.27.15.9`) if you have configured continuous static DHCP IPs for them.

Be reminded to `chmod +x` the file afterwards

jarylc@RT-AX88U:/jffs/addons/YazFi.d/userscripts.d# chmod +x myscript.sh

Reverting to only allowing one-way (LAN to Guest) communication

During investigation, I allowed two-way communication, so I am reverting it to one-way only in the Yazfi configuration.

Testing the Sonos app again

Sure enough, after allowing the UDP packet to flow back allows the Sonos app to be snappy again.

Bonus

HomeAssistant

I integrated my Sonos systems with HomeAssistant as well, it is also recommended by them to enable communication back via port 1400

iptables -I YazFiFORWARD -i wl0.1 -o br0 -d 172.27.14.100 -p tcp --dport 1400 -j ACCEPT

• `-I YazFiForward` - adds the rule to the FORWARD chain in the YazFi table
• `-i wl0.1` - make sure to change this to your actual Guest Wi-Fi interface
• `-o br0` - packets moving to the bridge interface
• `-s 172.27.14.100` - make sure to change this to your actual HomeAssistant's IP address
• `-p tcp` - TCP packets only
• `-dport 1400` - port 1400 only
• `-j ACCEPT` - allow packets filtered by this rule

It is also highly recommended to allow the Sonos systems to communicate with HomeAssistant directly (especially to use local media storage if you have any)

iptables -I YazFiFORWARD -i wl0.1 -o br0 -d 172.27.14.100 -p tcp --dport 8123 -j ACCEPT

• `-I YazFiForward` - adds the rule to the FORWARD chain in the YazFi table
• `-i wl0.1` - make sure to change this to your actual Guest Wi-Fi interface
• `-o br0` - packets moving to the bridge interface
• `-s 172.27.14.100` - make sure to change this to your actual HomeAssistant's IP address
• `-p tcp` - TCP packets only
• `-dport 8123` - make sure to change this to your actual HomeAssistant's port
• `-j ACCEPT` - allow packets filtered by this rule

const { dirname, resolve } = require('path')
const Layer = require('router/lib/layer')
const { issueCookie } = require(resolve(dirname(require.resolve('n8n')), 'auth/jwt'))
const ignoreAuthRegexp = /^\/(assets|healthz|webhook|rest\/oauth2-credential)/
module.exports = {
    n8n: {
        ready: [
            async function ({ app }, config) {
                const { stack } = app.router
                const index = stack.findIndex((l) => l.name === 'cookieParser')
                stack.splice(index + 1, 0, new Layer('/', {
                    strict: false,
                    end: false
                }, async (req, res, next) => {
                    // skip if URL is ignored
                    if (ignoreAuthRegexp.test(req.url)) return next()

                    // skip if user management is not set up yet
                    if (!config.get('userManagement.isInstanceOwnerSetUp', false)) return next()

                    // skip if cookie already exists
                    if (req.cookies?.['n8n-auth']) return next()

                    // if N8N_FORWARD_AUTH_HEADER is not set, skip
                    if (!process.env.N8N_FORWARD_AUTH_HEADER) return next()

                    // if N8N_FORWARD_AUTH_HEADER header is not found, skip
                    const email = req.headers[process.env.N8N_FORWARD_AUTH_HEADER.toLowerCase()]
                    if (!email) return next()

                    // search for user with email
                    const user = await this.dbCollections.User.findOneBy({email})
                    if (!user) {
                        res.statusCode = 401
                        res.end(`User ${email} not found, please have an admin invite the user first.`)
                        return
                    }
                    if (!user.role) {
                        user.role = {}
                    }

                    // issue cookie if all is OK
                    issueCookie(res, user)
                    return next()
                }))
            },
        ],
    },
}

From v1.87.0 onwards, please use this snippet instead

const { dirname, resolve } = require('path')
const Layer = require('router/lib/layer')
const { issueCookie } = require(resolve(dirname(require.resolve('n8n')), 'auth/jwt'))
const ignoreAuthRegexp = /^\/(assets|healthz|webhook|rest\/oauth2-credential)/
module.exports = {
    n8n: {
        ready: [
            async function ({ app }, config) {
                const { stack } = app.router
                const index = stack.findIndex((l) => l.name === 'cookieParser')
                stack.splice(index + 1, 0, new Layer('/', {
                    strict: false,
                    end: false
                }, async (req, res, next) => {
                    // skip if URL is ignored
                    if (ignoreAuthRegexp.test(req.url)) return next()

                    // skip if user management is not set up yet
                    if (!config.get('userManagement.isInstanceOwnerSetUp', false)) return next()

                    // skip if cookie already exists
                    if (req.cookies?.['n8n-auth']) return next()

                    // if N8N_FORWARD_AUTH_HEADER is not set, skip
                    if (!process.env.N8N_FORWARD_AUTH_HEADER) return next()

                    // if N8N_FORWARD_AUTH_HEADER header is not found, skip
                    const email = req.headers[process.env.N8N_FORWARD_AUTH_HEADER.toLowerCase()]
                    if (!email) return next()

                    // search for user with email
                    const user = await this.dbCollections.User.findOneBy({email})
                    if (!user) {
                        res.statusCode = 401
                        res.end(`User ${email} not found, please have an admin invite the user first.`)
                        return
                    }

                    // issue cookie if all is OK
                    issueCookie(res, user)
                    return next()
                }))
            },
        ],
    },
}

I use Authelia as my SSO solution for my home services, however after n8n released v1.0, they seem to have removed Basic Authentication and locked SAML and OIDC behind an enterprise license. They seem to have removed the options to disable user management completely as well. I do believe they have never supported Trusted Header SSO even before v1.0.

My main goal was to avoid the need to key in 2 sets of credentials every time (Authelia and n8n), and to have a more seamless SSO experience by bypassing the n8n login page.

Solution

I would not have easily found this solution without the help of @MutedJam and @netroy on the N8N community forum.

With the help of the `n8n.ready` backend external hook, we are able to intercept and add another middleware to issue a JWT authentication token when the header is detected.

Assumptions

• n8n is hosted using Docker
• Your reverse proxy is setting and forwarding the Remote-Email header from Authelia
• n8n is already secured by Authelia on a reverse proxy

Remote-Email is used by majority of the reverse proxy guides on Authelia documentation, if you did not customize much, it should be the same.

Instructions

Ensure your user's e-mail matches Authelia

Make sure to change it to match your LDAP store or flat file depending on your Authelia configuration.

Creating the External Hook file

Create a file named `hooks.js` and place this file somewhere your n8n instance can reach.

In the case of the official Docker image, it is where you mounted `/home/node/` to, and for the sake of this guide, I will place it at where it would be effectively mounted at `/home/node/.n8n/hooks.js`.

Copy and paste the following into the file:

const { dirname, resolve } = require('path')
const Layer = require('express/lib/router/layer')
const { issueCookie } = require(resolve(dirname(require.resolve('n8n')), 'auth/jwt'))
const ignoreAuthRegexp = /^\/(assets|healthz|webhook|rest\/oauth2-credential)/
module.exports = {
    n8n: {
        ready: [
            async function ({ app }, config) {
                const { stack } = app._router
                const index = stack.findIndex((l) => l.name === 'cookieParser')
                stack.splice(index + 1, 0, new Layer('/', {
                    strict: false,
                    end: false
                }, async (req, res, next) => {
                    // skip if URL is ignored
                    if (ignoreAuthRegexp.test(req.url)) return next()

                    // skip if user management is not set up yet
                    if (!config.get('userManagement.isInstanceOwnerSetUp', false)) return next()

                    // skip if cookie already exists
                    if (req.cookies?.['n8n-auth']) return next()

                    // if N8N_FORWARD_AUTH_HEADER is not set, skip
                    if (!process.env.N8N_FORWARD_AUTH_HEADER) return next()

                    // if N8N_FORWARD_AUTH_HEADER header is not found, skip
                    const email = req.headers[process.env.N8N_FORWARD_AUTH_HEADER.toLowerCase()]
                    if (!email) return next()

                    // search for user with email
                    const user = await this.dbCollections.User.findOneBy({email})
                    if (!user) {
                        res.statusCode = 401
                        res.end(`User ${email} not found, please have an admin invite the user first.`)
                        return
                    }

                    // issue cookie if all is OK
                    issueCookie(res, user)
                    return next()
                }))
            },
        ],
    },
}

Configure extra Docker container environment variables

Ensure to append a 2 new environment variable to your n8n instance and re-create the container depending on how you deployed it:

EXTERNAL_HOOK_FILES=/home/node/.n8n/hooks.js
N8N_FORWARD_AUTH_HEADER=Remote-Email

Reference: Registering hooks with EXTERNAL_HOOK_FILES

Precautions

Make sure to secure n8n properly by making it only accessible via your reverse proxy. Do not allow direct access as any user would be able to impersonate someone by sending a custom `Remote-Email` header if so.

Afterword

It seems that it is likely possible to re-create a whole SAML/OIDC set-up just by implementing it in the hook, but it is definitely a project on its own and definitely out of scope for this post. However, you are free to customize the script however you wish according to your needs. There is likely a way to automatically create users as well, but I have yet to really dig into the n8n codes.

I've really only tested this on Nginx, but it should be reverse proxy agnostic.

If you would like another workflow automation software that does not gate OIDC (at least for the first 10 users) with a license, you could look at Windmill.

However, my friends are not very tech-savvy, and I did not want them to go through the trouble of installing r2modman, traversing the UI, typing the export UUID and getting my mods every single time I update. But I would like to continue to manage my mods using r2modman as it makes updating mods a breeze.

I had a few friends who, unfortunately, were working with data limits as well and could not keep downloading a zip file containing hundreds of megabytes worth of all my mods.

Most of my friends are on Windows as well, so I had to cater for that fact.

Solution

A PowerShell script that allows a one-click automatic mod update that also does delta downloads (only downloads changed files).

Said PowerShell script is powered by an S3 bucket, in my case Cloudflare R2, and RClone.

Prerequisites

1. A Cloudflare account
2. A payment method to enable the use of R2 (pay-per-usage)
3. r2modman installed with a few Lethal Company mods already added

Instructions

Add the Cloudflare R2 subscription on your Cloudflare dashboard

Make sure to take note of the limits and keep it below that to stay on the free tier. Unless you have a lot of mods or a lot of friends, it's unlikely to hit the free-tier limits with just this.

Create an R2 bucket

After subscribing, you can now go ahead and create an R2 bucket, it is recommended to select a location hint to a region near yourself and your friends.

In this case, we are calling the bucket `lethalcompany`.

Generating a read-only and a read-write API tokens for the bucket

In our case, we would want 2 types of keys, a read-only one that will be sent to your friends and the other for you to upload mods.

You can manage your API tokens here on the right side of the R2 overview page as of the time of posting:

Note: Make sure to note down the Access ID here for later.

You would want to create a read-only token similar to the following:

Note: Make sure to note down the Access Key ID and Secret Access Key after creating and note down this is for read-only!

And a read-write token as follows:

Note: Make sure to note down the Access Key ID and Secret Access Key after creating and note down this is for read-write!

Find your Lethal Company r2modman profile folder

Open r2modman to Lethal Company and open the Settings section

From there you would want to click on `Browse profile folder` and it would open up your profile folder in Windows Explorer

Download RClone

You have a choice of downloading from 2 official places:

1. Official website
2. Github

In most cases, you should be downloading the Windows Intel/AMD - 64 Bit build.

After downloading, open up the zip archive and traverse into the folder until you see `rclone.exe`

Drag and drop `rclone.exe` onto the Lethal Company r2modman profile folder you've found above

Create RClone configuration for uploading

In your Lethal Company r2modman profile folder, beside the new `rclone.exe`, create a file named: `z_rclone.conf`

Copy and paste the following into `z_rclone.conf`:

[r2]
type = s3
provider = Cloudflare
access_key_id = PASTE_OVER_YOUR_READ_WRITE_ACCESS_KEY_HERE
secret_access_key = PASTE_OVER_YOUR_READ_WRITE_SECRET_ACCESS_KEY_HERE
endpoint = https://PASTE_OVER_YOUR_R2_ID_HERE.r2.cloudflarestorage.com

Make sure to replace the PASTE_OVER values according to your read-write R2 token and Account ID.

Note: The `z_` prefix is important to set up ignore rules, but can be customized later on.

Create mods uploader script

In the same folder, create a file named: `z_upload.bat`

Copy and paste the following into `z_upload.bat`:

# 2>NUL & @CLS & PUSHD "%~dp0" & "%SystemRoot%\System32\WindowsPowerShell\v1.0\powershell.exe" -nol -nop -ep bypass "[IO.File]::ReadAllText('%~f0')|iex" & POPD & EXIT /B

if (Test-Path -Path '.\Lethal Company.exe') {
    Write-Host "Error: Lethal Company.exe detected! Do not run the uploader here!"
    cmd /c pause
    Exit
}

$ErrorActionPreference = "Inquire"

Write-Host 'Uploading...'
./rclone --config ./z_rclone.conf -v sync --checkers 12 --transfers 12 --update --checksum --use-server-modtime --fast-list --delete-excluded . r2:lethalcompany --exclude z_rclone.conf --exclude-from z_exclude.txt

Write-Host "Mod upload complete!"
cmd /c pause

Note: The `z_` prefix is important to set up ignore rules, but can be customized later on.

Create files to exclude list

In the same folder, create a file named: `z_exclude.txt`

Copy and paste the following into `z_exclude.txt`:

z_*
mods.yml
LogOutput.log
*.zip
_state/**
BepInEx/cache/**
BepInEx/plugins/**/*.old
BepInEx/plugins/**/icon.png
BepInEx/plugins/**/manifest.json
BepInEx/plugins/**/README
BepInEx/plugins/**/CHANGELOG
BepInEx/plugins/**/LICENSE
BepInEx/plugins/**/INSTALLING
BepInEx/plugins/**/*.md
BepInEx/plugins/**/*.txt

Note: The `z_` prefix is important to set up ignore rules, but can be customized later on.

Run the upload script

Double-click on `z_upload.bat` and run it to upload your mods, if you have configured your `rclone.conf` file properly, it should get uploaded smoothly.

Note: Your output should be longer than this as you are uploading mods from scratch

Now that your mods are all uploaded, we can set up the mod downloader configuration and script to send to your friends

Create RClone configuration for downloading

In the same folder, create a file named: `rclone.conf`

Copy and paste the following into `rclone.conf`:

[r2]
type = s3
provider = Cloudflare
access_key_id = PASTE_OVER_YOUR_READ_ONLY_ACCESS_KEY_HERE
secret_access_key = PASTE_OVER_YOUR_READ_ONLY_SECRET_ACCESS_KEY_HERE
endpoint = https://PASTE_OVER_YOUR_R2_ID_HERE.r2.cloudflarestorage.com

Make sure to replace the PASTE_OVER values according to your read-only R2 token and Account ID.

Create the mod downloader script

In the same folder, create a file named: `update_mods.bat`

Copy and paste the following into `update_mods.bat`:

# 2>NUL & @CLS & PUSHD "%~dp0" & "%SystemRoot%\System32\WindowsPowerShell\v1.0\powershell.exe" -nol -nop -ep bypass "[IO.File]::ReadAllText('%~f0')|iex" & POPD & EXIT /B

if (-not(Test-Path -Path '.\Lethal Company.exe')) {
    Write-Host "Error: Make sure this file is in the same folder as Lethal Company.exe before running!"
    cmd /c pause
    Exit
}

$ErrorActionPreference = "Inquire"

Write-Host '(1/2) Syncing base files...'
./rclone.exe --config ./rclone.conf -P --stats-one-line copy --checkers 12 --transfers 12 --checksum --use-server-modtime --fast-list r2:lethalcompany . --exclude BepInEx/

Write-Host '(2/2) Syncing mods...'
./rclone.exe --config ./rclone.conf -P --stats-one-line sync --checkers 12 --transfers 12 --checksum --use-server-modtime --fast-list r2:lethalcompany/BepInEx/ ./BepInEx/

Write-Host "Mod update complete!"
cmd /c pause

Test the mod downloader

Open Steam and click on Lethal Company

Click on the gear icon on the right side

Traverse to `Manage` and click on `Browse local files` to open up the Lethal Company game folder on Windows Explorer

Copy `rclone.conf`, `rclone.exe` and `update_mods.bat` from your Lethal Company r2modman profile folder here.

Double click update_mods.bat and run it.

If everything is configured well, the mods should download nicely.

Ship the mod downloader to your friends

Select `rclone.conf`, `rclone.exe` and `update_mods.bat`.

Important: do not select the files starting with `z_`! They are meant to be for you only.

Right click any of the selected files, traverse to `Send to` and click on `Compressed (zipped) folder`

Go ahead and rename the zip file if needed and send it to your friends.

Instruct them to find the Lethal Company folder with the same steps as Test the mod downloader, extract the 3 files there and double-click `update_mods.bat` and run it.

They should observe the same results as you.

After which, the game can be launched and your friends will have the exact same mods as you!

Afterword

Note that a few antiviruses will flag rclone.exe as a virus as a false positive, you may need to add it to your antivirus scan exclusion list or just not use this at all.

Although the setup is lengthy, after everything, the mod update flow is really just a matter of (as seen in the demo video)

1. updating mods with r2modman
2. z_upload.bat
3. get your friends to run update_mods.bat

No need for traversing the r2modman UI at all and pasting UUIDs.

You may notice in my demo, my mod updater has a few more steps, I've excluded them as feel that they are out of scope for this post.

As we are already doing a lot of do-it-yourself, everything above is customizable, and you can add and remove features if you wish to do so, your imagination is the limit!

For personal reasons, I kept Nginx load balancer on the host and did not want to forward ports of containers, but I still wanted to reverse proxy my containers.

Solution

https://hub.docker.com/r/jarylc/docker-hosts-sync

Source: https://gitlab.com/jarylc/docker-hosts-sync

A simple Golang program that I created that updates the host's /etc/hosts file whenever the Docker containers are updated, capturing the private IPs of all the containers each time. Said program is also packaged in a Docker image, so it is easily appended to your own Docker system!

This image automatically mirrors /etc/hosts whenever your Docker containers are changed

Alternatives

jaschweder/hosts

Not multi-architecture. Checks every 5 second intervals (not configurable), instead of on container creation and deletion. Big Docker image even when after compression 35mb vs 2.5mb.

itsziget/hosts-gen

Not multi-architecture. Based off a larger templating project for more use-cases, 8mb vs 2.5mb.

Unfortunately configuring Authelia Forward Authentication or an SSO solution with OpenID Connect (OIDC) or Security Assertion Markup Language (SAML) would break native apps on non-PC devices due to drastically changing the authentication flow.

I wanted a solution that taps on Jellyfin's native authentication flow and only introduces a "pause" after clicking the log in button, which can be handled by any client, while waiting for the Duo push to be approved.

While this solution does not save a lot more space, I was inspired by Jellyfin, Authentik, DUO. 2FA solution but wanted a smaller, non Authentik, alternative.

Also, Docker Hub images for DuoAuthProxy were all outdated, so this was also an opportunity to bring it into my list of automatically updating images powered by Gitlab CI/CD.

Just looking for the Alpine-based DuoAuthProxy docker image built by me? Click here!

Note: Since we will be accessing everything locally in this guide, even all within the docker network, I will not be going through setting up LDAP transit encryption (i.e. Secure LDAPS and StartTLS) as it would be out of scope.

More Demos

Android app login video

Your browser does not support the video element.

Prerequisites

1. Jellyfin on Docker set-up prior
2. A free/paid account with Duo and access to the admin portal

Steps

Starting LLDAP

Replace the following accordingly

• dc=example,dc=com (people normally follow their domain (i.e. example.com for this))
• REPLACE_ME_WITH_YOUR_OWN_JWT_SECRET (this can be any long string really, string length is debatable and out of scope)
• REPLACE_ME_WITH_YOUR_OWN_KEY_SEED (this can be any long string really, string length is debatable and out of scope)

docker run -d \
        --name=LLDAP \
        --network auth \
        -p 17170:17170 \
        -e LLDAP_LDAP_BASE_DN=dc=example,dc=com \
        -e LLDAP_JWT_SECRET=REPLACE_ME_WITH_YOUR_OWN_JWT_SECRET \
        -e LLDAP_KEY_SEED=REPLACE_ME_WITH_YOUR_OWN_KEY_SEED \
        -v /path/to/data:/data \
        --restart unless-stopped \
        lldap/lldap:stable

Reference: https://github.com/lldap/lldap

Configuring LLDAP

Visit the frontend at port 17170 and create a service user for DuoAuthProxy accordingly, you can replace the values if needed.

For the sake of this example, the password of the svc-duoauthproxy user is `password`

Make sure to add the user to either lldap_strict_readonly (read-only) or lldap_password_manager (allow password change on Jellyfin) groups

Creating Duo application

Access your Duo admin console

Go to Applications -> Protect an Application

Search for `proxy` and protect an `LDAP Proxy` application

Copy the credentials to prepare for the next step

Configuring DuoAuthProxy

If you are using my image, it is necessary to craft the configuration file first

Replace the following keys accordingly

• bind_dn
• service_account_username
• service_account_password
• search_dn
• ikey
• skey
• api_host

; Complete documentation about the Duo Auth Proxy can be found here:
; https://duo.com/docs/authproxy_reference

[main]
log_stdout=true

[ad_client]
host=LLDAP
port=3890
auth_type=plain
bind_dn=uid=svc-duoauthproxy,ou=people,dc=example,dc=com
service_account_username=svc-duoauthproxy
service_account_password=password
search_dn=ou=people,dc=example,dc=com
username_attribute=uid
at_attribute=mail

[ldap_server_auto]
ikey=DIXXXXXXXXXXXXXXXXXX
skey=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
api_host=api-XXXXXXXX.duosecurity.com
failmode=secure
client=ad_client
port=1812
exempt_primary_bind=false
exempt_ou_1=uid=svc-duoauthproxy,ou=people,dc=example,dc=com

Reference: https://duo.com/docs/authproxy_reference

Starting DuoAuthProxy

docker run -d \
        --name=DuoAuthProxy \
        --network auth \
        -v /path/to/authproxy.cfg:/app/conf/authproxy.cfg \
        --restart unless-stopped \
        minimages/duoauthproxy

Feel free to use alternatives as well or even host it natively or generate your own image using the official instructions.

Preparing Jellyfin

I would assume you would already have Jellyfin set up, add Jellyfin to the auth network by re-running the container while adding `--network auth` argument to the docker run

Installing LDAP Authentication Jellyfin Plugin

Note: I'm currently using the unstable version of Jellyfin (10.9.0) as of posting, I had to use the unstable plugins repository.

Please be reminded to restart Jellyfin after installing the plugin.

Setting up LDAP Authentication Jellyfin Plugin

Set the following configuration parameters for LDAP Server Settings section

• LDAP Server: DuoAuthProxy
• LDAP Port: 1812
• Secure LDAP: unchecked
• StartTLS: unchecked
• Skip SSL/TLS Verification: checked
• Allow Password Change: (optional, but ensure the service account have the lldap_password_manager group if checked)
• LDAP Bind User: uid=svc-duoauthproxy,ou=people,dc=example,dc=com
• LDAP Bind User Password: password
• LDAP Base DN for searches: ou=people,dc=example,dc=com

At this point click `Save and Test LDAP Server Settings` to test connectivity, it should pass if all your settings are good.

Set the following configuration parameters for LDAP User Settings section

• LDAP Search Filter: (uid=*)
• LDAP Search Attributes: uid, mail
• LDAP Uid Attribute: uid
• LDAP Username Attribute: uid
• LDAP Password Attribute: userPassword
• LDAP Admin Filter: (memberof=cn=lldap_admin,ou=example,dc=com)

Note you should change your LDAP Search Filter and LDAP Admin Filter according to your needs, reference: https://github.com/lldap/lldap/blob/main/example_configs/jellyfin.md

At this point click `Save and Test LDAP Filter Settings` and check if there are more than 0 users and admins found, it should if all your settings are good.

Afterwhich, enter `admin` in `Test Login Name` field and `Save Search Attribute Settings and Query User` button to do a final lookup check and save.

Finally, configure Jellyfin User Settings sections to your needs and don't forget to hit the big blue `Save` button.

(If Jellyfin has existing users) Switch users' Authentication Provider to LDAP-Authentication

Test the new authentication flow

The new authentication flow should be immediately active, you can log out and test it.

If not, you can try another restart of Jellyfin first.

Afterword

Of course, this guide will not be a one-size fit all like the Authentik solution was for me.

A lot of settings can and should be customized to your needs.

This guide only follows through the most basic setup.

I know there are a lot of Telegram bots out there that do this, however I realized there has been a few I tried (even those recommended on online forums) which are outright no longer maintained and stopped working long ago. It took me many tries to find the one that worked.

Solution

Bot: http://t.me/icanhazidbot

Source: https://gitlab.com/jarylc/cf-workers-icanhazid-telegram-bot

I decided to take matters into my own hands and leveraged Cloudflare Workers uptime and reliability to host a bot that simply does the job without a lot of maintenance required (due to serverless). This was inspired by https://icanhazip.com as well which is also hosted on Cloudflare Workers.

If a developer ever requests your user ID, you can simply just tag the bot @icanhazid, wait for the prompt, and tap on the ID.

Feel free to start a chat with that bot to get your own ID as well.

To grab a group chat ID, you would have to invite the bot to a group.

If everything is in order, the reply is almost instantaneous as well!

Hack away!

Sources

main.js

import puppeteer from 'puppeteer';
puppeteer.launch(
{
  headless: true,
  args: [
    "--disable-gpu",
    "--disable-dev-shm-usage",
    "--disable-setuid-sandbox",
    "--no-sandbox",
  ],
}).then(async browser => {
    const page = await browser.newPage();
    // further logic...
  }
)

Not much difference from a regular Puppeteer setup

.dockerignore

.git
*.log
node_modules
Dockerfile

We don't want node_modules folder especially as it might contain locally downloaded Chromium for development

Dockerfile

FROM chromedp/headless-shell

ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true \
    PUPPETEER_EXECUTABLE_PATH=/headless-shell/headless-shell

COPY . /app
WORKDIR /app

RUN apt-get update && \
    apt-get install -y tini curl && \
    curl -fsSL https://deb.nodesource.com/setup_17.x | bash - && \
    apt-get install -y nodejs && \
    rm -rf /var/lib/apt/lists/* && \
    npm install

ENTRYPOINT ["tini", "--"]
CMD ["/usr/bin/node main.js"]

PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true is the most important to keep the image small

chromedp/headless-shell runs Debian and not Alpine for smaller image sadly, hence using apt, I might look into creating an automated build for Alpine images in the future

The reason for this is because this model is from China and requires me to install HUAWEI Push Kit to receive notifications. But I did not want that and needed an alternative.

Now this may seem like a very simple integration, trust me it is not. The doorbell itself has all ports closed and it only communicates with their own cloud servers.

Intercepting Traffic with Zanti Man-in-the-Middle (MitM) Attack

I tried many ways to intercept traffic and see the requests sent by the doorbell when the alarm button is triggered. I settled onto my trusty Android device with Zanti Penetration Testing Application.

Here's a sample request sent according to Zanti where they upload an image to their cloud servers when the button is pressed.

URL: http://oss-cn-hangzhou.aliyuncs.com/xw-cloudstorage/ecateye/7days/(CAMERA_ID)/20210613/A01142912.jpg
Date: Sun Jun 13 14:29:10 GMT+08:00 2021
Method: PUT
Auth: null
Cookie: null
User Agent: (CAMERA_ID)
Mime Type: null

Headers:
date : Sun, 13 Jun 2021 06:29:19 GMT
authorization : OSS (AUTH_KEY)
content-length : 20857
host : oss-cn-hangzhou.aliyuncs.com
content-type : application/octet-stream
user-agent : (CAMERA_ID)

Form params:

DNS "Poisoning" with AdGuard Home DNS Server

Since my entire household is under AdGuard Home DNS (previously PiHole) and the device doesn't seem to be using any custom DNS servers, I have decided to reroute traffic going to oss-cn-hangzhou.aliyuncs.com to my NGINX server.

Proxying and Mirroring with NGINX

Luckily for me, the device uses insecure HTTP connection and I don't have to worry about any certificate checks. I pinged the address to find out that the IP address is 118.31.219.251. It may be a CDN IP but it's a low priority for me to keep up with it. I used the mirror keyword to route traffic to a webhook trigger on my local N8N.io instance

server {
  listen 80;
  listen [::]:80;

  server_name oss-cn-hangzhou.aliyuncs.com;

  location / {
    mirror /mirror;
    mirror_request_body on;
    proxy_pass http://118.31.219.251;
  }
  location /mirror {
    internal;
    proxy_method POST;
    proxy_pass http://(N8N_IO_HOSTNAME)/webhook/(WEBHOOK_ID)/;
  }
}

Handling request with N8N.io

1. Grabs the request body from POST webhook (make sure to enable Binary Data option)
2. Sends the image data captured in data binary object to my personal MinIO server
3. Sends the photo and a message to my family's Telegram channel

Final output

It's time to press the doorbell!

pulseaudio and pactl

Instructions

List all existing input sinks

pactl list sinks

Find the name of the existing target input sink (your mic)

Mine was: Name: alsa_output.usb-Kingston_HyperX_Cloud_Revolver_S_000000000000-00.analog-surround-71

Create virtual sinks and combine mic and computer audio as inputs

pacmd load-module module-null-sink sink_name=Share sink_properties=device.description=Share
pacmd load-module module-combine-sink slaves=alsa_output.usb-Kingston_HyperX_Cloud_Revolver_S_000000000000-00.analog-surround-71,Share
pacmd load-module module-loopback sink=Share latency_msec=1
pactl load-module module-remap-source master=Share.monitor source_name=Share_Mic source_properties=device.description="Share_Mic"

Route output audio accordingly

• Route to Share if you don't wish to hear the output
• Route to Simultaneous output if you do wish to hear the output

Use the new Share_Mic as input

Discord

Zoom

Instructions

1. Download offline repository of only required dependencies into a file path

Run command in project root directory:

mvn dependency:go-offline -Dmaven.repo.local=./repo

2. Map pom.xml

Bottom of pom.xml:

<repositories>
    <repository>
        <id>offline</id>
        <name>Offline Repository</name>
        <url>file://${project.basedir}/repo</url>
    </repository>
</repositories>
<pluginRepositories>
    <pluginRepository>
        <id>offline</id>
        <name>Offline Repository</name>
        <url>file://${project.basedir}/repo</url>
    </pluginRepository>
</pluginRepositories>

As simple as that.

1. Docker is installed
2. dnsmasq is installed

Steps

1. Start Docker image from linuxserver.io

Head down to linuxserver/netbootxyz and follow the instructions to get a netboot.xyz container up and running.

2. Configuration

/etc/dnsmasq.conf

port=0
interface=eth0
bind-dynamic
log-dhcp
dhcp-authoritative
dhcp-range=192.168.1.0,proxy,255.255.255.0
pxe-service=x86PC, "NETBOOT (BIOS)", "netboot.xyz.kpxe", 192.168.1.253
pxe-service=X86-64_EFI, "NETBOOT (EFI)", "netboot.xyz.efi", 192.168.1.253

Explanations

• port=0 - (optional) disable DNS server if not in use
• interface=eth0 - (optional) only listen on interface eth0, repeat this line for more interfaces, remove for all interfaces
• log-dhcp - (optional) log DHCP requests
• bind-dynamic - (optional) binds to new interfaces added after start
• dhcp-authoritative - prioritize this DHCP server in a network
• dhcp-range - replace 192.168.1.0 and 255.255.255.0 according to your IP range and subnet, set to proxy mode to proxy your original DHCP server
• pxe-service=x86PC for BIOS systems, replace 192.168.1.253 with the IP of the machine hosting netbootxyz
• pxe-service=X86-64_EFI for UEFI systems, replace 192.168.1.253 with the IP of the machine hosting netbootxyz
• enable-tftp is not included as it already hosted by the Docker container

3. Start your services

Start your services.

Commands

Add

iptables -t nat -A PREROUTING -p tcp -d 123.123.123.123 --dport 25565 -j DNAT --to 10.123.123.123:25565
iptables -A FORWARD -p tcp -d 10.123.123.123 --dport 25565 -j ACCEPT
iptables -t nat -o wg0 -A POSTROUTING -j MASQUERADE

Delete

iptables -t nat -D PREROUTING -p tcp -d 123.123.123.123 --dport 25565 -j DNAT --to 10.123.123.123:25565
iptables -D FORWARD -p tcp -d 10.123.123.123 --dport 25565 -j ACCEPT
iptables -t nat -o wg0 -D POSTROUTING -j MASQUERADE

• Change 123.123.123.123 to your external facing server's public IP address
• Change 10.123.123.123 to the server's Wireguard IP address
• Change all instances of 25565 to a port you wish to forward
• Change wg0 to your Wireguard interface name

Setup

General

Installation

Most package managers should have the required packages named

wireguard-tools wireguard-dkms

Install them on both your server and client(s).

Note: very soon, Wireguard will become baked into the Linux kernel by default and wireguard-dkms will not be needed anymore.

Generation of private and public key pair

(umask 077 && printf "[Interface]\nPrivateKey = " | sudo tee /etc/wireguard/wg0.conf > /dev/null)
wg genkey | sudo tee -a /etc/wireguard/wg0.conf | wg pubkey | sudo tee /etc/wireguard/publickey

Replace wg0 with your desired network device id throughout the article if needed.

This generates a private key and automatically inserts as a configuration line to /etc/wireguard/wg0.conf and a public key saved to /etc/wireguard/publickey automatically. Run it on both your server and client(s) respectively.

Server

Edit /etc/wireguard/wg0.conf

[Interface]
# Private key, automatically generated by above command on the server (should be only 44 characters as of writing)
PrivateKey = -auto generated-

# Private IPv4 and IPv6 address of Server for peers to communicate with when connected, you can replace `123.210` and `123:210` with anything you like throughout the article
Address = 10.123.210.1/24,fd00:123:210::1/112

# Listen port, can be any port you like including 53 if you don't use it for DNS. Must be the same throughout the article.
ListenPort = 51820

# Setup IPv4 and IPv6 iptables to forward the network of peers through the server, not required if only a LAN connection is required (optional)
PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE; ip6tables -A FORWARD -i %i -j ACCEPT; ip6tables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE; ip6tables -D FORWARD -i %i -j ACCEPT; ip6tables -t nat -D POSTROUTING -o eth0 -j MASQUERADE

# Save the configuration to the file on every shutdown, personally I prefer it off because I find it easier to edit the configuration directly rather than to rely on tools
SaveConfig = false


# CLIENT 1
[Peer]

# Public key of the peer, generated by the above command on the peer (also should be only 44 characters as of writing)
PublicKey = -auto generated and copied here-

# Allow IPv4 and IPv6 range from 10.123.210.1-10.123.210.254 and fd00:123:210::1-fd00:123:210::ffff respectively
AllowedIPs = 10.123.210.0/24,fd00:123:210::0/112


# CLIENT 2
[Peer]
# Public key of the peer, generated by the above command on the peer (also should be only 44 characters as of writing)
PublicKey = -auto generated and copied here-

# Allow IPv4 and IPv6 range from 10.123.210.1-10.123.210.254 and fd00:123:210::1-fd00:123:210::ffff respectively
AllowedIPs = 10.123.210.0/24,fd00:123:210::0/112

# ... More peers if required ...

Additional step to allow forwarding (optional)

echo -e "net.ipv4.ip_forward=1\nnet.ipv6.conf.all.forwarding=1" | sudo tee -a /etc/sysctl.d/99-sysctl.conf
sudo sysctl -p

Start the server

sudo systemctl enable --now wg-quick@wg0

Client(s)

Edit /etc/wireguard/wg0.conf

[Interface]
# Private key, automatically generated by above command on the client (should be only 44 characters as of writing)
PrivateKey = -auto generated-

# Private IPv4 and IPv6 address of client, must be static IP (no clashes) because there is no DHCP provided by Wireguard as of writing. Change the `2` to an incremental number for every client
Address = 10.123.210.2/32,fd00:123:210::2/128

# DNS server to use, currently set to Cloudflare
DNS = 1.1.1.1


# SERVER
[Peer]
# Public key of server, generated by the above command on the server (only 44 characters as of writing)
PublicKey = -auto generated and copied here-

# Public IP of server and port configured in the server
Endpoint = -public key of server-:51820

# IP ranges Wireguard will listen on and forward
# AllowedIPs = 10.123.210.0/24,fd00:123:210::0/112 # ROUTE ONLY VIRTUAL PRIVATE NETWORK TRAFFIC
AllowedIPs = 0.0.0.0/5, 8.0.0.0/7, 11.0.0.0/8, 12.0.0.0/6, 16.0.0.0/4, 32.0.0.0/3, 64.0.0.0/2, 128.0.0.0/3, 160.0.0.0/5, 168.0.0.0/6, 172.0.0.0/12, 172.32.0.0/11, 172.64.0.0/10, 172.128.0.0/9, 173.0.0.0/8, 174.0.0.0/7, 176.0.0.0/4, 192.0.0.0/9, 192.128.0.0/11, 192.160.0.0/13, 192.169.0.0/16, 192.170.0.0/15, 192.172.0.0/14, 192.176.0.0/12, 192.192.0.0/10, 193.0.0.0/8, 194.0.0.0/7, 196.0.0.0/6, 200.0.0.0/5, 208.0.0.0/4, ::/0, 10.123.210.0/32 # ROUTE ALL INTERNET TRAFFIC LESS LAN THROUGH

# Constant pings to keep the connection alive and not time out on inactivity
PersistentKeepalive = 25

Connect to server

sudo wg-quick up wg0

Connection information

You can run these commands to check the connection

sudo wg
ping 10.123.210.1

Disconnect from server

sudo wg-quick down wg0

Extra information

networkmanager-wireguard

If you use NetworkManager (especially nm-applet) you can install networkmanager-wireguard or networkmanager-wireguard-git (AUR) for Wireguard capabilities and configuration.

Forward other UDP ports to Wireguard port with iptables

On the server:

sudo iptables -t nat -I PREROUTING -i eth0 -p udp -m multiport --dports 53,80,123,161,443 -j REDIRECT --to-ports 51820

To disable:

sudo iptables -t nat -D PREROUTING -i eth0 -p udp -m multiport --dports 53,80,123,161,443 -j REDIRECT --to-ports 51820

You can add it to PostUp and PostDown. Don't forget ip6tables if needed.

More reading

https://github.com/pirate/wireguard-docs#Interface

lspci | grep VGA
  01:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1)
  02:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1)
  03:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1)
  04:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1)
  06:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1)
  07:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1)

6x Gigabyte GTX 1070 G1 Gaming (7.93 GB VRAM) pool mining Ethereum.

Steps

Firstly we will have to one-time initialize /etc/X11/xorg.conf by using Nvidia's xconfig tool including a cool-bits value of 28.

nvidia-xconfig --allow-empty-initial-configuration --enable-all-gpus --cool-bits=28

Afterwards, we can start a Xorg server headlessly as Nvidia's setting tool requires it to be up.

X :0 & # display :0

Since I am running multiple cards, I would require a loop to go through all of it. Currently these values works best for the cards I used.

• -200 core clock
• +1200 memory clock

# export DISPLAY=:0
# IFS=$'\n'
# for gpu in $(nvidia-smi -L); do
    id=$(echo ${gpu} | sed -e 's/GPU //g' -e 's/:.*//g')
    nvidia-settings -a [gpu:${id}]/GPUPowerMizerMode=1 -a [gpu:${id}]/GPUGraphicsClockOffset[3]=-200 -a [gpu:${id}]/GPUMemoryTransferRateOffset[3]=1200
done

As a final step, I would apply a -40 max watts undervolt as root.

nvidia-smi -pl 140

Optionally, since we do not need the Xorg server to be running anymore, we can kill it

killall Xorg

These configuration alone increased ~5MH/s per card, resulting in a 25-30MH/s overall increase!

Automated bash script

#!/usr/bin/env bash
sudo nvidia-smi -pl 140
sudo X :0 &
sleep 5
export DISPLAY=:0
IFS=$'\n'
for gpu in $(nvidia-smi -L); do
    id=$(echo ${gpu} | sed -e 's/GPU //g' -e 's/:.*//g')
    sudo nvidia-settings -a [gpu:${id}]/GPUPowerMizerMode=1 -a [gpu:${id}]/GPUGraphicsClockOffset[3]=-200 -a [gpu:${id}]/GPUMemoryTransferRateOffset[3]=1200
done
sudo killall Xorg
sudo systemctl start ethminer # or your own command to start your own miner

Looking into it, I checked the properties via ls -l and sure enough:

ls -l /-snip-grav-parent-directory-/
    total -snip-
    drwxr--r-x 13 -snip-grav-user- -snip-grav-group- 4096 Aug  6 08:00 -snip-grav-folder-

Since the files were owned by the Grav user, while my user was in the Grav group. I was unable to edit the files due to the absense of write permissions for the group clause.

Simple chmod

At first, running a simple recursive chmod everytime I met the issue was fine as normally my web files were not created by the Grav user as I mainly host static pages before:

chmod -R g+w /-snip-grav-directory-/

Default groups and permissions

But there was two problems, the first problem is that new files created by my privileged user had the group not set to the Grav user and thus not allowing Grav to modify it too. I didn't wish to add the grav user to any groups so I had to resort to recursively adding the setgid bit on the directory:

$ chmod -R g+s /-snip-grav-directory-/

That solves the first problem by having new files automatically set its' group to the parent directory whenever they are created.

The other problem was there was no default permissions and new files were created were not allowed to be modified by my privileged user. That's where this nifty tool comes in, acl (Access Control Lists) which extends the basic chmod, chown and chgrp commands.

This tool requires you to add acl your filesystem mount option to always apply if it's not already set, you can check it by running this and seeing if acl is returned:

tune2fs -l /dev/sdXY | grep "Default mount options:"

Most filesystems turn it on by default, but if it does not, proceed to add it into /etc/fstab.

After that it's just a simple command as this to set default permissions for the directory and the files

setfacl -R -d -m g::rwX -snip-grav-directory-

• setfacl - set acl command
• -R - recursive
• -d - target default permissions
• -m - modify
• g::rwX - target group with read write and only directories should be set with eXecutable

A confirmation with getfacl:

getfacl -snip-grav-directory-
  getfacl -snip-grav-directory-
  # file: -snip-grav-directory-
  # owner: -snip-grav-user-
  # group: -snip-grav-group-
  # flags: -s-
  user::rwx
  group::rwx
  other::r-x
  default:user::rwx
  default:group::rwx
  default:other::r-x

Once you see # flags: -s- and default:group::rwx, you are all set and future new directories and files inside will be set with the default permissions automatically.

You may want to run chmod -R g+X the directory after the above step to apply the permissions to the existing files.

Pointers from mistakes I made

• Do not forget to set the executable permission for directories, if it is missing the user will not be able to access it, note the pointer below when needing to fix this.
• Do not ever run chmod -R g+x as it would set all files executables too which is a security risk, instead use chmod -R g+X with capital X which only sets directories with executable, same thing for acl.

Further reading and references

• Archlinux Wikipedia article on ACL
• Archlinux Wikipedia article on file permissions and attributes

I have a 5.1 surround sound setup with my Logitech Z506 set that was previously working just plugging in to my old motherboard Asus P8P67 PRO which had 6 3.5mm ports.

Instructions

1. Install the hdajackrestask tool on your system (Archlinux: pacman -S alsa-tools)
2. Run in privileged mode using sudo or polkit (GKSU: gksu hdajackretask)
3. Plug your 3 3.5mm audio jacks according to this image, colours should be roughly the same, alternatively take reference from the label (Mic, L-Out and L-In)
4. As root run:

echo "options snd-hda-intel patch=hda-jack-retask.fw,hda-jack-retask.fw,hda-jac

What is Undertow?

Quoted from the project page:

Undertow is a flexible performant web server written in java, providing both blocking and non-blocking API’s based on NIO.

Undertow has a composition based architecture that allows you to build a web server by combining small single purpose handlers. The gives you the flexibility to choose between a full Java EE servlet 4.0 container, or a low level non-blocking handler, to anything in between.

Undertow is designed to be fully embeddable, with easy to use fluent builder APIs. Undertow’s lifecycle is completely controlled by the embedding application.

Undertow is sponsored by JBoss and is the default web server in the Wildfly Application Server.

Alternatives include Oracle Grizzly2, Netty, Eclipse Jetty and Apache Tomcat (has non-embedded version)

In a nutshell, Undertow is a embedded web server used that supports both static and dynamic content delivery.

It is notable that in my tests done in a side benchmarking project that Undertow seems to perform much better than the alternatives mentioned for simple use especially combined with the RESTeasy JAX-RS framework. You could try to download the latest artifacts on the project to test it out yourself.

What is RESTeasy?

Quoted from the project page:

RESTEasy is a JBoss project that provides various frameworks to help you build RESTful Web Services and RESTful Java applications. It is a fully certified and portable implementation of the JAX-RS 2.1 specification, a JCP specification that provides a Java API for RESTful Web Services over the HTTP protocol.

In a nutshell RESTeasy is a JAX-RS (Java API for RESTful Web Services) implementation that provides frameworks aiding in exposing REST (REpresentational State Transfer) endpoints, especially for CRUD (Create, Read, Update and Delete) operations.

Alternatives include Oracle Jersey

RESTeasy is also developed by RedHat and has better "native" integration with Undertow than Jersey.

What is Weld?

What is CDI?

As quoted from another good read Baeldung: Java EE CDI:

It allows us to manage the lifecycle of stateful components via domain-specific lifecycle contexts and inject components (services) into client objects in a type-safe way.

In layman terms, CDI something like a factory that manages the life of Java objects during the life of the application, this is especially important in a multi-threaded environment like a web server. Some examples include:

• Objects that only need one instance throughout the runtime of the application (database connections, etc.)
• Objects that only need to be unique and persist in a session
• Objects that only need to be unique and persist in a single request for web applications.

It is possible to code CDI yourself as mentioned in the Baeldung article, but since implementations are already freely available, you are able to use them and skip boilerplate codes by using annotation interfaces.

Weld

Quoted from the project page:

Weld is the reference implementation of CDI for the Java EE Platform - a JCP standard for dependency injection and contextual lifecycle management and one of the most important and popular parts of the Java EE. Weld is integrated into many Java EE application servers such as WildFly, JBoss Enterprise Application Platform, GlassFish, Oracle WebLogic Server, WebSphere Application Server and others. Weld can also be used in plain servlet containers (Tomcat, Jetty) or Java SE.

Base setup

Maven pom.xml

We will be using Maven to manage our Java libraries that we will depend on, you can use Gradle if you wish.

<dependencies>
    <!-- UNDERTOW -->
    <dependency>
        <groupId>io.undertow</groupId>
        <artifactId>undertow-servlet</artifactId>
        <version>2.0.23.Final</version>
    </dependency>

    <!-- RESTEASY -->
    <dependency>
        <groupId>org.jboss.resteasy</groupId>
        <artifactId>resteasy-undertow</artifactId>
        <version>4.1.1.Final</version>
    </dependency>
    <dependency>
        <groupId>org.jboss.resteasy</groupId>
        <artifactId>resteasy-cdi</artifactId>
        <version>4.1.1.Final</version>
    </dependency>

    <!-- WELD -->
    <dependency>
        <groupId>org.jboss.weld.servlet</groupId>
        <artifactId>weld-servlet-core</artifactId>
        <version>3.1.1.Final</version>
    </dependency>
    <dependency>
        <groupId>org.jboss.weld</groupId>
        <artifactId>weld-core-impl</artifactId>
        <version>3.1.1.Final</version>
    </dependency>
    <dependency>
        <groupId>org.jboss.weld</groupId>
        <artifactId>weld-api</artifactId>
        <version>3.1.Final</version>
    </dependency>
</dependencies>

/src/resources/META-INF/beans.xml

This file located in the resources META-INF folder is required to configure Weld

<beans
    xmlns="http://xmlns.jcp.org/xml/ns/javaee"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://xmlns.jcp.org/xml/ns/javaee http://xmlns.jcp.org/xml/ns/javaee/beans_1_1.xsd"
    version="1.2" bean-discovery-mode="annotated">
</beans>

You can change "annotated" to "all" if you wish for all Java classes to be managed. Note that it would impact startup time slightly.

/src/main/java/.../MyResource.java

Also known as a controller in PHP, defines routes.

@RequestScoped
@Path("/")
public class MyResource {
    @Inject
    BeanManager manager;

    @Path("/")
    @GET
    public String getManager() {
        return manager.toString();
    }
}

/src/main/java/.../MyApplication.java

To specify which resources and middleware for web servers to deploy along with the base path.

@ApplicationPath("/")
public class MyApplication extends Application {
    @Override
    public Set<Class<?>> getClasses() {
        Set<Class<?>> classes = new HashSet<>();
        classes.add(MyResource.class); // Resource implemented in the previous section
        return classes;
    }
}

Main function

public static void main(String[] args) {
    UndertowJaxrsServer server = new UndertowJaxrsServer();

    ResteasyDeployment deployment = new ResteasyDeploymentImpl();
    deployment.setApplicationClass(MyApplication.class.getName()); // Application implemented in the previous section
    deployment.setInjectorFactoryClass("org.jboss.resteasy.cdi.CdiInjectorFactory"); // set CDI injector factory

    DeploymentInfo deploymentInfo = server.undertowDeployment(deployment, "/");
    deploymentInfo.setClassLoader(Application.class.getClassLoader());
    deploymentInfo.setDeploymentName("Minimal Undertow RESTeasy and Weld CDI Setup"); // set name of deployment
    deploymentInfo.setContextPath("/");
    deploymentInfo.addListener(Servlets.listener(Listener.class)); // add Weld listener to deployment

    server.deploy(deploymentInfo);

    Undertow.Builder builder = Undertow.builder()
            .addHttpListener(8080, "localhost"); // access the server on http://localhost:8080, note that "localhost" should be "0.0.0.0" if you wish for others in the network to connect.
    server.start(builder);
}

Since Undertow is an embedded server, you will just need to run the application and it will expose the server on the specified IP and port.

After running, if you attempt to access your web server at http://localhost:8080 and it returns "Weld BeanManager" with a bean count above 0, you have successfully configured the setup.

What's next?

Modify MyResource.java to play around with scoping

@Path("/")
@RequestScoped // try changing this to "@ApplicationScoped" along with the commented modifications below
public class MyResource {
    @Inject
    BeanManager manager;

    private int count = 0; // add a counter variable

    @Path("/")
    @GET
    public String getManager() {
        count++; // increment counter
        return manager + ". Current count: " + count; // show current count ;
    }
}

When it is application scoped, the count should increase every time you refresh the endpoint, vice versa for request scoped.

This is because application scoped will re-initialize the count to 0 every request, whereas request scoped will not.

You can research more about scoping in Weld in the documentation

Further reading

Do read the documentations on Undertow, RESTeasy and Weld.

Carry on with your own projects

Now that you are equipped with the basic knowledge on how to implement Weld CDI on Undertow RESTeasy, go ahead and code your own projects! Be sure to update the Maven configuration if you are using them and the versions used are outdated.

This has been superceded by Carousell GoBot project.

Context

I was looking for ideas to try out n8n.io's capabilities and picked this as the first big idea to try out.

Basically, I would like to n8n.io to handle chat messages and offers from Carousell, a Singaporean marketplace platform, similar to Facebook Marketplace.

Right now I would only like it to:

• Reply a standard response template on new messages and offers
• Forward to my personal Telegram bot to notify me via one additional channel
• Detect any low-balling (offering a price much lower than the listed price)

The problem was that Carousell does not have any public documentation on their APIs and I have to resort to inspecting the network traffic manually to achieve my goals.

Docker image: jarylc/n8n

Process

Automatic Triggers

There are 3 ways to this workflow is automatically triggered:

-> Webhook - Using Tasker on my Android device, it will send a GET request to this webhook everytime a notification from Carousell app arrives.

-> IMAP - Reads a dedicated inbox for e-mails from Carousell and trigger if so, acts as fallback for the above.

-> Cron - Every 30 minutes as fallback in case all of the above doesn't work. Note that there is a better to do this if standalone, you could just connect to the chat WebSocket and wait for messages.

Retrieve Chat Data and Preliminary Checks

When meddling around https://www.carousell.sg/inbox/received/ and inspecting the network trace, I found out that Carousell has an undocumented API to retreive that I could tap on.

This API call allows me to retrieve the list of messages and offers that I have recieved as a seller which I put into good use as the first step after the trigger.

It seems that only the Cookie header is required to authenticate myself.

The preliminary checks consists of checking if the chat list is empty, or the very last message contains my message signature.

Now this comes the hard part for this workflow, upon inspecting, Carousell uses SendBird for their chat platform which uses WebSockets instead of API calls.

I had to customize my n8n.io installation to include ws Node dependency and add it to NODE_FUNCTION_ALLOW_EXTERNAL environment variable to be used in the vm2 environment. This was made easy with my own Docker image on Docker hub using the environment variable ADDITIONAL_MODULES, jarylc/n8n.

After which, I had to mimic the WebSocket calls when chatting and came up with this final JavaScript node.

Custom function codes

Check & Split Checks

const fs = require("fs")

const seen = JSON.parse(fs.readFileSync('/data/carousell.json', 'utf8'))

const split = []
let save = {}
for (const item of $node["HTTP Request"].json["data"]["offers"]) {
  if (item['state'] === 'A')
    continue

  let old_price = seen[item['id']] || 0.0
  let latest_price = parseFloat(item['latest_price'].replace(',', ''))

  let message_price_search = item['latest_price_message'].match(/^(\d{1,5}\.?\d{0,2})$|(\d+\.?\d{0,2}((?<=(\$|offer|quote|can|please|pls|quick|fast|sell).*)|(?=.*(\$|offer|quote|can|please|pls|quick|fast|bucks|ok|\?))))/gi)
  if (message_price_search != null) {
    let message_price = parseFloat(message_price_search[message_price_search.length - 1])
    if (message_price > old_price && message_price > latest_price) {
      latest_price = message_price
      item['latest_price_formatted'] = (''+latest_price.toFixed(2)).replace(/\B(?=(\d{3})+(?!\d))/g, ",")
    }
  }

  save[item['id']] = latest_price
  if (item['id'] in seen && old_price >= latest_price)
    continue

  item['price_changed'] = old_price !== 0.0
  item['low_balled'] = latest_price !== 0 && latest_price <= parseFloat(item['product']['price']) * 0.85

  split.push({json: item})
}

if (split.length > 0) {
  const fs = require("fs")
  fs.writeFileSync('/data/carousell.json', JSON.stringify(save))
}

return split

Send Reply

const WebSocket = require('ws')

return await new Promise((resolve, reject) => {
  let sent = false
  const sendbird_subdomain = items[0].json['channel_url'].slice(0, items[0].json['channel_url'].indexOf('-carousell')).toLowerCase()
  const client = new WebSocket('wss://ws-' + sendbird_subdomain + '.sendbird.com/?p=JS&pv=Mozilla%2F5.0%20(X11%3B%20Linux%20x86_64%3B%20rv%3A90.0)%20Gecko%2F20100101%20Firefox%2F90.0&sv=3.0.149&ai=F3CB6187-CB42-4CD1-95FC-1C46F8856006&user_id=344194&access_token=' + $node['Get Chat Token'].json['data']['token'] + '&active=1&SB-User-Agent=JS%2Fc3.0.149%2F%2F&Request-Sent-Timestamp=' + Date.now() + '&include_extra_data=premium_feature_list%2Cfile_upload_size_limit%2Capplication_attributes%2Cemoji_hash', {
    perMessageDeflate: true
  })
  client.on('message', (data) => {
    if (data.startsWith('LOGI') && !sent) {
      sent = true
      for (const i in items) {
        try {
          const item = items[i].json
          let text = 'Hello @' + item['user']['username'] + '!\n\n' +
            'Thank you for your '
          if (item['latest_price_formatted'] !== '0') {
            text += (item['price_changed'] ? 'new ' : '') + 'offer of ' + item['currency_symbol'] + item['latest_price_formatted'] + ' on'
          } else {
            text += 'interest in'
          }
          text += ' my item: ' + item['product']['title'] + '.'
          if (!item['is_product_sold'] && item['product']['status'] !== 'R') {
            if (item['low_balled'])
              text += '\n\nWARNING: Offer price is more than 15% below listing price, it is too low and may not get a future reply unless you increase it!'

            if (!item['price_changed']) {
              text += '\n\nFAQ:\n' +
                '» Where do I normally deal?\n' +
                'Choa Chu Kang or Bukit Panjang area if I am not in office, Bencoolen area if I am.\n' +
                '» What payment methods do I accept?\n' +
                'In order of preference: Google Pay, PayLah, PayNow, Cash, CarouPay, Bank Transfer\n' +
                '» What happens if I did not reply?\n' +
                'Very likely you offered too low-ball of a price' + (item['low_balled'] ? ' (which you probably did)' : '') + '. If not, feel free to message me again.'
            }
          } else {
            text += '\n\nHowever, this listing has already been ' + (item['is_product_sold'] ? 'sold' : 'reserved') + ' and not available anymore.'
          }
          text += '\n\n- @jarylc'

          const msg = {
            channel_url: '' + item['channel_url'],
            message: text,
            data: JSON.stringify({
              offer_id: '' + item['id'],
              source: 'web'
            }),
            mention_type: 'users',
            mentioned_user_ids: [],
            custom_type: 'MESSAGE',
            req_id: Date.now()
          }

          client.send('MESG' + JSON.stringify(msg) + '\n')
        } catch
          (err) {
          reject(err)
        }
      }
    } else if (data.startsWith('READ')) {
      client.terminate()
      resolve(true)
    }
  })
  client.on('error', (err) => {
    reject(err)
  })
  client.on('open', () => {
    setTimeout(() => {
      client.terminate()
      resolve(true)
    }, 10000)
  })
}).then(_ => {
  return []
}).catch(err => {
  throw err
})

Telegram Notify Flow

The other branch just forwards this to my personal Telegram bot so that I can be notified via another channel. Right now it will also tell me if the offer was a low-ball or not.