Skip to content

NodeBB Global Tchat : Forum with many users performance issues

Solved Configure
  • Like you know I use glolbal chat plugin
    We had a massive influx of registrations and new members which means that they all integrate the chat room of the plugin

    We see this topic on github plugin page :

    https://github.com/NodeBB/nodebb-plugin-global-chat/issues/5

    I see maybe a solution here dating from 2019 : https://github.com/NodeBB/nodebb-plugin-global-chat/issues/5#issuecomment-492242438

    7bc1e6f9-542e-41d9-96aa-8a717d6bd538-image.png

    I then asked the question to the staff without an answer :

    Is this still relevant?
    Where in the file should I put the code?

    So I’m opening this topic to find out what you think about it with your expert eye 😉

  • @DownPW I’d say that this was still relevant, and would solve the short term issue (in most cases) but if anything, you’ll still experience slowness if you have say 600 users online.

    The right way to handle this in my view would be to pass the notification stream to a message queuing service (like RabbitMQ or Redis) and let that process in the back end rather than cause locking on the forum itself. I’m surprised that this is in fact the case as standard emails are being queued and aren’t sent in real time.

    I personally don’t use this plugin as you know, so I’d need to review the code first in order to understand it’s structure.

  • @phenomlab said in NodeBB Global Tchat : Forum with many users performance issues:

    I’d say that this was still relevant, and would solve the short term issue (in most cases) but if anything, you’ll still experience slowness if you have say 600 users online.
    The right way to handle this in my view would be to pass the notification stream to a message queuing service (like RabbitMQ or Redis) and let that process in the back end rather than cause locking on the forum itself. I’m surprised that this is in fact the case as standard emails are being queued and aren’t sent in real time.
    I personally don’t use this plugin as you know, so I’d need to review the code first in order to understand it’s structure.

    Thank you for your analyse 👍

    I would have liked to try the thing all the same, just to see the result

    600 users is better than 4000 😉

    Here is the content of the /src/messsagin/notification.js file

    I put it here because I don’t see where to integrate the given code for test :

    'use strict';
    
    const winston = require('winston');
    
    const user = require('../user');
    const notifications = require('../notifications');
    const sockets = require('../socket.io');
    const plugins = require('../plugins');
    const meta = require('../meta');
    
    module.exports = function (Messaging) {
    	Messaging.notifyQueue = {}; // Only used to notify a user of a new chat message, see Messaging.notifyUser
    
    	Messaging.notifyUsersInRoom = async (fromUid, roomId, messageObj) => {
    		let uids = await Messaging.getUidsInRoom(roomId, 0, -1);
    		uids = await user.blocks.filterUids(fromUid, uids);
    
    		let data = {
    			roomId: roomId,
    			fromUid: fromUid,
    			message: messageObj,
    			uids: uids,
    		};
    		data = await plugins.hooks.fire('filter:messaging.notify', data);
    		if (!data || !data.uids || !data.uids.length) {
    			return;
    		}
    
    		uids = data.uids;
    		uids.forEach((uid) => {
    			data.self = parseInt(uid, 10) === parseInt(fromUid, 10) ? 1 : 0;
    			Messaging.pushUnreadCount(uid);
    			sockets.in(`uid_${uid}`).emit('event:chats.receive', data);
    		});
    		if (messageObj.system) {
    			return;
    		}
    		// Delayed notifications
    		let queueObj = Messaging.notifyQueue[`${fromUid}:${roomId}`];
    		if (queueObj) {
    			queueObj.message.content += `\n${messageObj.content}`;
    			clearTimeout(queueObj.timeout);
    		} else {
    			queueObj = {
    				message: messageObj,
    			};
    			Messaging.notifyQueue[`${fromUid}:${roomId}`] = queueObj;
    		}
    
    		queueObj.timeout = setTimeout(async () => {
    			try {
    				await sendNotifications(fromUid, uids, roomId, queueObj.message);
    			} catch (err) {
    				winston.error(`[messaging/notifications] Unabled to send notification\n${err.stack}`);
    			}
    		}, meta.config.notificationSendDelay * 1000);
    	};
    
    	async function sendNotifications(fromuid, uids, roomId, messageObj) {
    		const isOnline = await user.isOnline(uids);
    		uids = uids.filter((uid, index) => !isOnline[index] && parseInt(fromuid, 10) !== parseInt(uid, 10));
    		if (!uids.length) {
    			return;
    		}
    
    		const { displayname } = messageObj.fromUser;
    
    		const isGroupChat = await Messaging.isGroupChat(roomId);
    		const notification = await notifications.create({
    			type: isGroupChat ? 'new-group-chat' : 'new-chat',
    			subject: `[[email:notif.chat.subject, ${displayname}]]`,
    			bodyShort: `[[notifications:new_message_from, ${displayname}]]`,
    			bodyLong: messageObj.content,
    			nid: `chat_${fromuid}_${roomId}`,
    			from: fromuid,
    			path: `/chats/${messageObj.roomId}`,
    		});
    
    		delete Messaging.notifyQueue[`${fromuid}:${roomId}`];
    		notifications.push(notification, uids);
    	}
    };
    
  • @phenomlab

    An idea for where to put this code before I open a thread an NodeBB communauty ?

  • @DownPW not specifically, no, as there is an existing function with the same name. The comma at the end of the revised function would indicate part of an existing array but I’m not entirely sure of where it should be placed - or if it should override the existing function altogether (which I don’t think is the case).

  • @phenomlab arf I hope i have an answers in nodeBB 😞

    But it’s an async function maybe here :

    345a9b2c-26cf-491d-8ffe-047afa529d61-image.png

  • @DownPW you can always experiment 👍

  • @DownPW what have you tried?

  • @phenomlab lot of things 😉

  • @DownPW can you provide some brief examples?

  • @phenomlab .

    'use strict';
    
    const winston = require('winston');
    
    const user = require('../user');
    const notifications = require('../notifications');
    const sockets = require('../socket.io');
    const plugins = require('../plugins');
    const meta = require('../meta');
    
    module.exports = function (Messaging) {
    	Messaging.notifyQueue = {}; // Only used to notify a user of a new chat message, see Messaging.notifyUser
    	
    	Messaging.notifyUsersInRoom = async (fromUid, roomId, messageObj) => {
    		let uids = await Messaging.getUidsInRoom(roomId, 0, -1);
    		uids = await user.blocks.filterUids(fromUid, uids);
    
    		let data = {
    			roomId: roomId,
    			fromUid: fromUid,
    			message: messageObj,
    			uids: uids,
    		};
    		data = await plugins.hooks.fire('filter:messaging.notify', data);
    		if (!data || !data.uids || !data.uids.length) {
    			return;
    		}
    
    		uids = data.uids;
    		uids.forEach((uid) => {
    			data.self = parseInt(uid, 10) === parseInt(fromUid, 10) ? 1 : 0;
    			Messaging.pushUnreadCount(uid);
    			sockets.in(`uid_${uid}`).emit('event:chats.receive', data);
    		});
    		if (messageObj.system) {
    			return;
    		}
    		// Delayed notifications
    		let queueObj = Messaging.notifyQueue[`${fromUid}:${roomId}`];
    		if (queueObj) {
    			queueObj.message.content += `\n${messageObj.content}`;
    			clearTimeout(queueObj.timeout);
    		} else {
    			queueObj = {
    				message: messageObj,
    			};
    			Messaging.notifyQueue[`${fromUid}:${roomId}`] = queueObj;
    		}
    
    		queueObj.timeout = setTimeout(async () => {
    			try {
    				await sendNotifications(fromUid, uids, roomId, queueObj.message);
    			} catch (err) {
    				winston.error(`[messaging/notifications] Unabled to send notification\n${err.stack}`);
    			}
    		}, meta.config.notificationSendDelay * 1000);
    		
    		
               if (roomId != 11) { // 5 Is the ID of the ID of the global chat room.
                   Messaging.getUidsInRoom(roomId, 0, -1); // Proceed as normal.
                } else {
                    user.getUidsFromSet('users:online', 0, -1); // Only notify online users.
                }
    	};
    
    	async function sendNotifications(fromuid, uids, roomId, messageObj) {
    		const isOnline = await user.isOnline(uids);
    		uids = uids.filter((uid, index) => !isOnline[index] && parseInt(fromuid, 10) !== parseInt(uid, 10));
    		if (!uids.length) {
    			return;
    		}
    
    
    		const { displayname } = messageObj.fromUser;
    
    		const isGroupChat = await Messaging.isGroupChat(roomId);
    		const notification = await notifications.create({
    			type: isGroupChat ? 'new-group-chat' : 'new-chat',
    			subject: `[[email:notif.chat.subject, ${displayname}]]`,
    			bodyShort: `[[notifications:new_message_from, ${displayname}]]`,
    			bodyLong: messageObj.content,
    			nid: `chat_${fromuid}_${roomId}`,
    			from: fromuid,
    			path: `/chats/${messageObj.roomId}`,
    		});
    
    		delete Messaging.notifyQueue[`${fromuid}:${roomId}`];
    		notifications.push(notification, uids);
    	}
    	
    
    		
    };
    

    nodebb build is ok with this code but I don’t see any diiference of latencies

  • 'use strict';
    
    const winston = require('winston');
    
    const user = require('../user');
    const notifications = require('../notifications');
    const sockets = require('../socket.io');
    const plugins = require('../plugins');
    const meta = require('../meta');
    
    module.exports = function (Messaging) {
    	Messaging.notifyQueue = {}; // Only used to notify a user of a new chat message, see Messaging.notifyUser
    	
    	Messaging.notifyUsersInRoom = async (fromUid, roomId, messageObj) => {
    		let uids = await Messaging.getUidsInRoom(roomId, 0, -1);
    		uids = await user.blocks.filterUids(fromUid, uids);
    
    		let data = {
    			roomId: roomId,
    			fromUid: fromUid,
    			message: messageObj,
    			uids: uids,
    		};
    		data = await plugins.hooks.fire('filter:messaging.notify', data);
    		if (!data || !data.uids || !data.uids.length) {
    			return;
    		}
    
    		uids = data.uids;
    		uids.forEach((uid) => {
    			data.self = parseInt(uid, 10) === parseInt(fromUid, 10) ? 1 : 0;
    			Messaging.pushUnreadCount(uid);
    			sockets.in(`uid_${uid}`).emit('event:chats.receive', data);
    		});
    		if (messageObj.system) {
    			return;
    		}
    		// Delayed notifications
    		let queueObj = Messaging.notifyQueue[`${fromUid}:${roomId}`];
    		if (queueObj) {
    			queueObj.message.content += `\n${messageObj.content}`;
    			clearTimeout(queueObj.timeout);
    		} else {
    			queueObj = {
    				message: messageObj,
    			};
    			Messaging.notifyQueue[`${fromUid}:${roomId}`] = queueObj;
    		}
    
    		queueObj.timeout = setTimeout(async () => {
    			try {
    				await sendNotifications(fromUid, uids, roomId, queueObj.message);
    			} catch (err) {
    				winston.error(`[messaging/notifications] Unabled to send notification\n${err.stack}`);
    			}
    		}, meta.config.notificationSendDelay * 1000);
    		
    	};
    
    	async function sendNotifications(fromuid, uids, roomId, messageObj) {
    		const isOnline = await user.isOnline(uids);
    		uids = uids.filter((uid, index) => !isOnline[index] && parseInt(fromuid, 10) !== parseInt(uid, 10));
    		if (!uids.length) {
    			return;
    		}
    		
    		if (roomId != 11) { // 5 Is the ID of the ID of the global chat room.
            Messaging.getUidsInRoom(roomId, 0, -1); // Proceed as normal.
            } else {
            user.getUidsFromSet('users:online', 0, -1); // Only notify online users.
            }
    
    
    		const { displayname } = messageObj.fromUser;
    
    		const isGroupChat = await Messaging.isGroupChat(roomId);
    		const notification = await notifications.create({
    			type: isGroupChat ? 'new-group-chat' : 'new-chat',
    			subject: `[[email:notif.chat.subject, ${displayname}]]`,
    			bodyShort: `[[notifications:new_message_from, ${displayname}]]`,
    			bodyLong: messageObj.content,
    			nid: `chat_${fromuid}_${roomId}`,
    			from: fromuid,
    			path: `/chats/${messageObj.roomId}`,
    		});
    
    		delete Messaging.notifyQueue[`${fromuid}:${roomId}`];
    		notifications.push(notification, uids);
    	}
    	
    
    		
    };
    
  • DownPWundefined DownPW has marked this topic as solved on

Did this solution help you?
Did you find the suggested solution useful? Why not buy me a coffee? It's a nice gesture, and a great way to show your appreciation 💗

Related Topics
  • Composer options on nodebb

    Solved Configure
    8
    3 Votes
    8 Posts
    410 Views

    @Panda You should be able to expose the CSS for these using F12 to get into console

    3591518c-e3a3-4ada-a43c-6b32a5e0359c-image.png

    a2b8ed46-4157-4ff2-85f0-576543380107-image.png

    That should then expose the element once selected

    89d9c545-a47a-40d1-98f4-80cf3b958e8f-image.png

    Here’s the below CSS you need based on the screenshot provided.

    .composer .formatting-bar .formatting-group li[data-format="picture-o"], .composer .formatting-bar .formatting-group li[data-format="spoiler"] { display: none; }
  • NodeBB: Favicon upload issue

    Solved Configure
    12
    3 Votes
    12 Posts
    522 Views

    @phenomlab I am on a Mac, so I used the “Option + Command + I”, and then performed the steps. It loaded my favicon! I checked on Firefox which I haven’t used before, and it showed my favicon also! That’s fantastic and thank you for the help!

  • NodeBB: The global Search option

    Solved Configure
    5
    0 Votes
    5 Posts
    292 Views

    @mventures Yes, exactly. The other icon will restart NodeBB whilst the first icon I referenced will rebuild (recompile) it.

    The huge strength of NodeBB over Flarum (for example) is that the code is precompiled, and called once at boot. PHP’s code has to repeatedly reload code from source making it much slower.

  • Gettin Erors NodeBB

    Solved Configure
    7
    0 Votes
    7 Posts
    433 Views

    @phenomlab no forum is working goods.
    there is no eror message since yestarday.

  • Virtualmin SQL problem

    Solved Performance
    39
    3 Votes
    39 Posts
    2k Views

    @DownPW said in Virtualmin SQL problem:

    Finally problem solved simply but I still don’t understand why this service was installed.

    Glad to hear (and see) that this issue is now resolved. Virtualmin and Webmin are both derived from the origin of LAMP (Linux, Apache, MySQL, and PHP) - the earliest form would have been a project which was a fork of the original concept called “WAMP” (Windows, Apache, MySQL, and PHP)

    https://www.wampserver.com/en/

    Scroll to the bottom, and you’ll see the packages it comes with

    cf3c0965-a699-4c6f-b89f-65e7bb381bbc-image.png

    Over time, activity on this project dropped somewhat due to the rise of Virtualmin and Webmin - acting as “full blown” platforms designed to manage an entire web server from start to finish, and providing an easy way to do so with a GUI interface. Over time, the LAMP bundle became LEMP (Linux, NGINX [actually pronounced “engine X”], MySQL, and PHP). The issue with Apache is that it had limits, and compared to NGINX, was in fact much slower.

    Virtualmin and Webmin do not actually use MySQL for their core operation - they don’t even use Apache or NGINX for the core web services, which is why the admin port is set to a default of 10000 when you first complete the setup.

    Essentially, you can think of Virtualmin and Webmin as a central pane of glass to manage the underlying components that are required to run a website. MySQL doesn’t need to be installed if you are not actually using it, but is there as PHP typically is paired with either MySQL or MariaDB, so it makes sense to offer the installation of this itself, as well as support for managing it.

  • Configure SMTP for Nodebb

    Solved Configure
    14
    5 Votes
    14 Posts
    973 Views

    @marusaky based on the work completed thus far (in relation to PM exchanges), I’m going to mark this completed. Sending email from the server itself works fine without issue, and DNS appears to be clean (valid SPF, DMARC, and DKIM records).

    It appears that only Gmail marks incoming messages from your domain as spam - perhaps because of the domain age, which there is nothing we can do to prevent this. Mail delivery to all other domains appears to work fine in al of my tests.

  • NodeBB metadata

    Solved Configure
    4
    2 Votes
    4 Posts
    434 Views

    @phenomlab said in NodeBB metadata:

    @jac Are you sure ?

    https://www.google.co.uk/search?q=site%3Astockportcounty.fans&sxsrf=AOaemvLwnaZL-PliU_2dBOg_Eo1pMVhBjg%3A1638982328139&source=hp&ei=uOKwYeatBcOsad3yp7AE&iflsig=ALs-wAMAAAAAYbDwyLBSDcG5XYoFCKwQFhgz94wTxOcV&ved=0ahUKEwjm6dX71NT0AhVDVhoKHV35CUYQ4dUDCAk&uact=5&oq=site%3Astockportcounty.fans&gs_lcp=Cgdnd3Mtd2l6EAM6BAgjECc6CwgAEIAEELEDEIMBOg4ILhCABBCxAxDHARCjAjoRCC4QgAQQsQMQgwEQxwEQowI6BQguEIAEOggIABCABBCxAzoFCAAQgAQ6CAguELEDEIMBOgsILhCABBDHARCvAToICC4QgAQQsQM6BQgAELEDOgsILhCABBDHARDRAzoLCAAQgAQQsQMQyQM6BQgAEJIDUABYySZg0CdoAHAAeACAAW2IAa0NkgEEMjMuMpgBAKABAQ&sclient=gws-wiz

    Fair enough 🤪🤪😁.

  • Iframely (Nodebb)

    Solved Configure
    40
    4 Votes
    40 Posts
    3k Views

    @DownPW This is now resolved. The issue was an incorrect URL specified in the Nodebb plugin. I’ve corrected this, and now it works as intended.