Skip to content

NODEBB: Nginx error performance & High CPU

Solved Performance
  • @phenomlab

    I don’t think it’s DDOS attack.
    Indeed, I am already equipped with anti DDOS musers with crowdsec software. This one did not detect DDOS on the server

    On 2 days ago, we have just two HTTP bad user agent scan on the server but not ddos attack

    585ae0ca-6ccb-4951-8578-46d0c3c1b4e6-image.png

    In fact, for the record, a very reputable illegal site closed with a text file pointing to one of our topics that talked about it. This caused a massive influx of users. We could see it in the number of connections as well as in the numbers of people coming to register permanently.

    I don’t really want to undo all the nginx modifications because it’s thanks to this one that the server is stabilized

    After if you have tweak against the ddos attack, I really want to know them and apply them
    but for me it’s really not that.

    I’m taking all possible concrete steps but I think the focus should be on optimizing nodeBB too

    Actually, we have 758 members online 😵

    3f04b2f3-a66b-4d22-96af-b1cddb79ed4f-image.png

    You can go the server if you want

  • @DownPW said in NODEBB: Nginx error performance & High CPU:

    very reputable illegal site

    That statement alone is enough to cause concern in my view. How can it be reputable if it’s an illegal site 😂

    I’m still going to reserve judgement on this until I’ve fully reviewed the logs. Nginx was designed to automatically scale to hundreds and thousands of users per second and that is it’s main advantage over Apache. The client body size should not have to be adjusted unless an upload is being performed on the site which would have me concerned as this could be an attempt to deposit a malicious payload.

    If you’re confident that this isn’t a DDoS attack, then ok, but you may have just opened the floodgates in terms of removing built in restrictions with nginx. Given the very nature of that site, I’d be exercising caution and would probably recommend enabling the “I’m under attack” . If this is organic traffic and legitimate in intent, then they’ll complete the captcha. If they don’t, then they are bots.

    Have you checked random IP addresses in the nginx logs to see if they are listed in the IP abuse database? And if you enable the “I’m under attack” setting I wouldn’t mind bettibg that the number of active users will start to freefall.

    Finally, how many new users do you have registered on the site? Presently, I wouldn’t be comfortable with this scenario without a full analysis and review, but I can only recommend here.

  • @DownPW I’ve just started reviewing your nginx.log and a couple of things immediately stand out for me

    1. Literally all of the IP addresses are within Cloudflare’s subnet ranges. This means that you cannot possibly tell if the traffic is legitimate or not as you do not know what the originating IP addresses are
    2. The TCP established times are all within seconds of each other but from different IP addresses. This very much lends itself towards the traffic not being organic in my view, but you will never really know the true identity of these connections without first attempting to unpack the TCP headers - see the below

    https://support.cloudflare.com/hc/en-us/articles/200170786-Restoring-original-visitor-IPs

    24c3670c-c9a2-4da0-a4d2-ded8c65d033a-image.png

    You could temporarily disable Cloudflare on your site to get a quick analysis, then keep an eye on your own DDoS implementation to determine if this traffic is legitimate or not.

    The bottom line is this. Don’t be lulled into a false sense of security just because Cloudflare passes the traffic to your site.

  • @phenomlab

    I don’t understand all you say.
    Finally what we can do ?

    Actually we have 1.1k users online

    We have a lot of inscriptions

  • @phenomlab 362 user inscription in two days and many user on just read forum

  • @DownPW said in NODEBB: Nginx error performance & High CPU:

    I don’t understand all you say.
    Finally what we can do ?

    My point here is that the traffic, whilst legitimate in the sense that it’s from another site that has closed, could still be nefarious in nature so you should keep your guard up. However, a number of signups can’t be wrong - particularly if they are actually posting content and not performing requests that actually do not pertain to available URL’s on your site.

    I see no indication of that, so the comfort level in the sense that it’s legitimate traffic does increase somewhat accompanied by the seemingly legitimate registrations. However, because all the source IP addresses and within the Cloudflare ranges, you have no ability to tell really who they are without performing the steps I outlined in the previous post.

    The good news is that your site just got a huge increase in popularity, but with that will always be a need to keep a close eye on activity. It would only take one nefarious actor to potentially bring down your site.

    The nginx configuration you’ve applied will indeed alleviate the stress placed on the server but is a double edged sword in the sense that it does make the goalpost much wider in terms of any potential attack.

    My advice herein would be to not scale these settings too high. Use sane judgement.

    For the NodeBB side, I know they have baked rate limiting into the product but I’m sure you can actually change that behaviour.

    Have a look at

    /admin/settings/advanced#traffic-management
    

    You’ll probably need to play with the values here to get a decent balance, but this is where I’d start.

  • @phenomlab

    I think you’re right Mark and that’s why I come here looking for your valuable advice and expertise 😉

    Basically, the illegal site that closed was a movie download site A topic was opened on our forum to talk about it and many came looking for answers on why and how.

    You’re actually right about the fact that we can’t be sure of anything and there are bot attacks or ddos in the lot of connexions

    I activated the under attack mode on Cloudflare as you advised me to see (just now.) and we will see like you said

    As you advised, I also reset the default nginx configuration values ​​and removed my nginx modifications specified above.

    I would like to take advantage of your expertise, see a hand from you to properly configure nginx for ddos ​​and high traffic. (What precise modifications to specify as well as the precise values.)

  • @DownPW ok, good. Let’s see what the challenge does to the site traffic. Those whom are legitimate users won’t mind having to perform a one time additional authentication step, but bots of course will simply stumble at this hurdle.

  • @phenomlab

    number of user is better (408) but a lot of loose connexion. navigation is hard

  • I have chaneg nginx conf with :

    worker_rlimit_nofile 70000;

    events {
    worker_connections 65535;
    multi_accept on;
    }

    CF is under attack mode

  • @DownPW I still have access to your Cloudflare tenant so will have a look shortly.

    EDIT: I am in now - personally, I would also enable this (and configure it)

    d85820ea-6643-49bd-98da-a8537e970f04-image.png

    b6a188a9-deba-4980-b4a1-99df7975160d-image.png

  • @phenomlab I have already activate it and add a waf rules for russian country

    2e3108dc-8d68-4e48-91c7-e1c3bc00c229-image.png

    With this bots settings :
    e12409dd-df54-4def-b998-470786c3afa9-image.png

    and this settings for ddos protection :

    2e176374-f7d1-48e4-b38f-83360a0f182a-image.png

  • @DownPW said in NODEBB: Nginx error performance & High CPU:

    I have already activate it

    Are you sure? When I checked your tenant it wasn’t active - it’s from where I took the screenshot 😁

  • @phenomlab

    yep I activate it after 😉

  • For your information @phenomlab ,

    • I have ban via iptables suspicious ip address find on /etc/nginx/accesss.log and virtualhost access.log like this : iptables -I INPUT -s IPADDRESS -j DROP
    • Activate bot option on CF
    • Create contry rules (Russie and China) on CF WAF
    • I left under attack mode option activated on CF
    • I have just change nginx.conf like this for test (If you have best value, I take it ! ) :
    worker_rlimit_nofile 70000; 
    
    events {
    
    	worker_connections 65535;
    	multi_accept on; 
    }
    
    http {
    
    	##
    	# Basic Settings
    	##
    
    	limit_req_zone $binary_remote_addr zone=flood:10m rate=100r/s; 
    	limit_req zone=flood burst=100 nodelay; 
    	limit_conn_zone $binary_remote_addr zone=ddos:10m; 
    	limit_conn ddos 100;
    

    100r/s iit’s already a lot !!

    and for vhost file :

    server {
    	.....
    
            location / {
    				
    				limit_req zone=flood; #Test 
    				limit_conn ddos 100; #Test 
    }
    

    –> If you have other ideas, I’m interested
    –> If you have better values ​​to use in what I modified, please let me know.

  • @DownPW my only preference would be to not set worker_connections so high

  • @phenomlab

    Ok so what value do you advise?

  • @DownPW you should base it on the output of ulimit - see below

    https://linuxhint.com/what-are-worker-connections-nginx/#:~:text=The worker_connections are the maximum,to accommodate a higher value

    With that high value you run the risk of overwhelming your server.

  • @phenomlab

    Thanks mark 😉

    My ulimit is 1024, so I will set it to 1024

  • @DownPW And the worker_processes value ? I expect this to be between 1 and 4 ?


Did this solution help you?
Did you find the suggested solution useful? Why not buy me a coffee? It's a nice gesture, and a great way to show your appreciation 💗

Related Topics
  • Bug Report

    Solved Bugs
    47
    26 Votes
    47 Posts
    2k Views

    @crazycells Good points, thanks. I completely forgot that classes are added - makes life much simpler!

    EDIT - seems this is pretty straightforward, and only needs the below CSS

    .upvoted i { color: var(--bs-user-level) !important; }

    This then yields

    3f072f8a-ebfa-4910-8723-73c493b8e4eb-image.png

    However, the caveat here is that the .upvoted class will only show for your upvotes, and nobody else’s. However, this does satisfy the original request

    however I would love to see my upvoted posts more clearly, because currently, when I upvote, nothing on the post tool is changing, it would be nicer if there is an indication that I have upvoted (like a filled or colored triangle?)

  • NodeBB socket with CloudFlare

    Unsolved Performance
    23
    1 Votes
    23 Posts
    2k Views

    @DownPW it’s your only realistic option at this stage.

  • error with v3 in browser console

    Solved Performance
    4
    0 Votes
    4 Posts
    249 Views

    @DownPW it’s in relation to the response I provided above

  • Is nginx necessary to use?

    Moved Solved Hosting
    2
    1 Votes
    2 Posts
    381 Views

    @Panda said in Cloudflare bot fight mode and Google search:

    Basic question again, is nginx necessary to use?

    No, but you’d need something at least to handle the inbound requests, so you could use Apache, NGINX, Caddy… (there are plenty of them, but I tend to prefer NGINX)

    @Panda said in Cloudflare bot fight mode and Google search:

    Do these two sites need to be attached to different ports, and the ports put in the DNS record?

    No. They will both use ports 80 (HTTP) and 443 (HTTPS) by default.

    @Panda said in Cloudflare bot fight mode and Google search:

    Its not currently working, but how would the domain name know which of the two sites to resolve to without more info?
    Currently it only says the IP of the whole server.

    Yes, that’s correct. Domain routing is handled (for example) at the NGINX level, so whatever you have in DNS will be presented as the hostname, and NGINX will expect a match which once received, will then be forwarded onto the relevant destination.

    As an example, in your NGINX config, you could have (at a basic level used in reverse proxy mode - obviously, the IP addresses here are redacted and replaced with fakes). We assume you have created an A record in your DNS called “proxy” which resolves to 192.206.28.1, so fully qualified, will be proxy.sudonix.org in this case.

    The web browser requests this site, which is in turn received by NGINX and matches the below config

    server { server_name proxy.sudonix.org; listen 192.206.28.1; root /home/sudonix.org/domains/proxy.sudonix.org/ogproxy; index index.php index.htm index.html; access_log /var/log/virtualmin/proxy.sudonix.org_access_log; error_log /var/log/virtualmin/proxy.sudonix.org_error_log; location / { proxy_set_header Access-Control-Allow-Origin *; proxy_set_header Host $host; proxy_pass http://localhost:2000; proxy_redirect off; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Api-Key $http_x_api_key; } location /images { index index.php index.htm index.html; root /home/sudonix.org/domains/proxy.sudonix.org/ogproxy; } fastcgi_split_path_info "^(.+\.php)(/.+)$"; listen 192.206.28.1:443 ssl http2; ssl_certificate /home/sudonix.org/domains/proxy.sudonix.org/ssl.combined; ssl_certificate_key /home/sudonix.org/ssl.key; }

    The important part here is server_name proxy.sudonix.org; as this is used to “map” the request to the actual domain name, which you can see in the root section as root /home/sudonix.org/domains/proxy.sudonix.org/ogproxy;

    As the DNS record you specified matches this hostname, NGINX then knows what to do with the request when it receives it.

  • Optimum config for NodeBB under NGINX

    Performance
    4
    3 Votes
    4 Posts
    826 Views

    @crazycells hi - no security reason, or anything specific in this case. However, the nginx.conf I posted was from my Dev environment which uses this port as a way of not interfering with production.

    And yes, I use clustering on this site with three instances.

  • Nord VPN & Bitdefender

    Solved Performance
    8
    1 Votes
    8 Posts
    538 Views

    @JAC been working fine. No complaints from me

  • NodeBB slow after DB recovery

    Solved Performance
    1
    5 Votes
    1 Posts
    293 Views
    No one has replied
  • NodeBB 1.19.3

    Solved Performance
    33
    4 Votes
    33 Posts
    3k Views

    @phenomlab

    I find the problem Mark 😉

    The error message indicated this path :

    http://localhost:4567/assets/plugins/nodebb-plugin-emoji/emoji/styles.css?v=6983dobg16u

    I change the path url on config.json

    47bacc80-f141-41e4-a261-3f8d650cc6f6-image.png

    And all it’s good 🙂

    Weird, I didn’t have to change that path before 1.19.3

    But this does not prevent the problem from a clean install with Emoji Plugin

    EDIT: After test, that resolv the problem installation for 1.18.x but not for 1.19.x (I have other error message when I run ./nodebb Setup

    For resume: NodeJS 16_x with 1.18.x is ok