Skip to content

NODEBB: Nginx error performance & High CPU

Solved Performance
69 3 62.9k 2
  • @DownPW said in NODEBB: Nginx error performance & High CPU:

    I don’t understand all you say.
    Finally what we can do ?

    My point here is that the traffic, whilst legitimate in the sense that it’s from another site that has closed, could still be nefarious in nature so you should keep your guard up. However, a number of signups can’t be wrong - particularly if they are actually posting content and not performing requests that actually do not pertain to available URL’s on your site.

    I see no indication of that, so the comfort level in the sense that it’s legitimate traffic does increase somewhat accompanied by the seemingly legitimate registrations. However, because all the source IP addresses and within the Cloudflare ranges, you have no ability to tell really who they are without performing the steps I outlined in the previous post.

    The good news is that your site just got a huge increase in popularity, but with that will always be a need to keep a close eye on activity. It would only take one nefarious actor to potentially bring down your site.

    The nginx configuration you’ve applied will indeed alleviate the stress placed on the server but is a double edged sword in the sense that it does make the goalpost much wider in terms of any potential attack.

    My advice herein would be to not scale these settings too high. Use sane judgement.

    For the NodeBB side, I know they have baked rate limiting into the product but I’m sure you can actually change that behaviour.

    Have a look at

    /admin/settings/advanced#traffic-management
    

    You’ll probably need to play with the values here to get a decent balance, but this is where I’d start.

    wrote on last edited by DownPW
    #9

    @phenomlab

    I think you’re right Mark and that’s why I come here looking for your valuable advice and expertise 😉

    Basically, the illegal site that closed was a movie download site A topic was opened on our forum to talk about it and many came looking for answers on why and how.

    You’re actually right about the fact that we can’t be sure of anything and there are bot attacks or ddos in the lot of connexions

    I activated the under attack mode on Cloudflare as you advised me to see (just now.) and we will see like you said

    As you advised, I also reset the default nginx configuration values ​​and removed my nginx modifications specified above.

    I would like to take advantage of your expertise, see a hand from you to properly configure nginx for ddos ​​and high traffic. (What precise modifications to specify as well as the precise values.)

  • @phenomlab

    I think you’re right Mark and that’s why I come here looking for your valuable advice and expertise 😉

    Basically, the illegal site that closed was a movie download site A topic was opened on our forum to talk about it and many came looking for answers on why and how.

    You’re actually right about the fact that we can’t be sure of anything and there are bot attacks or ddos in the lot of connexions

    I activated the under attack mode on Cloudflare as you advised me to see (just now.) and we will see like you said

    As you advised, I also reset the default nginx configuration values ​​and removed my nginx modifications specified above.

    I would like to take advantage of your expertise, see a hand from you to properly configure nginx for ddos ​​and high traffic. (What precise modifications to specify as well as the precise values.)

    wrote on last edited by
    #10

    @DownPW ok, good. Let’s see what the challenge does to the site traffic. Those whom are legitimate users won’t mind having to perform a one time additional authentication step, but bots of course will simply stumble at this hurdle.

  • @DownPW ok, good. Let’s see what the challenge does to the site traffic. Those whom are legitimate users won’t mind having to perform a one time additional authentication step, but bots of course will simply stumble at this hurdle.

    wrote on last edited by DownPW
    #11

    @phenomlab

    number of user is better (408) but a lot of loose connexion. navigation is hard

  • @phenomlab

    number of user is better (408) but a lot of loose connexion. navigation is hard

    wrote on last edited by
    #12

    I have chaneg nginx conf with :

    worker_rlimit_nofile 70000;

    events {
    worker_connections 65535;
    multi_accept on;
    }

    CF is under attack mode

  • I have chaneg nginx conf with :

    worker_rlimit_nofile 70000;

    events {
    worker_connections 65535;
    multi_accept on;
    }

    CF is under attack mode

    wrote on last edited by phenomlab
    #13

    @DownPW I still have access to your Cloudflare tenant so will have a look shortly.

    EDIT: I am in now - personally, I would also enable this (and configure it)

    d85820ea-6643-49bd-98da-a8537e970f04-image.png

    b6a188a9-deba-4980-b4a1-99df7975160d-image.png

  • @DownPW I still have access to your Cloudflare tenant so will have a look shortly.

    EDIT: I am in now - personally, I would also enable this (and configure it)

    d85820ea-6643-49bd-98da-a8537e970f04-image.png

    b6a188a9-deba-4980-b4a1-99df7975160d-image.png

    wrote on last edited by DownPW
    #14

    @phenomlab I have already activate it and add a waf rules for russian country

    2e3108dc-8d68-4e48-91c7-e1c3bc00c229-image.png

    With this bots settings :
    e12409dd-df54-4def-b998-470786c3afa9-image.png

    and this settings for ddos protection :

    2e176374-f7d1-48e4-b38f-83360a0f182a-image.png

  • @phenomlab I have already activate it and add a waf rules for russian country

    2e3108dc-8d68-4e48-91c7-e1c3bc00c229-image.png

    With this bots settings :
    e12409dd-df54-4def-b998-470786c3afa9-image.png

    and this settings for ddos protection :

    2e176374-f7d1-48e4-b38f-83360a0f182a-image.png

    wrote on last edited by
    #15

    @DownPW said in NODEBB: Nginx error performance & High CPU:

    I have already activate it

    Are you sure? When I checked your tenant it wasn’t active - it’s from where I took the screenshot 😁

  • @DownPW said in NODEBB: Nginx error performance & High CPU:

    I have already activate it

    Are you sure? When I checked your tenant it wasn’t active - it’s from where I took the screenshot 😁

    wrote on last edited by
    #16

    @phenomlab

    yep I activate it after 😉

  • @phenomlab

    yep I activate it after 😉

    wrote on last edited by DownPW
    #17

    For your information @phenomlab ,

    • I have ban via iptables suspicious ip address find on /etc/nginx/accesss.log and virtualhost access.log like this : iptables -I INPUT -s IPADDRESS -j DROP
    • Activate bot option on CF
    • Create contry rules (Russie and China) on CF WAF
    • I left under attack mode option activated on CF
    • I have just change nginx.conf like this for test (If you have best value, I take it ! ) :
    worker_rlimit_nofile 70000; 
    
    events {
    
    	worker_connections 65535;
    	multi_accept on; 
    }
    
    http {
    
    	##
    	# Basic Settings
    	##
    
    	limit_req_zone $binary_remote_addr zone=flood:10m rate=100r/s; 
    	limit_req zone=flood burst=100 nodelay; 
    	limit_conn_zone $binary_remote_addr zone=ddos:10m; 
    	limit_conn ddos 100;
    

    100r/s iit’s already a lot !!

    and for vhost file :

    server {
    	.....
    
            location / {
    				
    				limit_req zone=flood; #Test 
    				limit_conn ddos 100; #Test 
    }
    

    –> If you have other ideas, I’m interested
    –> If you have better values ​​to use in what I modified, please let me know.

  • For your information @phenomlab ,

    • I have ban via iptables suspicious ip address find on /etc/nginx/accesss.log and virtualhost access.log like this : iptables -I INPUT -s IPADDRESS -j DROP
    • Activate bot option on CF
    • Create contry rules (Russie and China) on CF WAF
    • I left under attack mode option activated on CF
    • I have just change nginx.conf like this for test (If you have best value, I take it ! ) :
    worker_rlimit_nofile 70000; 
    
    events {
    
    	worker_connections 65535;
    	multi_accept on; 
    }
    
    http {
    
    	##
    	# Basic Settings
    	##
    
    	limit_req_zone $binary_remote_addr zone=flood:10m rate=100r/s; 
    	limit_req zone=flood burst=100 nodelay; 
    	limit_conn_zone $binary_remote_addr zone=ddos:10m; 
    	limit_conn ddos 100;
    

    100r/s iit’s already a lot !!

    and for vhost file :

    server {
    	.....
    
            location / {
    				
    				limit_req zone=flood; #Test 
    				limit_conn ddos 100; #Test 
    }
    

    –> If you have other ideas, I’m interested
    –> If you have better values ​​to use in what I modified, please let me know.

    wrote on last edited by phenomlab
    #18

    @DownPW my only preference would be to not set worker_connections so high

  • @DownPW my only preference would be to not set worker_connections so high

    wrote on last edited by
    #19

    @phenomlab

    Ok so what value do you advise?

  • @phenomlab

    Ok so what value do you advise?

    wrote on last edited by phenomlab
    #20

    @DownPW you should base it on the output of ulimit - see below

    https://linuxhint.com/what-are-worker-connections-nginx/#:~:text=The worker_connections are the maximum,to accommodate a higher value

    With that high value you run the risk of overwhelming your server.

  • @DownPW you should base it on the output of ulimit - see below

    https://linuxhint.com/what-are-worker-connections-nginx/#:~:text=The worker_connections are the maximum,to accommodate a higher value

    With that high value you run the risk of overwhelming your server.

    wrote on last edited by DownPW
    #21

    @phenomlab

    Thanks mark 😉

    My ulimit is 1024, so I will set it to 1024

  • @phenomlab

    Thanks mark 😉

    My ulimit is 1024, so I will set it to 1024

    wrote on last edited by
    #22

    @DownPW And the worker_processes value ? I expect this to be between 1 and 4 ?

  • @DownPW And the worker_processes value ? I expect this to be between 1 and 4 ?

    wrote on last edited by DownPW
    #23

    @phenomlab

    worker_processes auto;
    
  • @phenomlab

    worker_processes auto;
    
    wrote on last edited by
    #24

    @DownPW ok. You should refer to that some article I previously provided. You can probably set this to a static value.

  • @DownPW ok. You should refer to that some article I previously provided. You can probably set this to a static value.

    wrote on last edited by DownPW
    #25

    @phenomlab

    Ok I will see it for better worker_processes value

    I add a rate limite request and limit_conn_zone on http block and vhost block :

    – nginx.conf:

    http {
    
    	#Requete maximun par ip 
    	limit_req_zone $binary_remote_addr zone=flood:10m rate=100r/s; 
    	#Connexions maximum par ip 
    	limit_conn_zone $binary_remote_addr zone=ddos:1m;
    
    -- vhost.conf :
            location / {
    				
    				limit_req zone=flood burst=100 nodelay; 
    				limit_conn ddos 10;
    

    –> I have test other value for rate and burst but they cause problem access to the forum. If you have better, I take it

    I add today a proxy_read_timeout on vhost.conf (60 by default)

    proxy_read_timeout 180;
    

    I have deactivate underattack mode on CF and change for high Level

    I have add other rules on CF waf :

    9059db6c-f4f3-4415-bb47-ab88b2b41861-image.png

  • @phenomlab

    Ok I will see it for better worker_processes value

    I add a rate limite request and limit_conn_zone on http block and vhost block :

    – nginx.conf:

    http {
    
    	#Requete maximun par ip 
    	limit_req_zone $binary_remote_addr zone=flood:10m rate=100r/s; 
    	#Connexions maximum par ip 
    	limit_conn_zone $binary_remote_addr zone=ddos:1m;
    
    -- vhost.conf :
            location / {
    				
    				limit_req zone=flood burst=100 nodelay; 
    				limit_conn ddos 10;
    

    –> I have test other value for rate and burst but they cause problem access to the forum. If you have better, I take it

    I add today a proxy_read_timeout on vhost.conf (60 by default)

    proxy_read_timeout 180;
    

    I have deactivate underattack mode on CF and change for high Level

    I have add other rules on CF waf :

    9059db6c-f4f3-4415-bb47-ab88b2b41861-image.png

    wrote on last edited by
    #26

    @DownPW what settings do you have in advanced (in settings) for rate limit etc?

  • @DownPW what settings do you have in advanced (in settings) for rate limit etc?

    wrote on last edited by
    #27

    @phenomlab said in NODEBB: Nginx error performance & High CPU:

    @DownPW what settings do you have in advanced (in settings) for rate limit etc?

    In cloudflare ?

  • wrote on last edited by DownPW
    #28

    @phenomlab

    I wanted to test awstats on virtualmin with root account and it hasn’t updated since August 2022.

    I wanted to regenerate the files but I have a problem of rights.

    df12b039-076d-4e64-9a8d-ad6580ed57dc-image.png
    4030eec2-29ce-4dcf-83f6-adc60f87be5e-image.png

    What do you think ? and how t ore-generate a rapport correctly

    I would like to use it to better manage the @ips that connect to the server


Did this solution help you?
Did you find the suggested solution useful? Support 💗 Sudonix with a coffee ☕
If your organisation needs deeper expertise around infrastructure, security, or technology leadership, learn more about Phenomlab Ltd. Many of the deeper technical guides behind Sudonix are published there.

Related Topics
  • Spam spam spam

    Solved Configure nodebb
    6
    2 Votes
    6 Posts
    3k Views
    phenomlab
    @Panda said in Spam spam spam: ok, yes Ive seen the queue, it shows IP, but doesnt have a field for comments from registrant. It’s not designed for that. It merely serves as a gateway between posts appearing on your form or not. @Panda said in Spam spam spam: It would be better if nodebb had this plugin included in ACP list, as not only then do you know its approved and should work, but many people cant or dont want to use CLI on the server That’s a question for the NodeBB devs but in all honesty you can’t not use the CLI when installing nodebb so to be this isn’t a big deal.
  • error with v3 in browser console

    Solved Performance socket error csrf
    4
    1
    0 Votes
    4 Posts
    2k Views
    phenomlab
    @DownPW it’s in relation to the response I provided above
  • SEO and Nodebb

    Performance nodebb seo
    2
    2 Votes
    2 Posts
    2k Views
    phenomlab
    @Panda It’s the best it’s ever been to be honest. I’ve used a myriad of systems in the past - most notably, WordPress, and then Flarum (which for SEO, was absolutely dire - they never even had SEO out of the box, and relied on a third party extension to do it), and NodeBB easily fares the best - see below example https://www.google.com/search?q=site%3Asudonix.org&oq=site%3Asudonix.org&aqs=chrome..69i57j69i60j69i58j69i60l2.9039j0j3&sourceid=chrome&ie=UTF-8#ip=1 However, this was not without significant effort on my part once I’d migrated from COM to ORG - see below posts https://community.nodebb.org/topic/17286/google-crawl-error-after-site-migration/17?_=1688461250365 And also https://support.google.com/webmasters/thread/221027803?hl=en&msgid=221464164 It was painful to say the least - as it turns out, there was an issue in NodeBB core that prevented spiders from getting to content, which as far as I understand, is now fixed. SEO in itself is a dark art - a black box that nobody really fully understands, and it’s essentially going to boil down to one thing - “content”. Google’s algorithm for indexing has also changed dramatically over the years. They only now crawl content that has value, so if it believes that your site has nothing to offer, it will simply skip it.
  • Email validation NodeBB

    Bugs nodebb
    21
    3 Votes
    21 Posts
    10k Views
    phenomlab
    @Panda said in Email validation NodeBB: Did you configure that as a custom change to the usual quote icon. How do you do that? I notice on NodeBB site its a solid blue quotes Yes, I changed it. NodeBB by default users the free font awesome library whereas I use the pro (paid) version SDK have access to a wider set of icons, and at different thicknesses etc. The change of colour is just simple CSS.
  • Optimum config for NodeBB under NGINX

    Performance nginx performance
    4
    3 Votes
    4 Posts
    2k Views
    phenomlab
    @crazycells hi - no security reason, or anything specific in this case. However, the nginx.conf I posted was from my Dev environment which uses this port as a way of not interfering with production. And yes, I use clustering on this site with three instances.
  • NodeBB slow after DB recovery

    Solved Performance performance nodebb
    1
    5 Votes
    1 Posts
    967 Views
    No one has replied
  • Iframely (Nodebb)

    Solved Configure
    40
    4 Votes
    40 Posts
    17k Views
    phenomlab
    @DownPW This is now resolved. The issue was an incorrect URL specified in the Nodebb plugin. I’ve corrected this, and now it works as intended.
  • RSS parser script [NodeBB]

    Customisation nodebb rss script
    47
    9 Votes
    47 Posts
    23k Views
    JAC
    @phenomlab said in RSS parser script [NodeBB]: @jac Not yet. Sorry. Have some other work commitments I need to prioritise. Hoping to get to this over the weekend. No worries at all mate, thanks