Skip to content
Sudonix

Mongodb or Redis

General
  • Madchatthewundefined

    @phenomlab Yeah I found this one too that says the same thing. When I do the command to see if avx is on there, it doesn’t show anything. Also, my computer is 10 years old.

    https://superuser.com/questions/1814515/mongodb-error-failed-with-result-core-dump

    0
  • Madchatthewundefined

    I was able to figure it out with the help of this stackoverflow post linked below. Following these instructions I was able to get mongodb running and avx showed up in the cpu info. So I have this setup in virtualbox so on a hosted server, hopefully this wouldn’t be an issue, but if anyone else runs across this, hopefully it helps.

    To summarize the instructions:

Open a Command Prompt in Windows Host as Administrator.

Find the Command Prompt icon and right-click. Choose Run As Administrator.
Disable Hypervisor lunch

bcdedit /set hypervisorlaunchtype off

Disable Microsoft Hyper-V

DISM /Online /Disable-Feature:Microsoft-Hyper-V

Shutdown quick shutdown -s -t 2

Wait a few seconds before turn it on.

    https://stackoverflow.com/questions/65780506/how-to-enable-avx-avx2-in-virtualbox-6-1-16-with-ubuntu-20-04-64bit

    1
  • Madchatthewundefined

    Now the text task will be to migrate data from Redis to Mongodb and setup the multithread that @phenomlab has setup for this site.

    1
  • Madchatthewundefined

    @phenomlab how do you have it setup to have this sight start up automatically if the server gets rebooted? I attempted to follow nodebb’s instructions but can’t seem to get the systemd nodebb.service instructions to work. I have also tried a few others that I googled for and it just doesn’t seem to want to run as a service.

    0
  • phenomlabundefined

    @Madchatthew can you post the configuration you have for the service?

    Author of OGProxy, Swatch, Threaded, Card, and most of Sudonix's bugs…

    0
  • Madchatthewundefined

    @phenomlab

    This is the Nginx conf file

    upstream io_nodes {
        ip_hash;
        server 127.0.0.1:4567;
        server 127.0.0.1:4568;
        server 127.0.0.1:4569;
}

server {

        listen 80;
        server_name nodebb.lan;
        root /home/nodebb/public_html;
        index index.php index.html index.htm;

        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Host $http_host;
        proxy_set_header X-NginX-Proxy true;
        proxy_redirect off;

        # Socket.IO Support
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";

        location @nodebb {
                proxy_pass http://io_nodes;
        }

        location ~ ^/assets/(.*) {
                root /home/nodebb/public_html/;
                try_files /build/public/$1 /public/$1 @nodebb;
        }

        location / {
                proxy_pass http://io_nodes;
        }
}

    This is the current nodebb.service file

    [Unit]
Description=NodeBB forum for Node.js.
Documentation=http://nodebb.readthedocs.io/en/latest/
After=system.slice multi-user.target

[Service]
Type=simple
User=nodebb

StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=nodebb

Environment=NODE_ENV=production
WorkingDirectory=/home/nodebb/public_html/

# Call with exec to be able to redirect output: http://stackoverflow.com/a/33036589/1827734
ExecStart=/usr/bin/node /home/nodebb/public_html/loader.js
Restart=on-failure

[Install]
Alias=forum
WantedBy=multi-user.target

    I started with this code and the above code is from a google search. This code below, I did also try it with the correct path for node.

    [Unit]
Description=NodeBB
Documentation=https://docs.nodebb.org
After=system.slice multi-user.target mongod.service

[Service]
Type=forking
User=nodebb

WorkingDirectory=/path/to/nodebb
PIDFile=/path/to/nodebb/pidfile
ExecStart=/usr/bin/env node loader.js --no-silent
Restart=always

[Install]
WantedBy=multi-user.target

    The “/usr/bin/env node” is where node is located that I found from using a command.
    I also get a 502 Bad Gateway when I start either ./nodebb start or node loader.js.

    When I do “sudo systemctl status nodebb” after using “sudo systemctl start nodebb”, I get the following.

    ○ nodebb.service - NodeBB forum for Node.js.
     Loaded: loaded (/etc/systemd/system/nodebb.service; disabled; preset: disabled)
     Active: inactive (dead)
       Docs: http://nodebb.readthedocs.io/en/latest/

Aug 11 20:53:16 morecowbell.lan systemd[1]: /etc/systemd/system/nodebb.service:10: Standard output type syslog is obsolete, automatically updating to journal. Please update your unit file, and consider removing the setting altogether.
Aug 11 20:53:16 morecowbell.lan systemd[1]: /etc/systemd/system/nodebb.service:11: Standard output type syslog is obsolete, automatically updating to journal. Please update your unit file, and consider removing the setting altogether.
Aug 11 20:53:16 morecowbell.lan systemd[1]: Started NodeBB forum for Node.js..
Aug 11 20:53:16 morecowbell.lan nodebb[2605]: Process "1298" from pidfile already running, exiting
Aug 11 20:53:16 morecowbell.lan systemd[1]: nodebb.service: Deactivated successfully.
Aug 11 20:53:24 morecowbell.lan systemd[1]: /etc/systemd/system/nodebb.service:10: Standard output type syslog is obsolete, automatically updating to journal. Please update your unit file, and consider removing the setting altogether.
Aug 11 20:53:24 morecowbell.lan systemd[1]: /etc/systemd/system/nodebb.service:11: Standard output type syslog is obsolete, automatically updating to journal. Please update your unit file, and consider removing the setting altogether.

    Sorry this post is so long and thank you for your help.

    Update: I just ran the status for nginx and received the below. So the 192.168.1.10 is my windows PC and the connection was refused. Would I need to add the port to ufw in order for this to work, or is this a different issue.

    Aug 11 20:49:54 morecowbell.lan nginx[576]: 2024/08/11 20:49:54 [error] 576#576: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.1.10, server: nodebb.lan, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:4568/", host: "nodebb.lan"
Aug 11 20:49:54 morecowbell.lan nginx[576]: 2024/08/11 20:49:54 [error] 576#576: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.1.10, server: nodebb.lan, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:4569/", host: "nodebb.lan"
Aug 11 20:49:54 morecowbell.lan nginx[576]: 2024/08/11 20:49:54 [error] 576#576: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.1.10, server: nodebb.lan, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:4567/", host: "nodebb.lan"
Aug 11 20:49:55 morecowbell.lan nginx[576]: 2024/08/11 20:49:55 [error] 576#576: *1 no live upstreams while connecting to upstream, client: 192.168.1.10, server: nodebb.lan, request: "GET /favicon.ico HTTP/1.1", upstream: "http://io_nodes/favicon.ico", host: "nodebb.lan", referrer: "http://nodebb.lan/"
Aug 11 20:51:57 morecowbell.lan nginx[577]: 2024/08/11 20:51:57 [error] 577#577: *6 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.1.10, server: nodebb.lan, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:4568/", host: "nodebb.lan"
Aug 11 20:51:57 morecowbell.lan nginx[577]: 2024/08/11 20:51:57 [error] 577#577: *6 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.1.10, server: nodebb.lan, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:4569/", host: "nodebb.lan"
Aug 11 20:51:57 morecowbell.lan nginx[577]: 2024/08/11 20:51:57 [error] 577#577: *6 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.1.10, server: nodebb.lan, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:4567/", host: "nodebb.lan"
Aug 11 20:51:57 morecowbell.lan nginx[577]: 2024/08/11 20:51:57 [error] 577#577: *6 no live upstreams while connecting to upstream, client: 192.168.1.10, server: nodebb.lan, request: "GET /favicon.ico HTTP/1.1", upstream: "http://io_nodes/favicon.ico", host: "nodebb.lan", referrer: "http://nodebb.lan/"
Aug 11 20:51:59 morecowbell.lan nginx[577]: 2024/08/11 20:51:59 [error] 577#577: *6 no live upstreams while connecting to upstream, client: 192.168.1.10, server: nodebb.lan, request: "GET / HTTP/1.1", upstream: "http://io_nodes/", host: "nodebb.lan"
Aug 11 20:51:59 morecowbell.lan nginx[577]: 2024/08/11 20:51:59 [error] 577#577: *6 no live upstreams while connecting to upstream, client: 192.168.1.10, server: nodebb.lan, request: "GET /favicon.ico HTTP/1.1", upstream: "http://io_nodes/favicon.ico", host: "nodebb.lan", referrer: "http://nodebb.lan/"

    Update: adding those ports to ufw did not fix it.

    0
  • phenomlabundefined

    @Madchatthew connection refused typically means that port is being used elsewhere.

    What’s the output of

    ./nodebb log

    Author of OGProxy, Swatch, Threaded, Card, and most of Sudonix's bugs…

    0
  • Madchatthewundefined

    @phenomlab

    [nodebb@morecowbell public_html]$ ./nodebb log

Hit Ctrl-C to exit


tail: cannot open './logs/output.log' for reading: No such file or directory
    0
  • phenomlabundefined

    @Madchatthew hmm. That doesn’t look like it’s properly installed, or you have a permissions issue. What user are launching NodeBB as?

    Can you try

    ./nodebb dev

    Author of OGProxy, Swatch, Threaded, Card, and most of Sudonix's bugs…

    0
  • Madchatthewundefined

    @phenomlab said in Mongodb or Redis:

    @Madchatthew hmm. That doesn’t look like it’s properly installed, or you have a permissions issue. What user are launching NodeBB as?

    Can you try

    ./nodebb dev

    I will give this a try this evening. At work now.

    1
  • phenomlabundefined

    @Madchatthew thanks. Let me know.

    Author of OGProxy, Swatch, Threaded, Card, and most of Sudonix's bugs…

    0
  • Madchatthewundefined

    @phenomlab here are those results. It is a little long but I wanted to make sure you were getting the whole picture.

    [nodebb@morecowbell public_html]$ ./nodebb dev

NodeBB v3.8.4 Copyright (C) 2013-2024 NodeBB Inc.
This program comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it under certain conditions.
For the full license, please visit: http://www.gnu.org/copyleft/gpl.html

Clustering enabled: Spinning up 3 process(es).

2024-08-13T01:03:42.527Z [4568/7624] - verbose: * using configuration stored in: /home/nodebb/public_html/config.json
2024-08-13T01:03:42.539Z [4567/7623] - verbose: * using configuration stored in: /home/nodebb/public_html/config.json
2024-08-13T01:03:42.547Z [4567/7623] - info: Initializing NodeBB v3.8.4 http://localhost:4567
2024-08-13T01:03:42.548Z [4567/7623] - verbose: * using mongo store at 127.0.0.1:27017
2024-08-13T01:03:42.548Z [4567/7623] - verbose: * using themes stored in: /home/nodebb/public_html/node_modules
2024-08-13T01:03:42.556Z [4569/7625] - verbose: * using configuration stored in: /home/nodebb/public_html/config.json
2024-08-13T01:03:42.961Z [4568/7624] - error: ReplyError: NOAUTH Authentication required.
    at parseError (/home/nodebb/public_html/node_modules/redis-parser/lib/parser.js:179:12)
    at parseType (/home/nodebb/public_html/node_modules/redis-parser/lib/parser.js:302:14)
2024-08-13T01:03:42.966Z [4568/7624] - error: uncaughtException: NOAUTH Authentication required.
ReplyError: NOAUTH Authentication required.
    at parseError (/home/nodebb/public_html/node_modules/redis-parser/lib/parser.js:179:12)
    at parseType (/home/nodebb/public_html/node_modules/redis-parser/lib/parser.js:302:14) {"date":"Mon Aug 12 2024 20:03:42 GMT-0500 (Central Daylight Time)","error":{"command":{"args":[],"name":"info"}},"exception":true,"os":{"loadavg":[0.12,0.1,0.08],"uptime":84039.1},"process":{"argv":["/usr/bin/node","/home/nodebb/public_html/app.js"],"cwd":"/home/nodebb/public_html","execPath":"/usr/bin/node","gid":1006,"memoryUsage":{"arrayBuffers":18535346,"external":20792656,"heapTotal":55267328,"heapUsed":25627808,"rss":141299712},"pid":7624,"uid":1006,"version":"v22.6.0"},"stack":"ReplyError: NOAUTH Authentication required.\n    at parseError (/home/nodebb/public_html/node_modules/redis-parser/lib/parser.js:179:12)\n    at parseType (/home/nodebb/public_html/node_modules/redis-parser/lib/parser.js:302:14)","trace":[{"column":12,"file":"/home/nodebb/public_html/node_modules/redis-parser/lib/parser.js","function":"parseError","line":179,"method":null,"native":false},{"column":14,"file":"/home/nodebb/public_html/node_modules/redis-parser/lib/parser.js","function":"parseType","line":302,"method":null,"native":false}]}
2024-08-13T01:03:42.967Z [4568/7624] - error: ReplyError: NOAUTH Authentication required.
    at parseError (/home/nodebb/public_html/node_modules/redis-parser/lib/parser.js:179:12)
    at parseType (/home/nodebb/public_html/node_modules/redis-parser/lib/parser.js:302:14)
2024-08-13T01:03:42.970Z [4567/7623] - error: ReplyError: NOAUTH Authentication required.
    at parseError (/home/nodebb/public_html/node_modules/redis-parser/lib/parser.js:179:12)
    at parseType (/home/nodebb/public_html/node_modules/redis-parser/lib/parser.js:302:14)
2024-08-13T01:03:42.973Z [4569/7625] - error: ReplyError: NOAUTH Authentication required.
    at parseError (/home/nodebb/public_html/node_modules/redis-parser/lib/parser.js:179:12)
    at parseType (/home/nodebb/public_html/node_modules/redis-parser/lib/parser.js:302:14)
2024-08-13T01:03:42.974Z [4567/7623] - error: uncaughtException: NOAUTH Authentication required.
ReplyError: NOAUTH Authentication required.
    at parseError (/home/nodebb/public_html/node_modules/redis-parser/lib/parser.js:179:12)
    at parseType (/home/nodebb/public_html/node_modules/redis-parser/lib/parser.js:302:14) {"date":"Mon Aug 12 2024 20:03:42 GMT-0500 (Central Daylight Time)","error":{"command":{"args":[],"name":"info"}},"exception":true,"os":{"loadavg":[0.12,0.1,0.08],"uptime":84039.11},"process":{"argv":["/usr/bin/node","/home/nodebb/public_html/app.js"],"cwd":"/home/nodebb/public_html","execPath":"/usr/bin/node","gid":1006,"memoryUsage":{"arrayBuffers":18535345,"external":20792655,"heapTotal":55267328,"heapUsed":25973952,"rss":142553088},"pid":7623,"uid":1006,"version":"v22.6.0"},"stack":"ReplyError: NOAUTH Authentication required.\n    at parseError (/home/nodebb/public_html/node_modules/redis-parser/lib/parser.js:179:12)\n    at parseType (/home/nodebb/public_html/node_modules/redis-parser/lib/parser.js:302:14)","trace":[{"column":12,"file":"/home/nodebb/public_html/node_modules/redis-parser/lib/parser.js","function":"parseError","line":179,"method":null,"native":false},{"column":14,"file":"/home/nodebb/public_html/node_modules/redis-parser/lib/parser.js","function":"parseType","line":302,"method":null,"native":false}]}
2024-08-13T01:03:42.975Z [4567/7623] - error: ReplyError: NOAUTH Authentication required.
    at parseError (/home/nodebb/public_html/node_modules/redis-parser/lib/parser.js:179:12)
    at parseType (/home/nodebb/public_html/node_modules/redis-parser/lib/parser.js:302:14)
2024-08-13T01:03:42.978Z [4569/7625] - error: uncaughtException: NOAUTH Authentication required.
ReplyError: NOAUTH Authentication required.
    at parseError (/home/nodebb/public_html/node_modules/redis-parser/lib/parser.js:179:12)
    at parseType (/home/nodebb/public_html/node_modules/redis-parser/lib/parser.js:302:14) {"date":"Mon Aug 12 2024 20:03:42 GMT-0500 (Central Daylight Time)","error":{"command":{"args":[],"name":"info"}},"exception":true,"os":{"loadavg":[0.12,0.1,0.08],"uptime":84039.12},"process":{"argv":["/usr/bin/node","/home/nodebb/public_html/app.js"],"cwd":"/home/nodebb/public_html","execPath":"/usr/bin/node","gid":1006,"memoryUsage":{"arrayBuffers":18535345,"external":20792655,"heapTotal":55005184,"heapUsed":25626160,"rss":141709312},"pid":7625,"uid":1006,"version":"v22.6.0"},"stack":"ReplyError: NOAUTH Authentication required.\n    at parseError (/home/nodebb/public_html/node_modules/redis-parser/lib/parser.js:179:12)\n    at parseType (/home/nodebb/public_html/node_modules/redis-parser/lib/parser.js:302:14)","trace":[{"column":12,"file":"/home/nodebb/public_html/node_modules/redis-parser/lib/parser.js","function":"parseError","line":179,"method":null,"native":false},{"column":14,"file":"/home/nodebb/public_html/node_modules/redis-parser/lib/parser.js","function":"parseType","line":302,"method":null,"native":false}]}
2024-08-13T01:03:42.980Z [4569/7625] - error: ReplyError: NOAUTH Authentication required.
    at parseError (/home/nodebb/public_html/node_modules/redis-parser/lib/parser.js:179:12)
    at parseType (/home/nodebb/public_html/node_modules/redis-parser/lib/parser.js:302:14)
2024-08-13T01:03:44.687Z [4567/7623] - verbose: [minifier] utilizing a maximum of 2 additional threads
2024-08-13T01:03:44.707Z [4569/7625] - verbose: [minifier] utilizing a maximum of 2 additional threads
2024-08-13T01:03:44.712Z [4567/7623] - info: [app] Shutdown (SIGTERM/SIGINT/SIGQUIT) Initialised.
2024-08-13T01:03:44.720Z [4569/7625] - info: [app] Shutdown (SIGTERM/SIGINT/SIGQUIT) Initialised.
2024-08-13T01:03:44.767Z [4568/7624] - verbose: [minifier] utilizing a maximum of 2 additional threads
2024-08-13T01:03:44.780Z [4568/7624] - info: [app] Shutdown (SIGTERM/SIGINT/SIGQUIT) Initialised.
(node:7625) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
2024-08-13T01:03:45.068Z [4569/7625] - error: Error [ERR_SERVER_NOT_RUNNING]: Server is not running.
    at Server.close (node:net:2278:12)
    at Object.onceWrapper (node:events:634:28)
    at Server.emit (node:events:520:28)
    at emitCloseNT (node:net:2338:8)
    at process.processTicksAndRejections (node:internal/process/task_queues:81:21)
(node:7623) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
2024-08-13T01:03:45.073Z [4567/7623] - error: Error [ERR_SERVER_NOT_RUNNING]: Server is not running.
    at Server.close (node:net:2278:12)
    at Object.onceWrapper (node:events:634:28)
    at Server.emit (node:events:520:28)
    at emitCloseNT (node:net:2338:8)
    at process.processTicksAndRejections (node:internal/process/task_queues:81:21)
[cluster] Child Process (7625) has exited (code: 1, signal: null)
[cluster] Spinning up another process...
[cluster] Child Process (7623) has exited (code: 1, signal: null)
[cluster] Spinning up another process...
(node:7624) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
2024-08-13T01:03:45.129Z [4568/7624] - error: Error [ERR_SERVER_NOT_RUNNING]: Server is not running.
    at Server.close (node:net:2278:12)
    at Object.onceWrapper (node:events:634:28)
    at Server.emit (node:events:520:28)
    at emitCloseNT (node:net:2338:8)
    at process.processTicksAndRejections (node:internal/process/task_queues:81:21)
[cluster] Child Process (7624) has exited (code: 1, signal: null)
[cluster] Spinning up another process...
    1
  • Madchatthewundefined

    Woot, I figured it out man!!! First off, in the config.json, i forgot to put the redis password on it. Then I was having trouble connecting to mongodb. I was googling, changing the config and finally did a sudo systemctl status mongodb to find out it wasn’t even running. So I enabled it, then started it and boom using ./nodebb dev started it right now.

    So then I went back in and configured my nodebb.service file to match what nodebb had on their site and BOOOM it is working. By using the scaling method, nodebb runs even faster and I don’t get the 503 error when refreshing.

    I have learned a lot by going through all of this and it is pretty awesome! Now having this complete, I have some more testing to do, but I am many steps closer to using Arch as my production server.

    🎆
    1
  • phenomlabundefined

    @Madchatthew Excellent! I looked at the error message above, and it does seem that your MongoDB configuration has been set to require authentication (which is correct), but seems the password was missing.

    Author of OGProxy, Swatch, Threaded, Card, and most of Sudonix's bugs…

    1
  • Madchatthewundefined

    @phenomlab yeah I looked through the error log more carefully and found the Redis auth part first, then when that was cleared up I found the Mongodb auth error. Probably should have figured that one out first, but oh well hahah

    1
  • phenomlabundefined

    @Madchatthew still a great catch.

    Author of OGProxy, Swatch, Threaded, Card, and most of Sudonix's bugs…

    👍
    1

Related Topics
  • ahmedundefined

    Is my Mongodb installation correct?

    Solved General
    27
    13 Votes
    27 Posts
    3k Views
    ahmedundefined

    yuuuuu.png

  • ahmedundefined

    mongodb backup problem

    Moved Configure
    3
    1 Votes
    3 Posts
    250 Views
    phenomlabundefined

    You might want to also review this post

    https://sudonix.org/topic/389/mongodb-backup-script

  • veronikyaundefined

    mongodb replica set

    Configure
    11
    2 Votes
    11 Posts
    526 Views
    phenomlabundefined

    @veronikya said in mongodb replica set:

    The host’s local dns resolution is not configured. The problem of the host’s hosts being unable to be resolved in docker has been solved. Surprisingly
    Solution:
    Edit the /etc/resovel.conf file
    Add 127.0.0.53

    One immediate issue I can think of here is that editing resolv.conf directly is no longer supported and not recommended (because the changes do not survive a reboot) - unless you install the resolvconf package?

  • DownPWundefined

    Nodebb: failed to restore a mongo dump

    Solved Configure
    2
    1 Votes
    2 Posts
    219 Views
    DownPWundefined

    @phenomlab

    In fact I specified the sub rep and not the rep

    DON’T DO THIS:

    nodebb@nodebbpwclonedb:~/nodebb$ sudo mongorestore --username admin --password XXXXXXXXXXXXXX --nsInclude nodebb.objects --drop /home/nodebb/nodebb_DB_20230107/nodebb/

    BUT THIS :

    nodebb@nodebbpwclonedb:~/nodebb$ sudo mongorestore --username admin --password XXXXXXXXXXXXXX --nsInclude nodebb.objects --drop /home/nodebb/nodebb_DB_20230107/

    🙂

  • Sampo2910undefined

    Mongodb Authorisation.

    Solved General
    17
    2 Votes
    17 Posts
    767 Views
    phenomlabundefined

    @Sampo2910 I just saw this on the NodeBB community site and it seems very much in line with the issue you are experiencing. Worth a look I think

    https://community.nodebb.org/topic/16826/does-nodebb-work-with-mongodb-6-0

  • phenomlabundefined

    MongoDB backup script

    Moved Let's Build It
    4
    6 Votes
    4 Posts
    454 Views
    phenomlabundefined

    After playing with the original code in the first post, I’ve modified it a little to suit my needs - posting it here in it’s full glory in case it’s of use for anyone else. Obviously, you’ll need to change the important parts

    #!/bin/bash #set -eo pipefail # # MongoDB Backup Script # VER. 0.20 # More Info: http://github.com/micahwedemeyer/automongobackup # Note, this is a lobotomized port of AutoMySQLBackup # (http://sourceforge.net/projects/automysqlbackup/) for use with # MongoDB. # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # #===================================================================== #===================================================================== # Set the following variables to your system needs # (Detailed instructions below variables) #===================================================================== # Database name to specify a specific database only e.g. myawesomeapp # Unnecessary if backup all databases # DBNAME="" # Collections name list to include e.g. system.profile users # DBNAME is required # Unecessary if backup all collections # COLLECTIONS="" # Collections to exclude e.g. system.profile users # DBNAME is required # Unecessary if backup all collections # EXCLUDE_COLLECTIONS="" # Username to access the mongo server e.g. dbuser # Unnecessary if authentication is off DBUSERNAME="" # Password to access the mongo server e.g. password # Unnecessary if authentication is off DBPASSWORD="" # Database for authentication to the mongo server e.g. admin # Unnecessary if authentication is off DBAUTHDB="admin" # Host name (or IP address) of mongo server e.g localhost DBHOST="127.0.0.1" # Port that mongo is listening on DBPORT="27017" # Backup directory location e.g /backups BACKUPDIR="" # Mail setup # What would you like to be mailed to you? # - log : send only log file # - files : send log file and sql files as attachments (see docs) # - stdout : will simply output the log to the screen if run manually. # - quiet : Only send logs if an error occurs to the MAILADDR. MAILCONTENT="log" # Set the maximum allowed email size in k. (4000 = approx 5MB email [see docs]) export MAXATTSIZE="4000" # Email Address to send mail to? (user@domain.com) MAILADDR="" # ============================================================================ # === SCHEDULING AND RETENTION OPTIONS ( Read the doc's below for details )=== #============================================================================= # Do you want to do hourly backups? How long do you want to keep them? DOHOURLY="no" HOURLYRETENTION=24 # Do you want to do daily backups? How long do you want to keep them? DODAILY="yes" DAILYRETENTION=7 # Which day do you want weekly backups? (1 to 7 where 1 is Monday) DOWEEKLY="no" WEEKLYDAY=6 WEEKLYRETENTION=4 # Do you want monthly backups? How long do you want to keep them? DOMONTHLY="no" MONTHLYRETENTION=4 # ============================================================ # === ADVANCED OPTIONS ( Read the doc's below for details )=== #============================================================= # Choose Compression type. (gzip or bzip2) COMP="gzip" # Choose if the uncompressed folder should be deleted after compression has completed CLEANUP="yes" # Additionally keep a copy of the most recent backup in a seperate directory. LATEST="yes" # Make Hardlink not a copy LATESTLINK="yes" # Use oplog for point-in-time snapshotting. OPLOG="no" # Choose other Server if is Replica-Set Master REPLICAONSLAVE="no" # Allow DBUSERNAME without DBAUTHDB REQUIREDBAUTHDB="yes" # Maximum files of a single backup used by split - leave empty if no split required # MAXFILESIZE="" # Command to run before backups (uncomment to use) # PREBACKUP="" # Command run after backups (uncomment to use) # POSTBACKUP="" #===================================================================== # Options documentation #===================================================================== # Set DBUSERNAME and DBPASSWORD of a user that has at least SELECT permission # to ALL databases. # # Set the DBHOST option to the server you wish to backup, leave the # default to backup "this server".(to backup multiple servers make # copies of this file and set the options for that server) # # You can change the backup storage location from /backups to anything # you like by using the BACKUPDIR setting.. # # The MAILCONTENT and MAILADDR options and pretty self explanatory, use # these to have the backup log mailed to you at any email address or multiple # email addresses in a space seperated list. # # (If you set mail content to "log" you will require access to the "mail" program # on your server. If you set this to "files" you will have to have mutt installed # on your server. If you set it to "stdout" it will log to the screen if run from # the console or to the cron job owner if run through cron. If you set it to "quiet" # logs will only be mailed if there are errors reported. ) # # # Finally copy automongobackup.sh to anywhere on your server and make sure # to set executable permission. You can also copy the script to # /etc/cron.daily to have it execute automatically every night or simply # place a symlink in /etc/cron.daily to the file if you wish to keep it # somwhere else. # # NOTE: On Debian copy the file with no extention for it to be run # by cron e.g just name the file "automongobackup" # # Thats it.. # # # === Advanced options === # # To set the day of the week that you would like the weekly backup to happen # set the WEEKLYDAY setting, this can be a value from 1 to 7 where 1 is Monday, # The default is 6 which means that weekly backups are done on a Saturday. # # Use PREBACKUP and POSTBACKUP to specify Pre and Post backup commands # or scripts to perform tasks either before or after the backup process. # # #===================================================================== # Backup Rotation.. #===================================================================== # # Hourly backups are executed if DOHOURLY is set to "yes". # The number of hours backup copies to keep for each day (i.e. 'Monday', 'Tuesday', etc.) is set with DHOURLYRETENTION. # DHOURLYRETENTION=0 rotates hourly backups every day (i.e. only the most recent hourly copy is kept). -1 disables rotation. # # Daily backups are executed if DODAILY is set to "yes". # The number of daily backup copies to keep for each day (i.e. 'Monday', 'Tuesday', etc.) is set with DAILYRETENTION. # DAILYRETENTION=0 rotates daily backups every week (i.e. only the most recent daily copy is kept). -1 disables rotation. # # Weekly backups are executed if DOWEEKLY is set to "yes". # WEEKLYDAY [1-7] sets which day a weekly backup occurs when cron.daily scripts are run. # Rotate weekly copies after the number of weeks set by WEEKLYRETENTION. # WEEKLYRETENTION=0 rotates weekly backups every week. -1 disables rotation. # # Monthly backups are executed if DOMONTHLY is set to "yes". # Monthy backups occur on the first day of each month when cron.daily scripts are run. # Rotate monthly backups after the number of months set by MONTHLYRETENTION. # MONTHLYRETENTION=0 rotates monthly backups upon each execution. -1 disables rotation. # #===================================================================== # Please Note!! #===================================================================== # # I take no resposibility for any data loss or corruption when using # this script. # # This script will not help in the event of a hard drive crash. You # should copy your backups offline or to another PC for best protection. # # Happy backing up! # #===================================================================== # Restoring #===================================================================== # ??? # #===================================================================== # Change Log #===================================================================== # VER 0.11 - (2016-05-04) (author: Claudio Prato) # - Fixed bugs in select_secondary_member() with authdb enabled # - Fixed bugs in Compression function by removing the * symbol # - Added incremental backup feature # - Added option to select the collections to backup # # VER 0.10 - (2015-06-22) (author: Markus Graf) # - Added option to backup only one specific database # # VER 0.9 - (2011-10-28) (author: Joshua Keroes) # - Fixed bugs and improved logic in select_secondary_member() # - Fixed minor grammar issues and formatting in docs # # VER 0.8 - (2011-10-02) (author: Krzysztof Wilczynski) # - Added better support for selecting Secondary member in the # Replica Sets that can be used to take backups without bothering # busy Primary member too much. # # VER 0.7 - (2011-09-23) (author: Krzysztof Wilczynski) # - Added support for --journal dring taking backup # to enable journaling. # # VER 0.6 - (2011-09-15) (author: Krzysztof Wilczynski) # - Added support for --oplog during taking backup for # point-in-time snapshotting. # - Added filter for "mongodump" writing "connected to:" # on the standard error, which is not desirable. # # VER 0.5 - (2011-02-04) (author: Jan Doberstein) # - Added replicaset support (don't Backup on Master) # - Added Hard Support for 'latest' Copy # # VER 0.4 - (2010-10-26) # - Cleaned up warning message to make it clear that it can # usually be safely ignored # # VER 0.3 - (2010-06-11) # - Added the DBPORT parameter # - Changed USERNAME and PASSWORD to DBUSERNAME and DBPASSWORD # - Fixed some bugs with compression # # VER 0.2 - (2010-05-27) (author: Gregory Barchard) # - Added back the compression option for automatically creating # tgz or bz2 archives # - Added a cleanup option to optionally remove the database dump # after creating the archives # - Removed unnecessary path additions # # VER 0.1 - (2010-05-11) # - Initial Release # # VER 0.2 - (2015-09-10) # - Added configurable backup rentention options, even for # monthly backups. # #===================================================================== #===================================================================== #===================================================================== # # Should not need to be modified from here down!! # #===================================================================== #===================================================================== #===================================================================== shellout () { if [ -n "$1" ]; then echo "$1" exit 1 fi exit 0 } # External config - override default values set above for x in default sysconfig; do if [ -f "/etc/$x/automongobackup" ]; then # shellcheck source=/dev/null source /etc/$x/automongobackup fi done # Include extra config file if specified on commandline, e.g. for backuping several remote dbs from central server # shellcheck source=/dev/null [ ! -z "$1" ] && [ -f "$1" ] && source ${1} #===================================================================== PATH=/usr/local/bin:/usr/bin:/bin timestamp=$(date +%F-%T) DATE=$(date +%F-%H%M%S) # Datestamp e.g 2002-09-21 HOD=$(date +%s) # Current timestamp for PITR backup DOW=$(date +%A) # Day of the week e.g. Monday DNOW=$(date +%u) # Day number of the week 1 to 7 where 1 represents Monday DOM=$(date +%d) # Date of the Month e.g. 27 M=$(date +%B) # Month e.g January W=$(date +%V) # Week Number e.g 37 VER=0.11 # Version Number LOGFILE=$BACKUPDIR/$DBHOST-$(date +%F-%H%M%S).log # Logfile Name LOGERR=$BACKUPDIR/ERRORS_$DBHOST-$(date +%F-%H%M%S).log # Logfile Name OPT="" # OPT string for use with mongodump OPTSEC="" # OPT string for use with mongodump in select_secondary_member function QUERY="" # QUERY string for use with mongodump HOURLYQUERY="" # HOURLYQUERY string for use with mongodump # Do we need to use a username/password? if [ "$DBUSERNAME" ]; then OPT="$OPT --username=$DBUSERNAME --password=$DBPASSWORD" if [ "$REQUIREDBAUTHDB" = "yes" ]; then OPT="$OPT --authenticationDatabase=$DBAUTHDB" fi fi # Do we need to use a username/password for ReplicaSet Secondary Members Selection? if [ "$DBUSERNAME" ]; then OPTSEC="$OPTSEC --username=$DBUSERNAME --password=$DBPASSWORD" if [ "$REQUIREDBAUTHDB" = "yes" ]; then OPTSEC="$OPTSEC --authenticationDatabase=$DBAUTHDB" fi fi # Do we use oplog for point-in-time snapshotting? if [ "$OPLOG" = "yes" ] && [ -z "$DBNAME" ]; then OPT="$OPT --oplog" fi # Do we need to backup only a specific database? if [ "$DBNAME" ]; then OPT="$OPT -d $DBNAME" fi # Do we need to backup only a specific collections? if [ "$COLLECTIONS" ]; then for x in $COLLECTIONS; do OPT="$OPT --collection $x" done fi # Do we need to exclude collections? if [ "$EXCLUDE_COLLECTIONS" ]; then for x in $EXCLUDE_COLLECTIONS; do OPT="$OPT --excludeCollection $x" done fi # Do we use a filter for hourly point-in-time snapshotting? if [ "$DOHOURLY" == "yes" ]; then # getting PITR START timestamp # shellcheck disable=SC2012 [ "$COMP" = "gzip" ] && HOURLYQUERY=$(ls -t $BACKUPDIR/hourly | head -n 1 | cut -d '.' -f3) # setting the start timestamp to NOW for the first execution if [ -z "$HOURLYQUERY" ]; then QUERY="" else # limit the documents included in the output of mongodump # shellcheck disable=SC2016 QUERY='{ "ts" : { $gt : Timestamp('$HOURLYQUERY', 1) } }' fi fi # Create required directories mkdir -p $BACKUPDIR/{hourly,daily,weekly,monthly} || shellout 'failed to create directories' if [ "$LATEST" = "yes" ]; then rm -rf "$BACKUPDIR/latest" mkdir -p "$BACKUPDIR/latest" || shellout 'failed to create directory' fi # Do we use a filter for hourly point-in-time snapshotting? if [ "$DOHOURLY" == "yes" ]; then # getting PITR START timestamp # shellcheck disable=SC2012 [ "$COMP" = "gzip" ] && HOURLYQUERY=$(ls -t $BACKUPDIR/hourly | head -n 1 | cut -d '.' -f3) # setting the start timestamp to NOW for the first execution if [ -z "$HOURLYQUERY" ]; then QUERY="" else # limit the documents included in the output of mongodump # shellcheck disable=SC2016 QUERY='{ "ts" : { $gt : Timestamp('$HOURLYQUERY', 1) } }' fi fi # Check for correct sed usage if [ "$(uname -s)" = 'Darwin' ] || [ "$(uname -s)" = 'FreeBSD' ]; then SED="sed -i ''" else SED="sed -i" fi # IO redirection for logging. touch "$LOGFILE" exec 6>&1 # Link file descriptor #6 with stdout. # Saves stdout. exec > "$LOGFILE" # stdout replaced with file $LOGFILE. touch "$LOGERR" exec 7>&2 # Link file descriptor #7 with stderr. # Saves stderr. exec 2> "$LOGERR" # stderr replaced with file $LOGERR. # When a desire is to receive log via e-mail then we close stdout and stderr. [ "x$MAILCONTENT" == "xlog" ] && exec 6>&- 7>&- # Functions # Database dump function dbdump () { if [ -n "$QUERY" ]; then # filter for point-in-time snapshotting and if DOHOURLY=yes # shellcheck disable=SC2086 mongodump --quiet --host=$DBHOST:$DBPORT --out="$1" $OPT -q "$QUERY" MDUMPSTATUS=$? else # all others backups type # shellcheck disable=SC2086 mongodump --quiet --host=$DBHOST:$DBPORT --out="$1" $OPT MDUMPSTATUS=$? fi if [ $MDUMPSTATUS -ne 0 ]; then echo "$timestamp - ERROR: mongodump failed: $1" >&2 return 1 fi [ -e "$1" ] && return 0 echo "$timestamp - ERROR: mongodump failed to create dumpfile: $1" >&2 return 1 } # # Select first available Secondary member in the Replica Sets and show its # host name and port. # function select_secondary_member { # We will use indirect-reference hack to return variable from this function. local __return=$1 # Return list of with all replica set members # shellcheck disable=SC2086 members=( $(mongo --quiet --host $DBHOST:$DBPORT --eval 'rs.conf().members.forEach(function(x){ print(x.host) })' $OPTSEC ) ) # Check each replset member to see if it's a secondary and return it. if [ ${#members[@]} -gt 1 ]; then for member in "${members[@]}"; do is_secondary=$(mongo --quiet --host "$member" --eval 'rs.isMaster().secondary' $OPTSEC ) case "$is_secondary" in 'true') # First secondary wins ... secondary=$member break ;; 'false') # Skip particular member if it is a Primary. continue ;; *) # Skip irrelevant entries. Should not be any anyway ... continue ;; esac done fi if [ -n "$secondary" ]; then # Ugly hack to return value from a Bash function ... # shellcheck disable=SC2086 eval $__return="'$secondary'" fi } if [ -n "$MAXFILESIZE" ]; then write_file() { split --bytes "$MAXFILESIZE" --numeric-suffixes - "${1}-" } else write_file() { cat > "$1" } fi # Compression function plus latest copy compression () { SUFFIX="" dir=$(dirname "$1") file=$(basename "$1") if [ -n "$COMP" ]; then [ "$COMP" = "gzip" ] && SUFFIX=".tgz" [ "$COMP" = "bzip2" ] && SUFFIX=".tar.bz2" echo $timestamp - Tar and $COMP to "$file$SUFFIX" cd "$dir" || return 1 tar -cf - "$file" | $COMP --stdout | write_file "${file}${SUFFIX}" cd - >/dev/null || return 1 else echo "$timestamp - No compression option set, check advanced settings" fi if [ "$LATEST" = "yes" ]; then if [ "$LATESTLINK" = "yes" ];then COPY="ln" else COPY="cp" fi $COPY "$1$SUFFIX" "$BACKUPDIR/latest/" fi if [ "$CLEANUP" = "yes" ]; then echo $timestamp - Cleaning up folder at "$1" rm -rf "$1" fi return 0 } # Run command before we begin if [ "$PREBACKUP" ]; then echo "$timestamp - Prebackup command output." eval "$PREBACKUP" echo fi # Hostname for LOG information if [ "$DBHOST" = "localhost" ] || [ "$DBHOST" = "127.0.0.1" ]; then HOST=$(hostname) if [ "$SOCKET" ]; then OPT="$OPT --socket=$SOCKET" fi else HOST=$DBHOST fi # Try to select an available secondary for the backup or fallback to DBHOST. if [ "x${REPLICAONSLAVE}" == "xyes" ]; then # Return value via indirect-reference hack ... select_secondary_member secondary if [ -n "$secondary" ]; then DBHOST=${secondary%%:*} DBPORT=${secondary##*:} else SECONDARY_WARNING="WARNING: No suitable Secondary found in the Replica Sets. Falling back to ${DBHOST}." fi fi if [ ! -z "$SECONDARY_WARNING" ]; then echo echo "$SECONDARY_WARNING" fi echo "$timestamp - Database Server - $HOST on $DBHOST" echo "$timestamp - Backup started $(date)" # Monthly Full Backup of all Databases if [[ $DOM = "01" ]] && [[ $DOMONTHLY = "yes" ]]; then echo "$timestamp - Monthly Full Backup" echo # Delete old monthly backups while respecting the set rentention policy. if [[ $MONTHLYRETENTION -ge 0 ]] ; then NUM_OLD_FILES=$(find $BACKUPDIR/monthly -depth -not -newermt "$MONTHLYRETENTION month ago" -type f | wc -l) if [[ $NUM_OLD_FILES -gt 0 ]] ; then echo $timestamp - Deleting "$NUM_OLD_FILES" global setting backup file\(s\) older than "$MONTHLYRETENTION" month\(s\) old. find $BACKUPDIR/monthly -not -newermt "$MONTHLYRETENTION month ago" -type f -delete fi fi FILE="$BACKUPDIR/monthly/$DATE.$M" # Weekly Backup elif [[ "$DNOW" = "$WEEKLYDAY" ]] && [[ "$DOWEEKLY" = "yes" ]] ; then echo Weekly Backup echo if [[ $WEEKLYRETENTION -ge 0 ]] ; then # Delete old weekly backups while respecting the set rentention policy. NUM_OLD_FILES=$(find $BACKUPDIR/weekly -depth -not -newermt "$WEEKLYRETENTION week ago" -type f | wc -l) if [[ $NUM_OLD_FILES -gt 0 ]] ; then echo $timestamp - Deleting "$NUM_OLD_FILES" global setting backup file\(s\) older than "$WEEKLYRETENTION" week\(s\) old. find $BACKUPDIR/weekly -not -newermt "$WEEKLYRETENTION week ago" -type f -delete fi fi FILE="$BACKUPDIR/weekly/week.$W.$DATE" # Daily Backup elif [[ $DODAILY = "yes" ]] ; then echo $timestamp - Daily Backup of Databases echo # Delete old daily backups while respecting the set rentention policy. if [[ $DAILYRETENTION -ge 0 ]] ; then NUM_OLD_FILES=$(find $BACKUPDIR/daily -depth -not -newermt "$DAILYRETENTION days ago" -type f | wc -l) if [[ $NUM_OLD_FILES -gt 0 ]] ; then echo Deleting "$NUM_OLD_FILES" global setting backup file\(s\) made in previous weeks. find "$BACKUPDIR/daily" -not -newermt "$DAILYRETENTION days ago" -type f -delete fi fi # Below changed to remove .$DOW as this is not required FILE="$BACKUPDIR/daily/$DATE" # Hourly Backup elif [[ $DOHOURLY = "yes" ]] ; then echo $timestamp - Hourly Backup of Databases echo # Delete old hourly backups while respecting the set rentention policy. if [[ $HOURLYRETENTION -ge 0 ]] ; then NUM_OLD_FILES=$(find $BACKUPDIR/hourly -depth -not -newermt "$HOURLYRETENTION hour ago" -type f | wc -l) if [[ $NUM_OLD_FILES -gt 0 ]] ; then echo "$timestamp - Deleting $NUM_OLD_FILES global setting backup file\(s\) made in previous weeks." find $BACKUPDIR/hourly -not -newermt "$HOURLYRETENTION hour ago" -type f -delete fi fi FILE="$BACKUPDIR/hourly/$DATE.$DOW.$HOD" # convert timestamp to date: echo $TIMESTAMP | gawk '{print strftime("%c", $0)}' fi # FILE will not be set if no frequency is selected. if [[ -z "$FILE" ]] ; then echo "$timestamp - ERROR: No backup frequency was chosen." echo "$timestamp - Please set one of DOHOURLY,DODAILY,DOWEEKLY,DOMONTHLY to \"yes\"" exit 1 fi dbdump "$FILE" && compression "$FILE" echo "$timestamp - Backup completed $(date)" echo echo "Total disk space used for backup storage" echo echo "Size - Location " du -hs "$BACKUPDIR" echo # Run command when we're done if [ "$POSTBACKUP" ]; then echo "$timestamp - Postbackup command output." echo eval "$POSTBACKUP" echo fi # Clean up IO redirection if we plan not to deliver log via e-mail. [ ! "x$MAILCONTENT" == "xlog" ] && exec 1>&6 2>&7 6>&- 7>&- if [ -s "$LOGERR" ]; then eval "$SED" "/^connected/d" "$LOGERR" fi if [ "$MAILCONTENT" = "log" ]; then mail -s "Mongo Backup Log for $HOST - $DATE" "$MAILADDR" < "$LOGFILE" if [ -s "$LOGERR" ]; then cat "$LOGERR" mail -s "ERRORS REPORTED: Mongo Backup error Log for $HOST - $DATE" "$MAILADDR" < "$LOGERR" fi else if [ -s "$LOGERR" ]; then cat "$LOGFILE" echo echo "$timestamp - ###### WARNING ######" echo "$timestamp - STDERR written to during mongodump execution." echo "$timestamp - The backup probably succeeded, as mongodump sometimes writes to STDERR, but you may wish to scan the error log below:" cat "$LOGERR" else cat "$LOGFILE" fi fi # TODO: Would be nice to know if there were any *actual* errors in the $LOGERR STATUS=0 if [ -s "$LOGERR" ]; then STATUS=1 fi # Clean up Logfile rm -f "$LOGFILE" "$LOGERR" exit $STATUS

    One other thing I forgot to mention here is that you’ll need mailutils installed on your server for the mail option to work 🙂

  • WesleyMouraundefined

    How to restrict external access to Redis instance installed in a docker image

    Solved Configure
    8
    4 Votes
    8 Posts
    1k Views
    WesleyMouraundefined

    @phenomlab
    Sorry to delay in responding, yes as i mentioned above, i had to remove my redis from docker and reinstall a new image with this command

    docker run --name=redis -p 127.0.0.1:6379:6379 -d -t redis:alpine

    and now when i test my ip and port on
    https://www.yougetsignal.com/tools/open-ports/

    the status of my redis port is closed. I think which to configure firewall in droplet digital ocean is a good idea too, and i will configure soon.
    Thanks for the help!

  • phenomlabundefined

    mongorestore "E11000 duplicate key error collection"

    Solved Configure
    1
    1 Votes
    1 Posts
    791 Views
    No one has replied