Skip to content

Mongodb Authorisation.

Solved General
  • Error: couldn’t add user: there are no users authenticated :

    Above error is what I receive when i try to add a user or showdbs for example.

    This is a clean install of ubuntu 20.04 and all on a new drive.

    Things is I thought I am sure I already added a user and db previously. Now this when i try to install nodebb again. Any ideas? Security is auth=true in mongodb.conf

  • @Sampo2910 this is because you have the below in your mongod.conf

    security:
      authorization: enabled
    

    This forces the database to only accept connections or direct commands if you authenticate first. If you look at the guide, the ordering of steps is to add the users in the database before enabling the authorization.

    https://docs.nodebb.org/installing/os/ubuntu/

    Not that this matters as such but can be confusing if you enable this setting before creating the users.

  • @Sampo2910 What do you get from the command below

    mongo -u admin -p password --authenticationDatabase=admin
    

    Obviously, replace password with the real one.

  • @phenomlab
    When i used that and replaced details I get

    MongoDB shell version v3.6.8
    connecting to: mongodb://127.0.0.1:27017
    Implicit session: session { "id" : UUID("33e5efda-709f-43a9-ab1f-6c2f5be24199") }
    MongoDB server version: 3.6.8
    Server has startup warnings:
    2022-12-02T18:33:17.917+0200 I STORAGE  [initandlisten]
    2022-12-02T18:33:17.917+0200 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
    2022-12-02T18:33:17.917+0200 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
    > show dbs
    admin          0.000GB
    cod4xwebadmin  0.002GB
    config         0.000GB
    local          0.000GB
    >
    
    

    So i can go in there that way and i managed to create a user etc. Previously i just used ‘mongo’ and I could do all. What did change? I am using a newer version of Mongo than before. I know that.

  • @Sampo2910 this is because you have the below in your mongod.conf

    security:
      authorization: enabled
    

    This forces the database to only accept connections or direct commands if you authenticate first. If you look at the guide, the ordering of steps is to add the users in the database before enabling the authorization.

    https://docs.nodebb.org/installing/os/ubuntu/

    Not that this matters as such but can be confusing if you enable this setting before creating the users.

  • phenomlabundefined phenomlab has marked this topic as solved on
  • @phenomlab

    I do not really understand because I have always had it enabled in the past and was still able to use commands by just using ‘mongo’ in the shell. Whilst enabled I was also able to install NodeBB and create users. Just this time round I cannot even though I have followed instruction the same way.

    The other app i use instructs me to enable and at no point says to turn it off. That’s the same with nodebb install instructions.

    I do not have mongod.conf but mongodb.conf. The security looks like this:

    # Turn on/off security.  Off is currently the default
    #noauth = true
    auth = true
    
  • @Sampo2910 interesting. What OS are you using?

  • @phenomlab

    Ubuntu 20.04
    MongoDB shell version v3.6.8

  • @Sampo2910 that conf file doesn’t look right to me. From where did you install MongoDB?

  • @phenomlab It is the one i have used for years and the way i did it for years.

    https://linuxhint.com/install_mongodb_ubuntu_20_04/

  • @Sampo2910 hmm. Nothing wrong with that guide, but I’ve never seen a conf file from MongoDB that looks like it.

    Can you send me the remainder of the file?

  • @phenomlab

    # mongodb.conf
    
    # Where to store the data.
    dbpath=/var/lib/mongodb
    
    #where to log
    logpath=/var/log/mongodb/mongodb.log
    
    logappend=true
    
    bind_ip = 127.0.0.1
    #port = 27017
    
    # Enable journaling, http://www.mongodb.org/display/DOCS/Journaling
    journal=true
    
    # Enables periodic logging of CPU utilization and I/O wait
    #cpu = true
    
    # Turn on/off security.  Off is currently the default
    #noauth = true
    auth = true
    
    # Verbose logging output.
    #verbose = true
    
    # Inspect all client data for validity on receipt (useful for
    # developing drivers)
    #objcheck = true
    
    # Enable db quota management
    #quota = true
    
    # Set diagnostic logging level where n is
    #   0=off (default)
    #   1=W
    #   2=R
    #   3=both
    #   7=W+some reads
    #diaglog = 0
    
    # Diagnostic/debugging option
    #nocursors = true
    
    # Ignore query hints
    #nohints = true
    
    # Disable the HTTP interface (Defaults to localhost:27018).
    #nohttpinterface = true
    
    # Turns off server-side scripting.  This will result in greatly limited
    # functionality
    #noscripting = true
    
    # Turns off table scans.  Any query that would do a table scan fails.
    #notablescan = true
    
    # Disable data file preallocation.
    #noprealloc = true
    
    # Specify .ns file size for new databases.
    # nssize = <size>
    
    # Accout token for Mongo monitoring server.
    #mms-token = <token>
    
    # Server name for Mongo monitoring server.
    #mms-name = <server-name>
    
    # Ping interval for Mongo monitoring server.
    #mms-interval = <seconds>
    
    # Replication Options
    
    # in replicated mongo databases, specify here whether this is a slave or master
    #slave = true
    #source = master.example.com
    # Slave only: specify a single database to replicate
    #only = master.example.com
    # or
    #master = true
    #source = slave.example.com
    
    # Address of a server to pair with.
    #pairwith = <server:port>
    # Address of arbiter server.
    #arbiter = <server:port>
    # Automatically resync if slave data is stale
    #autoresync
    # Custom size for replication operation log.
    #oplogSize = <MB>
    # Size limit for in-memory storage of op ids.
    #opIdMem = <bytes>
    
    # SSL options
    # Enable SSL on normal ports
    #sslOnNormalPorts = true
    # SSL Key file and password
    #sslPEMKeyFile = /etc/ssl/mongodb.pem
    #sslPEMKeyPassword = pass
    
    
  • @Sampo2910 I’ve only ever installed MongoDB using the NodeBB guide

    https://docs.nodebb.org/installing/os/ubuntu/

  • @phenomlab

    All broken at the moment. Will figure it out 🙂 Using that guide above stopped my other app and a ton of errors on install 🙂 Also this is now in nodebb. All for another day 🙂

    2022-12-03 01_31_24-Extend _ Plugins _ NodeBB Admin Control Panel.png

  • @Sampo2910 what were the errors? I’ve never had an issue with it?

  • @phenomlab Only way all works is if i use node v12 anything more i get errors like that on the page.

    MongoDB shell version v5.0.14 and MongoDB server version: 3.6.8 is what I am using. I get a warning that they do not match but i cant seem to update mongo server version either.

    I am not at a level where I understand to be honest. Node, npm, nvm mongo…things work on some versions and not others. Updating to the latest just breaks things.

    smiley@fmjcod4:~$ mongo
    MongoDB shell version v5.0.14
    connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb
    Implicit session: session { "id" : UUID("b7326d9b-2817-403e-b072-2f0f5330761d") }
    MongoDB server version: 3.6.8
    WARNING: shell and server versions do not match
    ================
    Warning: the "mongo" shell has been superseded by "mongosh",
    which delivers improved usability and compatibility.The "mongo" shell has been deprecated and will be removed in
    an upcoming release.
    For installation instructions, see
    https://docs.mongodb.com/mongodb-shell/install/
    ================
    > show dbs
    uncaught exception: Error: Failed to acquire database information from privileges :
    _getErrorWithCode@src/mongo/shell/utils.js:25:13
    Mongo.prototype._getDatabaseNamesFromPrivileges@src/mongo/shell/mongo.js:69:15
    Mongo.prototype.getDBs/<@src/mongo/shell/mongo.js:134:31
    Mongo.prototype.getDBs@src/mongo/shell/mongo.js:97:12
    shellHelper.show@src/mongo/shell/utils.js:956:13
    shellHelper@src/mongo/shell/utils.js:838:15
    @(shellhelp2):1:1
    > exit
    bye
    
  • @Sampo2910 yes, that does indeed look a mess! I don’t think nodeJS at version 12 for NodeBB is even supported anymore. I asked a similar question but so long ago

    https://community.nodebb.org/topic/16532/supported-version-of-nodejs

    Plus, the version of MongoDB started (3.x) is also really old. I seem to recall you using an app which is considered legacy and requires older versions of software, but I think the two together are going to cause you significant issues - clearly which you’re seeing now.

    My suggestion would be to keep these on separate machines.

  • @Sampo2910 I just saw this on the NodeBB community site and it seems very much in line with the issue you are experiencing. Worth a look I think

    https://community.nodebb.org/topic/16826/does-nodebb-work-with-mongodb-6-0


Did this solution help you?
Did you find the suggested solution useful? Why not buy me a coffee? It's a nice gesture, and a great way to show your appreciation 💗

Related Topics
  • mongodb replica set

    Configure
    11
    2 Votes
    11 Posts
    370 Views

    @veronikya said in mongodb replica set:

    The host’s local dns resolution is not configured. The problem of the host’s hosts being unable to be resolved in docker has been solved. Surprisingly
    Solution:
    Edit the /etc/resovel.conf file
    Add 127.0.0.53

    One immediate issue I can think of here is that editing resolv.conf directly is no longer supported and not recommended (because the changes do not survive a reboot) - unless you install the resolvconf package?

  • Nodebb design

    Solved General
    2
    1 Votes
    2 Posts
    226 Views

    @Panda said in Nodebb design:

    One negative is not being so good for SEO as more Server side rendered forums, if web crawlers dont run the JS to read the forum.

    From recollection, Google and Bing have the capability to read and process JS, although it’s not in the same manner as a physical person will consume content on a page. It will be seen as plain text, but will be indexed. However, it’s important to note that Yandex and Baidu will not render JS, although seeing as Google has a 90% share of the content available on the web in terms of indexing, this isn’t something you’ll likely lose sleep over.

    @Panda said in Nodebb design:

    The “write api” is preferred for server-to-server interactions.

    This is mostly based around overall security - you won’t typically want a client machine changing database elements or altering data. This is why you have “client-side” which could be DOM manipulation etc, and “server-side” which performs more complex operations as it can communicate directly with the database whereas the client cannot (and if it can, then you have a serious security flaw). Reading from the API is perfectly acceptable on the client-side, but not being able to write.

    A paradigm here would be something like SNMP. This protocol exists as a UDP (UDP is very efficient, as it is “fire and forget” and does not wait for a response like TCP does) based service which reads performance data from a remote source, thus enabling an application to parse that data for use in a monitoring application. In all cases, SNMP access should be “RO” (Read Only) and not RW (Read Write). It is completely feasible to assume complete control over a firewall for example by having RW access to SNMP and then exposing it to the entire internet with a weak passphrase.

    You wouldn’t do it (at least, I hope you wouldn’t) and the same ethic applies to server-side rendering and the execution of commands.

  • 1 Votes
    4 Posts
    245 Views

    @Vijay-Kumavat-0 I think you’d be better off customising the plugin above to be honest. It seems like you’d be reinventing the wheel otherwise.

    I’d modify

    /nodebb/node_modules/nodebb-rewards-essentials/lib/conditions.js
    /nodebb/node_modules/nodebb-rewards-essentials/lib/rewards.js

    These seem to be the two files that control what is in the dropdown lists.

  • Sidebar Widget is no longer on the side!

    Moved Solved General
    4
    2 Votes
    4 Posts
    251 Views

    @Panda said in Sidebar Widget is no longer on the side!:

    Ah, so sidebar wont work on mobile?

    Correct. If you review the docs on bootstrap, you’ll notice that it is designed on a grid system

    https://getbootstrap.com/docs/5.0/layout/grid/

    What I mean by changing the category is moving it on here to general as you posted it in bugs, when it isn’t.

  • 3 Votes
    2 Posts
    143 Views

    @DownPW This isn’t something I have readily available, and because I’m not entirely familiar with the database structure of NodeBB, it’s something that would require research and development in terms of script.

    I think it’d be quicker to ask this in the NodeBB forums.

  • MongoDB backup script

    Moved Let's Build It
    4
    6 Votes
    4 Posts
    288 Views

    After playing with the original code in the first post, I’ve modified it a little to suit my needs - posting it here in it’s full glory in case it’s of use for anyone else. Obviously, you’ll need to change the important parts

    #!/bin/bash #set -eo pipefail # # MongoDB Backup Script # VER. 0.20 # More Info: http://github.com/micahwedemeyer/automongobackup # Note, this is a lobotomized port of AutoMySQLBackup # (http://sourceforge.net/projects/automysqlbackup/) for use with # MongoDB. # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # #===================================================================== #===================================================================== # Set the following variables to your system needs # (Detailed instructions below variables) #===================================================================== # Database name to specify a specific database only e.g. myawesomeapp # Unnecessary if backup all databases # DBNAME="" # Collections name list to include e.g. system.profile users # DBNAME is required # Unecessary if backup all collections # COLLECTIONS="" # Collections to exclude e.g. system.profile users # DBNAME is required # Unecessary if backup all collections # EXCLUDE_COLLECTIONS="" # Username to access the mongo server e.g. dbuser # Unnecessary if authentication is off DBUSERNAME="" # Password to access the mongo server e.g. password # Unnecessary if authentication is off DBPASSWORD="" # Database for authentication to the mongo server e.g. admin # Unnecessary if authentication is off DBAUTHDB="admin" # Host name (or IP address) of mongo server e.g localhost DBHOST="127.0.0.1" # Port that mongo is listening on DBPORT="27017" # Backup directory location e.g /backups BACKUPDIR="" # Mail setup # What would you like to be mailed to you? # - log : send only log file # - files : send log file and sql files as attachments (see docs) # - stdout : will simply output the log to the screen if run manually. # - quiet : Only send logs if an error occurs to the MAILADDR. MAILCONTENT="log" # Set the maximum allowed email size in k. (4000 = approx 5MB email [see docs]) export MAXATTSIZE="4000" # Email Address to send mail to? (user@domain.com) MAILADDR="" # ============================================================================ # === SCHEDULING AND RETENTION OPTIONS ( Read the doc's below for details )=== #============================================================================= # Do you want to do hourly backups? How long do you want to keep them? DOHOURLY="no" HOURLYRETENTION=24 # Do you want to do daily backups? How long do you want to keep them? DODAILY="yes" DAILYRETENTION=7 # Which day do you want weekly backups? (1 to 7 where 1 is Monday) DOWEEKLY="no" WEEKLYDAY=6 WEEKLYRETENTION=4 # Do you want monthly backups? How long do you want to keep them? DOMONTHLY="no" MONTHLYRETENTION=4 # ============================================================ # === ADVANCED OPTIONS ( Read the doc's below for details )=== #============================================================= # Choose Compression type. (gzip or bzip2) COMP="gzip" # Choose if the uncompressed folder should be deleted after compression has completed CLEANUP="yes" # Additionally keep a copy of the most recent backup in a seperate directory. LATEST="yes" # Make Hardlink not a copy LATESTLINK="yes" # Use oplog for point-in-time snapshotting. OPLOG="no" # Choose other Server if is Replica-Set Master REPLICAONSLAVE="no" # Allow DBUSERNAME without DBAUTHDB REQUIREDBAUTHDB="yes" # Maximum files of a single backup used by split - leave empty if no split required # MAXFILESIZE="" # Command to run before backups (uncomment to use) # PREBACKUP="" # Command run after backups (uncomment to use) # POSTBACKUP="" #===================================================================== # Options documentation #===================================================================== # Set DBUSERNAME and DBPASSWORD of a user that has at least SELECT permission # to ALL databases. # # Set the DBHOST option to the server you wish to backup, leave the # default to backup "this server".(to backup multiple servers make # copies of this file and set the options for that server) # # You can change the backup storage location from /backups to anything # you like by using the BACKUPDIR setting.. # # The MAILCONTENT and MAILADDR options and pretty self explanatory, use # these to have the backup log mailed to you at any email address or multiple # email addresses in a space seperated list. # # (If you set mail content to "log" you will require access to the "mail" program # on your server. If you set this to "files" you will have to have mutt installed # on your server. If you set it to "stdout" it will log to the screen if run from # the console or to the cron job owner if run through cron. If you set it to "quiet" # logs will only be mailed if there are errors reported. ) # # # Finally copy automongobackup.sh to anywhere on your server and make sure # to set executable permission. You can also copy the script to # /etc/cron.daily to have it execute automatically every night or simply # place a symlink in /etc/cron.daily to the file if you wish to keep it # somwhere else. # # NOTE: On Debian copy the file with no extention for it to be run # by cron e.g just name the file "automongobackup" # # Thats it.. # # # === Advanced options === # # To set the day of the week that you would like the weekly backup to happen # set the WEEKLYDAY setting, this can be a value from 1 to 7 where 1 is Monday, # The default is 6 which means that weekly backups are done on a Saturday. # # Use PREBACKUP and POSTBACKUP to specify Pre and Post backup commands # or scripts to perform tasks either before or after the backup process. # # #===================================================================== # Backup Rotation.. #===================================================================== # # Hourly backups are executed if DOHOURLY is set to "yes". # The number of hours backup copies to keep for each day (i.e. 'Monday', 'Tuesday', etc.) is set with DHOURLYRETENTION. # DHOURLYRETENTION=0 rotates hourly backups every day (i.e. only the most recent hourly copy is kept). -1 disables rotation. # # Daily backups are executed if DODAILY is set to "yes". # The number of daily backup copies to keep for each day (i.e. 'Monday', 'Tuesday', etc.) is set with DAILYRETENTION. # DAILYRETENTION=0 rotates daily backups every week (i.e. only the most recent daily copy is kept). -1 disables rotation. # # Weekly backups are executed if DOWEEKLY is set to "yes". # WEEKLYDAY [1-7] sets which day a weekly backup occurs when cron.daily scripts are run. # Rotate weekly copies after the number of weeks set by WEEKLYRETENTION. # WEEKLYRETENTION=0 rotates weekly backups every week. -1 disables rotation. # # Monthly backups are executed if DOMONTHLY is set to "yes". # Monthy backups occur on the first day of each month when cron.daily scripts are run. # Rotate monthly backups after the number of months set by MONTHLYRETENTION. # MONTHLYRETENTION=0 rotates monthly backups upon each execution. -1 disables rotation. # #===================================================================== # Please Note!! #===================================================================== # # I take no resposibility for any data loss or corruption when using # this script. # # This script will not help in the event of a hard drive crash. You # should copy your backups offline or to another PC for best protection. # # Happy backing up! # #===================================================================== # Restoring #===================================================================== # ??? # #===================================================================== # Change Log #===================================================================== # VER 0.11 - (2016-05-04) (author: Claudio Prato) # - Fixed bugs in select_secondary_member() with authdb enabled # - Fixed bugs in Compression function by removing the * symbol # - Added incremental backup feature # - Added option to select the collections to backup # # VER 0.10 - (2015-06-22) (author: Markus Graf) # - Added option to backup only one specific database # # VER 0.9 - (2011-10-28) (author: Joshua Keroes) # - Fixed bugs and improved logic in select_secondary_member() # - Fixed minor grammar issues and formatting in docs # # VER 0.8 - (2011-10-02) (author: Krzysztof Wilczynski) # - Added better support for selecting Secondary member in the # Replica Sets that can be used to take backups without bothering # busy Primary member too much. # # VER 0.7 - (2011-09-23) (author: Krzysztof Wilczynski) # - Added support for --journal dring taking backup # to enable journaling. # # VER 0.6 - (2011-09-15) (author: Krzysztof Wilczynski) # - Added support for --oplog during taking backup for # point-in-time snapshotting. # - Added filter for "mongodump" writing "connected to:" # on the standard error, which is not desirable. # # VER 0.5 - (2011-02-04) (author: Jan Doberstein) # - Added replicaset support (don't Backup on Master) # - Added Hard Support for 'latest' Copy # # VER 0.4 - (2010-10-26) # - Cleaned up warning message to make it clear that it can # usually be safely ignored # # VER 0.3 - (2010-06-11) # - Added the DBPORT parameter # - Changed USERNAME and PASSWORD to DBUSERNAME and DBPASSWORD # - Fixed some bugs with compression # # VER 0.2 - (2010-05-27) (author: Gregory Barchard) # - Added back the compression option for automatically creating # tgz or bz2 archives # - Added a cleanup option to optionally remove the database dump # after creating the archives # - Removed unnecessary path additions # # VER 0.1 - (2010-05-11) # - Initial Release # # VER 0.2 - (2015-09-10) # - Added configurable backup rentention options, even for # monthly backups. # #===================================================================== #===================================================================== #===================================================================== # # Should not need to be modified from here down!! # #===================================================================== #===================================================================== #===================================================================== shellout () { if [ -n "$1" ]; then echo "$1" exit 1 fi exit 0 } # External config - override default values set above for x in default sysconfig; do if [ -f "/etc/$x/automongobackup" ]; then # shellcheck source=/dev/null source /etc/$x/automongobackup fi done # Include extra config file if specified on commandline, e.g. for backuping several remote dbs from central server # shellcheck source=/dev/null [ ! -z "$1" ] && [ -f "$1" ] && source ${1} #===================================================================== PATH=/usr/local/bin:/usr/bin:/bin timestamp=$(date +%F-%T) DATE=$(date +%F-%H%M%S) # Datestamp e.g 2002-09-21 HOD=$(date +%s) # Current timestamp for PITR backup DOW=$(date +%A) # Day of the week e.g. Monday DNOW=$(date +%u) # Day number of the week 1 to 7 where 1 represents Monday DOM=$(date +%d) # Date of the Month e.g. 27 M=$(date +%B) # Month e.g January W=$(date +%V) # Week Number e.g 37 VER=0.11 # Version Number LOGFILE=$BACKUPDIR/$DBHOST-$(date +%F-%H%M%S).log # Logfile Name LOGERR=$BACKUPDIR/ERRORS_$DBHOST-$(date +%F-%H%M%S).log # Logfile Name OPT="" # OPT string for use with mongodump OPTSEC="" # OPT string for use with mongodump in select_secondary_member function QUERY="" # QUERY string for use with mongodump HOURLYQUERY="" # HOURLYQUERY string for use with mongodump # Do we need to use a username/password? if [ "$DBUSERNAME" ]; then OPT="$OPT --username=$DBUSERNAME --password=$DBPASSWORD" if [ "$REQUIREDBAUTHDB" = "yes" ]; then OPT="$OPT --authenticationDatabase=$DBAUTHDB" fi fi # Do we need to use a username/password for ReplicaSet Secondary Members Selection? if [ "$DBUSERNAME" ]; then OPTSEC="$OPTSEC --username=$DBUSERNAME --password=$DBPASSWORD" if [ "$REQUIREDBAUTHDB" = "yes" ]; then OPTSEC="$OPTSEC --authenticationDatabase=$DBAUTHDB" fi fi # Do we use oplog for point-in-time snapshotting? if [ "$OPLOG" = "yes" ] && [ -z "$DBNAME" ]; then OPT="$OPT --oplog" fi # Do we need to backup only a specific database? if [ "$DBNAME" ]; then OPT="$OPT -d $DBNAME" fi # Do we need to backup only a specific collections? if [ "$COLLECTIONS" ]; then for x in $COLLECTIONS; do OPT="$OPT --collection $x" done fi # Do we need to exclude collections? if [ "$EXCLUDE_COLLECTIONS" ]; then for x in $EXCLUDE_COLLECTIONS; do OPT="$OPT --excludeCollection $x" done fi # Do we use a filter for hourly point-in-time snapshotting? if [ "$DOHOURLY" == "yes" ]; then # getting PITR START timestamp # shellcheck disable=SC2012 [ "$COMP" = "gzip" ] && HOURLYQUERY=$(ls -t $BACKUPDIR/hourly | head -n 1 | cut -d '.' -f3) # setting the start timestamp to NOW for the first execution if [ -z "$HOURLYQUERY" ]; then QUERY="" else # limit the documents included in the output of mongodump # shellcheck disable=SC2016 QUERY='{ "ts" : { $gt : Timestamp('$HOURLYQUERY', 1) } }' fi fi # Create required directories mkdir -p $BACKUPDIR/{hourly,daily,weekly,monthly} || shellout 'failed to create directories' if [ "$LATEST" = "yes" ]; then rm -rf "$BACKUPDIR/latest" mkdir -p "$BACKUPDIR/latest" || shellout 'failed to create directory' fi # Do we use a filter for hourly point-in-time snapshotting? if [ "$DOHOURLY" == "yes" ]; then # getting PITR START timestamp # shellcheck disable=SC2012 [ "$COMP" = "gzip" ] && HOURLYQUERY=$(ls -t $BACKUPDIR/hourly | head -n 1 | cut -d '.' -f3) # setting the start timestamp to NOW for the first execution if [ -z "$HOURLYQUERY" ]; then QUERY="" else # limit the documents included in the output of mongodump # shellcheck disable=SC2016 QUERY='{ "ts" : { $gt : Timestamp('$HOURLYQUERY', 1) } }' fi fi # Check for correct sed usage if [ "$(uname -s)" = 'Darwin' ] || [ "$(uname -s)" = 'FreeBSD' ]; then SED="sed -i ''" else SED="sed -i" fi # IO redirection for logging. touch "$LOGFILE" exec 6>&1 # Link file descriptor #6 with stdout. # Saves stdout. exec > "$LOGFILE" # stdout replaced with file $LOGFILE. touch "$LOGERR" exec 7>&2 # Link file descriptor #7 with stderr. # Saves stderr. exec 2> "$LOGERR" # stderr replaced with file $LOGERR. # When a desire is to receive log via e-mail then we close stdout and stderr. [ "x$MAILCONTENT" == "xlog" ] && exec 6>&- 7>&- # Functions # Database dump function dbdump () { if [ -n "$QUERY" ]; then # filter for point-in-time snapshotting and if DOHOURLY=yes # shellcheck disable=SC2086 mongodump --quiet --host=$DBHOST:$DBPORT --out="$1" $OPT -q "$QUERY" MDUMPSTATUS=$? else # all others backups type # shellcheck disable=SC2086 mongodump --quiet --host=$DBHOST:$DBPORT --out="$1" $OPT MDUMPSTATUS=$? fi if [ $MDUMPSTATUS -ne 0 ]; then echo "$timestamp - ERROR: mongodump failed: $1" >&2 return 1 fi [ -e "$1" ] && return 0 echo "$timestamp - ERROR: mongodump failed to create dumpfile: $1" >&2 return 1 } # # Select first available Secondary member in the Replica Sets and show its # host name and port. # function select_secondary_member { # We will use indirect-reference hack to return variable from this function. local __return=$1 # Return list of with all replica set members # shellcheck disable=SC2086 members=( $(mongo --quiet --host $DBHOST:$DBPORT --eval 'rs.conf().members.forEach(function(x){ print(x.host) })' $OPTSEC ) ) # Check each replset member to see if it's a secondary and return it. if [ ${#members[@]} -gt 1 ]; then for member in "${members[@]}"; do is_secondary=$(mongo --quiet --host "$member" --eval 'rs.isMaster().secondary' $OPTSEC ) case "$is_secondary" in 'true') # First secondary wins ... secondary=$member break ;; 'false') # Skip particular member if it is a Primary. continue ;; *) # Skip irrelevant entries. Should not be any anyway ... continue ;; esac done fi if [ -n "$secondary" ]; then # Ugly hack to return value from a Bash function ... # shellcheck disable=SC2086 eval $__return="'$secondary'" fi } if [ -n "$MAXFILESIZE" ]; then write_file() { split --bytes "$MAXFILESIZE" --numeric-suffixes - "${1}-" } else write_file() { cat > "$1" } fi # Compression function plus latest copy compression () { SUFFIX="" dir=$(dirname "$1") file=$(basename "$1") if [ -n "$COMP" ]; then [ "$COMP" = "gzip" ] && SUFFIX=".tgz" [ "$COMP" = "bzip2" ] && SUFFIX=".tar.bz2" echo $timestamp - Tar and $COMP to "$file$SUFFIX" cd "$dir" || return 1 tar -cf - "$file" | $COMP --stdout | write_file "${file}${SUFFIX}" cd - >/dev/null || return 1 else echo "$timestamp - No compression option set, check advanced settings" fi if [ "$LATEST" = "yes" ]; then if [ "$LATESTLINK" = "yes" ];then COPY="ln" else COPY="cp" fi $COPY "$1$SUFFIX" "$BACKUPDIR/latest/" fi if [ "$CLEANUP" = "yes" ]; then echo $timestamp - Cleaning up folder at "$1" rm -rf "$1" fi return 0 } # Run command before we begin if [ "$PREBACKUP" ]; then echo "$timestamp - Prebackup command output." eval "$PREBACKUP" echo fi # Hostname for LOG information if [ "$DBHOST" = "localhost" ] || [ "$DBHOST" = "127.0.0.1" ]; then HOST=$(hostname) if [ "$SOCKET" ]; then OPT="$OPT --socket=$SOCKET" fi else HOST=$DBHOST fi # Try to select an available secondary for the backup or fallback to DBHOST. if [ "x${REPLICAONSLAVE}" == "xyes" ]; then # Return value via indirect-reference hack ... select_secondary_member secondary if [ -n "$secondary" ]; then DBHOST=${secondary%%:*} DBPORT=${secondary##*:} else SECONDARY_WARNING="WARNING: No suitable Secondary found in the Replica Sets. Falling back to ${DBHOST}." fi fi if [ ! -z "$SECONDARY_WARNING" ]; then echo echo "$SECONDARY_WARNING" fi echo "$timestamp - Database Server - $HOST on $DBHOST" echo "$timestamp - Backup started $(date)" # Monthly Full Backup of all Databases if [[ $DOM = "01" ]] && [[ $DOMONTHLY = "yes" ]]; then echo "$timestamp - Monthly Full Backup" echo # Delete old monthly backups while respecting the set rentention policy. if [[ $MONTHLYRETENTION -ge 0 ]] ; then NUM_OLD_FILES=$(find $BACKUPDIR/monthly -depth -not -newermt "$MONTHLYRETENTION month ago" -type f | wc -l) if [[ $NUM_OLD_FILES -gt 0 ]] ; then echo $timestamp - Deleting "$NUM_OLD_FILES" global setting backup file\(s\) older than "$MONTHLYRETENTION" month\(s\) old. find $BACKUPDIR/monthly -not -newermt "$MONTHLYRETENTION month ago" -type f -delete fi fi FILE="$BACKUPDIR/monthly/$DATE.$M" # Weekly Backup elif [[ "$DNOW" = "$WEEKLYDAY" ]] && [[ "$DOWEEKLY" = "yes" ]] ; then echo Weekly Backup echo if [[ $WEEKLYRETENTION -ge 0 ]] ; then # Delete old weekly backups while respecting the set rentention policy. NUM_OLD_FILES=$(find $BACKUPDIR/weekly -depth -not -newermt "$WEEKLYRETENTION week ago" -type f | wc -l) if [[ $NUM_OLD_FILES -gt 0 ]] ; then echo $timestamp - Deleting "$NUM_OLD_FILES" global setting backup file\(s\) older than "$WEEKLYRETENTION" week\(s\) old. find $BACKUPDIR/weekly -not -newermt "$WEEKLYRETENTION week ago" -type f -delete fi fi FILE="$BACKUPDIR/weekly/week.$W.$DATE" # Daily Backup elif [[ $DODAILY = "yes" ]] ; then echo $timestamp - Daily Backup of Databases echo # Delete old daily backups while respecting the set rentention policy. if [[ $DAILYRETENTION -ge 0 ]] ; then NUM_OLD_FILES=$(find $BACKUPDIR/daily -depth -not -newermt "$DAILYRETENTION days ago" -type f | wc -l) if [[ $NUM_OLD_FILES -gt 0 ]] ; then echo Deleting "$NUM_OLD_FILES" global setting backup file\(s\) made in previous weeks. find "$BACKUPDIR/daily" -not -newermt "$DAILYRETENTION days ago" -type f -delete fi fi # Below changed to remove .$DOW as this is not required FILE="$BACKUPDIR/daily/$DATE" # Hourly Backup elif [[ $DOHOURLY = "yes" ]] ; then echo $timestamp - Hourly Backup of Databases echo # Delete old hourly backups while respecting the set rentention policy. if [[ $HOURLYRETENTION -ge 0 ]] ; then NUM_OLD_FILES=$(find $BACKUPDIR/hourly -depth -not -newermt "$HOURLYRETENTION hour ago" -type f | wc -l) if [[ $NUM_OLD_FILES -gt 0 ]] ; then echo "$timestamp - Deleting $NUM_OLD_FILES global setting backup file\(s\) made in previous weeks." find $BACKUPDIR/hourly -not -newermt "$HOURLYRETENTION hour ago" -type f -delete fi fi FILE="$BACKUPDIR/hourly/$DATE.$DOW.$HOD" # convert timestamp to date: echo $TIMESTAMP | gawk '{print strftime("%c", $0)}' fi # FILE will not be set if no frequency is selected. if [[ -z "$FILE" ]] ; then echo "$timestamp - ERROR: No backup frequency was chosen." echo "$timestamp - Please set one of DOHOURLY,DODAILY,DOWEEKLY,DOMONTHLY to \"yes\"" exit 1 fi dbdump "$FILE" && compression "$FILE" echo "$timestamp - Backup completed $(date)" echo echo "Total disk space used for backup storage" echo echo "Size - Location " du -hs "$BACKUPDIR" echo # Run command when we're done if [ "$POSTBACKUP" ]; then echo "$timestamp - Postbackup command output." echo eval "$POSTBACKUP" echo fi # Clean up IO redirection if we plan not to deliver log via e-mail. [ ! "x$MAILCONTENT" == "xlog" ] && exec 1>&6 2>&7 6>&- 7>&- if [ -s "$LOGERR" ]; then eval "$SED" "/^connected/d" "$LOGERR" fi if [ "$MAILCONTENT" = "log" ]; then mail -s "Mongo Backup Log for $HOST - $DATE" "$MAILADDR" < "$LOGFILE" if [ -s "$LOGERR" ]; then cat "$LOGERR" mail -s "ERRORS REPORTED: Mongo Backup error Log for $HOST - $DATE" "$MAILADDR" < "$LOGERR" fi else if [ -s "$LOGERR" ]; then cat "$LOGFILE" echo echo "$timestamp - ###### WARNING ######" echo "$timestamp - STDERR written to during mongodump execution." echo "$timestamp - The backup probably succeeded, as mongodump sometimes writes to STDERR, but you may wish to scan the error log below:" cat "$LOGERR" else cat "$LOGFILE" fi fi # TODO: Would be nice to know if there were any *actual* errors in the $LOGERR STATUS=0 if [ -s "$LOGERR" ]; then STATUS=1 fi # Clean up Logfile rm -f "$LOGFILE" "$LOGERR" exit $STATUS

    One other thing I forgot to mention here is that you’ll need mailutils installed on your server for the mail option to work 🙂

  • mongorestore "E11000 duplicate key error collection"

    Solved Configure
    1
    1 Votes
    1 Posts
    674 Views
    No one has replied
  • Facebook goes offline 🤭

    General
    51
    26 Votes
    51 Posts
    3k Views

    @JAC yeah, I’ve never seen the appeal. I tried to get into LinkedIn a while back and then gave up because it’s just chock full of people who have far too much time on their hands telling anyone who will listen about how great they are and what they have achieved.

    Actions speak louder than words.