Skip to content
Sudonix

Mongo Completely Broken.

General
  • Sampo2910undefined

    Hi All,

    I have been running a Gaming Server and Site from home for many years. I have not really touched it in a while and switched on again today for some nostalgia and basically take a look.

    My site uses mongo to store the information and i had one db on there. I updated my system and all seems completely broken. I can no longer see the site. Sorry for the long code text but you can see from when I use sudo apt update then upgrade that it suggests to fix. But I get this.

    I am happy for someone to take dive in and take a look for the price of a coffee or two of course 🙂 I guess it may be something simple but I am far from the level of most of the guys here. I am wondering if my db and data is lost 😞 Any help much appreciated. Thank you in advance.

    smiley@fmjcod4:~$ sudo apt update
Hit:1 http://fi.archive.ubuntu.com/ubuntu focal InRelease
Hit:2 http://fi.archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:3 https://dl.google.com/linux/chrome/deb stable InRelease
Hit:4 http://fi.archive.ubuntu.com/ubuntu focal-backports InRelease
Hit:5 http://fi.archive.ubuntu.com/ubuntu focal-security InRelease
Hit:6 http://ppa.launchpad.net/maxmind/ppa/ubuntu focal InRelease
Hit:7 https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/8.0 InRelease
Hit:8 https://downloads.sourceforge.net/project/d-apt d-apt InRelease
Reading package lists... Done
Building dependency tree
Reading state information... Done
22 packages can be upgraded. Run 'apt list --upgradable' to see them.
smiley@fmjcod4:~$ sudo apt upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 mongodb-org-database : Depends: mongodb-org-server but it is not installed
                        Depends: mongodb-org-mongos but it is not installed
 mongodb-org-tools : Depends: mongodb-database-tools but it is not installed
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).
smiley@fmjcod4:~$ sudo apt --fix-broken install
Reading package lists... Done
Building dependency tree
Reading state information... Done
Correcting dependencies... Done
The following packages were automatically installed and are no longer required:
  libboost-program-options1.71.0 libgoogle-perftools4 libpcrecpp0v5 libphobos2-108 libphobos2-108:i386 libsnappy1v5 libtcmalloc-minimal4 libyaml-cpp0.6 mongo-tools mongodb-server-core
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
  mongodb-database-tools mongodb-org-mongos mongodb-org-server
The following NEW packages will be installed:
  mongodb-database-tools mongodb-org-mongos mongodb-org-server
0 upgraded, 3 newly installed, 0 to remove and 22 not upgraded.
6 not fully installed or removed.
Need to get 0 B/114 MB of archives.
After this operation, 386 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
(Reading database ... 198688 files and directories currently installed.)
Preparing to unpack .../mongodb-org-server_8.0.4_amd64.deb ...
Unpacking mongodb-org-server (8.0.4) ...
dpkg: error processing archive /var/cache/apt/archives/mongodb-org-server_8.0.4_amd64.deb (--unpack):
 trying to overwrite '/usr/bin/mongod', which is also in package mongodb-server-core 1:3.6.9+really3.6.8+90~g8e540c0b6d-0ubuntu5.3
dpkg-deb: error: paste subprocess was killed by signal (Broken pipe)
Preparing to unpack .../mongodb-org-mongos_8.0.4_amd64.deb ...
Unpacking mongodb-org-mongos (8.0.4) ...
dpkg: error processing archive /var/cache/apt/archives/mongodb-org-mongos_8.0.4_amd64.deb (--unpack):
 trying to overwrite '/usr/bin/mongos', which is also in package mongodb-server-core 1:3.6.9+really3.6.8+90~g8e540c0b6d-0ubuntu5.3
dpkg-deb: error: paste subprocess was killed by signal (Broken pipe)
Preparing to unpack .../mongodb-database-tools_100.10.0_amd64.deb ...
Unpacking mongodb-database-tools (100.10.0) ...
dpkg: error processing archive /var/cache/apt/archives/mongodb-database-tools_100.10.0_amd64.deb (--unpack):
 trying to overwrite '/usr/bin/bsondump', which is also in package mongo-tools 3.6.3-0ubuntu1
dpkg-deb: error: paste subprocess was killed by signal (Broken pipe)
Errors were encountered while processing:
 /var/cache/apt/archives/mongodb-org-server_8.0.4_amd64.deb
 /var/cache/apt/archives/mongodb-org-mongos_8.0.4_amd64.deb
 /var/cache/apt/archives/mongodb-database-tools_100.10.0_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)
    1
  • phenomlabundefined

    @Sampo2910 can you try stopping the mongodb service then try the update process again? Either use

    Upstart:

    sudo service mongod stop

    Or

    Sysvinit:

    sudo /etc/init.d/mongod stop

    Author of OGProxy, Swatch, Threaded, Card, and most of Sudonix's bugs…

    Sampo2910undefined 1 Reply
    0
  • Sampo2910undefined

    Just for the record after I reboot server I can see my page again. But same errors are there.

    I use the method for it form a friend. Its a bit old but still works.

    1
  • Sampo2910undefined

    @phenomlab

    smiley@fmjcod4:~$ sudo service mongod stop
Failed to stop mongod.service: Unit mongod.service not loaded.
smiley@fmjcod4:~$ sudo /etc/init.d/mongod stop
sudo: /etc/init.d/mongod: command not found


    smiley@fmjcod4:~$ sudo service mongod stop
Failed to stop mongod.service: Unit mongod.service not loaded.
smiley@fmjcod4:~$ sudo service mongodb stop
smiley@fmjcod4:~$ sudo service mongodb start
smiley@fmjcod4:~$ mongo

Command 'mongo' not found, but can be installed with:

sudo apt install mongodb-clients

smiley@fmjcod4:~$ sudo apt install mongodb-clients
Reading package lists... Done
Building dependency tree
Reading state information... Done
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 libssl-dev : Depends: libssl1.1 (= 1.1.1f-1ubuntu2.23) but 1.1.1f-1ubuntu2 is to be installed
 libssl1.1 : Breaks: libssl1.1:i386 (!= 1.1.1f-1ubuntu2) but 1.1.1f-1ubuntu2.23 is to be installed
 libssl1.1:i386 : Breaks: libssl1.1 (!= 1.1.1f-1ubuntu2.23) but 1.1.1f-1ubuntu2 is to be installed
 mongodb-org : Conflicts: mongodb-clients but 1:3.6.9+really3.6.8+90~g8e540c0b6d-0ubuntu5.3 is to be installed
 mongodb-org-database : Depends: mongodb-org-server but it is not going to be installed
                        Depends: mongodb-org-mongos but it is not going to be installed
                        Conflicts: mongodb-clients but 1:3.6.9+really3.6.8+90~g8e540c0b6d-0ubuntu5.3 is to be installed
 mongodb-org-database-tools-extra : Conflicts: mongodb-clients but 1:3.6.9+really3.6.8+90~g8e540c0b6d-0ubuntu5.3 is to be installed
 mongodb-org-shell : Conflicts: mongodb-clients but 1:3.6.9+really3.6.8+90~g8e540c0b6d-0ubuntu5.3 is to be installed
 mongodb-org-tools : Depends: mongodb-database-tools but it is not going to be installed
                     Conflicts: mongodb-clients but 1:3.6.9+really3.6.8+90~g8e540c0b6d-0ubuntu5.3 is to be installed
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).
    0
  • phenomlabundefined

    @Sampo2910 sorry. Try with mongodb instead of mongo

    Author of OGProxy, Swatch, Threaded, Card, and most of Sudonix's bugs…

    0
  • Sampo2910undefined

    @phenomlab

    I stopped the mongodb ok. The update goes ok but then to upgrade get the following.

    smiley@fmjcod4:~$ sudo apt upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 libssl-dev : Depends: libssl1.1 (= 1.1.1f-1ubuntu2.23) but 1.1.1f-1ubuntu2 is installed
 libssl1.1 : Breaks: libssl1.1:i386 (!= 1.1.1f-1ubuntu2) but 1.1.1f-1ubuntu2.23 is installed
 libssl1.1:i386 : Breaks: libssl1.1 (!= 1.1.1f-1ubuntu2.23) but 1.1.1f-1ubuntu2 is installed
 mongodb-org-database : Depends: mongodb-org-server but it is not installed
                        Depends: mongodb-org-mongos but it is not installed
 mongodb-org-tools : Depends: mongodb-database-tools but it is not installed
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).
smiley@fmjcod4:~$

    Trying to fix broken install does not work and gives this.

    smiley@fmjcod4:~$ sudo apt --fix-broken install
Reading package lists... Done
Building dependency tree
Reading state information... Done
Correcting dependencies... Done
The following packages were automatically installed and are no longer required:
  libboost-program-options1.71.0 libgoogle-perftools4 libpcrecpp0v5 libphobos2-108 libphobos2-108:i386 libsnappy1v5 libtcmalloc-minimal4
  libyaml-cpp0.6 mongo-tools mongodb-server-core
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
  libssl1.1 mongodb-database-tools mongodb-org-mongos mongodb-org-server
The following NEW packages will be installed:
  mongodb-database-tools mongodb-org-mongos mongodb-org-server
The following packages will be upgraded:
  libssl1.1
1 upgraded, 3 newly installed, 0 to remove and 22 not upgraded.
8 not fully installed or removed.
Need to get 116 MB of archives.
After this operation, 386 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://fi.archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl1.1 amd64 1.1.1f-1ubuntu2.23 [1,323 kB]
Get:2 https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/8.0/multiverse amd64 mongodb-org-server amd64 8.0.4 [38.6 MB]
Get:3 https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/8.0/multiverse amd64 mongodb-org-mongos amd64 8.0.4 [29.3 MB]
Get:4 https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/8.0/multiverse amd64 mongodb-database-tools amd64 100.10.0 [46.5 MB]
Fetched 116 MB in 3s (43.1 MB/s)
Preconfiguring packages ...
(Reading database ... 198688 files and directories currently installed.)
Preparing to unpack .../libssl1.1_1.1.1f-1ubuntu2.23_amd64.deb ...
Unpacking libssl1.1:amd64 (1.1.1f-1ubuntu2.23) over (1.1.1f-1ubuntu2) ...
dpkg: error processing archive /var/cache/apt/archives/libssl1.1_1.1.1f-1ubuntu2.23_amd64.deb (--unpack):
 trying to overwrite shared '/usr/share/doc/libssl1.1/changelog.Debian.gz', which is different from other instances of package libssl1.1:amd64
Preparing to unpack .../mongodb-org-server_8.0.4_amd64.deb ...
Unpacking mongodb-org-server (8.0.4) ...
dpkg: error processing archive /var/cache/apt/archives/mongodb-org-server_8.0.4_amd64.deb (--unpack):
 trying to overwrite '/usr/bin/mongod', which is also in package mongodb-server-core 1:3.6.9+really3.6.8+90~g8e540c0b6d-0ubuntu5.3
dpkg-deb: error: paste subprocess was killed by signal (Broken pipe)
Preparing to unpack .../mongodb-org-mongos_8.0.4_amd64.deb ...
Unpacking mongodb-org-mongos (8.0.4) ...
dpkg: error processing archive /var/cache/apt/archives/mongodb-org-mongos_8.0.4_amd64.deb (--unpack):
 trying to overwrite '/usr/bin/mongos', which is also in package mongodb-server-core 1:3.6.9+really3.6.8+90~g8e540c0b6d-0ubuntu5.3
dpkg-deb: error: paste subprocess was killed by signal (Broken pipe)
Preparing to unpack .../mongodb-database-tools_100.10.0_amd64.deb ...
Unpacking mongodb-database-tools (100.10.0) ...
dpkg: error processing archive /var/cache/apt/archives/mongodb-database-tools_100.10.0_amd64.deb (--unpack):
 trying to overwrite '/usr/bin/bsondump', which is also in package mongo-tools 3.6.3-0ubuntu1
No apport report written because MaxReports is reached already
                                                              dpkg-deb: error: paste subprocess was killed by signal (Broken pipe)
Errors were encountered while processing:
 /var/cache/apt/archives/libssl1.1_1.1.1f-1ubuntu2.23_amd64.deb
 /var/cache/apt/archives/mongodb-org-server_8.0.4_amd64.deb
 /var/cache/apt/archives/mongodb-org-mongos_8.0.4_amd64.deb
 /var/cache/apt/archives/mongodb-database-tools_100.10.0_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

    This is the same output with mongodb started or stopped.

    0
  • phenomlabundefined

    @Sampo2910 can you switch to root and try again? Seems like a permissions issue.

    Author of OGProxy, Swatch, Threaded, Card, and most of Sudonix's bugs…

    0
  • Sampo2910undefined

    Same output 😞

    Is it possible that there are two mongo instances installed. I think it tried to update. I was using an old version ( 3 something ) which was fine for my needs.

    0
  • phenomlabundefined

    @Sampo2910 quite possible, yes. What is the output of

    sudo dpkg -l | grep mongodb

    or

    sudo apt list --installed | grep mongodb

    Author of OGProxy, Swatch, Threaded, Card, and most of Sudonix's bugs…

    0
  • Sampo2910undefined

    @phenomlab Here is the output for both.

    smiley@fmjcod4:~$ sudo dpkg -l | grep mongodb
[sudo] password for smiley:
iU  mongodb-mongosh                            2.3.7                                         amd64        MongoDB Shell CLI REPL Package
iU  mongodb-org                                8.0.4                                         amd64        MongoDB open source document-oriented database system (metapackage)
iU  mongodb-org-database                       8.0.4                                         amd64        MongoDB open source document-oriented database system (metapackage)
iU  mongodb-org-database-tools-extra           8.0.4                                         amd64        Extra MongoDB database tools
iU  mongodb-org-shell                          8.0.4                                         amd64        MongoDB shell client
iU  mongodb-org-tools                          8.0.4                                         amd64        MongoDB tools
rc  mongodb-server                             1:3.6.9+really3.6.8+90~g8e540c0b6d-0ubuntu5.3 all          object/document-oriented database (managed server package)
ii  mongodb-server-core                        1:3.6.9+really3.6.8+90~g8e540c0b6d-0ubuntu5.3 amd64        object/document-oriented database (server binaries package)
smiley@fmjcod4:~$ ^C
smiley@fmjcod4:~$ sudo apt list --installed | grep mongodb

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

mongodb-mongosh/focal/mongodb-org/8.0,now 2.3.7 amd64 [installed,automatic]
mongodb-org-database-tools-extra/focal/mongodb-org/8.0,now 8.0.4 amd64 [installed,automatic]
mongodb-org-database/focal/mongodb-org/8.0,now 8.0.4 amd64 [installed,automatic]
mongodb-org-shell/focal/mongodb-org/8.0,now 8.0.4 amd64 [installed,automatic]
mongodb-org-tools/focal/mongodb-org/8.0,now 8.0.4 amd64 [installed,automatic]
mongodb-org/focal/mongodb-org/8.0,now 8.0.4 amd64 [installed]
mongodb-server-core/focal-updates,focal-security,now 1:3.6.9+really3.6.8+90~g8e540c0b6d-0ubuntu5.3 amd64 [installed,auto-removable]
    0
  • Sampo2910undefined

    It is running but I still cannot do any updates as mentioned.

    smiley@fmjcod4:~$ sudo service mongodb status
● mongodb.service - LSB: An object/document-oriented database
     Loaded: loaded (/etc/init.d/mongodb; generated)
     Active: active (running) since Tue 2024-12-31 02:13:22 EET; 1h 8min ago
       Docs: man:systemd-sysv-generator(8)
    Process: 14698 ExecStart=/etc/init.d/mongodb start (code=exited, status=0/SUCCESS)
      Tasks: 34 (limit: 16594)
     Memory: 149.6M
     CGroup: /system.slice/mongodb.service
             └─14716 /usr/bin/mongod --config /etc/mongodb.conf

Dec 31 02:13:21 fmjcod4 systemd[1]: Starting LSB: An object/document-oriented database...
Dec 31 02:13:21 fmjcod4 mongodb[14698]:  * Starting database mongodb
Dec 31 02:13:22 fmjcod4 mongodb[14698]:    ...done.
Dec 31 02:13:22 fmjcod4 systemd[1]: Started LSB: An object/document-oriented database.
    0
  • phenomlabundefined

    @Sampo2910 I can take a look at this today at some point. Can you PM me the login details?

    Author of OGProxy, Swatch, Threaded, Card, and most of Sudonix's bugs…

    Sampo2910undefined 1 Reply
    0
  • DownPWundefined

    Repo problem

    PW and WS Spirit alive

    phenomlabundefined 1 Reply
    👍
    0
  • Sampo2910undefined

    @phenomlab I will PM you .Thank you.

    0
  • phenomlabundefined

    @DownPW Seems not

    Hit:10 https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/8.0 InRelease

    Author of OGProxy, Swatch, Threaded, Card, and most of Sudonix's bugs…

    0

Related Topics
  • Madchatthewundefined

    Mongodb or Redis

    General
    29
    11 Votes
    29 Posts
    581 Views
    phenomlabundefined

    @Madchatthew still a great catch.

  • ahmedundefined

    Is my Mongodb installation correct?

    Solved General
    27
    13 Votes
    27 Posts
    3k Views
    ahmedundefined

    yuuuuu.png

  • ahmedundefined

    mongodb backup problem

    Moved Configure
    3
    1 Votes
    3 Posts
    257 Views
    phenomlabundefined

    You might want to also review this post

    https://sudonix.org/topic/389/mongodb-backup-script

  • veronikyaundefined

    mongodb replica set

    Configure
    11
    2 Votes
    11 Posts
    528 Views
    phenomlabundefined

    @veronikya said in mongodb replica set:

    The host’s local dns resolution is not configured. The problem of the host’s hosts being unable to be resolved in docker has been solved. Surprisingly
    Solution:
    Edit the /etc/resovel.conf file
    Add 127.0.0.53

    One immediate issue I can think of here is that editing resolv.conf directly is no longer supported and not recommended (because the changes do not survive a reboot) - unless you install the resolvconf package?

  • DownPWundefined

    Nodebb: failed to restore a mongo dump

    Solved Configure
    2
    1 Votes
    2 Posts
    221 Views
    DownPWundefined

    @phenomlab

    In fact I specified the sub rep and not the rep

    DON’T DO THIS:

    nodebb@nodebbpwclonedb:~/nodebb$ sudo mongorestore --username admin --password XXXXXXXXXXXXXX --nsInclude nodebb.objects --drop /home/nodebb/nodebb_DB_20230107/nodebb/

    BUT THIS :

    nodebb@nodebbpwclonedb:~/nodebb$ sudo mongorestore --username admin --password XXXXXXXXXXXXXX --nsInclude nodebb.objects --drop /home/nodebb/nodebb_DB_20230107/

    🙂

  • Sampo2910undefined

    Mongodb Authorisation.

    Solved General
    17
    2 Votes
    17 Posts
    769 Views
    phenomlabundefined

    @Sampo2910 I just saw this on the NodeBB community site and it seems very much in line with the issue you are experiencing. Worth a look I think

    https://community.nodebb.org/topic/16826/does-nodebb-work-with-mongodb-6-0

  • phenomlabundefined

    MongoDB backup script

    Moved Let's Build It
    4
    6 Votes
    4 Posts
    457 Views
    phenomlabundefined

    After playing with the original code in the first post, I’ve modified it a little to suit my needs - posting it here in it’s full glory in case it’s of use for anyone else. Obviously, you’ll need to change the important parts

    #!/bin/bash #set -eo pipefail # # MongoDB Backup Script # VER. 0.20 # More Info: http://github.com/micahwedemeyer/automongobackup # Note, this is a lobotomized port of AutoMySQLBackup # (http://sourceforge.net/projects/automysqlbackup/) for use with # MongoDB. # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # #===================================================================== #===================================================================== # Set the following variables to your system needs # (Detailed instructions below variables) #===================================================================== # Database name to specify a specific database only e.g. myawesomeapp # Unnecessary if backup all databases # DBNAME="" # Collections name list to include e.g. system.profile users # DBNAME is required # Unecessary if backup all collections # COLLECTIONS="" # Collections to exclude e.g. system.profile users # DBNAME is required # Unecessary if backup all collections # EXCLUDE_COLLECTIONS="" # Username to access the mongo server e.g. dbuser # Unnecessary if authentication is off DBUSERNAME="" # Password to access the mongo server e.g. password # Unnecessary if authentication is off DBPASSWORD="" # Database for authentication to the mongo server e.g. admin # Unnecessary if authentication is off DBAUTHDB="admin" # Host name (or IP address) of mongo server e.g localhost DBHOST="127.0.0.1" # Port that mongo is listening on DBPORT="27017" # Backup directory location e.g /backups BACKUPDIR="" # Mail setup # What would you like to be mailed to you? # - log : send only log file # - files : send log file and sql files as attachments (see docs) # - stdout : will simply output the log to the screen if run manually. # - quiet : Only send logs if an error occurs to the MAILADDR. MAILCONTENT="log" # Set the maximum allowed email size in k. (4000 = approx 5MB email [see docs]) export MAXATTSIZE="4000" # Email Address to send mail to? (user@domain.com) MAILADDR="" # ============================================================================ # === SCHEDULING AND RETENTION OPTIONS ( Read the doc's below for details )=== #============================================================================= # Do you want to do hourly backups? How long do you want to keep them? DOHOURLY="no" HOURLYRETENTION=24 # Do you want to do daily backups? How long do you want to keep them? DODAILY="yes" DAILYRETENTION=7 # Which day do you want weekly backups? (1 to 7 where 1 is Monday) DOWEEKLY="no" WEEKLYDAY=6 WEEKLYRETENTION=4 # Do you want monthly backups? How long do you want to keep them? DOMONTHLY="no" MONTHLYRETENTION=4 # ============================================================ # === ADVANCED OPTIONS ( Read the doc's below for details )=== #============================================================= # Choose Compression type. (gzip or bzip2) COMP="gzip" # Choose if the uncompressed folder should be deleted after compression has completed CLEANUP="yes" # Additionally keep a copy of the most recent backup in a seperate directory. LATEST="yes" # Make Hardlink not a copy LATESTLINK="yes" # Use oplog for point-in-time snapshotting. OPLOG="no" # Choose other Server if is Replica-Set Master REPLICAONSLAVE="no" # Allow DBUSERNAME without DBAUTHDB REQUIREDBAUTHDB="yes" # Maximum files of a single backup used by split - leave empty if no split required # MAXFILESIZE="" # Command to run before backups (uncomment to use) # PREBACKUP="" # Command run after backups (uncomment to use) # POSTBACKUP="" #===================================================================== # Options documentation #===================================================================== # Set DBUSERNAME and DBPASSWORD of a user that has at least SELECT permission # to ALL databases. # # Set the DBHOST option to the server you wish to backup, leave the # default to backup "this server".(to backup multiple servers make # copies of this file and set the options for that server) # # You can change the backup storage location from /backups to anything # you like by using the BACKUPDIR setting.. # # The MAILCONTENT and MAILADDR options and pretty self explanatory, use # these to have the backup log mailed to you at any email address or multiple # email addresses in a space seperated list. # # (If you set mail content to "log" you will require access to the "mail" program # on your server. If you set this to "files" you will have to have mutt installed # on your server. If you set it to "stdout" it will log to the screen if run from # the console or to the cron job owner if run through cron. If you set it to "quiet" # logs will only be mailed if there are errors reported. ) # # # Finally copy automongobackup.sh to anywhere on your server and make sure # to set executable permission. You can also copy the script to # /etc/cron.daily to have it execute automatically every night or simply # place a symlink in /etc/cron.daily to the file if you wish to keep it # somwhere else. # # NOTE: On Debian copy the file with no extention for it to be run # by cron e.g just name the file "automongobackup" # # Thats it.. # # # === Advanced options === # # To set the day of the week that you would like the weekly backup to happen # set the WEEKLYDAY setting, this can be a value from 1 to 7 where 1 is Monday, # The default is 6 which means that weekly backups are done on a Saturday. # # Use PREBACKUP and POSTBACKUP to specify Pre and Post backup commands # or scripts to perform tasks either before or after the backup process. # # #===================================================================== # Backup Rotation.. #===================================================================== # # Hourly backups are executed if DOHOURLY is set to "yes". # The number of hours backup copies to keep for each day (i.e. 'Monday', 'Tuesday', etc.) is set with DHOURLYRETENTION. # DHOURLYRETENTION=0 rotates hourly backups every day (i.e. only the most recent hourly copy is kept). -1 disables rotation. # # Daily backups are executed if DODAILY is set to "yes". # The number of daily backup copies to keep for each day (i.e. 'Monday', 'Tuesday', etc.) is set with DAILYRETENTION. # DAILYRETENTION=0 rotates daily backups every week (i.e. only the most recent daily copy is kept). -1 disables rotation. # # Weekly backups are executed if DOWEEKLY is set to "yes". # WEEKLYDAY [1-7] sets which day a weekly backup occurs when cron.daily scripts are run. # Rotate weekly copies after the number of weeks set by WEEKLYRETENTION. # WEEKLYRETENTION=0 rotates weekly backups every week. -1 disables rotation. # # Monthly backups are executed if DOMONTHLY is set to "yes". # Monthy backups occur on the first day of each month when cron.daily scripts are run. # Rotate monthly backups after the number of months set by MONTHLYRETENTION. # MONTHLYRETENTION=0 rotates monthly backups upon each execution. -1 disables rotation. # #===================================================================== # Please Note!! #===================================================================== # # I take no resposibility for any data loss or corruption when using # this script. # # This script will not help in the event of a hard drive crash. You # should copy your backups offline or to another PC for best protection. # # Happy backing up! # #===================================================================== # Restoring #===================================================================== # ??? # #===================================================================== # Change Log #===================================================================== # VER 0.11 - (2016-05-04) (author: Claudio Prato) # - Fixed bugs in select_secondary_member() with authdb enabled # - Fixed bugs in Compression function by removing the * symbol # - Added incremental backup feature # - Added option to select the collections to backup # # VER 0.10 - (2015-06-22) (author: Markus Graf) # - Added option to backup only one specific database # # VER 0.9 - (2011-10-28) (author: Joshua Keroes) # - Fixed bugs and improved logic in select_secondary_member() # - Fixed minor grammar issues and formatting in docs # # VER 0.8 - (2011-10-02) (author: Krzysztof Wilczynski) # - Added better support for selecting Secondary member in the # Replica Sets that can be used to take backups without bothering # busy Primary member too much. # # VER 0.7 - (2011-09-23) (author: Krzysztof Wilczynski) # - Added support for --journal dring taking backup # to enable journaling. # # VER 0.6 - (2011-09-15) (author: Krzysztof Wilczynski) # - Added support for --oplog during taking backup for # point-in-time snapshotting. # - Added filter for "mongodump" writing "connected to:" # on the standard error, which is not desirable. # # VER 0.5 - (2011-02-04) (author: Jan Doberstein) # - Added replicaset support (don't Backup on Master) # - Added Hard Support for 'latest' Copy # # VER 0.4 - (2010-10-26) # - Cleaned up warning message to make it clear that it can # usually be safely ignored # # VER 0.3 - (2010-06-11) # - Added the DBPORT parameter # - Changed USERNAME and PASSWORD to DBUSERNAME and DBPASSWORD # - Fixed some bugs with compression # # VER 0.2 - (2010-05-27) (author: Gregory Barchard) # - Added back the compression option for automatically creating # tgz or bz2 archives # - Added a cleanup option to optionally remove the database dump # after creating the archives # - Removed unnecessary path additions # # VER 0.1 - (2010-05-11) # - Initial Release # # VER 0.2 - (2015-09-10) # - Added configurable backup rentention options, even for # monthly backups. # #===================================================================== #===================================================================== #===================================================================== # # Should not need to be modified from here down!! # #===================================================================== #===================================================================== #===================================================================== shellout () { if [ -n "$1" ]; then echo "$1" exit 1 fi exit 0 } # External config - override default values set above for x in default sysconfig; do if [ -f "/etc/$x/automongobackup" ]; then # shellcheck source=/dev/null source /etc/$x/automongobackup fi done # Include extra config file if specified on commandline, e.g. for backuping several remote dbs from central server # shellcheck source=/dev/null [ ! -z "$1" ] && [ -f "$1" ] && source ${1} #===================================================================== PATH=/usr/local/bin:/usr/bin:/bin timestamp=$(date +%F-%T) DATE=$(date +%F-%H%M%S) # Datestamp e.g 2002-09-21 HOD=$(date +%s) # Current timestamp for PITR backup DOW=$(date +%A) # Day of the week e.g. Monday DNOW=$(date +%u) # Day number of the week 1 to 7 where 1 represents Monday DOM=$(date +%d) # Date of the Month e.g. 27 M=$(date +%B) # Month e.g January W=$(date +%V) # Week Number e.g 37 VER=0.11 # Version Number LOGFILE=$BACKUPDIR/$DBHOST-$(date +%F-%H%M%S).log # Logfile Name LOGERR=$BACKUPDIR/ERRORS_$DBHOST-$(date +%F-%H%M%S).log # Logfile Name OPT="" # OPT string for use with mongodump OPTSEC="" # OPT string for use with mongodump in select_secondary_member function QUERY="" # QUERY string for use with mongodump HOURLYQUERY="" # HOURLYQUERY string for use with mongodump # Do we need to use a username/password? if [ "$DBUSERNAME" ]; then OPT="$OPT --username=$DBUSERNAME --password=$DBPASSWORD" if [ "$REQUIREDBAUTHDB" = "yes" ]; then OPT="$OPT --authenticationDatabase=$DBAUTHDB" fi fi # Do we need to use a username/password for ReplicaSet Secondary Members Selection? if [ "$DBUSERNAME" ]; then OPTSEC="$OPTSEC --username=$DBUSERNAME --password=$DBPASSWORD" if [ "$REQUIREDBAUTHDB" = "yes" ]; then OPTSEC="$OPTSEC --authenticationDatabase=$DBAUTHDB" fi fi # Do we use oplog for point-in-time snapshotting? if [ "$OPLOG" = "yes" ] && [ -z "$DBNAME" ]; then OPT="$OPT --oplog" fi # Do we need to backup only a specific database? if [ "$DBNAME" ]; then OPT="$OPT -d $DBNAME" fi # Do we need to backup only a specific collections? if [ "$COLLECTIONS" ]; then for x in $COLLECTIONS; do OPT="$OPT --collection $x" done fi # Do we need to exclude collections? if [ "$EXCLUDE_COLLECTIONS" ]; then for x in $EXCLUDE_COLLECTIONS; do OPT="$OPT --excludeCollection $x" done fi # Do we use a filter for hourly point-in-time snapshotting? if [ "$DOHOURLY" == "yes" ]; then # getting PITR START timestamp # shellcheck disable=SC2012 [ "$COMP" = "gzip" ] && HOURLYQUERY=$(ls -t $BACKUPDIR/hourly | head -n 1 | cut -d '.' -f3) # setting the start timestamp to NOW for the first execution if [ -z "$HOURLYQUERY" ]; then QUERY="" else # limit the documents included in the output of mongodump # shellcheck disable=SC2016 QUERY='{ "ts" : { $gt : Timestamp('$HOURLYQUERY', 1) } }' fi fi # Create required directories mkdir -p $BACKUPDIR/{hourly,daily,weekly,monthly} || shellout 'failed to create directories' if [ "$LATEST" = "yes" ]; then rm -rf "$BACKUPDIR/latest" mkdir -p "$BACKUPDIR/latest" || shellout 'failed to create directory' fi # Do we use a filter for hourly point-in-time snapshotting? if [ "$DOHOURLY" == "yes" ]; then # getting PITR START timestamp # shellcheck disable=SC2012 [ "$COMP" = "gzip" ] && HOURLYQUERY=$(ls -t $BACKUPDIR/hourly | head -n 1 | cut -d '.' -f3) # setting the start timestamp to NOW for the first execution if [ -z "$HOURLYQUERY" ]; then QUERY="" else # limit the documents included in the output of mongodump # shellcheck disable=SC2016 QUERY='{ "ts" : { $gt : Timestamp('$HOURLYQUERY', 1) } }' fi fi # Check for correct sed usage if [ "$(uname -s)" = 'Darwin' ] || [ "$(uname -s)" = 'FreeBSD' ]; then SED="sed -i ''" else SED="sed -i" fi # IO redirection for logging. touch "$LOGFILE" exec 6>&1 # Link file descriptor #6 with stdout. # Saves stdout. exec > "$LOGFILE" # stdout replaced with file $LOGFILE. touch "$LOGERR" exec 7>&2 # Link file descriptor #7 with stderr. # Saves stderr. exec 2> "$LOGERR" # stderr replaced with file $LOGERR. # When a desire is to receive log via e-mail then we close stdout and stderr. [ "x$MAILCONTENT" == "xlog" ] && exec 6>&- 7>&- # Functions # Database dump function dbdump () { if [ -n "$QUERY" ]; then # filter for point-in-time snapshotting and if DOHOURLY=yes # shellcheck disable=SC2086 mongodump --quiet --host=$DBHOST:$DBPORT --out="$1" $OPT -q "$QUERY" MDUMPSTATUS=$? else # all others backups type # shellcheck disable=SC2086 mongodump --quiet --host=$DBHOST:$DBPORT --out="$1" $OPT MDUMPSTATUS=$? fi if [ $MDUMPSTATUS -ne 0 ]; then echo "$timestamp - ERROR: mongodump failed: $1" >&2 return 1 fi [ -e "$1" ] && return 0 echo "$timestamp - ERROR: mongodump failed to create dumpfile: $1" >&2 return 1 } # # Select first available Secondary member in the Replica Sets and show its # host name and port. # function select_secondary_member { # We will use indirect-reference hack to return variable from this function. local __return=$1 # Return list of with all replica set members # shellcheck disable=SC2086 members=( $(mongo --quiet --host $DBHOST:$DBPORT --eval 'rs.conf().members.forEach(function(x){ print(x.host) })' $OPTSEC ) ) # Check each replset member to see if it's a secondary and return it. if [ ${#members[@]} -gt 1 ]; then for member in "${members[@]}"; do is_secondary=$(mongo --quiet --host "$member" --eval 'rs.isMaster().secondary' $OPTSEC ) case "$is_secondary" in 'true') # First secondary wins ... secondary=$member break ;; 'false') # Skip particular member if it is a Primary. continue ;; *) # Skip irrelevant entries. Should not be any anyway ... continue ;; esac done fi if [ -n "$secondary" ]; then # Ugly hack to return value from a Bash function ... # shellcheck disable=SC2086 eval $__return="'$secondary'" fi } if [ -n "$MAXFILESIZE" ]; then write_file() { split --bytes "$MAXFILESIZE" --numeric-suffixes - "${1}-" } else write_file() { cat > "$1" } fi # Compression function plus latest copy compression () { SUFFIX="" dir=$(dirname "$1") file=$(basename "$1") if [ -n "$COMP" ]; then [ "$COMP" = "gzip" ] && SUFFIX=".tgz" [ "$COMP" = "bzip2" ] && SUFFIX=".tar.bz2" echo $timestamp - Tar and $COMP to "$file$SUFFIX" cd "$dir" || return 1 tar -cf - "$file" | $COMP --stdout | write_file "${file}${SUFFIX}" cd - >/dev/null || return 1 else echo "$timestamp - No compression option set, check advanced settings" fi if [ "$LATEST" = "yes" ]; then if [ "$LATESTLINK" = "yes" ];then COPY="ln" else COPY="cp" fi $COPY "$1$SUFFIX" "$BACKUPDIR/latest/" fi if [ "$CLEANUP" = "yes" ]; then echo $timestamp - Cleaning up folder at "$1" rm -rf "$1" fi return 0 } # Run command before we begin if [ "$PREBACKUP" ]; then echo "$timestamp - Prebackup command output." eval "$PREBACKUP" echo fi # Hostname for LOG information if [ "$DBHOST" = "localhost" ] || [ "$DBHOST" = "127.0.0.1" ]; then HOST=$(hostname) if [ "$SOCKET" ]; then OPT="$OPT --socket=$SOCKET" fi else HOST=$DBHOST fi # Try to select an available secondary for the backup or fallback to DBHOST. if [ "x${REPLICAONSLAVE}" == "xyes" ]; then # Return value via indirect-reference hack ... select_secondary_member secondary if [ -n "$secondary" ]; then DBHOST=${secondary%%:*} DBPORT=${secondary##*:} else SECONDARY_WARNING="WARNING: No suitable Secondary found in the Replica Sets. Falling back to ${DBHOST}." fi fi if [ ! -z "$SECONDARY_WARNING" ]; then echo echo "$SECONDARY_WARNING" fi echo "$timestamp - Database Server - $HOST on $DBHOST" echo "$timestamp - Backup started $(date)" # Monthly Full Backup of all Databases if [[ $DOM = "01" ]] && [[ $DOMONTHLY = "yes" ]]; then echo "$timestamp - Monthly Full Backup" echo # Delete old monthly backups while respecting the set rentention policy. if [[ $MONTHLYRETENTION -ge 0 ]] ; then NUM_OLD_FILES=$(find $BACKUPDIR/monthly -depth -not -newermt "$MONTHLYRETENTION month ago" -type f | wc -l) if [[ $NUM_OLD_FILES -gt 0 ]] ; then echo $timestamp - Deleting "$NUM_OLD_FILES" global setting backup file\(s\) older than "$MONTHLYRETENTION" month\(s\) old. find $BACKUPDIR/monthly -not -newermt "$MONTHLYRETENTION month ago" -type f -delete fi fi FILE="$BACKUPDIR/monthly/$DATE.$M" # Weekly Backup elif [[ "$DNOW" = "$WEEKLYDAY" ]] && [[ "$DOWEEKLY" = "yes" ]] ; then echo Weekly Backup echo if [[ $WEEKLYRETENTION -ge 0 ]] ; then # Delete old weekly backups while respecting the set rentention policy. NUM_OLD_FILES=$(find $BACKUPDIR/weekly -depth -not -newermt "$WEEKLYRETENTION week ago" -type f | wc -l) if [[ $NUM_OLD_FILES -gt 0 ]] ; then echo $timestamp - Deleting "$NUM_OLD_FILES" global setting backup file\(s\) older than "$WEEKLYRETENTION" week\(s\) old. find $BACKUPDIR/weekly -not -newermt "$WEEKLYRETENTION week ago" -type f -delete fi fi FILE="$BACKUPDIR/weekly/week.$W.$DATE" # Daily Backup elif [[ $DODAILY = "yes" ]] ; then echo $timestamp - Daily Backup of Databases echo # Delete old daily backups while respecting the set rentention policy. if [[ $DAILYRETENTION -ge 0 ]] ; then NUM_OLD_FILES=$(find $BACKUPDIR/daily -depth -not -newermt "$DAILYRETENTION days ago" -type f | wc -l) if [[ $NUM_OLD_FILES -gt 0 ]] ; then echo Deleting "$NUM_OLD_FILES" global setting backup file\(s\) made in previous weeks. find "$BACKUPDIR/daily" -not -newermt "$DAILYRETENTION days ago" -type f -delete fi fi # Below changed to remove .$DOW as this is not required FILE="$BACKUPDIR/daily/$DATE" # Hourly Backup elif [[ $DOHOURLY = "yes" ]] ; then echo $timestamp - Hourly Backup of Databases echo # Delete old hourly backups while respecting the set rentention policy. if [[ $HOURLYRETENTION -ge 0 ]] ; then NUM_OLD_FILES=$(find $BACKUPDIR/hourly -depth -not -newermt "$HOURLYRETENTION hour ago" -type f | wc -l) if [[ $NUM_OLD_FILES -gt 0 ]] ; then echo "$timestamp - Deleting $NUM_OLD_FILES global setting backup file\(s\) made in previous weeks." find $BACKUPDIR/hourly -not -newermt "$HOURLYRETENTION hour ago" -type f -delete fi fi FILE="$BACKUPDIR/hourly/$DATE.$DOW.$HOD" # convert timestamp to date: echo $TIMESTAMP | gawk '{print strftime("%c", $0)}' fi # FILE will not be set if no frequency is selected. if [[ -z "$FILE" ]] ; then echo "$timestamp - ERROR: No backup frequency was chosen." echo "$timestamp - Please set one of DOHOURLY,DODAILY,DOWEEKLY,DOMONTHLY to \"yes\"" exit 1 fi dbdump "$FILE" && compression "$FILE" echo "$timestamp - Backup completed $(date)" echo echo "Total disk space used for backup storage" echo echo "Size - Location " du -hs "$BACKUPDIR" echo # Run command when we're done if [ "$POSTBACKUP" ]; then echo "$timestamp - Postbackup command output." echo eval "$POSTBACKUP" echo fi # Clean up IO redirection if we plan not to deliver log via e-mail. [ ! "x$MAILCONTENT" == "xlog" ] && exec 1>&6 2>&7 6>&- 7>&- if [ -s "$LOGERR" ]; then eval "$SED" "/^connected/d" "$LOGERR" fi if [ "$MAILCONTENT" = "log" ]; then mail -s "Mongo Backup Log for $HOST - $DATE" "$MAILADDR" < "$LOGFILE" if [ -s "$LOGERR" ]; then cat "$LOGERR" mail -s "ERRORS REPORTED: Mongo Backup error Log for $HOST - $DATE" "$MAILADDR" < "$LOGERR" fi else if [ -s "$LOGERR" ]; then cat "$LOGFILE" echo echo "$timestamp - ###### WARNING ######" echo "$timestamp - STDERR written to during mongodump execution." echo "$timestamp - The backup probably succeeded, as mongodump sometimes writes to STDERR, but you may wish to scan the error log below:" cat "$LOGERR" else cat "$LOGFILE" fi fi # TODO: Would be nice to know if there were any *actual* errors in the $LOGERR STATUS=0 if [ -s "$LOGERR" ]; then STATUS=1 fi # Clean up Logfile rm -f "$LOGFILE" "$LOGERR" exit $STATUS

    One other thing I forgot to mention here is that you’ll need mailutils installed on your server for the mail option to work 🙂

  • phenomlabundefined

    mongorestore "E11000 duplicate key error collection"

    Solved Configure
    1
    1 Votes
    1 Posts
    801 Views
    No one has replied