You wouldn't think this happens very often, but then you would be surprised...
This week someone's sharded MongoDB cluster was running with one config server instead of three (you are "required" to have three but for testing purposes the system "allows" you to run with one with a warning). Wouldn't you know it, somehow the database directory got deleted. Oops.
Last week, someone was asking how they can recover from accidentally deleted files on a node which was running stand-alone (not replica set!)
The week before, someone accidentally let loose a script which deleted a bunch of database files on nodes (primaries and secondaries were affected, making it impossible to "just sync from unaffected node").
Here's a little trick you might keep in the back of your mind, in case something like this is needed to save your bacon some day, when someone deletes some database files that you were in the process of actually using!
What I say applies to Linux because Windows is a little more pedantic about letting users mess around with files that other users or processes happen to have open. But in Linux, I can have a file open, and you can delete it. If I'm in the editor it might give me a message like "Hey, this file seems to have changed on disk, what do you want to do - load this new thing? Save your work on top of it? Something else?"
If you know a little about MongoDB internals, you know that each database has its own set of files inside the "dbpath" - by default "/data/db" directory. If your database is called "mydb" then the files will look something like:
This week someone's sharded MongoDB cluster was running with one config server instead of three (you are "required" to have three but for testing purposes the system "allows" you to run with one with a warning). Wouldn't you know it, somehow the database directory got deleted. Oops.
Last week, someone was asking how they can recover from accidentally deleted files on a node which was running stand-alone (not replica set!)
The week before, someone accidentally let loose a script which deleted a bunch of database files on nodes (primaries and secondaries were affected, making it impossible to "just sync from unaffected node").
Here's a little trick you might keep in the back of your mind, in case something like this is needed to save your bacon some day, when someone deletes some database files that you were in the process of actually using!
What I say applies to Linux because Windows is a little more pedantic about letting users mess around with files that other users or processes happen to have open. But in Linux, I can have a file open, and you can delete it. If I'm in the editor it might give me a message like "Hey, this file seems to have changed on disk, what do you want to do - load this new thing? Save your work on top of it? Something else?"
If you know a little about MongoDB internals, you know that each database has its own set of files inside the "dbpath" - by default "/data/db" directory. If your database is called "mydb" then the files will look something like:
As you can see, there is a "namespace" file, and two "data" files, each double the size of the previous one (up to 2GB) and there is always one more preallocated than the one you are currently writing to. I just created this database, so I've got a little bit of data in mydb.0 and just lots of zeros in mydb.1. Let's change that.
Now that I've got some useful data (or at least *some* data) I'm going to simulate some idiot "cleaning up" and let's say he deletes all my files that are bigger than 100MB. Hey, we know that sort of thing happens...
What a jerk!!! But notice that we can still see the files "out there" even though we can't see them in the directory listing... And when we get back to the database, we see the strangest thing - our fileSize dropped from 2239MB to 64MB, but our data and index counts haven't changed. In fact, if we play around, we will find that it's not noticeable in a running 'mongod' that the physical files are not on disk!
So let's take advantage of that. There are many ways to rebuild the database files if they are still memory mapped, the simplest one would be the database command "repairDatabase()"
We did it! We took advantage of the fact that 'mongod' was still running, we connected to it and asked it to run a command which rewrites all of the records in all collections of a db anew (and then drops the old database, and its old files).
Please promise me, that if you are about to run out and try this out yourself that you won't try it on a live production database. Sure, you can restore the files with db.repairDatabase() if mongod is still running, but do you really want to run that experiment and find out that in the middle of it the power went out and you lost your files anyway? (Of course you didn't, because you run a replica set, right? Right?)
RSS Feed
