Over the last few weeks, users of MongoDB all over the world have managed to employ what one of my co-workers calls a "Foot-Gun" - that's when you shoot yourself in the foot, needlessly and usually carelessly causing yourself very much pain.

I've worked with databases for more years than I care to think about - open source, closed source, big and well known, small and obscure, OLTP and OLAP.  I've been on the vendor side, and I've been on the customer side, as a developer and as a DBA.

In each of them there were ways to employ the foot-gun and depending on when, how and how many times you shot yourself you might recover completely, or you might limp along for a long time painfully trying to recreate the data from backups, replay logs and anything else you could get your hands on.

In a perfect world, a database would have a solid trigger lock on the foot-gun that requires bio-recognition, two-factor authentication and some other magical incantation before allowing you to mess with its innards, but we live in the real world where 'sudo' allows an OS user run 'rm' on arbitrary set of files, and where rushed and overworked DBAs and developers sometimes try to fix things "ad hoc" in the heat of a fire drill (which is when the worst self-mutilations usually occur, IME).

The most I realistically hope for is that we (as database makers/vendors/champions) don't load the gun, take the safety off, and place it within easy reach of every database and OS user.  When we do that, it's our fault even when the user picked up the gun and shot themselves in the foot. Twice.

I've debated whether or not it's a good idea to document exact failings that led to such painful outcomes.  On one hand, schadenfreude may be an enjoyable pastime; on the other hand, I don't think it ever helped anyone in a constructive manner, except when it included the detailed explanation of either how to avoid such circumstances, or how to recover with least amount of long-term damage.

How about it readers?  Have you shot yourself with the foot-gun?  Do you think you would have been able to avoid it had you read about someone else doing the exact same thing you wished you hadn't done?  Or was in caused in part by reading on the internet "how to fix" something and getting it wrong?

P.S. One of the best footgun metaphors I've ever seen is by Tom \"spot\" Calloway on Fedora developers mailing list where he described always logging in as root like this:    

         "What you're doing is analogous to using a loaded shotgun as a golf club, and what you're 
           suggesting is that we take the safety off, because it interferes with your golf game."

In an early post I mention that the reports of demise of MongoDB DBA have been greatly exaggerated, in my opinion.  NoSQL does not mean "no DBA".  If anyone tries to convince you otherwise, they probably have something to sell you.

This does not mean that you have a team or even a person who has the title "DBA" - however, if you have a database, whether it's relational, or non-relational, then someone has the role of "DBA" - and if they don't know that they do, then a whole bunch of things aren't being done or thought about before problems happen.

Over the last couple of weeks, I've worked with users who ran into major problems, some of which could have been prevented if they had better clarity around "DBA" role.  In all but one of those cases, they were able to recover successfully partly because they had recent backups.  Some of the tasks owned by the DBA role were happening, even if others were not.

In the remaining case, it turned out the most recent backup was more than three months old.  There is no excuse for that, unless you can easily regenerate all of your data!

Luckily, we were able to recover the lost data for this user (straight from the free list in the database files, but you really cannot count of that being possible - they got very lucky in that regard) but I'm willing to bet that they will start keeping track of whose job it is to make sure that the database is backed up.

Sometimes people forget that to measure performance you need to measure throughput (how many requests are you servicing in parallel) and latency (how long is it taking to service each request).  

It boggles my mind when someone runs a single threaded test and sends work to the database serially and then complains about bad throughput.  In order to measure throughput you have to throw as much work at the database server as you can get to it, and usually it means running many clients or multi-threaded clients.

Recently on the MongoDB Google Group someone asked if adding more shards to their cluster would make a particular MapReduce job faster.   Here is how I answered them:

Make sense?

MongoDB Replica Sets are an awesome included feature that gives you High Availability with automatic failover via a consensus election.  Having been a pager-carrying DBA in my previous life, I can tell you that if failover involves a human (usually me) being woken up in the middle of the night to promote a secondary to be the new master, I do not consider that "high" availability at all!   Working in the middle of the night is when I make all the errors that catch up with me weeks or months later, so I'd rather the system "do the right thing", and replica sets do just that.

Of course you also get extra copies of your data (for Disaster Recovery or just for compliance reasons) and other benefits, like ability to do maintenance without scheduling downtime for the database, but what if you don't need any of that?

What if you just want to have an extra copy of your data, but you don't want to have high availability?

I occasionally want to have that myself, when I need to test replication of data but don't want to deal with elections, arbiters or for whatever other reason want to pretend that my "primary" is basically a stand-alone node, it just happens to be replicating to another node (that might even be running on the same laptop).

In MongoDB the replica set architecture is set up to prevent two primaries from existing at the same time (this allows the data to stay consistent and avoids needing to conflict-resolve in cases of network partition) - consensus elections where a node must receive >50% of the votes to assume a PRIMARY role guarantees that two nodes cannot be elected at the same time.  But that also means that if you have an even number of nodes (like two) you must get both votes to become or stay a primary.

In order to avoid losing the primary whenever I happen to take down my secondary node, I adjust the replica set configuration to make sure that only one node gets to participate in elections.  Having only one vote means that majority is one, and my intended primary can "unanimously" elect itself every time - as long as it's up!

Here is the configuration:

The "priority" field says whether a node can be elected as a primary - zero means not eligible.  Other numbers would influence the nodes relative standing against other equally eligible nodes during an election;  "hidden" means the node's presence won't be revealed to clients/drivers that are connecting to the replica set as a whole; "votes" expresses whether the node gets a vote in the election - note that whether a node can vote and whether it can become a primary are two separate settings!

This means that if I lose the primary node, the secondary will remain as a secondary since it cannot elect itself - and I won't have high availability, and that's exactly what I was trying to achieve here.

You wouldn't think this happens very often, but then you would be surprised...

This week someone's sharded MongoDB cluster was running with one config server instead of three (you are "required" to have three but for testing purposes the system "allows" you to run with one with a warning).   Wouldn't you know it, somehow the database directory got deleted.  Oops.

Last week, someone was asking how they can recover from accidentally deleted files on a node which was running stand-alone (not replica set!)

The week before, someone accidentally let loose a script which deleted a bunch of database files on nodes (primaries and secondaries were affected, making it impossible to "just sync from unaffected node").

Here's a little trick you might keep in the back of your mind, in case something like this is needed to save your bacon some day, when someone deletes some database files that you were in the process of actually using!

What I say applies to Linux because Windows is a little more pedantic about letting users mess around with files that other users or processes happen to have open.  But in Linux, I can have a file open, and you can delete it.  If I'm in the editor it might give me a message like "Hey, this file seems to have changed on disk, what do you want to do - load this new thing?  Save your work on top of it?  Something else?"

If you know a little about MongoDB internals, you know that each database has its own set of files inside the "dbpath" - by default "/data/db" directory.  If your database is called "mydb" then the files will look something like:

As you can see, there is a "namespace" file, and two "data" files, each double the size of the previous one (up to 2GB) and there is always one more preallocated than the one you are currently writing to.  I just created this database, so I've got a little bit of data in mydb.0 and just lots of zeros in mydb.1.  Let's change that.

Now that I've got some useful data (or at least *some* data) I'm going to simulate some idiot "cleaning up" and let's say he deletes all my files that are bigger than 100MB.  Hey, we know that sort of thing happens...

What a jerk!!!  But notice that we can still see the files "out there" even though we can't see them in the directory listing...  And when we get back to the database, we see the strangest thing - our fileSize dropped from 2239MB to 64MB, but our data and index counts haven't changed.   In fact, if we play around, we will find that it's not noticeable in a running 'mongod' that the physical files are not on disk!

So let's take advantage of that.  There are many ways to rebuild the database files if they are still memory mapped, the simplest one would be the database command "repairDatabase()"

We did it!   We took advantage of the fact that 'mongod' was still running, we connected to it and asked it to run a command which rewrites all of the records in all collections of a db anew (and then drops the old database, and its old files).

Please promise me, that if you are about to run out and try this out yourself that you won't try it on a live production database.  Sure, you can restore the files with db.repairDatabase() if mongod is still running, but do you really want to run that experiment and find out that in the middle of it the power went out and you lost your files anyway?  (Of course you didn't, because you run a replica set, right?  Right?)

Everyone knows that MongoDB is incredibly easy to get started with.

Most people also agree that MongoDB is very easy to prototype with.  I'm talking especially about the kind of prototyping where you don't really know what the stress points of your application are going to be, what data will be most critical to access fast, what sort of questions you will need to ask of the data after the fact...  In other words, exactly the situation you're in when you first start developing any new application.

MongoDB is easy to install, easy to use with your choice of development languages and it has some wonderful operational features.  Okay, we already know all this, so what is this trap that I am talking about?

I've seen time and time again what happens to a super-fast little MongoDB backed app as the demand on it suddenly skyrockets: if there was no performance/load testing before launch, it's likely that the bottlenecks are a surprise and it's not always clear what they are, where they are, and how to "fix" them.

Here is where people can fall into one of two traps:
  
1. assume that their application load is not a good fit for MongoDB at scale.  
2. assume that MongoDB can handle this if they can figure out how to "tune" it properly.

"Wait a minute," I hear you say.  "Isn't it going to be the case that one of those things is true?"

Absolutely true.  It's possible that the requirements of the application at scale are not a good fit for a document database.  It's also possible that it is the use of MongoDB (schema design, indexing, hardware configuration) that was not done with high scale in mind, and MongoDB is perfectly suited to the workload being thrown at it.

"The trap" is when the assumption about which scenario you're in and reality don't match:

1. Scenario: Assumption is made that application is not a good fit for MongoDB but the application load is actually a perfect fit for MongoDB but poor schema design or bad indexing strategy or suboptimal topology or hardware choices are the cause of bad performance.  
    Result: the team will spend tremendous effort architecting a new system to store their data when they could have improved their MongoDB performance by several orders of magnitude by fixing their design or hardware or cluster topology.

Here are some examples of this:
- There is a missing index.  Solution: you don't fix the slow query by moving the task into a different system, you simply add the missing index.
- Indexes created are not best for the queries/updates running on the system, they take up RAM without providing much of a benefit.  Solution: review and fix your indexes.
- The schema split a number of things into separate collections that should be stored together and every request for the object takes multiple accesses to the database instead of just one.  Solution: reconsider/redesign parts of your schema.
- You have multiple shards but every request is sent to every shard, increasing the overall number of requests that the system must handle.  Sharding scales best when each shard is only required to do work on its portion of the dataset, not the entire dataset.
- You have system that does heavy writes but you chose extremely slow storage system.  Solution: get faster disks (after making sure that your writes aren't unnecessarily inefficient).
- You've made some inappropriate assumptions about how various parts of the system will benefit your use case, or followed outdated or flat-out wrong advice about how to use MongoDB.  Solution: examine your assumptions that don't seem to be holding up, and remove any inefficiency that was created by following the incorrect assumption.

A variant of these scenarios is when an assumption is made about how MongoDB will work with the application and then time is wasted trying to "tune" the wrong thing or worse, "pre"-optimizations are made to cater to some rumored limitation of MongoDB and it's the "optimization" that ends up killing the performance, not the original thing that it was meant to "correct".

2. Scenario: The application load was a terrible fit for MongoDB but for whatever reason an assumption is made that MongoDB will be able to handle it, if only the right "tuning" is made.  
    Result:   The team will spend tremendous effort trying to improve performance of a square peg in a round hole, instead of finding a round peg.

Examples of this include any scenario where you find yourself implementing more database work in your  application than you are asking the database to do for you.   Some examples out in the real world include situations where after migrating from MongoDB to another datastore, the application ended up having a lot less code - that tends to tell me that MongoDB was a poor choice from the start.   

To be honest, I've seen a lot more examples of 1. than 2.  Because MongoDB is quite flexible, it can be a good fit for an extremely wide range of application needs, but if no thought is given to proper schema, indexes and cluster configuration to serve those needs, there is no end to the number of ways that it can fall short of your expectations.

The worst of them all is the assumption that because MongoDB was so easy to install and so easy to get started with (not to mention so fast when it was being run with just some test data) that it will somehow tune itself at scale, without the developer having to give any thought to it.  It would be wonderful if that were true, but at the end of the day, MongoDB is a database, and there is no magic pixie dust that you can sprinkle on it to say "just go faster" - it is my humble opinion, that reports of death of the role of MongoDB DBA have been greatly exaggerated.