The EXPERT’s VOIce ® in Open Source The Definitive Guide to MongoDB The NoSQL Database for Cloud and Desktop Computing...

Author: Peter Membrey | Wouter Thielen | Eelco Plugge | Tim Hawkins

64 downloads 870 Views 2MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!

Report copyright / DMCA form

DOWNLOAD PDF

$document[Title]

"; print "By $author on $date"; print "

$document[Message]

"; // Show clickable link to view the comments when found, and count them $postid = new MongoId($document['_id']); if(isset($document["Comments"])) { $count = count($document["Comments"]); } else { $count = 0; } $count = count($document["Comments"]); print "View Comments ($count)"; }

170

CHAPTER 8 ■ CREATING A BLOG APPLICATION WITH THE PHP DRIVER

That should look a little better. At this point, you should be able to see the last 10 items added to the posts collection, including the date and name of the person who has added it.

Paging with PHP and MongoDB Now assume more than 10 posts have been added. Obviously, the first posts added will no longer be displayed (note the sorting happens on _id in descending order), so you will need to make sure that older posts remain visible in case you want to find these later. The skip() function makes it easy to do this. Begin by adding a check to the PHP page that will determine whether a certain variable called $page has been set. If $page has been set, then that variable will be used to define the offset; in other words, it will specify how many posts should be skipped before you start requesting posts and printing them out. After all, if you know which page a document is on, and the limit per page has been set to 10, then it’s a simple matter to determine how many records you need to skip. For example, assume you are on page 3. A limit of 10 means that you want to skip the first 20 posts, and retrieve the 10 posts (or less if no more are found) immediately prior to the skipped posts. If the $page variable has not been set, however, you can force it to carry the value of 1, so it will display the latest 10 posts added by default. To do this, add the following code block either just under or just above your currently defined variables. The following example places this code just under them: // Set up some PHP code to check if the $page is set and if not, set $page to '1'. if(isset($_GET['page'])){ $page = $_GET['page']; } else { $page = 1; } Next, you’ll need to define one more variable to specify the offset, and two more variables that can either increase or decrease the $page value. You can place these directly under the if/else statement shown previously: // Let's define some more variables for paging $offset = ($page -1) * $limit; $nextpage = ($page +1); $prevpage = ($page -1); We’re almost finished. The $offset parameter has been specified (the page number minus one times the limit, 10), so you can pass this integer on to the skip() function. The previously defined $cursor value was derived like this: $cursor = $c->$db->$col_posts->find()->limit($limit)->sort(array('_id'=>-1)); You need to update it so it looks like this: $cursor = $c->$db->$col_posts->find()->skip($offset)->limit($limit)->sort(array('_id'=>-1)); The final step is to add two hyperlinks at the bottom of your page. These links let you navigate either one page back, or one page forward, whichever is applicable. Note that the links will not be displayed if the number of posts found can be displayed on a single page: // Perform a count on all posts within the collection and add links based upon the outcome $posts = $c->$db->$col_posts->find()->count(); if ($page > 1) { print "Previous Page";

171

CHAPTER 8 ■ CREATING A BLOG APPLICATION WITH THE PHP DRIVER

if ($page * $limit < $posts) { print "Next page"; } } else { if ($page * $limit < $posts) { print "Next page"; } } ?> That’s it for paging. In the next section, you’ll learn how to use another paging method that relies on the $slice operator for the comments array. For now, you are done with this page, and you can save it as posts.php.

■ Tip Don’t worry if you lose track about where you are in the example. You can use the code listing at the end of this chapter to review the entire code for the project.

Looking at a Single Post So far you’ve gotten a basic overview of all posts. Next, you will learn how to view a single post instead. Generally speaking, you want this to work in the same way that viewing multiple documents does. One difference is that viewing a single document also lets you see comments associated with that post (if any), as well as a form for adding any new comments. Let’s start off with a new document called view.php. You start the document with a number of variables that define the connection details that you’ll use throughout this page, as in the following example:
// // // //

Used This This This

to is is is

connect to the db the name of the database the name of the authors collection the name of the posts collection

Now that you’ve defined your variables, you need to add an if statement that checks whether the post ID has been specified in the URL (e.g., view.php?id=xxx). If so, you need to perform another check to see whether any page variable has been specified for paging the comments. If the paging variable $_GET[p] hasn’t been specified, then it will be set to 1, assuming this is the default page you’d want to start at for adding any comments: if (isset($_GET['id'])) { if (isset($_GET['p'])) { $commentpage = $_GET['p']; } else {

172

CHAPTER 8 ■ CREATING A BLOG APPLICATION WITH THE PHP DRIVER

$commentpage = 1; } You probably noticed that this code block isn’t finished. At this stage, it will do nothing useful, unless you enjoy observing the error messages.

Specifying Additional Variables Let’s continue building on this code block. Begin by specifying a number of additional variables needed for the paging: // Add some more variables for paging the comments $id = new MongoId($_GET['id']); $limit = 5; // $offset = ($commentpage -1) * $limit; // $nextpage = ($commentpage +1); // $prevpage = ($commentpage -1); //

Used This Used Used

to is to to

define the max no. of comments our comments offset increasing the commentspage decreasing the commentspage

Next, you need to specify the query, the $slice operator, and the operator’s parameters. The query is rather straightforward: you specify that the only post that you would like to see returned must have the _id field set to whatever the $id variable is set to (this variable was specified earlier in the URL of the page). Because the _id is unique, you can be quite certain that the resulting document is the correct one. Using the $slice operator enables you to tell the database to query for only a small amount of items in the nested comments, as well as to skip a certain number of them depending on the page set. You use the $offset parameter to specify the number of items to skip, and you use the $limit parameter to specify the number of items to show (i.e., limit): // Define the query parameters that we want to use $query = array('_id' => $id); $cond = (object)array('Comments' => array('$slice' => array($offset, $limit) ) ); You’ve specified the query parameters, so now you can perform the search operation itself. Of course, you’ll use the findOne() function to accomplish this. This code looks simple now that you’ve specified the query parameters: // Connect to the database, and perform the findOne $document = $c->$db->$col_posts->findOne($query, $cond); Next, you use DBRef to retrieve the author’s username. You can also print the document’s contents in the $document array from the previously executed findOne() function: // Now get the author's name via MongoDBRef, and translate the date $ref = $c->$db->getDBRef($document["Author"]); $author = $ref["Name"]; $date = date('M d, Y @ h:i', $document["Date"]->sec); // ..and print all the keys and values out print "

$document[Title]

"; print "By $author on $date"; print "

$document[Message]

"; You’re halfway there, but so far the page wouldn’t look much different to what you’d see on the previously created page. The next step is to add the code that enables you to view and add paging for the comments (assuming there are any). Adding another if statement can help you accomplish this: you simply tell it to display any comments that may exist. Note that at this stage the code block is not yet

173

CHAPTER 8 ■ CREATING A BLOG APPLICATION WITH THE PHP DRIVER

finished so it will not be closed off yet with a closing bracket at the end; instead this is done in the next code block. Here is the code snippet: // If there are any comments, then show them here if (isset($document['Comments'])) { // Show a header to announce the comments print "

Comments

"; // Set each comment that gets found in the $document array to $comm foreach ($document['Comments'] as $comm) { // For each of these, print the name and comment print "$comm[Name]
"; print "$comm[Comment]
"; }

Viewing and Adding Comments The next step is to add hyperlinks so that the comments can be navigated. At the beginning of the first PHP page, you specified a limit for these comments by setting the $limit parameter to 5 (see the “Specifying Additional Variables” section for the full context). You also defined the offset using the $p variable, which is set to 1 by default, unless otherwise modified using the hyperlinks that will be added shortly. To calculate the number of comment pages to list, you will first need to perform a count of the comments. Based on this count, you will need to let the PHP code decide whether the navigation links need to be added. You will also add an HTML form at the end of the page. This page will allow visiting users to add their own comments to the post. The form will be quite simple. It will contain a total of two visible input fields, one for the user’s name and another for the comment. The form will also include a hidden third field that contains the Object ID of the post. Finally, you need to add a submit button for submitting the information to the current PHP page itself. Here is the code snippet that does this: // Perform a count on the comments to create paging $doc = $c->$db->$col_posts->findOne($query); $count = count($doc["Comments"]); // Add some links for paging the comments if ($commentpage > 1) { print "Previous Page "; if ($commentpage * $limit < $count) { print "Next page
"; } } else { if ($commentpage * $limit < $count) { print "Next page
"; } } }

174

CHAPTER 8 ■ CREATING A BLOG APPLICATION WITH THE PHP DRIVER

// Let's add a simple HTML form print ""; } You still need to specify the submit behavior of this form. This will require creating two more variables: $criteria and $update. You will use $criteria to specify the criteria a document must match to be updated, based on the submitted Object ID information. You will use $update to contain the information that performs the update itself. The information in the $update variable will consist of an array that contains the $addToSet variable for one, as well as an embedded array that contains the submitted data. Once you gather all information, you will process it using an if statement that calls MongoDB’s update() function to update the document, as in the following example: if(isset($_POST["addcomment"])) { // Specify the update criteria $criteria = array("_id" => new MongoId($_POST['id'])); // Specify the update that should be performed $update = array( '$addToSet' => array ( "Comments" => array ( "Name" => $_POST['commentauthor'], "Comment" => addslashes($_POST['comment']) ) ) ); // Perform the update $c->$db->$col_posts->update($criteria, $update); // Thanks print "Thanks for your comment!"; } ?>

Searching the Posts So far you’ve learned how to view all posts or a single one. You’ve also learned how to implement paging for both examples. The more posts you get, the harder it is to find them again. You can simplify the process of finding such posts again by creating an HTML form that passes on the input to the find() function. This is also a case where Regular Expression can help. Let’s begin by adding another if statement to the original posts.php page saved earlier. You can place this statement under the first listing of defined variables, and it will work like this: if a search has been performed using the HTML form, then the search value will be used to specify the query. If no search parameter has been specified, then the search query will remain empty. The code to accomplish this is as follows:

175

CHAPTER 8 ■ CREATING A BLOG APPLICATION WITH THE PHP DRIVER

if(isset($_POST["search"])) { // Use MongoRegEx to specify the search criteria $regex = new MongoRegex("/$_POST[search]/i"); $query = array("Message" => $regex); } else { // If no search criteria is specified, $query will be set to an empty array $query = array(); } You also need to make the find() command itself aware of this change. After all, the $cursor currently looks like this, so it isn’t aware of any $query: $cursor = $c->$db->$col_posts->find()->skip($offset)->limit($limit)->sort(array('_id'=>-1)); The next step is to change $cursor variable so that it takes the $query variable as a parameter for the find() function, as in the following example: $cursor = $c->$db->$col_posts->find($query)->skip($offset)->limit($limit)->sort(array('_id'=>1)); You still need to add the form to input the search criteria. You can do this by adding a HTML form with one input field and naming it search. Add this field somewhere at the top of the page, before any of the documents will be printed. For example, you might place it after the paging variables and before the $c variable: // Add a form for searching the posts print "";

Adding, Deleting, and Modifying Posts So far you’ve created the end-user code. The next step is to work on the code that administers the blog itself. Specifically, you need to write code that creates, updates, and deletes posts. Of course, you’ll also make some cosmetic changes, such as adding links that initiate these actions. Begin by creating a new folder called admin. You also want to create a new PHP file (also called posts.php, just to keep things simple). The first thing you will want to see when going into Admin mode is an overview of the posts themselves, as well as links to either delete or edit individual posts. It would also be a good idea to create a link that lets you add an additional post. It’s best to be “lazy” and avoid doing work that you’ve already done before. Therefore, writing another page from scratch to show all the posts available would be unnecessary because you’ve already written code that does this. A simple copy/paste will suffice to start. Once you do this, you can augment the page with a few additional if statements, as well as a few hyperlinks that add, modify, or delete a post. Begin by creating a copy of the posts.php page and ensure that it gets saved in the previously mentioned admin folder. For now, it’s enough to add the links for adding, removing, or modifying a post. For example, you could add a link that adds new posts under the search box, as in the following example:

176

CHAPTER 8 ■ CREATING A BLOG APPLICATION WITH THE PHP DRIVER

// Add a form for searching the posts print ""; // Add a link to add a post print "Add a post"; You will write the add.php page momentarily, and you will use this page to add documents (i.e., posts) to the collection by combining an HTML form and the insert() function. Before you write that code, however, first you want to add the two links for either modifying or deleting a post. A good place to put these links would be next to the Comments link, which will also need to be changed slightly: print "View Comments ($count) "; // Show clickable links to modify or delete the post print "Modify "; print "Delete"; Later in this chapter, you’ll define actual actions to be performed whenever these links are clicked. For now, you can leave things like this and continue on to the PHP page that adds a new post.

Adding a New Post You will need to create a new page so you can add any new posts through a simple HTML form. You already know what the fieldnames will be (the title, the message, and a dropdown box to identify who will add the post), so it’s easy to construct the HTML form in a new PHP page named add.php. Before creating this HTML form, however, you can make your life a little easier by first adding a handful of new variables: $db->$col_authors->find(); The authors’ names are now listed under the cursor, so you can now build the HTML form: print ""; To submit the post itself, you need to add a section in the file that gets triggered by a $_GET[‘addpost’]. This section will create a new array called $arr. You will fill this array with a set of keys called Title, Message, Author, and Date. The values of these keys will be set to whatever has been inserted into the HTML form previously, with one exception: the date value will be added through the MongoDate() function, which will set the date field to current date and time. Once all information has been successfully inserted into the array, the data will be injected into the posts collection with MongoDB’s insert() function. This function takes the array name ($arr) as an argument, as shown in the following example: // Specify the behavior when clicking on Add if(isset($_GET["addpost"])) { // Create a new array to store the changed values in $arr = array(); $arr['Title'] = addslashes($_GET['Title']); $arr['Message'] = addslashes($_GET['Message']); $arr['Author'] = MongoDBRef::create( $c->$db->$col_authors->getName(), new MongoId($_GET['names']) ); $arr['Date'] = new MongoDate(); $c->$db->$col_posts->insert($arr); print "Post added. You can leave this page."; } ?> Et voilà! Notice that the resulting output will be extremely basic. It consists only of a message stating that the post has been added. Of course, you can make this fancier as you go along. At this point, you’re ready to save this document; name it add.php.

Editing a Post Sometimes you may need to modify an existing post. For example, perhaps the post’s author used language inappropriate for your site, and you want to strip that out, but leave the post itself. In this circumstance, you’d probably prefer to modify the original post without removing it, editing it, adding it back again, and then manually adding the comments back again. The preceding approach isn’t exactly flexible, nor is it a good idea. For these kinds of actions, you may prefer to use the update() function instead. In the posts.php page, you previously added a link to deal with changing a document. Currently, this link doesn’t do anything other than respond when the $_GET['modify'] variable is called; however, you’re now ready to add in some code to deal with editing a post. Begin by creating an HTML form and fetching the document from the collection. You will place the values retrieved from the collection into the fields of the HTML form, which will make it easier to change any existing data. Once you do this, you will be only a click away from using the update() function to update the document. The update() function takes two parameters: _id, which specifies the ID of the document that needs to be changed; and $set, which holds the new information the document needs to show instead. You need to add the following code at the bottom of the posts.php page, just beneath the if statement and just before the PHP closing tag:

178

CHAPTER 8 ■ CREATING A BLOG APPLICATION WITH THE PHP DRIVER

// Specify what ought to happen when the Modify link has been clicked if(isset($_GET["modify"])) { // Use the posted ID to retrieve the data, and input it in a form $filter = array("_id" => new MongoId($_GET["modify"])); $post = $c->$db->$col_posts->findOne($filter); // Start writing the form print ""; } You’ve added the HTML form; next, you can specify the submission behavior: // Specify the behavior when clicking on Change else if(isset($_POST["modifypost"])) { // Create a new array to store the changed values in $arr = array(); $arr['Title'] = addslashes($_POST['Title']); $arr['Message'] = addslashes($_POST['Message']); $id = new MongoId($_POST['id']); $c->$db->$col_posts->update(array("_id" => $id),array('$set' => $arr)); // Hooray print "The post has been updated. Hooray"; }

Deleting a Post Deleting a post doesn’t take much effort. There’s hardly any server-client communication involved because the process of deleting a post doesn’t require that you query and return the values from all fields in the post. This means that it’s a relatively low-bandwidth task to perform. Similarly, the code itself is simple and can be covered quickly. You probably recall that you also added a Delete link to the post.php page, but didn’t supply any implementation code at the time. To delete a post, you set the URL to ?del=, where specifies the _id field’s value from the post you want to delete. The ID value is also perfect for removing documents; all that you need to do to remove a document is another if statement. This new if statement will convert the received ID into a MongoDB type ID value, and store this in the $id variable. Next, you need to establish a connection to the database, and then use the remove() function to complete the removal process. The remove() function takes the $id variable as an argument. What follows is the last piece of PHP code; you can place this under the same if() statement that you used to modify the posts: // Specify what ought to happen when the Delete link has been clicked if(isset($_GET["del"])) { // Use the posted ID as the _id, and convert it via MongoId $id = array("_id" => new MongoId($_GET["del"])); // Specify the options $options = array('justOne' => true, 'safe' => true); // Connect to the database and remove the document

179

CHAPTER 8 ■ CREATING A BLOG APPLICATION WITH THE PHP DRIVER

$c->$db->$col_posts->remove($id, $options); // In case the document was deleted successfully, report a success print "Post removed successfully"; }

Creating the Index Pages You’ve finished creating the basic code of your blog application. At this point, you should be able to add, modify, and delete posts. You should also be able to use paging to navigate the posts, as well as to read, add, modify, and delete comments. The only problem is that all your code is spread across a couple of pages, and clicking certain actions (e.g., submitting a post) currently navigates you away from the page, which is undesirable behavior. To counter this, you can design a simple index page (index.html) that contains a frameset, or table with the earlier pages embedded. Or you can add all the blog’s contents into a single page—the choice is yours. This section walks you through the easier of the two approaches: creating a frameset where a single page holds two frames. The first frame contains another HTML page with some navigation links, while the second frame contains the posts overview (posts.php). You can place the following lines of HTML code in either the root’s index or in the admin directory: MongoDB PHP Example The page containing the navigation links isn’t exactly rocket science. But remember: the aim in this chapter is to keep things simple and show what MongoDB can do for you, not to become a PHP guru. The following snippet shows the contents from the nav.html file, which you place in the root: View Posts
Manage posts
You can adjust the hyperlinks in the admin section to your liking; the following snippet shows an example of the nav.html page for the admin folder: View Posts
Add Posts

180

CHAPTER 8 ■ CREATING A BLOG APPLICATION WITH THE PHP DRIVER

At this point, you have a basic sample application that allows you to add, modify, and delete documents through a simple PHP interface. You are free to expand this sample application with additional code, as your preferences and needs require.

Recapping the blog Application Throughout this chapter, you’ve seen code examples designed to help you get the blog application up and running. These code examples rely on basic, yet practical functions. Because it can be hard to keep track of which code goes where—and because the code doesn’t always do what you expect it to—the following sections will provide the complete and finished versions of the code examples again, along with a short commentary that explains what you see in each code sample. The first code example you’ll review (Listing 8–1) shows the code for the index.html file that contains the frontpage information. The index file is generally the first file to be loaded; therefore, it includes the information a visitor must see first when visiting the page. This following code snippet includes a standard, uncomplicated frameset that loads both the navigation menu (nav.html) on the left part of the screen and the posts overview (posts.php) on the right. You can adjust the percentages in the frameset tag based on your preferences: Listing 8–1. index.html MongoDB PHP Next comes the nav.html file shown in Listing 8–2, which contains two hyperlinks to other pages. The first link is a reference to the aforementioned posts.php page, and it ensures that this page will be loaded in the right-hand frame. The second is a link to the admin section, which lets you modify and/or add posts: Listing 8–2. nav.html View Posts
Manage posts
The next page, posts.php (Listing 8–3), contains the necessary PHP and MongoDB code needed to get a list of all available posts and to display their contents. At the top of this page, you define a set of variables to control a variety of tasks, such as which database and collections need to be connected to successfully obtain the posts. You also specify the limit, which indicates the maximum number of posts to display per page.

181

CHAPTER 8 ■ CREATING A BLOG APPLICATION WITH THE PHP DRIVER

This page also contains a number of if statements that control the behavior of the PHP page. The first one specifies that, if a search is being requested through the search form, then the query parameters must be changed to match the terms specified in the search box. Next, the MongoDB find() command is used to fetch and display all the posts. You use this function to match the search criteria specified and limit the results to the value specified previously. Finally, the posts.php page adds hyperlinks to facilitate navigating the blog posts if the find() function returns more than the maximum number of posts per page: Listing 8–3. posts.php
// // // //

This This This This

is is is is

the the the the

name of the database name of the authors collection name of the posts collection total number of posts displayed

// Set up some PHP code to check if the $page is set and if not, set $page to '1'. if(isset($_GET['page'])){ $page = $_GET['page']; } else { $page = 1; } if(isset($_POST["search"])) { // Use MongoRegEx to specify the search criteria $regex = new MongoRegex("/$_POST[search]/i"); $query = array("Message" => $regex); } else { $query = array(); } // Let's define some more variables for paging $offset = ($page -1) * $limit; $nextpage = ($page +1); $prevpage = ($page -1); // Add a form for searching the posts print ""; // Connect to the database $c = new Mongo(); // Execute a search and store the posts under the cursor variable

182

CHAPTER 8 ■ CREATING A BLOG APPLICATION WITH THE PHP DRIVER

$cursor = $c->$db->$col_posts->find($query)->skip($offset)->limit($limit)->sort(array('_id'=>1)); // For each document it finds within the collection, print it’s contents while ($document = $cursor->getNext()) { // Get the author's name via MongoDBRef $ref = $c->$db->getDBRef($document["Author"]); $author = $ref["Name"]; // Translate the date back using date() $date = date('M d, Y @ h:i', $document["Date"]->sec); // Show the title, author, date and message print "

$document[Title]

"; print "By $author on $date"; print "

$document[Message]

"; // Show clickable link to view the comments, and count them $postid = new MongoId($document['_id']); if(isset($document["Comments"])) { $count = count($document["Comments"]); } else { $count = 0; } print "View Comments ($count)"; } $posts = $c->$db->$col_posts->find()->count(); if ($page > 1) { print "Previous Page"; if ($page * $limit < $posts) { print "Next page"; } } else { if ($page * $limit < $posts) { print "Next page"; } } ?> You use the next page, view.php (Listing 8–4), solely to display a single post, as well as its associated comments. This page also contains a set of variables that you use to connect to and obtain the right information from the right database. If the Object ID is given in the URL, then the code in view.php implements paging for the comments added to the post, assuming the number of comments to display exceeds the limit specified per page. The page also includes hyperlinks to navigate between comments when the number of comments exceeds the maximum number of comments per page.

183

CHAPTER 8 ■ CREATING A BLOG APPLICATION WITH THE PHP DRIVER

This page also includes an additional query that uses DBRef to target the authors collection. Specifically, this query obtains information about the author of a given post. Once all the appropriate information has been gathered, the post and its comments are displayed on the screen, followed by an HTML form that allows the user to add new comments to the post: Listing 8–4. view.php
// // // //

Used This This This

to is is is

connect to the db the name of the database the name of the authors collection the name of the posts collection

if (isset($_GET['id'])) { if (isset($_GET['p'])) { $commentpage = $_GET['p']; } else { $commentpage = 1; } // Add some more variables for paging the comments $id = new MongoId($_GET['id']); $limit = 5; // Used to define the max no. of comments $offset = ($commentpage -1) * $limit; // This is our comments offset $nextpage = ($commentpage +1); $prevpage = ($commentpage -1); // Define the query parameters that we want to use $query = array('_id' => $id); $cond = (object)array('Comments' => array('$slice' => array($offset, $limit) ) ); // Connect to the database, and perform the findOne $document = $c->$db->$col_posts->findOne($query, $cond); // Now get the author's name via MongoDBRef, and translate the date $ref = $c->$db->getDBRef($document["Author"]); $author = $ref["Name"]; $date = date('M d, Y @ h:i', $document["Date"]->sec); // ..and print all the keys and values out print "

$document[Title]

"; print "By $author on $date"; print "

$document[Message]

"; // If there are any comments, then show them here if (isset($document['Comments']))

184

CHAPTER 8 ■ CREATING A BLOG APPLICATION WITH THE PHP DRIVER

{ // Show a header to announce the comments print "

Comments

"; // Set each comment that gets found in the $document array to $comm foreach ($document['Comments'] as $comm) { // For each of these, print the name and comment print "$comm[Name]
"; print "$comm[Comment]
"; } // Perform a count on the comments to create paging $doc = $c->$db->$col_posts->findOne($query); $count = count($doc["Comments"]); // Add some links for paging the comments if ($commentpage > 1) { print "Previous Page "; if ($commentpage * $limit < $count) { print "Next page
"; } } else { if ($commentpage * $limit < $count) { print "Next page
"; } } } // Let's add a simple HTML form print ""; } if(isset($_POST["addcomment"])) { // Specify the update criteria $criteria = array("_id" => new MongoId($_POST['id'])); // Specify the update that should be performed $update = array( '$addToSet' => array ( "Comments" => array ( "Name" => $_POST['commentauthor'], "Comment" => addslashes($_POST['comment'])

185

CHAPTER 8 ■ CREATING A BLOG APPLICATION WITH THE PHP DRIVER

) ) ); // Perform the update $c->$db->$col_posts->update($criteria, $update); // Thanks print "Thanks for your comment!"; } ?> The admin/index.html page (Listing 8–5) is the first page to load when you navigate to the admin/ directory of the website. This page contains a frameset that consists of two columns. The first column embeds the Navigation menu, while the second column embeds the posts overview (posts.php): Listing 8–5. admin/index.html MongoDB PHP The admin/nav.html page(Listing 8–6) contains a set of hyperlinks that display in the left column of the website. One of these links navigates you back to the posts overview (assuming you have navigated away from it). The second link navigates you to a page used to add more posts to the database: Listing 8–6. admin/nav.html View Posts
Add Posts
The next page, admin.posts/php (Listing 8–7), does not differ much from the previously mentioned posts.php page that is used to display all posts: the difference is that this page adds the functionality to modify and/or delete a post. The page begins by defining a set of variables to specify the database information used to connect and retrieve all information. Next, the page includes code to search for specific posts. This code consists of regular expressions and a simple HTML form. The paging parameters are added next, including the functionality that enables you to find, retrieve, and list all posts, including the information about the authors. All this is followed by a section that defines what ought to happen when the modify link is clicked. Clicking this link shows an HTML form with the post’s information filled in, making it easy to modify the data and save it again. Naturally, the submit behavior is specified next, followed by code that implements a delete button. Clicking the delete button invokes MongoDB’s remove() function, which removes the document in question, based on the Object ID:

186

CHAPTER 8 ■ CREATING A BLOG APPLICATION WITH THE PHP DRIVER

Listing 8–7. admin/posts.php
// // // //

This This This This

is is is is

the the the the

name of the database name of the authors collection name of the posts collection total number of posts displayed

// Set up some PHP code to check if the $page is set and if not, set $page to '1'. if(isset($_GET['page'])){ $page = $_GET['page']; } else { $page = 1; } if(isset($_POST["search"])) { // Use MongoRegEx to specify the search criteria $regex = new MongoRegex("/$_POST[search]/i"); $query = array("Message" => $regex); } else { $query = array(); } // Let's define some more variables for paging $offset = ($page -1) * $limit; $nextpage = ($page +1); $prevpage = ($page -1); // Add a form for searching the posts print ""; // Add a link to add a post print "Add a post"; // Connect to the database $c = new Mongo(); // Execute a search and store the posts under the cursor variable $cursor = $c->$db->$col_posts->find($query)->skip($offset)->limit($limit)->sort(array('_id'=>1)); // For each document it finds within the collection, print it’s contents while ($document = $cursor->getNext())

187

CHAPTER 8 ■ CREATING A BLOG APPLICATION WITH THE PHP DRIVER

{ // Get the author's name via MongoDBRef $ref = $c->$db->getDBRef($document["Author"]); $author = $ref["Name"]; // Translate the date back using date() $date = date('M d, Y @ h:i', $document["Date"]->sec); // Show the title, author, date and message print "

$document[Title]

"; print "By $author on $date"; print "

$document[Message]

"; // Show clickable link to view the comments, and count them $postid = new MongoId($document['_id']); if(isset($document["Comments"])) { $count = count($document["Comments"]); } else { $count = 0; } print "View Comments ($count) "; // Show clickable links to modify or delete the post print "Modify "; print "Delete"; } $posts = $c->$db->$col_posts->find()->count(); if ($page > 1) { print "Previous Page"; if ($page * $limit < $posts) { print "Next page"; } } else { if ($page * $limit < $posts) { print "Next page"; } } // Specify what ought to happen when the Modify link has been clicked if(isset($_GET["modify"])) { // Use the posted ID to retrieve the data, and input it in a form $filter = array("_id" => new MongoId($_GET["modify"])); $post = $c->$db->$col_posts->findOne($filter); print "";

188

CHAPTER 8 ■ CREATING A BLOG APPLICATION WITH THE PHP DRIVER

} // Specify the behavior when clicking on Change if(isset($_POST["modifypost"])) { // Create a new array to store the changed values in $arr = array(); $arr['Title'] = addslashes($_POST['Title']); $arr['Message'] = addslashes($_POST['Message']); $id = new MongoId($_POST['id']); $c->$db->$col_posts->update(array("_id" => $id),array('$set' => $arr)); // Hooray print "The post has been updated. Hooray"; } // Specify what ought to happen when the Delete link has been clicked if(isset($_GET["del"])) { // Use the posted ID as the _id, and convert it via MongoId $id = array("_id" => new MongoId($_GET["del"])); // Specify the options $options = array('justOne' => true, 'safe' => true); // Connect to the database and remove the document $c->$db->$col_posts->remove($id, $options); // In case the document was deleted successfully, report a success print "Post removed successfully"; } ?> The final page of the application, admin/add.php (Listing 8–8), adds new documents to the collection. The code starts by specifying a number of variables that mostly contain database- and collection-specific information used to connect to the database. Next, the find() command fetches a list of existing authors added to the database previously. The code continues with an HTML form used to input the information for a new post. The add button submits this information; this button’s behavior is specified directly below an if statement. The if statement specifies that the information is to be added in the defined collection: Listing 8–8. admin/add.php $db->$col_authors->find(); print print print print while

"

Let's add a post!

"; ""; // Specify the behavior when clicking on Add if(isset($_GET["addpost"])) { // Create a new array to store the changed values in $arr = array(); $arr['Title'] = addslashes($_GET['Title']); $arr['Message'] = addslashes($_GET['Message']); $arr['Author'] = MongoDBRef::create( $c->$db->$col_authors->getName(), new MongoId($_GET['names']) ); $arr['Date'] = new MongoDate(); $c->$db->$col_posts->insert($arr); print "Post added. You can leave this page."; } ?>

Summary In this chapter, you learned how to use the PHP MongoDB driver to create a small and simple web application. This admittedly simple application demonstrated just how little code you need to use to add, delete, or modify documents in a collection. You also saw a practical example that showed how to use Regular Expression to perform searches, as well as how to use DBRef to reference information. In the next chapter, we’ll leave the PHP driver behind us and start focusing on administration of the MongoDB installation.

190

P A R T III ■■■

Advanced

CHAPTER 9 ■■■

Database Administration In this chapter, we will walk you through some of the basic administrative operations you can perform on a MongoDB server. We will also show you how to automate some of those activities, such as backing up your server. The nature of MongoDB means that many of the more traditional functions that a DB Administrator would perform are not required. For example, creating new databases, collections, and fields on the server are not necessary because MongoDB will create these elements on-the-fly as you access them. Therefore, managing databases and schemas is not required for the vast majority of cases. However, this freedom from having to predefine everything can result in the unintended creation of elements, such as extraneous collections and fields in documents. Administrators and developers will occasionally need to clear out unused data elements on databases, particularly during the development phase of a project where change is often rapid. They may have to try out many approaches before settling on a final solution before cleaning up the databases. MongoDB’s ease of use encourages this explorative mode of development; however, it can also result in clutter in your datastores because the amount of effort required to create data structures has fallen almost to nothing. A contributing factor to this clutter and one of the more significant differences between MongoDB and SQL databases is that all object and element names in MongoDB are case sensitive on all platforms. Thus, the foo and Foo database names refer to two completely different databases. Therefore, you need to be careful with your database and collection naming to avoid accidentally creating multiple databases that differ only in the case of the name. The different versions of these databases will fill up your disk, and they can create a lot of confusion for developers and end users of the system by allowing them to connect to incomplete or unintended sets of data. In this chapter, you will learn how to perform all the following tasks: •

Back up and restore your MongoDB system.

•

Use the supplied MongoDB shell to perform common tasks.

•

Control access to your server with authentication.

•

Monitor your database instances.

Before diving into those tasks, however, we’ll begin by looking at the tools used to carry out many of the tasks listed above.

■ Note The examples in this chapter are based on the Blog application introduced in Chapter 8.

193

CHAPTER 9 ■ DATABASE ADMINISTRATION

Using Administrative Tools An administrator needs tools suitable for performing the day-to-day tasks of keeping the server running smoothly. There are some very good tools available in the MongoDB package, as well as an evolving collection of useful third-party tools. The following sections cover some of the more important tools available, as well as how to use them.

mongo, the MongoDB Console The main tool you will use as an administrator is mongo, the MongoDB console tool. mongo is a command-line console utility based on JavaScript. It is similar to many of the query tools supplied with mainstream relational databases. However, mongo has one unique trick up its sleeve: it can run programs written in JavaScript that interact directly with the MongoDB database. This console allows you to script all of your interactions with MongoDB in JavaScript, and then store those scripts in .js files to run when you need them to. In fact, many of the built-in commands in the mongo console are themselves written in JavaScript. You can place any commands that you would type into the command shell into a file with a .js extension, and run them by simply adding the file name to the command line when starting up the shell, the shell will execute the contents of the file and then exit. This is useful for running lists of repetitive commands. We will be using the mongo console to demonstrate many of the administrative tasks that you can perform on a MongoDB server, and, because it is distributed with the MongoDB server, we can guarantee it will be there.

Using Third-Party Administration Tools There are a several third-party administration (admin) tools available for MongoDB. 10gen maintains a page on the MongoDB website that lists currently available third-party tools. You can find this list at www.mongodb.org/display/DOCS/Admin+UIs. Most of these tools are web-based and similar in principle to phpMyAdmin for Mysql.

Backing up the MongoDB Server As a new MongoDB administrator, the first skill you should learn is how to back up and restore your MongoDB server. Arming yourself with this knowledge should make you feel more comfortable with exploring some of the more advanced administrative functions. This is because you know that your precious data is stored away safely somewhere.

Creating a Backup 101 Let’s begin by performing a simple backup then restoring it. Along the way, you will make sure the backup is intact, and you will look at some practical examples that illustrate how the backup and restoration features work. Once you have a solid understanding of how to use these features, you will be able to move on to exploring the more advanced administrative features of MongoDB. In this simple backup example, we will assume the following:

194

CHAPTER 9 ■ DATABASE ADMINISTRATION

•

Your MongoDB server is running on the same machine that you are currently logged into.

•

You have enough disk space for dump files that could be, at most, the same size as your database.

•

Your backup will be made in your home directory. This means you won’t have to deal with any issues related to permissions.

The MongoDB backup utility is called mongodump; this utility is supplied as part of the standard distribution. The following example performs a simple backup of the running MongoDB server to a designated disk directory: $> $> $> $>

cd ~ mkdir testmongobackup cd testmongobackup mongodump When mongodump is running, you should see it output something that looks like the following:

$ mongodump connected to: 127.0.0.1 all dbs DATABASE: blog to dump/blog blog.authors to dump/blog/authors.bson 2 objects blog.system.indexes to dump/blog/system.indexes.bson 3 objects blog.posts to dump/blog/posts.bson 1 objects blog.tagcloud to dump/blog/tagcloud.bson 4 objects DATABASE: admin to dump/admin If your output doesn’t look very similar to this, then you should double-check that your environment matches up with the assumptions stated previously. If you do see the correct output, then your database has been backed up to the testmongobackup/dump directory. The following snippet restores your database to its state at the time you performed the backup: $> cd ~/testmongobackup $> mongorestore --drop connected to: 127.0.0.1 Thu Aug 12 13:31:20 dump/blog/authors.bson Thu Aug 12 13:31:20 going into namespace [blog.authors] Thu Aug 12 13:31:20 dropping Thu Aug 12 13:31:20 2 objects found Thu Aug 12 13:31:20 dump/blog/posts.bson Thu Aug 12 13:31:20 going into namespace [blog.posts] Thu Aug 12 13:31:20 dropping Thu Aug 12 13:31:20 1 objects found Thu Aug 12 13:31:20 dump/blog/system.indexes.bson Thu Aug 12 13:31:20 going into namespace [blog.system.indexes] Thu Aug 12 13:31:20 dropping Thu Aug 12 13:31:20 3 objects found

195

CHAPTER 9 ■ DATABASE ADMINISTRATION

Thu Thu Thu Thu

Aug Aug Aug Aug

12 12 12 12

13:31:20 13:31:20 13:31:20 13:31:20

dump/blog/tagcloud.bson going into namespace [blog.tagcloud] dropping 4 objects found

The --drop option tells the mongorestore utility to discard each collection in the database before restoring it. Consequently, the backed-up data replaces the data currently in the database. If you were to choose not to use the --drop option, then the restored data would be appended to the end of each collection, which would result in duplicated items. Let’s examine what happened in the preceding example more closely. By default, the mongodump utility connects to the local database using the default port, and pulls out all of the datafiles associated with each database and collection and stores them in a predefined folder structure. The default folder structure created by mongodump takes this form: ./dump/[databasename]/[collectionname].bson The database system used in the preceding example consists of a single database called blog. The blog database contains three collections: authors, posts, and tagcloud. The mongodump saves the data it retrieves from the database server in .bson files that are just a copy of the internal BSON format that MongoDB uses internally to store documents. You can also see the MongoTest.system.indexes collection backed up and restored in the preceding example. This collection is maintained by the MongoDB server, and it records which elements of each collection are indexed. It is this collection that allows you to rebuild the indexes when restoring from a backup. Once you have dumped the database, you can archive and store the folder on to any online or offline media, such as CD, USB Drive, Tape, or S3 format.

■ Note The mongodump utility does not empty the contents of the output directory before it writes the backup files to it. If you have existing contents in this directory, then these contents won’t be removed unless they match the name of a file (collectionname.bson) that mongodump has been instructed to back up. This is good if you want to add multiple collection dumps to the same dump directory; however, it could cause some issues if you use the same dump directory each time you back up data, but don’t clear it out. For example, assume you have a database that you back up regularly, and at some point you decide to delete a collection from this database, unless you either clear out the directory where you are performing your backup, or manually remove the file associated with the deleted collection, then the next time you restore the data the deleted collection will reappear. Unless you want to overlay data in your backups, you should make sure you clear out the destination directory before you use mongodump.

196

CHAPTER 9 ■ DATABASE ADMINISTRATION

Backing up a Single Database When you have multiple applications running on the same server, often you may find yourself wanting to back up each database individually, rather than all at once, as in the preceding example. With mongodump, you can do this by adding the -d databasename option to the command line. This causes mongodump to create the ./dump folder; however, this folder will contain only the backup files for a single database.

Backing up a Single Collection Imagine you have a blog site where the contents of the authors collection does not change much. Instead, the rapidly changing content of the blog site is contained in the posts and tagcloud collections. You might back up the entire database only once a day, but want to back up these two collections once per hour. Fortunately, you can do that easily with mongobackup by using the -c option to specify the collection you wish to back up. The mongodump utility does not clear its destination directories. This means that, for each collection you want to back up, you can call mongodump successively to add a given collection to your backup, as shown in the following example: $mkdir ~/backuptemp $cd ~/backuptemp $mongodump -d blog -c posts $mongodump -d blog -c tagcloud ... archive the dump folder ~/backuptemp away as a tar file ... $ cd ~ $ rm -rf backuptemp

Digging Deeper into Backups At this point, you know how to perform the rudimentary tasks of backing up your data and subsequently restoring that data. Now you’re ready to look at some of the powerful options that allow you to tailor MongoDB’s backup and restore functionality to suit your particular needs. The mongodump utility includes the following options:

197

CHAPTER 9 ■ DATABASE ADMINISTRATION

Most of the options listed here are self-explanatory, with the exception of last three, which are listed here: •

--dbpath arg: If you have a large amount of data to back up and fast drives, then it may be preferable to back up the database by copying the datafiles that the MongoDB server uses directly to the backup medium. This option allows you to back up directly from the server’s datafiles, but it can only be used if the server is offline or otherwise write frozen (see the “Backing up Large Databases” section later in this chapter for more information).

•

--directoryperdb: You use this command-line option in conjunction with the --dbpath option to specify that the MongoDB server that is being backed up was configured to place its datafiles for each database in a separate directory. By default, MongoDB places all of its datafiles in a single directory. You should use this option only if you have configured your server to operate in this mode.

•

-o [ --out ] arg: You use this option to specify the directory where you want the database dumps to be placed. By default, the mongodump utility creates a folder called /dump in the current directory and writes the dumps into that. You can use the -o/--out option to choose an alternative path to place the output dumps.

Restoring Individual Databases or Collections You’ve just seen how the mongodump utility can back up a single database or collection; the mongorestore utility can do the same thing. You can use mongorestore to restore an item if the dump directory it is restoring from has the backup files for the required collection or database in it, you don’t need to restore all the items present in the backup; if you wish, you can restore them individually. Let’s begin by looking at the options available in mongorestore:

198

CHAPTER 9 ■ DATABASE ADMINISTRATION

You probably recognize most of these options from the discussion on mongodump; however, the following two options are worthy of special mention: •

--drop: This option instructs mongorestore to drop the existing collection before restoring it. This helps ensure that there are no duplicates. If this option is not used, then the restored data is appended (inserted) into the target collection. A planned future version of MongoDB intends to support an upsert mode of restoring collections, where the mongorestore utility will check to see whether a document of the same _id already exists in the collection. If the _id already exists, MongoDB will update it; if the _id doesn’t already exist, MongoDB will insert it. This will remove the need to drop the target collection to guarantee data consistency.

•

--objcheck: This option instructs mongorestore to validate the object before inserting it into the destination collection. For example, if there is corruption in the BSON data used to construct the object, then MongoDB will not restore it.

Restoring a Single Database You can use the mongorestore utility’s -d option to restore a single database. As before, don’t forget to use the --drop option if the database already exists in your MongoDB server: $cd ~/testmongobackup $mongorestore -d blog --drop

Restoring a Single Collection You use similar syntax to restore a single collection to a database; the difference is that you also specify a collection name with the -c option, as shown in the following example: $cd ~/testmongobackup $mongorestore -d blog -c posts --drop

Automating Backups For small installations or developer setups, the simple act of running the mongodump utility and saving the results is a perfectly adequate method of performing ad-hoc backups. For example, a common practice on a Mac OS X workstation is to let Time Machine (the Mac backup utility) store the backups. For any kind of production setup, you will want to back up the server automatically. This is because regular backups can help you prevent or recover from trouble if you encounter any problems. This holds true not only with your installation (e.g., corrupted databases), but also if your users inadvertently damage or destroy data. Let’s look at some simple scripts that you can use to automate your backups.

Using a Local Datastore A simple backup script that creates archives in a specified directory will suffice if you have a large backup drive attached to your system or you can mount an external filesystem through NFS or SMB. The following backup script is easy to set up; simply edit the variables at the top of the script to match those of your local system:

199

CHAPTER 9 ■ DATABASE ADMINISTRATION

#!/bin/bash ########################################## # Edit these to define source and destinations MONGO_DBS="" BACKUP_TMP=~/tmp BACKUP_DEST=~/backups MONGODUMP_BIN=/usr/bin/mongodump TAR_BIN=/usr/bin/tar ########################################## BACKUPFILE_DATE=`date +%Y%m%d-%H%M` # _do_store_archive function _do_store_archive { mkdir -p $3 cd $2 tar -cvzf $3/$4 dump } # _do_backup function _do_backup { UNIQ_DIR="$BACKUP_TMP/$1"`date "+%s"` mkdir -p $UNIQ_DIR/dump echo "dumping Mongo Database $1" if [ "all" = "$1" ]; then $MONGODUMP_BIN -o $UNIQ_DIR/dump else $MONGODUMP_BIN -d $1 -o $UNIQ_DIR/dump fi KEY="database-$BACKUPFILE_DATE.tgz" echo "Archiving Mongo database to $BACKUP_DEST/$1/$KEY" DEST_DIR=$BACKUP_DEST/$1 _do_store_archive

$1 $UNIQ_DIR $DEST_DIR $KEY

rm -rf $UNIQ_DIR } # check to see if individual databases have been specified, otherwise backup the whole server # to "all" if [ "" = "$MONGO_DBS" ]; then MONGO_DB="all" _do_backup $MONGO_DB else

200

CHAPTER 9 ■ DATABASE ADMINISTRATION

for MONGO_DB in $MONGO_DBS; do _do_backup $MONGO_DB done fi Table 9–1 lists the variables you have to change to make this simple backup script work with your system. Table 9–1. The Variables Used in the Local Datastore Backup Script

Variable

Description

MONGO_DBS

Leave this variable empty ("") to back up all databases on the local server. Or you can place a list of databases into it to back up selective databases ("db1 db2 db3").

BACKUP_TMP

Set this variable to a temporary directory suitable for holding the dump files for the backup. After the archive has been created, the temporary data used in this directory is deleted. Be sure to choose a suitable directory that is relevant to using your script. For example, if you are using the script to create backups in your local account, then use ~/tmp; if you are using it as a system cronjob that runs under a system account, then use /tmp. On an Amazon EC2 instance, you should probably use /mnt/tmp, so that the folder is not created on the system root partition, which is quite small.

BACKUP_DEST

This variable holds the destination folder for the backups, and individual folders will be created below this folder. Again, place this directory at a point relevant to how you use your backup script.

MONGODUMP_BIN

Because your backup script may be running under an account that does not have a full set of paths set up, it’s wise to use this variable to specify the full path to this binary. You can determine the appropriate path on your system by typing “which mongodump” into a terminal window.

TAR_BIN

You use this variable to set the full path for the tar binary; use “which tar” in a terminal window to determine this path.

You can now use this script to back up your databases; doing so will create a set of archival backups in the specified BACKUP_DEST directory. The files created follow this naming format: [Database Name]/database-YYYYMMDD-HHMM.tgz

201

CHAPTER 9 ■ DATABASE ADMINISTRATION

For example, the following snippet shows the backup names for this chapter’s test database: Backups:$ tree . |-- blog | |-- database-20100611-1144.tgz | `-- database-20100611-1145.tgz `-- all |-- database-20100611-1210.tgz |-- database-20100611-1221.tgz |-- database-20100611-1222.tgz |-- database-20100611-1224.tgz `-- database-20100611-1233.tgz

Installing the Script If you want to run this script daily, just put it into /etc/cron.daily and restart the cron service to make it active. This approach will work on ubuntu, fedora, centos, and redhat. If you want less frequent backups, then just move the script to /etc/cron.weekly or /etc/cron.monthly. For more frequent backups, you can use /etc/cron.hourly.

Using a Remote (Cloud-Based) Datastore The script described in the previous section has a separate function for creating and storing the archive. This makes it relatively easy to modify the script so it uses an external datastore to store the backup archive. Table 9–2 provides a couple of examples, but many more other mechanisms are possible. Table 9–2. Remote (Cloud-Based) Backup Storage Options

Method

Description

rsync/ftp/tftp or scp to another server

You can use rsync to move the archive to a backup storage machine.

s3 storage

s3 storage is a good place to put your backups if you run your system on EC2 because storage costs are low and Amazon makes redundant copies.

We will examine the s3 method of storing your backup; however, the same principles apply to any of the other mechanisms. This example uses the s3cmd utility available from http://s3tools.org. On Ubuntu, you can install this script using the sudo apt-get install s3cmd command; on Mac OSX, this script is available from the macport collection. On fedora, centos, and redhat, you can acquire the yum package from http://s3tools.net, and then install it using yum. Once you have installed the package, run s3cmd –configure to set your Amazon s3 credentials. Note that you only need to supply two keys: AWS_ACCESS_KEY and AWS_SECRET_ACCESS_KEY. The s3cmd utility will create a config file that contains the information you need in this file: ~/.s3cfg. Here are the changes that you need to make for your backup script to work with S3: # _do_store_archive BACKUP_S3_CONFIG=~/.s3cfg

202

CHAPTER 9 ■ DATABASE ADMINISTRATION

BACKUP_S3_BUCKET=somecompany.somebucket S3CMD_BIN=/usr/bin/s3cmd function _do_store_archive { UNIQ_FILE="aws"`date "+%s"` cd $2 tar -cvzf $BACKUP_TMP/$UNIQ_FILE dump $S3CMD_BIN --config $BACKUP_S3_CONFIG put $BACKUP_TMP/$UNIQ_FILE \ s3://$BACKUP_S3_BUCKET/$1/$4 rm -f $BACKUP_TMP/$UNIQ_FILE } Table 9–3 lists some variables that you need to configure to make this adapted script work. Table 9–3. Configuring the Variables of Your Adapted Backup Script

Variable

Description

BACKUP_S3_CONFIG

The path to the s3cmd configuration file that was created when you ran s3cmd –configure to save the details of your s3 account.

BACKUP_S3_BUCKET

The name of the bucket that you want the script to store backups into.

S3CMD_BIN

The path to the s3cmd, executable program, again use “which s3cmd” to find it on your system.

Backing up Large Databases Creating effective backup solutions can become a problem when working with large database systems. Often the time taken to make a copy of the database is significant and may even require hours to complete. During that time, you have to maintain the database in a consistent state, so the backup does not contain files that were copied at different points in time. The holy grail of a database backup system is a point-in-time snapshot, which can be done very quickly. The faster the snapshot can be done, the smaller the window of time where the database server must be frozen.

Using a Slave Server for Backups One technique used to perform large backups is to make the backup from a replication slave server that can be frozen while the backup is taken. This slave server is then restarted to catch up with the application after the backup is complete. MongoDB makes it very simple to set up a slave server and have it track the master server. It’s also relatively easy to halt and then restart the replication process (see Chapter 10 for more details on how to set up this type of configuration).

Creating Snapshots with a Journaling Filesystem Many modern volume managers have the ability to create snapshots of the state of the drive at any particular point-in-time. While setting up one of these systems is beyond the scope of this book, we can

203 9

CHAPTER 9 ■ DATABASE ADMINISTRATION

show you how to place the MongoDB server in a state where all of its data is in a consistent state on the disk. We also show you how to block writes so that further changes are not written to the disk, but are instead buffered in memory. A snapshot is like a bookmark that allows you to read the drive exactly as it was when the snapshot was taken. A system’s volume or filesystem manager makes sure that any blocks of data on the disk that are changed after the snapshot is taken are not written back to the same place on the drive; this preserves all the data on the disk to be read. Generally, the procedure for using a snapshot goes something like this: 1.

Create a snapshot.

2.

Copy data from the snapshot or restore the snapshot to another volume, depending on your volume manager.

3.

Release the snapshot; doing so releases all preserved disk blocks that are no longer needed back into the free space chain on the drive.

4.

Back up the data from the copied data while the server is still running.

The great thing about the method just described is that reads against the data can continue unhindered while the snapshot is taken. Some volume managers that have this capability include: •

Linux and the LVM volume management system

•

Sun ZFS

•

Amazon EBS volumes

•

Windows Server using shadow copies

Most of the preceding volume managers have the ability to perform a snapshot in a very short period of time—often just a few seconds—even on very large amounts of data. The volume managers don’t actually copy the data out at this point; instead, they effectively insert a bookmark onto the drive, so that you can read the drive in the state it existed at the point in time the snapshot was taken. Once the backup system has read the drive from the snapshot, then the old blocks that have been subsequently changed can be released back to the drive’s free space chain (or whatever mechanism the filesystem uses to mark free space). To make this an effective method of creating a backup, we have to get MongoDB to flush all outstanding disk writes to the disk so we can take a snapshot. The function that forces MongoDB to do this is called fsync; the function that blocks further writes is called a lock. MongoDB has the ability to perform both operations at the same time, so that, after the fsync, no further writes are done to the disk until the lock is released. This makes the image of the database on the disk consistent, and ensures it stays consistent until we have completed the snapshot. You use the following commands to make MongoDB enter this state: $mongo >use admin >db.runCommand({fsync:1,lock:1}) { "info" : "now locked against writes", "ok" : 1 } You use these commands to check the current state of the lock: $mongo >use admin

204 q

CHAPTER 9 ■ DATABASE ADMINISTRATION

>db.currentOp() { "inprog" : [ ], "fsyncLock" : 1 } The "fsyncLock": 1 status indicates that MongoDB’s fsync process, which is responsible for writing changes to the disk, is currently blocked from performing writes. At this point, you can issue whatever commands are required to make your volume manager create the snapshot of the folders where MongoDB has its datafiles stored. Once the snapshot is completed, you can use the following commands to release the lock: $mongo >use admin >db.$cmd.sys.unlock.findOne(); { "ok" : 1, "info" : "unlock requested" } Note that there may be a small delay before the lock is released; however, you can use the db.currentOp() function to check the result. When the lock is finally cleared, db.currentOp() will return: $mongo >use admin >db.currentOp() { "inprog" : [] } The { "inprog" : [] } line means that the lock has been released and that MongoDB can start writing to the disk again. Now that you have the snapshot bookmark inserted, you can use the utilities associated with your volume manager to copy the contents of the snapshot to a suitable place so you can store your backup. Don’t forget to release the snapshot once your backup is complete. You can visit the following links for more information on snapshots: •

http://tldp.org/HOWTO/LVM-HOWTO/snapshots_backup.html

•

http://docs.huihoo.com/opensolaris/solaris-zfs-administrationguide/html/ch06.html

•

http://support.rightscale.com/09–Clouds/AWS/02Amazon_EC2/EBS/Create_an_EBS_Snapshot

Disk Layout to Use with Volume Managers Some volume managers can take a snapshot of subdirectories on a partition, but most can’t, so it is a good idea to mount the volume you are planning to store your MongoDB data on in a suitable place on your filesystem (e.g., /mnt/mongodb) and use the server configuration options to place the data directories, the configuration file, and any other MongoDB-related files (e.g., logfiles) solely on that mount. This means that, when you take a snapshot of the volume, you capture the complete state of the server, including its configuration. It may even be a good idea to place the binaries of the server distribution directly on that volume, so that your backup contains a completely coordinated set of components.

205

CHAPTER 9 ■ DATABASE ADMINISTRATION

Importing Data into MongoDB Sometimes, you need to load lots of bulk data into MongoDB for use as reference data. Such data might include zipcode tables, IP geolocation tables, parts catalogs, and so on. MongoDB includes a bulk “loader” mongoimport, designed to import data directly into a particular collection on the server. The mongoimport utility can load data from three different file formats: 1.

CSV: In this file format, each line represents a document, and the fields are separated by a comma.

2.

TSV: This file format is similar to CSV; however, it uses a tab character as the delimiter. This format is popular because it does not require the escaping of any text characters other than those for new lines.

3.

JSON: This format contains one block of JSON per line that represents a document. Unlike the other formats, JSON can support documents with variable schemas.

The use of this utility is fairly intuitive. For input, it takes a file in one of the three preceding formats, a string or a file with a set of column header names (these form the element names in a MongoDB document), and several options that are used to control how the data is interpreted. The following example shows how to use the mongoimport utility:

206

CHAPTER 9 ■ DATABASE ADMINISTRATION

The following options deserve a little bit more explanation: •

--headerline: Uses the first line of the file as the list of field names. Note that this applies only to CSV and TSV formats.

•

--ignoreblanks: Does not import empty fields. If a field is empty, then a corresponding element will not be created in the document for that row; if you don’t invoke this option, then an empty element with the column name is created.

•

--drop: Drops a collection and then recreates it with data only from this import; otherwise, the data is appended to the collection.

You also have to specify the database name and the collection name when you use mongoimport to import data with the -d and -c options, as in the following example: $mongoimport -d blog -c tagcloud --type csv --headerline < csvimportfile.csv

Exporting Data from MongoDB The mongoexport utility is similar to mongoimport, but mongoexport, as its name implies, creates export files from an existing MongoDB collection instead. The following example shows how to use the mongoexport utility:

Notable options from the mongoexport utility include the following:

207

CHAPTER 9 ■ DATABASE ADMINISTRATION

•

-q: Specifies the query used to locate the records to output. This query can be any JSON query string that you might use with the db.collection.find() function to select a subset of records. If you don’t specify this option or you set it to {}, then the mongoexport utility will output all records.

•

-f: Lists the database element names to be exported.

The following example illustrates how to use the options for the mongoexport utility: $mongoexport -d blog -c posts -q {} -f _id,Title,Message,Author --csv >blogposts.csv connected to: 127.0.0.1 exported 1 records

Securing Your Data In some cases, your applications may be dealing with sensitive data, user records in social networks, or payment details in eCommerce applications. In many cases, there are rules that mandate that you have to ensure restricted access to sensitive data in your database systems. MongoDB supports a simple authentication system that allows you to control who has access to each database, and what level of access they are granted.

Restricting Access to a MongoDB Server Most of the commands that change the configuration of or make major alterations to the structure of data on a MongoDB server are restricted to running only inside the special admin database that is created automatically during each new MongoDB installation. Before you can issue these commands, you have to switch to the admin database with the use admin command. Upcoming sections will note any command that is admin only, so you will always know you need to be in the admin database before you can use it. This chapter assumes that you can select the database and authenticate against it, if necessary. By default, MongoDB does not use any authentication methods. Anybody with access to the network connection can connect and issue commands to the server. However, you can add users to any database, and MongoDB can be configured to require both connection and console authentication to access the relevant database. This is the recommended mechanism for restricting access to admin functions.

Protecting Your Server with Authentication MongoDB supports a simple authentication model that allows the administrator to restrict access to databases on a per user basis. MongoDB supports individual access control records on each database that are stored in a special system.users collection. For normal users to have access to two databases (e.g., db1 and db2), their credentials and rights must be added to both databases. If you create individual logins and access rights for the same user on different databases, then there is no synchronization between those records. In other words, if you change a user password on one database, that does not change the password on any other database. There is one exception to this rule: any users added to the special admin database will have the same access rights on all databases; you do not need to assign rights to such users individually.

208

CHAPTER 9 ■ DATABASE ADMINISTRATION

■ Note Before you enable authentication on your server, make sure you have added your admin users to the admin database; if you enable authentication before adding the admin users, you will not be able to perform any actions on your database.

Adding an Admin User Adding the admin user is as simple as changing to the admin database and using the addUser() command: $mongo > use admin >db.addUser(“admin”, “adminpassword”) You only need to add a single admin user at this point; once that user is defined, you can use it to add other admin users to the admin database or normal users to any other database.

Enabling Authentication Now you need to alter your server’s configuration to enable authentication. Do so by stopping your server and adding --auth to the startup parameters. If you installed MongoDB with a packaged installer (e.g., yum or aptitude), then typically you can edit /etc/mongodb.conf to enable auth=true. Next, you can use the following command to restart the server and enable authentication: $sudo service mongodb restart

Authenticating in the mongo Console Before you can run restricted commands in the admin database, you will need to be authenticated as an admin user, as in the following example: $mongo > use admin switched to db admin >show collections Sun Jun 13 11:05:04 JS Error: uncaught exception: error: { "$err" : "unauthorized for db [admin] lock type: -1 " } >db.auth(“admin”, “adminpassword”); At this point, the mongo console will print either a 1 (a successful authentication) or a 0 (a failed authentication): 1 >show collections system.indexes system.users

209

CHAPTER 9 ■ DATABASE ADMINISTRATION

If your authentication was successful, then you will be able to perform any of the restricted operations available in the admin database, as well as any read/write actions required in any other database. If your authentication was unsuccessful, then you need to check whether your user/password is correct and whether the admin user has been correctly added to the admin database. Reset your server so it has no authentication, and then use the following command to list the contents of the system.users collection in the admin database: $mongo >use admin > db.system.users.find() { "_id" : ObjectId("4c1449ef5f9f8fe6ae7b057b"), "user" : "admin", "readOnly" : false, "pwd" : "c91e3736e2dae03682a25805b6f81d99" }

■ Note If you are using an admin credential to access databases other than admin, then you must first authenticate against the admin database. Otherwise, you will not be able to access any other databases in the system.

The mongo console shows the contents of the user collection, enabling you to see what the userid is, while the password is shown as an MD5 hash of the original password you supplied: $ mongo > use blog switched to db blog > show collections Sun Jun 13 12:02:36 JS Error: uncaught exception: error: { "$err" : "unauthorized for db [blog] lock type: -1 " } > db.auth("admin","adminpassword") 0 > use admin switched to db admin > db.auth("admin","adminpassword") 1 > use blog switched to db blog > show collections system.indexes system.users authors posts tagcloud

Changing a User’s Credentials It’s easy to change a user’s access rights or password. You do this by executing addUser() function again, which causes MongoDB to update the existing user record. Technically, you can use any normal data

210

CHAPTER 9 ■ DATABASE ADMINISTRATION

manipulation command to change a user’s record; however, only the addUser() function can create the password field. Regardless, you can see how addUser() works by listing its contents: $mongo >use admin > db.addUser function (username, pass, readOnly) { readOnly = readOnly || false; var c = this.getCollection("system.users"); var u = c.findOne({user:username}) || {user:username}; u.readOnly = readOnly; u.pwd = hex_md5(username + ":mongo:" + pass); print(tojson(u)); c.save(u); } addUser() is just a function defined in JavaScript. Knowing how the password is constructed is useful if you want to create a webform that allows you to add users to the database or you want to import users into the system en masse from another credential source. Most mongo console functions can be listed in this fashion, enabling you to inspect the details of how they work.

Adding a Read-Only User The addUser() function includes an additional parameter that allows you to create a user that has only read-only permissions. The MongoDB client will throw an exception if a process authenticated as the newly created user attempts to do anything that would result in a change to the contents of the database. The following example gives a user access to the database for status monitoring or reporting purposes: $mongo >use admin switched to db admin > db.auth("admin","adminpassword") 1 >use blog switched to db blog >db.addUser(“shadycharacter”,”shadypassword”, true)

Deleting a User To remove a user from a database, simply use the normal remove() function for a collection. The following example removes the user just added; note that you have to authenticate against the admin database before you can remove the user: $mongo >use admin switched to db admin > db.auth("admin","adminpassword") 1 >use blog switched to db blog >db.system.users.remove({username:”shadycharacter”})

211

CHAPTER 9 ■ DATABASE ADMINISTRATION

Using Authenticated Connections in a PHP Application In Chapter 2, you saw how to create a connection with PHP to a MongoDB server. Once you have enabled authentication on your server, PHP applications will also have to supply credentials before they can execute commands against the server. The following simple example shows how to open an authenticated connection to a database: selectDB(“admin”); $result = $db->authenticate(“admin”, “adminpassword”); if(!$result[‘ok’]){ // Your Error handling here die(“Authentication Error: {$result[‘errmsg’]}”); } // Your code here // Close the database connection $connection->close(); ?>

Managing Servers As an administrator, you must ensure the smooth and reliable running of your MongoDB servers. You will have to tune the servers to achieve maximum performance or reconfigure them to better match the environment you are operating in. To that end, you need to familiarize yourself with a number of procedures that enable you to manage and control your servers.

Starting a Server Most modern linux distributions now include a set of /etc/init.d scripts that are used to manage services. If you installed your MongoDB server using one of the distribution packages from the 10gen site (see Chapter 2 for more information on these packages), the init.d scripts for managing your server will already be installed. You can use the service command on ubuntu, fedora, centos, and redhat to start, stop, and restart your server, as shown in the following example: $sudo service mongodb start mongodb start/running, process 3474 $sudo service mongodb stop mongodb stop/waiting $sudo service mongodb restart mongodb start/running, process 3474

212

CHAPTER 9 ■ DATABASE ADMINISTRATION

If you don’t have an initialization script available, you can start the MongoDB server manually by opening a terminal window then typing the following: $sudo mongod -f /etc/mongodb.conf Sun Jun 13 13:33:52 Mongo DB : starting : pid = 3517 port = 27017 dbpath = /var/lib/mongodb master = 0 slave = 0 64-bit Sun Jun 13 13:33:52 db version v1.4.3, pdfile version 4.5 Sun Jun 13 13:33:52 git version: 47ffbdfd53f46edeb6ff54bbb734783db7abc8ca Sun Jun 13 13:33:52 sys info: Linux domU-12-31-39–06-79–A1 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_41 Sun Jun 13 13:33:52 waiting for connections on port 27017 Sun Jun 13 13:33:52 web admin interface listening on port 28017 The server will show all connections being made, as well as other information you can use to monitor how the server is working. To terminate the server in manual mode, just type ^C; this causes the server to shut down cleanly. If you don’t supply a configuration file, then MongoDB will start up with a default database path of /data/db and bind to localhost (127.0.0.1) using the default ports of 27017 and 28017, as in the following example: $ sudo mkdir -p /data/db $ sudo mongod mongod --help for help and startup options ... Sun Jun 13 13:38:00 waiting for connections on port 27017 Sun Jun 13 13:38:00 web admin interface listening on port 28017 ^C typed here Sun Jun 13 13:40:26 got kill or ctrl c signal 2 (Interrupt), will terminate after current cmd ends ... Sun Jun 13 13:40:26 dbexit: really exiting now

Reconfiguring a Server MongoDB supplies two main methods for configuring the server. First, you can use command-line options in conjunction with the mongod server daemon. Second, you can do so by loading a configuration file. Most of the prepackaged MongoDB installers use the latter method, using a file that is normally stored in /etc/mongodb.conf on *nix systems. You can change the configuration of your server by editing this file and restarting your server. The contents of this file look like this: # mongodb.conf dbpath=/var/lib/mongodb logpath=/var/log/mongodb/mongodb.log logappend=true auth = false #enable the rest interface rest =true

213

CHAPTER 9 ■ DATABASE ADMINISTRATION

You enable an option by removing the # code from the front of the option and setting its value as you require it. Placing any of the following option values in the configuration file is the same as specifying -- on the command line when starting up MongoDB: •

dbpath: Indicates where MongoDB will store your data; you should make sure it is on a fast storage volume that is large enough to support your database size.

•

logpath: Indicates the file that MongoDB will store its logs in. The normal place to put this is /var/logs/mongodb/mongodb.log; you need to use logrotate to rotate this logfile and prevent it from filling up your server’s drive.

•

logappend: Setting this option to false causes the logfile to be cleared each time MongoDB is started up. Setting this option to true causes all log entries to be appended to the end of any existing logfile.

•

auth: Enables or disables the authentication mode on MongoDB server; see the notes in this chapter for more information on authentication.

•

rest: Enables or disables the rest interface to MongoDB. You must enable this interface if you want to use the links from the web-based status display to show additional information.

Getting the Server’s Version You can use the version() command to get the build and version information for a server. This information is useful for determining whether upgrades are required or when reporting an issue to 10gen’s support mail list. The following snippet shows how to use this command: $mongo > use admin switched to db admin > db.version() version: 1.6.2

Getting the Server’s Status MongoDB provides a simple method for determining the status of a server. The following example shows the information returned, including such things as server uptime, the maximum number of connections, and so on: $mongo > use admin switched to db admin > db.serverStatus() { "version" : "1.6.2", "uptime" : 15685, "uptimeEstimate" : 0, "localTime" : "Thu Aug 12 2010 15:36:57 GMT+0800 (PHT)", "globalLock" : { "totalTime" : 15684704779, "lockTime" : 1731139,

214

CHAPTER 9 ■ DATABASE ADMINISTRATION

"ratio" : 0.00011037115612898206, "currentQueue" : { "total" : 0, "readers" : 0, "writers" : 0 } }, "mem" : { "bits" : 64, "resident" : 7, "virtual" : 2479, "supported" : true, "mapped" : 80 }, "connections" : { "current" : 1, "available" : 203 }, "extra_info" : { "note" : "fields vary by platform" }, "indexCounters" : { "btree" : { "accesses" : 1, "hits" : 1, "misses" : 0, "resets" : 0, "missRatio" : 0 } }, "backgroundFlushing" : { "flushes" : 261, "total_ms" : 2517, "average_ms" : 9.64367816091954, "last_ms" : 1, "last_finished" : "Thu Aug 12 2010 15:36:33 GMT+0800 (PHT)" }, "opcounters" : { "insert" : 2032, "query" : 48, "update" : 1, "delete" : 0, "getmore" : 0, "command" : 94 }, "asserts" : { "regular" : 0, "warning" : 0, "msg" : 0, "user" : 1, "rollovers" : 0 }, "ok" : 1 }

215

CHAPTER 9 ■ DATABASE ADMINISTRATION

You can find the two most important sections of the information returned by this function in the “opcounters” and “asserts” sections. The “opcounters” section shows the number of operations of each type that have been performed against the database server. You should have a good idea about what constitutes a normal balance for these counters for your particular application. If these counters start to move out of the normal ratio, then it may be an early warning that your application has a problem. For example, the preceding profile has an extremely high ratio of inserts to reads. This could be normal for a logging application; however, for a blogging application, it could indicate that either a spambot was hitting your “comments” section or a URL pattern that caused writes to the database was being repeatedly crawled by a search engine spider. In this case, it would be time to either put a captcha on your comments form or to block the particular URL pattern in your robots.tx file. The “asserts” section shows the number of server and client exceptions or warnings that have been thrown. If such exceptions or warnings start to rise rapidly, then it’s time to take a good look through your server’s logfiles to see whether a problem is developing. A high number of asserts may also indicate there is a problem with the data in the database, and you should consider using MongoDB’s validation functions to check whether your data is intact.

Shutting Down a Server If you have installed your MongoDB server from a package, then you can use the operating system’s service management scripts to shut down the server. For example, ubuntu, fedora, centos, and redhat let you shut down the server by issuing the following command: $sudo service mongod stop You can also shut down the server from the mongo console: $mongo >use admin >db.shutdownServer() You can use the posix process management commands to terminate a server, or you can use the SIG_TERM(-15) signal to shut down the server. If—and only if—the server fails to respond to the two methods described above, then you can use the following command: $sudo killall -15 mongod

■ Warning You must not use the SIG_KILL(-9) signal to terminate a server because this could result in a corrupted database, and you will probably have to repair the server.

It might be that you have a particularly active server with a lot of write activity, and you have reconfigured the server so it has a large sync delay. If that’s the case, then the server may not respond immediately to a termination request because it is writing out all the in-memory changes to the disk. A little patience goes a long way here.

216

CHAPTER 9 ■ DATABASE ADMINISTRATION

Using MongoDB Logfiles By default, MongoDB writes its entire log output to stdout; however, you can use the logpath option described previously to redirect the log output to a file instead. You can use the contents of the logfile to spot problems such as excessive connections from individual machines and asserts (exceptions) that may indicate problems with your data.

Validating and Repairing Your Data It is possible that your data will be left in a damaged or incomplete state if your server unexpectedly reboots or your MongoDB server crashes for any reason. Here are some indications that your data has been compromised: •

Your database server refuses to start, stating that the datafiles are corrupted.

•

You start seeing asserts in your server log files or a high assert count when using the db.serverStatus() command.

•

You get strange or unexpected results from queries.

•

The records counts on collections don’t match up with your expectations.

Any of the preceding signs can indicate corruption or inconsistency in your data. Fortunately, MongoDB ships with tools to assist you in repairing or recovering your database server. Nevertheless, you might still suffer the loss of some data, so please remember the golden rule of making sure you have either a good backup of your data or a replication slave.

Repairing a Server Before you can initiate the server repair process, you must first shut down the server (assuming it is not already offline). This is one of the best arguments for using replica pairs: you don’t have to stop your service if you have to take a machine offline to repair it. To initiate the repair process, just use the manual server startup process (as described previously in this chapter). However, this time you need to add the “--repair” option to the end of the command, as in the following example: $sudo mongod -f /etc/mongodb.conf --repair Sun Jun 13 14:18:45 Mongo DB : starting : pid = 3697 port = 27017 dbpath = /var/lib/mongodb master = 0 slave = 0 64-bit Sun Jun 13 14:18:45 **** Sun Jun 13 14:18:45 **** Sun Jun 13 14:18:45 need to upgrade database admin with pdfile version 4.5, new version: 4.5 Sun Jun 13 14:18:45 starting upgrade Sun Jun 13 14:18:45 admin repairDatabase admin Sun Jun 13 14:18:45 allocating new datafile /var/lib/mongodb/$tmp_repairDatabase_0/admin.ns, filling with zeroes... Sun Jun 13 14:18:45 done allocating datafile /var/lib/mongodb/$tmp_repairDatabase_0/admin.ns, size: 16MB, took 0.006 secs Sun Jun 13 14:18:45 allocating new datafile /var/lib/mongodb/$tmp_repairDatabase_0/admin.0, filling with zeroes... Sun Jun 13 14:18:45 done allocating datafile /var/lib/mongodb/$tmp_repairDatabase_0/admin.0, size: 64MB, took 0.026 secs

217

CHAPTER 9 ■ DATABASE ADMINISTRATION

Sun Jun 13 14:18:45 allocating new datafile /var/lib/mongodb/$tmp_repairDatabase_0/admin.1, filling with zeroes... Sun Jun 13 14:18:45 building new index on { _id: 1 } for admin.system.users Sun Jun 13 14:18:45 Buildindex admin.system.users idxNo:0 { name: "_id_", ns: "admin.system.users", key: { _id: 1 } } Sun Jun 13 14:18:45 done for 0 records 0secs Sun Jun 13 14:18:45 done allocating datafile /var/lib/mongodb/$tmp_repairDatabase_0/admin.1, size: 128MB, took 0.052 secs Sun Jun 13 14:18:45 finished checking dbs Sun Jun 13 14:18:45 dbexit: Sun Jun 13 14:18:45 shutdown: going to close listening sockets... Sun Jun 13 14:18:45 shutdown: going to flush oplog... Sun Jun 13 14:18:45 shutdown: going to close sockets... Sun Jun 13 14:18:45 shutdown: waiting for fs preallocator... Sun Jun 13 14:18:45 shutdown: closing all files... Sun Jun 13 14:18:45 closeAllFiles() finished Sun Jun 13 14:18:45 shutdown: removing fs lock... Sun Jun 13 14:18:45 dbexit: really exiting now In the preceding example, the repair() function detected that the admin database was probably created under an older version of MongoDB and that it required an upgrade of its storage format to match the currently running server.

■ Note It is normal for the server to exit after running mongod utility with the --repair option; to bring it back online, just start it up again without specifying the --repair option.

Once the repair process finishes, you should be able to start up the server as normal, and then restore any missing data from your backups. If you’re trying to repair a large database, you may find that your drive runs out of disk space because MongoDB may need to make a temporary copy of the database files on the same drive as the data (see the .../$tmp_repairDatabase_0/.. directory in the preceding example). To overcome this potential issue, the MongoDB repair utility supports an additional command-line parameter called --repairpath. You can use this parameter to specify a drive with enough space to hold the temporary files it creates during the rebuild process, as in the following example: $sudo mongod -f /etc/mongodb.conf --repair --repairpath /mnt/bigdrive/tempdir

Validating a Single Collection Occasionally, you may suspect there is a problem with the data on a running server. In this case, you can use a handful of tools that ship with MongoDB to help you determine whether the server in question is corrupted or damaged. You can use the validate option to validate the contents of a collection in a database. The next example shows how to run the validate option against a collection with one million records in it: $mongo > use blog switched to db blog

218

CHAPTER 9 ■ DATABASE ADMINISTRATION

>db.posts.ensureIndex({Author:1}) > db.posts.validate() { "ns" : "blog.posts", "result" : " validate firstExtent:0:24b00 ns:blog.posts lastExtent:0:24b00 ns:blog.posts # extents:1 datasize?:160 nrecords?:1 lastExtentSize:9984 padding:1 first extent: loc:0:24b00 xnext:null xprev:null nsdiag:blog.posts size:9984 firstRecord:0:24bb0 lastRecord:0:24bb0 1 objects found, nobj:1 176 bytes data w/headers 160 bytes data wout/headers deletedList: 0000000001000000000 deleted: n: 1 size: 9632 nIndexes:2 blog.posts.$_id_ keys:1 blog.posts.$Author_1 keys:1 ", "ok" : 1, "valid" : true, "lastExtentSize" : 9984 } The preceding example takes about 30 seconds to complete. By default, the validate option checks both the datafiles and the indexes, and it provides some statistics about the collection when it completes. The option will tell you if there are any problems with either the datafile or the indexes. You can also use the validate option if you have a very large database and you want only to validate the indexes. Unfortunately, there is no shell helper command for this in the current version (1.6.1). That said, you can easily accomplish this using the runCommand option: $mongo >use TestMongo >db.runCommand({validate:"testIndex", scandata:false}) In this case, the server does not scan the datafiles; instead, it merely reports the information stored about the collection.

Repairing Collection Validation Faults If running validation on your collections turns up an error, then you have several options for repairing the data. Again, it’s impossible to overstress the importance of having good backups.

219

CHAPTER 9 ■ DATABASE ADMINISTRATION

Repairing a Collection’s Indexes If the validation process shows that the indexes are damaged, then you use the reIndex() function to reindex the affected collection. In the example that follows, you use the reIndex() function to reindex the blog’s posts collection to which you added the author index previously: $mongo >use blog > db.posts.reIndex() { "nIndexesWas" : 2, "msg" : "indexes dropped for collection", "ok" : 1, "nIndexes" : 2, "indexes" : [ { "name" : "_id_", "ns" : "blog.posts", "key" : { "_id" : 1 } }, { "_id" : ObjectId("4c63a5d56ffe6e590cfe9b58"), "ns" : "blog.posts", "key" : { "Author" : 1 }, "name" : "Author_1" } ], "ok" : 1 } The MongoDB server will drop all the current indexes on the collection and rebuild them; however, if you use the database repair option, it will also run the reIndex() function on all the collections in the database.

Repairing a Collection’s Datafiles The best way to repair all of the datafiles in a database is to use either the server’s --repair option or the repairDatabase() command in the shell. This latter repairs all the collection files in an individual database, and then reindexes all of the defined indexes. However, repairDatabase() is not a suitable function to run on a live server because it will block any requests to the data while the datafiles are being rebuilt. This results in unindexed queries while the indexes are being reindexed. The following snippet shows the syntax for using the repairDatabase() function: $mongo >use blog >db.repairDatabase() { "ok" : 1 }

220

CHAPTER 9 ■ DATABASE ADMINISTRATION

Upgrading MongoDB Occasionally, 10gen releases versions of MongoDB that require you to upgrade the format of the database files. The guys and gals at 10gen are aware of the impact (including the resulting downtime) caused by running an upgrade on a running production service; however, there are times when the need to support heavily demanded new features requires that an upgrade take place. Upgrading a server is similar to the server repair process discussed earlier. The difference is that you use the --upgrade option instead of the --repair option to make that happen.

■ Warning It is essential that you make a complete backup of your data before attempting any upgrade process.

MongoDB’s developers try to anticipate every possible problem that will present itself during an upgrade; nevertheless, you must take steps to protect yourself as well. Upgrades will typically rewrite every piece of data in your system in a new format, which means that even the slightest problem with the process can have disastrous consequences. The following list walks you through the proper steps required to upgrade a database server: 1.

Back up your data and make sure that the backup is viable. If possible, restore the backup to another server and check whether it’s ok.

2.

Stop or divert your application to another server.

3.

Stop your MongoDB server.

4.

Upgrade the code of your MongoDB server to the desired version.

5.

Run the following command: sudo mongod -f --upgrade.

6.

When the upgrade completes, restart your MongoDB server.

7.

Use the shell to perform some initial sanity checks on the data.

8.

If anything looks suspicious, use the validation tools to check the data.

9.

Re-enable your application when you are satisfied that everything looks OK.

10. Test your application carefully before reopening the service or diverting traffic back to this server.

Monitoring MongoDB The MongoDB distribution contains a simple status-monitoring tool called mongostat. This tool is designed mainly to provide a simple overview of what is happening on your server (see Figure 9–1). The stats produced by this tool are not extensive, but they do provide a good overview on what is going on in your MongoDB installation. For example, this display lets you see how frequently database operations are being performed, the rate at which the index is hit, and how much time your application spends blocked as it waits for locks on the database to be released. The main columns of interest are the first six columns, which show the rate at which the mongod server is handling certain operations (e.g., insert or query). Other columns worth keeping an eye on when diagnosing problems with your installation include the following:

221

CHAPTER 9 ■ DATABASE ADMINISTRATION

•

%idx miss: Shows the percentage of queries that could not use an index. A high value here indicates that you may need to add some indexes to your system.

•

conn: Shows you how many connections are open to your mongod instance. High numbers here indicate that there is a problem with your application releasing connections after operations.

•

% locked: Shows the amount of time the server has had its collections locked. A high number here indicates that you are performing blocking operations that may be best left to a maintenance period.

Figure 9–1. Monitoring the status of MongoDB with the mongostat utility

Rolling Your Own Stat Monitoring Tool Much of the information provided by mongostat is the same information that you can get from the db.serverStatus() call. It would not be a big task to create a service that uses this API to poll the server every few seconds, and then places the results into a MongoDB collection. The application of a few indexes, some carefully crafted queries, and a graphing package would enable you to use such a simple realtime monitor to produce historical logs. There are also many third-party adapters available for MongoDB that let you use common opensource or commercial monitoring systems, including tools such as Nagios, ganglia, and cacti. As mentioned previously, 10gen maintains a page on its website that shares the latest information about monitoring interfaces available for MongoDB (for more information on this topic see www.mongodb.org/display/DOCS/Monitoring+and+Diagnostics).

222

CHAPTER 9 ■ DATABASE ADMINISTRATION

Using the mongod Web Interface Most of the statistical information discussed so far is also available through the mongod web interface. When you start up mongod, by default it will automatically create a web interface on a port number that is 1000 higher than the port number used to access the server. For example, mongod will create a web interface on port 28017 for the default server port of 27017. This port can be accessed through your web browser; if you are running a server on your local machine using the default port, then you need to use the address: http://localhost:28017 to access the web interface (see Figure 9–2).

Figure 9–2. Viewing statistics through a web browser

Summary Keeping your MongoDB installation running smoothly typically requires very little effort. In this chapter, you have seen how you can use the tools provided with the MongoDB distribution to manage and maintain your system, as well as to stay on top of any problems that may develop. However, it must be stressed that the most important lesson to take away from this chapter is this: as the administrator of a database system, your first responsibility is to ensure that a reliable backup and restoration scheme for your data is available.

223

C H A P T E R 10 ■■■

Optimization There is a tongue-in-cheek statement about MongoDB attributed to an unknown Twitter user: “If a MongoDB query runs for longer than 0ms, then something is wrong.” This is typical of the kind of buzz that surrounded the product when it first burst onto the scene about a year ago. The reality is that MongoDB is extraordinarily fast. But if you give it the wrong data structures, or you don’t set up the database or collections with the right indexes, then MongoDB can slow down dramatically, like any data storage system. MongoDB also contains some advanced features, such as grouping and map/reduce, that require some tuning to get them running with optimal efficiency. The design of your data schemas can also have a big impact on performance; in this chapter, we will look at some techniques to shape your data into a form that makes maximum use of MongoDB’s strengths and minimizes its weaknesses. Before we look at improving the performance of the queries being run on the server or the ways of optimizing the structure of the data, we’ll begin with a look at how MongoDB interacts with the hardware it runs on and the factors that affect performance.

Optimizing Your Server Hardware for Performance Often the quickest and cheapest optimization you can make to a database server is to right-size the hardware it runs on. If a database server has too little memory or uses slow drives, it can impact database performance significantly. And, while some of these constraints may be acceptable for a development environment where the server may be running on a developer’s local workstation, they may not be acceptable for production applications, where some care must be used in calculating the correct hardware configuration to achieve the best performance.

Understanding How MongoDB Uses Memory MongoDB uses memory-mapped file I/O to access its underlying data storage. This method of file I/O has some characteristics that you should be aware of because they can affect both the type of operating system (OS) you run it under and the amount of memory you install. The first notable characteristic of memory-mapped files is that, on 32-bit operating systems, the maximum file size that can be managed is 2 GB. This effectively limits the size of a MongoDB database on 32-bit operating systems. If you need to store more than 2 GB, then you should consider installing and running MongoDB on a 64-bit operating system where the limit is greater than 4 TB. The second notable characteristic is that memory-mapped files use the operating system’s virtual memory system to map the required parts of the database files into memory, as it needs them. This can result in the slightly alarming impression that MongoDB will use up your entire RAM. This is not really the case because MongoDB will share the virtual address space with other applications and the operating system, and it will release memory back to the operating system as it is needed. Using the free

225

CHAPTER 10 ■ OPTIMIZATION

memory total as an indicator of excessive memory consumption is not a good practice because a good OS will ensure there is little or no free memory. This is because all of your expensive memory is pressed into good use through caching or buffering disk I/O. Free memory is wasted memory. By providing a suitable amount of memory, MongoDB can keep more of its data mapped into memory, which reduces the need for expensive disk I/O. In general, the more memory you give to MongoDB, the faster it will run. However, if you have a 2 GB database, then adding more than 2–3 GB of memory will not make much difference because the whole database will sit in RAM anyway.

Choosing the Right Database Server Hardware There is a general pressure to move to lower-power (energy) systems for hosting services. However, many of the lower-power servers use laptop or notebook components to achieve the lower power consumption. Unfortunately, lower-quality server hardware can use less expensive disk drives in particular. Such drives are not suited for heavy-duty server applications due to their disks’ low rotation speed, which slows the rate at which data can be transferred to and from the drive. Also, make sure you use a reputable supplier, one that you trust to assemble a system that has been optimized for server operation. If you plan to use replication or any kind of frequent backup system that would have to read across the network connections, then you should consider putting in an extra network card and forming a separate network so the servers can talk with each other. This reduces the amount of data being transmitted and received on the network interface used to connect the application to the server, which also has an effect on an application’s performance.

Evaluating Query Performance MongoDB has two main tools for optimizing query performance: explain() and the MongoDB Profiler (the db profiler). The db profiler is a good tool for finding those queries that are not performing well and selecting candidate queries for further inspection, while explain() is good for investigating a single query, so you can determine how well it is performing. Those of you familiar with MySQL will probably also be familiar with the use of the “slow query log” which helps you find queries that are consuming a lot of time; MongoDB uses the db profiler to provide this capability.

MongoDB Profiler The MongoDB profiler is a tool that records statistical information and execution plan details for every query that meets the trigger criteria. You can enable this tool separately on each database. Once enabled, MongoDB inserts a document with information about the performance and execution details of each query submitted by your application into a special capped collection called system.profile. You can use this collection to inspect the details of each query logged using normal collection querying commands. The system.profile collection is limited to a maximum of 128 KB of data, so that the profiler will not fill the disk with logging information. This limit should be enough to capture a few thousand profiles of even the most complex queries.

226

CHAPTER 10 ■ OPTIMIZATION

■ Warning When the profiler is enabled, it will impact the performance of your server, so it is not a good idea to leave it running on a production server unless you are performing an analysis for some observed issue. Don’t be tempted to leave it running permanently to provide a window on recently executed queries.

Enabling and Disabling the DB Profiler It’s a simple matter to turn on the MongoDB profiler: $mongo >use blog >db.setProfilingLevel(2) It is an equally simple matter to disable the profiler: $mongo >use blog >db.setProfilingLevel(0) MongoDB can also enable the profiler only for queries that exceed a specified execution time. The following example logs only queries that take more than half a second to execute: $mongo >use blog >db.setProfilingLevel(1,500) For profiling level 1, you can supply a maximum query execution time value in milliseconds (ms). If the query runs for longer than this amount of time, it is profiled and logged; otherwise, it is ignored. This provides the same functionality seen in MySQL’s “slow query log.”

Finding Slow Queries A typical record in the system.profile collection looks like this: > db.system.profile.find() { "ts" : "Mon Aug 23 2010 22:06:09 GMT+0800 (PHT)", "info" : "query blog.posts reslen:21342 nscanned:202 nreturned:101 bytes:21326", "millis" : 15 }

\nquery: { Tags: \"even\" }

Each record contains fields, and the following list outlines what they are and what they do: •

ts: Displays a timestamp in UTC that indicates when the query was executed.

•

info: Shows statistical information about the query and its structure.

•

milis: Records the number of milliseconds it took to execute the query.

Because the system.profile collection is just a normal collection, you can use MongoDB’s query tools to home in quickly on problematic queries.

227

CHAPTER 10 ■ OPTIMIZATION

The next example finds all the queries that are taking longer than 10ms to execute. In this case, you can just query for cases where millis >10 in the system.profile collection, and then sort the results by the execution time in descending order: > db.system.profile.find({millis:{$gt:10}}).sort({millis:-1}) { "ts" : "Mon Aug 23 2010 22:06:09 GMT+0800 (PHT)", "info" : "query blog.posts reslen:21342 nscanned:202 \nquery: { Tags: \"even\" } nreturned:101 bytes:21326", "millis" : 15 } If you also know your problem occurred during a specific time range, then you can use the ts field to add query terms that restrict the range to the required slice.

Analyzing a Specific Query with explain() If you suspect a query is not performing as well as expected, then you can use the explain() modifier to break down how MongoDB is executing the query. When you add the explain() modifier to a query, MongoDB executes the query, returning a document that describes how the query was handled, rather than a cursor to the results. The following query runs against a database of blog posts and indicates that the query had to scan 13325 records to form a cursor to return all the posts: $mongo >use blog > db.posts.find().explain() { "cursor" : "BasicCursor", "indexBounds" : [ ], "nscanned" : 13235, "nscannedObjects" : 13235, "n" : 13235, "millis" : 3, "allPlans" : [ { "cursor" : "BasicCursor", "indexBounds" : [ ] } ] } You can see the fields returned by explain() listed in Table 10–1. Table 10–1. Elements Returned by explain()

228

Element

Description

Cursor

Indicates the type of cursor created to enumerate the results. Typically, this is BasicCursor, a natural order reading cursor.

indexBounds

Indicates the min/max values used on an indexed lookup.

nScanned

Indicates the number of references scanned to find all the objects in the query.

CHAPTER 10 ■ OPTIMIZATION

Element

Description

nScannedObjects

Indicates the number of Physical objects that “nScanned” resolves to.

n

Indicates the number of items on the cursor (i.e., the number of items to be returned).

millis

Indicates the number of milliseconds it took to execute the query.

allPlans

Indicates the number of alternative methods the optimizer could use to execute this query. If there are multiple indexes available that would satisfy the query, then those plans would be shown here. This information is useful if you want to use the hint() modifier to move the query to a different plan than the one currently selected.

Using Profile and explain() to Optimize a Query Now let’s walk through a real-world optimization scenario and look at how we can use MongoDB’s profile and explain() modifier tools to fix a problem with a real application. The example discussed in this chapter is based on the blog database from Chapter 8. This database has a function to get the posts associated with a particular tag; this chapter will use the “even” tag. Let’s assume that you have noticed this function runs slowly, so you want to determine whether there is a problem. Let’s begin by writing a little program to fill the aforementioned database with data so that we have something to run queries against, to demonstrate the optimiazaiton process. blog; // First let’s get the first AuthorsID // We are going to use this to fake a author $author = $db->authors->findOne(); if(!$author){ die("There are no authors in the database"); } for( $i = 1; $i < 10000; $i++){ $blogpost=array(); $blogpost['author'] = $author['_id']; $blogpost['Title'] = "Completely fake blogpost number {$i}"; $blogpost['Message'] = "Some fake text to create a database of blog posts"; $blogpost['Tags'] = array(); if($i%2){

229

CHAPTER 10 ■ OPTIMIZATION

// Odd numbered blogs $blogpost['Tags'] = array("blog", "post", "odd", "tag{$i}"); } else { // Even numbered blogs $blogpost['Tags'] = array("blog", "post", "even", "tag{$i}"); } $db->posts->insert($blogpost); } ?> The preceding program finds the first author in the blog database’s authors collection, and then pretends that the author has been extraordinarily productive. It creates 10,000 fake blog postings in the author’s name, all in the blink of an eye. The posts are not very interesting to read; nevertheless, they are alternatively assigned “odd” and “even” tags. These tags will serve to demonstrate how to optimize a simple query. The next step is to save the program as fastblogger.php and then run it using the command-line PHP tool: $php fastblogger.php Next, you need to enable the database profiler, which you will use to determine whether you can improve the example’s queries: $ mongo > use blog switched to db blog > show collections authors posts ... system.profile tagcloud ... users > db.setProfilingLevel(2) { "was" : 0, "ok" : 1 } Now wait a few moments for the command to take effect, open the required collections, and then perform its other tasks. Next, you want to simulate having the blog website access all of the blog posts with the even tag. Do so by executing a query that the site can use to implement this function: $Mongo use blog $db.posts.find({Tags:”even”}) ... If you query the profiler collection for results that exceed 5ms, then you should see something like this: >db.system.profile.find({millis:{$gt:5}}).sort({millis:-1}) {"ts" : "Mon Aug 23 2010 22:06:09 GMT+0800 (PHT)", "info" : "query blog.posts reslen:21342 nscanned:202 \nquery: { Tags: \"even\" } nreturned:101 bytes:21326", "millis" : 15 } ... The results returned in the preceding example show that some queries are taking longer than 0ms (remember the quote at the beginning of the chapter).

230

CHAPTER 10 ■ OPTIMIZATION

Next, you want to reconstruct the query for the first (and worst performing) query, so you can see what is being returned. The preceding output indicates that the poor performing query is querying blog.posts and that the query term is {Tags:"even"}. Finally, you can see that this query is taking a whopping 15ms to execute. The reconstructed query looks like this: >db.posts.find({Tags:"even"}) { "_id" : ObjectId("4c727cbd91a01b2a14010000"), "author" : ObjectId("4c637ec8b8642fea02000000"), "Title" : "Completly fake : "Some fake text to create a database of blog posts", "Tags" : "tag2" ] } { "_id" : ObjectId("4c727cbd91a01b2a14030000"), "author" : ObjectId("4c637ec8b8642fea02000000"), "Title" : "Completly fake : "Some fake text to create a database of blog posts", "Tags" : "tag4" ] } { "_id" : ObjectId("4c727cbd91a01b2a14050000"), "author" : ObjectId("4c637ec8b8642fea02000000"), "Title" : "Completly fake : "Some fake text to create a database of blog posts", "Tags" : "tag6" ] } ...

blogpost number 2", "Message" [ "blog", "post", "even", blogpost number 4", "Message" [ "blog", "post", "even", blogpost number 6", "Message" [ "blog", "post", "even",

The preceding output should come as no surprise; this query was created for the expressed purpose of demonstrating how to find and fix a slow query. The goal is to figure how to make the query run faster, so use the explain() command to determine how MongoDB is performing this query: > db.posts.find({Tags:"even"}).explain() { "cursor" : "BasicCursor", "nscanned" : 9999, "nscannedObjects" : 9999, "n" : 4999, "millis" : 14, "indexBounds" : { } } You can see from the preceding output that the query is not using any indexes. The explain() command shows that the query is using a “BasicCursor”, which means the query is just performing a simple scan of the collection’s records. Specifically, it’s scanning all of the records in the database one by one to find the tags (all 9999 of them); this process takes 14ms. This may not sound like a very long time, but, if you were to use this query on a popular page of your website, then it would be causing additional load to the disk I/O, as well as a tremendous amount of stress on the web server. Consequently, this query would cause the connection to the web browser to remain open for longer while the page is being created.

■ Note If you see a detailed query explanation that shows a significantly larger number of scanned records (nscanned) than it returns (n), then that query is probably a candidate for indexing.

231

CHAPTER 10 ■ OPTIMIZATION

The next step is to determine whether adding an index on the “Tags” field improves the query’s performance: > db.posts.ensureIndex({Tags:1}) Now run the explain() command again to see the effect of adding the index: > db.posts.find({Tags:"even"}).explain() { "cursor" : "BtreeCursor Tags_1", "nscanned" : 4999, "nscannedObjects" : 4999, "n" : 4999, "millis" : 4, "indexBounds" : { "Tags" : [ [ "even", "even" ] ] } } > The performance of the query has improved significantly. You can see that the query is now using a BtreeCursor driven by the Tags_1 index. The number of scanned records has been reduced from 9999 records to the same 4999 records you expect the query to return, while the execution time has dropped to 4ms.

■ Note The most common index type used by MongoDB is the btree (binary-tree type). A BtreeCursor is a MongoDB data cursor that uses the binary tree index to navigate from document to document. Btree indexes are very common in database systems because they provide fast inserts and deletes, yet also provide a reasonable performance when used to walk or sort data.

Managing Indexes You’ve now seen how much of an impact the introduction of carefully selected indexes can have. As you learned in Chapter 3, MongoDB’s indexes are used for both queries (find, findOne) and sorts. If you intend to use a lot of sorts on your collection, then you should add indexes that correspond to your sort specifications. If you use sort() on a collection where there are no indexes for the fields in the sort specification, then you may get an error message if you exceed the maximum size of the internal sort buffer. So it is a good idea to create indexes for sorts. In the following sections, we’ll touch again on the basics, but also add some details that relate to how to manage and manipulate the indexes in your system. We will also cover how such indexes relate to some of the samples. Each time you add an index to a collection, MongoDB must maintain it and update it every time you perform any write operation (e.g., updates, inserts, or deletes). If you have too many indexes on a collection, it can cause a negative impact on write performance.

232

CHAPTER 10 ■ OPTIMIZATION

Indexes are best used on collections where the majority of access is read access. For write-heavy collections such as those used in logging systems, introducing an index would reduce the peak documents per second that could be streamed into the collection.

■ Warning At this time, you can have a maximum of 40 indexes per collection.

Listing Indexes MongoDB maintains a special collection called system.indexes inside each database. This collection keeps track of all the indexes that have been created on all the collections in the databases, including which fields or elements they refer to. The system.indexes collection is just like any normal collection. This means you can list its contents, run queries against it, and otherwise perform the usual tasks you can accomplish with a typical collection. The following example lists the indexes in your simple database: $mongo >use blog >db.system.indexes.find() { "name" : "_id_", "ns" : "blog.posts", "key" : { "_id" : 1 } } { "name" : "_id_", "ns" : "blog.authors", "key" : { "_id" : 1 } } The blog database does not have any user-defined indexes, but you can see the two indexes created automatically for the _id field on your two collections: posts and authors. You don’t have to do anything to create or delete these identity indexes; MongoDB creates and drops them whenever a collection is created or removed. When you define an index on an element, MongoDB will construct an internal btree index, which it will use to locate documents efficiently. If no suitable index can be found, then MongoDB will scan all the documents in the collection to find the records that will satisfy the query.

Creating a Simple Index MongoDB provides the ensureIndex() function for adding new indexes to a collection. This function begins by checking whether an index has already been created with the same specification. If it has, then ensureIndex() just returns that index. This means you can call ensureIndex() as many times as you like, but it won’t result in a lot of extra indexes being created for your collection. The following example defines a simple index: $mongo >use blog >db.posts.ensureIndex({Tags:1}) This preceding example creates a simple ascending btree index on the Tags field. Creating a descending index instead would require only small change: >db.posts.ensureIndex({Tags:-1}) To index a field in an embedded document, you can use the normal dot notation addressing scheme; that is, if you have a count field that is inside a comments subdocument, then you can use this syntax to index it:

233

CHAPTER 10 ■ OPTIMIZATION

>db.posts.ensureIndex({"comments.count":1}) If you specify a document field, which is an array type, then the index will include all the elements of the array as separate index terms. This is known as a multi-key index, and each document is linked to multiple values in the index. MongoDB has a special operator for performing queries where you wish to select only documents that have all of the terms you supply. In the blog database example, you have a posts collection with an element called Tags. This element has all the tags associated with the posting inside it. The following query finds all articles that have both the sailor and moon tags: >db.posts.find({Tags:{$all: ['sailor', 'moon']}}) Without a multi-key index on the Tags field, the query engine would have to scan each document in the collection to see whether either term existed and, if so, to check whether both terms were present.

Creating a Compound Index It may be tempting to simply create a separate index for each field mentioned in any of your queries. While this may speed up queries without having to apply too much thought, it would unfortunately have a significant impact on adding and removing data from your database, as these indexes need to be updated each time. Compound indexes provide a good way to keep down the number of indexes you have on a collection, allowing you to combine multiple fields into a single index, so you should try to use compound indexes wherever possible. There are two main types of compound indexes: subdocument indexes and manually defined compound indexes. MongoDB has some rules that allow it to use compound indexes for queries that do not use all of the component keys. Understanding these rules enables you to construct a set of compound indexes that cover all of the queries you wish to perform against the collection, without having to individually index each element (thereby avoiding the attendant impact on insert/update performance mentioned earlier). One area where compound indexes may not be useful is when using the index in a sort. Sorting is not good at using the compound index unless the list of terms and sort directions exactly matches the index structure. In this case, individual simple indexes on each of the term fields may be a better choice.

Creating Subdocument Compound Indexes You can create a compound index using an entire subdocument; in this case, MongoDB will create a compound index where each element in the embedded document becomes part of the index. For example, assume you have an author subdocument with name and email elements inside it. The following snippet creates a compound index with two key terms in it: author.name and author.email: >db.posts.ensureIndex({Author:1}) The preceding serves as a nice shortcut for creating compound indexes. However, you lose the ability to set the order of the keys in the index in this example because you can’t set the direction of each.

Constructing a Compound Index Manually When you use a subdocument as an index key, the order of the elements used to build the multi-key index matches the order in which they appear in the subdocument’s internal BSON representation. In many cases, this does not give you sufficient control over the process of creating the index.

234

CHAPTER 10 ■ OPTIMIZATION

To get around this while guaranteeing that the query uses an index constructed in the desired fashion, you need to make sure you use the same subdocument structure to create the index that you used when forming the query, as in the following example: >db.articles.find({author:{name: 'joe', email: '[email protected]'})) You can also create a compound index explicitly by naming all the fields that you wish to combine in the index, and then specifying the order to combine them in. The following example illustrates how to construct a compound index manually: >db.posts.ensureIndex({"author.name":1, "author.email":1})

Specifying Index Options You can specify several interesting options when creating an index, such as creating unique indexes or enabling background indexing (you’ll learn more about these options in the upcoming sections). You specify these options as additional parameters to the ensureIndex() function, as in the following example: >db.posts.ensureIndex({author:1}, {option1:true, option2:true, ... })

Creating an Index in the Background with {background:true} When you instruct MongoDB to create an index by using the ensureIndex() function for the first time, the server must read all the data in the collection and create the specified index. By default, this is done in the foreground and all operations on the specified collection block until the index operation completes. MongoDB also includes a feature that allows indexes to be created in the background. Operations by other connections on that collection are not blocked while that index is being built. No queries will use the index until it has been built, but the server will allow read and write operations to continue. Once the index operation has been completed, all queries that require the index will immediately start using it.

■ Note The index will be built in the background. However, the connection that initiates the request will block, if you issue the command from the MongoDB shell. A command issued on this connection won’t return until the indexing operation is complete. At first sight, this appears to contradict the idea that it is a background index. However, if you have another MongoDB shell open at the same time, you will find that queries and updates on that collection run unhindered while the index build is in progress. It is only the initiating connection that is blocked. This differs from the non-background behavior you see with a simple ensureIndex() command, where operations on the second MongoDB shell would also be blocked.

Killing the Indexing Process You can also kill the current indexing process if you think it has hung or is otherwise taking too long. You can do this by invoking the killOp() function:

235

CHAPTER 10 ■ OPTIMIZATION

> db.killOp()

■ Note When invoking the killOp() command, the partial index will also be removed again. This prevents broken or irrelevant data from building up in the database.

Creating an Index with a Unique Key {unique:true} When you specify the unique option, MongoDB creates an index where all the keys must be different. This means that MongoDB will return an error if you try to insert a document where the index key matches the key of an existing document. This is useful for a field where you want to ensure that no two people can have the same identity (e.g., userid). However, if you want to add a unique index to an existing collection that is already populated with data, then you must make sure that you have deduped the key(s). In this case, your attempt to create the index will fail if any two keys are not unique. The unique option works for simple and compound indexes, but not for multi-key value indexes, where they wouldn’t make much sense. If a document is inserted with a field missing that is specified as a unique key, then MongoDB will automatically insert the field, but will set its value to null. This means that you can only insert one document with a missing key field into such a collection; any additional null values would mean the key is not unique, as required.

Dropping Duplicates Automatically with {dropdups:true} If you want to create a unique index for a field where you know that duplicate keys exist, then you can specify the dropdups option. This option instructs MongoDB to remove documents that would cause the index creation to fail. In this case, MongoDB will retain the first document it finds in its natural ordering of the collection, but then drop any other documents that would result in an index-constraint violation. You need to be very careful when using this option because it will result in documents being deleted from your collection. You should be extremely aware of the data in your collection before using this option; otherwise, you might get unexpected (not to mention unwanted) behavior. It is an extremely good idea to run a group query against a collection for which you intend to make a key unique; this will enable you to determine the number of documents that would be regarded as duplicates before you executed this option.

Dropping an Index You can elect to drop all indexes or just one specific index from a collection. Use the following function to remove all indexes from a collection: >db.posts.dropIndexes() To remove a single index from a collection, you use syntax that mirrors the syntax used to create the index with ensureIndex(): >db.posts.dropIndex({"author.name":1, "author.email":1});

236

CHAPTER 10 ■ OPTIMIZATION

Re-Indexing a Collection If you suspect that the indexes in a collection are damaged—for example, if you’re getting inconsistent results to your queries—then you can force a re-indexing of the affected collection. This will force MongoDB to drop and re-create all the indexes on the specified collection (see Chapter 9 for more information on how to detect and solve problems with your indexes), as in the following example: > db.posts.reIndex() { "nIndexesWas" : 2, "msg" : "indexes dropped for collection", "ok" : 1, "nIndexes" : 2, "indexes" : [ { "name" : "_id_", "ns" : "blog.posts", "key" : { "_id" : 1 } }, { "_id" : ObjectId("4c7282c6d36e34765e2e6741"), "ns" : "blog.posts", "key" : { "Tags" : 1 }, "name" : "Tags_1" } , "ok" : 1 } The preceding output lists all the indexes the command has rebuilt, including the keys. Also, the nIndexWas: field shows how many indexes existed before running the command, while the nIndex: field gives the total number of indexes after the command has completed. If the two values are not the same, the implication is that there was a problem re-creating some of the indexes in the collection.

How MongoDB Selects Which Indexes It Will Use When a database system needs to run a query, it has to assemble a query plan, which is a list of steps it must run to perform the query. Each query may have multiple query plans that could produce the same result equally well. However, each plan may have elements in it that are more expensive to execute than others. For example, a scan of all the records in a collection is an expensive operation, and any plan that incorporates such an approach could be slow. These plans can also include alternative lists of indexes to use for query and sort operations. Road directions serve as a good illustration of this concept. If you want to get to the diagonally opposite side of a block from one corner, then “take a left, then take a right” and “take a right, then take a left” are equally valid plans for getting to the opposite corner. However, if one of the routes has two stop signs and the other has none, then the former approach is a more expensive plan, while the latter

237

CHAPTER 10 ■ OPTIMIZATION

approach is the best plan to use. Collection scans would qualify as potential stop signs when executing your queries. MongoDB ships with a component called the query analyzer. This component takes a query and the collection the query targets. The component then produces a set of plans for MongoDB to execute. The explain() function described earlier in this chapter lists both the plan used and the set of alternative plans produced for a given query. MongoDB also ships with a query optimizer component. The job of this component is to select which execution plan is best suited to run a particular query. In most relational database systems, a query optimizer uses statistics about the distribution of keys in a table, the number of records, the available indexes, the effectiveness of previous choices, and an assortment of weighting factors to calculate the cost of each approach. It then selects the least expensive plan to use. The query optimizer in MongoDB is simultaneously dumber and smarter than the typical RDBMS query analyzer. For example, it does not use a cost-based method of selecting an execution plan; instead, it runs all of them in parallel and uses the one that returns the results fastest, terminating all the others after the winner crosses the finish line. Thus, the query analyzer in MongoDB uses a simple mechanism (dumb) to ensure that it always gets the fastest result (smart).

Using Hint() to Force Using a Specific Index The query optimizer in MongoDB selects the best index from the set of available indexes for a collection. It uses the methods just outlined to try to match the best index or set of indexes to a given query. However, there may be cases where the query optimizer does not make the correct choice, in which case it may be necessary to give the component a helping hand. You can provide a hint to the query optimizer in such cases, nudging the component into making a better choice. For example, if you have used explain() to show which indexes are being used by your query, and you think you would like it to use a different index for a given query, then you can force the query optimizer to do so. Let’s look at an example. Assume that you have an index on a subdocument called author with name and email fields inside it. Also assume that you have the following defined index: >db.posts.ensureIndex({Author:1}) You can use the following hint to force the query optimizer to use the preceding defined index: >db.posts.find({author:{name:'joe', email: '[email protected]'}}).hint({Author:1}) If for some reason you want to force a query to use no indexes, that is, if you want to use collection document scanning as the means of selecting records, then you can use the following hint to do this: >db.posts.find({author:{name: 'joe', email: '[email protected]'}}).hint({$natural:1})

Optimizing the Storage of Small Objects Indexes are the key to speeding up data queries. But another factor that can affect the performance of your application is the size of the data it accesses. Unlike database systems with fixed schemas, MongoDB stores all the schema data for each record inside the record itself. Thus, for large records with large data contents per field, the ratio of schema data to record data is low; however, for small records with small data values, this ratio can grow surprisingly large. Consider a common problem in one of the application types that MongoDB is well suited for: logging. MongoDB’s extraordinary write rate makes streaming events as small documents into a collection very efficient. However, if you want to optimize further the speed at which you can perform this functionality, you can do a couple of things.

238

CHAPTER 10 ■ OPTIMIZATION

First, you can consider batching your inserts. MongoDB ships with a multiple document insert() call. You can use this call to place several events into a collection at the same time. This results in fewer round-trips through the database interface API. Second (and more importantly), you can reduce the size of your field names. If you have smaller field names, then MongoDB can pack more event records into memory before it has to flush them out to disk. This makes the whole system more efficient. For example, assume you have a collection that is used to log three fields: a time stamp, a counter, and a four-character string used to indicate the source of the data. The total storage size of your data is shown in Table 10–2. Table 10–2. The Logging Example Collection Storage Size

Field

Size

Timestamp

8 bytes

Integer

4 bytes

string

4 bytes

Total

16 bytes

If you use ts, n, and src for the field names, then the total size of the field names is 6 bytes. This is a relatively small value compared to the data size. But now assume you decided to name the fields WhenTheEventHappened, NumberOfEvents, and SourceOfEvents. In this case, the total size of the field names is 48 bytes, or three times the size of the data itself. If you wrote 1 TB of data into a collection, then you would be storing 750 GB of field names, but only 250 GB of actual data. This does more than waste disk space. It also affects all other aspects of the system’s performance, including the index size, data transfer time, and (probably more importantly) the use of precious system RAM to cache the data files. Another recommendation for logging applications is that you need to avoid adding indexes on your collections when writing records; as explained earlier, indexes take time and resources to maintain. Instead, you should add the index immediately before you start analyzing the data. Also, you should consider using a schema that splits the event stream into multiple collections. For example, you might write each day’s events into a separate collection. Smaller collections take less time to index and analyze.

Summary In this chapter, we looked at some tools for tracking down slow performance in MongoDB queries, as well as potential solutions for speeding up the slow queries that surface as a result. We also looked at some of the ways for optimizing data storage. For example, we looked at ways to ensure that we are making full use of the resources available to the MongoDB server. The specific techniques described in this chapter enable you to optimize your data and tune the MongoDB system it is stored in. The best approach to take will vary from application to application, and it will be dependent on a lot of factors, including the application type, data access patterns, read/write ratios, and so on.

239

C H A P T E R 11 ■■■

Replication Like many of its relational cousins, MongoDB supports the replication of a database’s contents to another server in realtime or near realtime. MongoDB’s replication features are simple to set up and use. They are also one of the key features in MongoDB that bolster the claim that the database is both a Web 2.0 and a cloud-based datastore. There are many scenarios where you might want to use replication, so the replication support in MongoDB has to be sufficiently flexible that it can cope with all of them. 10gen, the company that publishes MongoDB, has gone to great lengths to make sure that its implementation of replication meets all of today’s needs. In this chapter, we will cover the following replication topologies that MongoDB supports: •

Single master/single slave replication

•

Single master/multiple slave replication

•

Multiple master/single slave replication

•

Cascade replication

•

Master/master replication

•

Interleaved replication

•

Replica pairs (legacy MongoDB clustering)

•

Replica sets (advanced MongoDB clustering)

■ Note Replication is a feature that continues to evolve in MongoDB, and you can expect some changes to occur in how replication works as the product develops. This is particularly true with respect to the clustering of database servers. 10gen is investing considerable effort in ensuring that MongoDB meets and exceeds everybody’s expectations for scalability and availability; replication support is one of the key features that 10gen is counting on to help it meet those expectations.

Before looking at the various replication setups in detail, let’s review the goals the various setups are designed to achieve. We’ll also look at the oplog and how it fits in with the communication between members of a replication topology. These two topics will provide the context for the rest of the chapter.

241

CHAPTER 11 ■ REPLICATION

Spelling Out MongoDB’s Replication Goals Among other things, replication can be used to achieve scalability, durability/reliability, and isolation. In the upcoming sections, we’ll explore how you can use replication to achieve these goals along the way, while pointing out potential traps and mistakes you want to avoid.

Improving Scalability For web applications in particular, scalability is a critical design requirement, especially those that rely heavily on backend databases. Replication can help you create more scalable applications in two ways: •

Improve performance: Replication can help you improve an application’s raw performance. This is particularly true for cases where you have a large web application with a predominantly read-based dataset, and you want to distribute queries to multiple database servers to increase parallelism.

•

Improve redundancy: Replication can help you improve redundancy by enabling you to host an application in several data centers. In this approach, you ensure that there is a local copy of the data in each data center, so that the application can have high-speed access to it. Users can then be connected to the data center that is closest to them, minimizing latency.

■ Note MongoDB also supports another feature called sharding that is designed to assist you in creating more scalable applications with or without replication (see Chapter 12 for more information on using sharding and replication together in MongoDB).

Improving Durability/Reliability Replication is commonly used to help guard against hardware failure or database corruption. Some specific examples where people use replication in this manner include the following:

242

•

When you want to have a duplicate of your database that is no more than 30 minutes old. You may want to protect yourself against flaws in your application or provide a simple mechanism to provide trend information by highlighting the differences between the results of queries against both datasets.

•

When you want to have an aggregate data server: You may want to maintain copies of data from multiple master servers, to save the cost of having dedicated slaves for each.

•

When you have a large amount of data that changes rapidly: You may want to run a replica as a backup in cases where normal backup schemes would take too long to make or restore the backup in the event of a system failure.

CHAPTER 11 ■ REPLICATION

Providing Isolation There are some processes that, if run against the production database, would significantly impact that database’s performance or availability. You can use replication to create synchronized copies that isolate processes from the production database, for example: •

When you want to run reports or backups without impacting the performance of your production system: Maintaining a read-only slave replica enables you to isolate queries from your reporting system and make sure that the end-of-month reports don’t bring your site to its knees.

•

When you have a development or staging system that you want to provide access to live data: Isolation enables you to provide this access without running the risk that a flaw or bug in a development version of your code will cause problems with your production system.

Drilling Down on the Oplog In simple terms, the oplog is a journal of all the changes that a master instance makes to its databases for the purpose of replaying those changes on a slave to ensure that the two databases are identical. The master server maintains the oplog, and the slaves query the master for new entries to apply to their own copies of the replicated databases. The oplog creates a timestamp for each entry. This enables a slave to track how far it has read from the oplog during a previous read, and what entries it needs to transfer to catch up. If you stop a slave and restart it at a later time, it will use the master’s oplog to retrieve all the changes it has missed while it has been offline. Because it is not practical to have an infinitely large oplog, the oplog is limited or capped at a particular size. You can think of the oplog as a window on the recent activity of your master instance; if that window is too small, then operations will be lost from the oplog before they can be applied to the slave. The oplogSize option allows you to set the size of your oplog in MB. For an average 64-bit system with average read/write patterns, the oplogSize should be set to at least five percent of the disk space available for data storage. If your system is write intensive, then you may need to increase this size to ensure that slaves can be offline for a reasonable amount of time without losing data. For example, if you have a daily backup that is taken from the slave and it takes an hour to complete, then the size of the oplog will have to be set to allow the slave to stay offline for that hour plus an additional amount of time to provide a safety margin.

■ Note If you are hosting multiple databases on the master server, but only replicating one database to the slave, then the master instance still places all update operations for all its databases into the oplog, even if the slave is not reading them all. You have to remember this when determining the optimum size of your oplog.

It’s critical that you take into account the update rate on all the databases present on the master when calculating a suitable size for the oplog. You can get some idea about a suitable size for your oplog by using the printReplicationInfo() command that runs on the master instance:

243

CHAPTER 11 ■ REPLICATION

$mongo >db.printReplicationInfo() configured oplog size: log length start to end: oplog first event time: oplog last event time: now:

192MB 12365secs (3.43hrs) Tue Jul 06 2010 21:11:02 GMT+0800 (PHT) Wed Jul 07 2010 00:37:07 GMT+0800 (PHT) Wed Jul 07 2010 00:37:08 GMT+0800 (PHT)

This preceding command shows you the current size of your oplog, as well as the amount of time it will take to fill up at the current update rate. From this, you can estimate whether you need to increase or decrease the size of your oplog.

Implementing Single Master/Single Slave Replication Single master/single slave replication is the simplest form of the replication/clustering configurations that MongoDB provides. It consists of only a single master instance and a single slave instance. This mode of replication does not provide any automatic failover capabilities, and the application is normally set to interact with the master database only. Should the master instance fail, then the application must be reconfigured to attach to the slave instance instead. This operating mode is typically used to isolate potentially long-running operations such as creating a backup or generating a substantial report. This mode enables you to perform those activities without significantly impacting the performance of a running application, while still providing a degree of redundancy in the case of server failure. The characteristics of this replication model are listed in Table 11–1. Table 11–1. Characteristics of Single Master/Single Slave Replication System

Property

Description

Number of masters

Single, fixed, role does not change

Number of slaves

Single, fixed, role does not change

Failover Mode

Manual

Figure 11–1 shows a typical master/slave replication setup.

244

CHAPTER 11 ■ REPLICATION

Figure 11–1. Setting up single master/single slave replication

Setting Up a Master/Slave Replication Configuration It’s easy to set up two instances of MongoDB in a single master/single slave replication mode. You can do this either on separate machines in a production environment or as two instances on the same machine in a testing and development environment. If you run two instances on the same computer, then you will need to ensure that they are using separate dbpaths and network ports (see Table 11–2). In this first example, you will learn how to set up two instances on the same server. However, the principles are the same for setting up master/slave replication on two different machines, but there is no need to ensure that there are different dbpaths and port numbers. Table 11–2. MongoDB Instances Used in a Single Master/Single Slave Example

Service

Daemon

Address

Dbpath

Master instance

mongod

localhost:27017

/db/master/data

Slave instance

mongod

localhost:27018

/db/slave/data

You should begin by setting up the master server. You need to use --master option to let the mongod service know what its role will be in this replication setup; this option also instructs mongod to start recording any write, update, and delete commands into its oplog. Open a new terminal window and type the following: $sudo mkdir –p /db/master/data $sudo mongod --master --dbpath /db/master/data Be sure to leave the terminal window open as you test out the replication. This will enable you to see the console messages that MongoDB uses to tell you what is going on. If you close the terminal window, then MongoDB will terminate. The preceding snippet will start a copy of MongoDB running on the default port (27017); this instance will use the /db/master/data directory to store its databases and collections. Next, you need to set up the slave server, open another terminal window and type:

245

CHAPTER 11 ■ REPLICATION

$sudo mkdir –p /db/slave/data $sudo mongod --slave --source localhost:27017 --dbpath /db/slave/data –port 27018 The --slave command-line option instructs this new instance that it is to take on the role of a slave. The --source option lets this instance know the address of the master it should synchronize its data from. The new instance will start and perform an initial synchronization of its databases with the master database, and then start tracking changes by polling the master’s oplog. At this point, you have two instances of MongoDB running: the master is running on the default port of 27017, while the slave is running on port 27018. Now, let’s check whether it’s working. Begin by connecting to the master: $ mongo > show dbs admin local > use local switched to db local > show collections oplog.$main slaves system.indexes > This example shows that MongoDB has created a special database in the master called local; this database has two special collections called oplog.$main and slaves, the initial contents of the slaves collection for our example master/slave setup is shown below. > db.slaves.find() { "_id" : ObjectId("4c332bbae1edee6634b251e5"), "host" : "localhost", "ns" : "local.oplog.$main", "syncedTo" : { "t" : 1278422286000, "i" : 1 } } >

■ Note If you do not see a slaves collection, then check the version number of your MongoDB system; this collection was added as part of the general migration to Replication Sets in Version 1.6.x; versions prior to this may not show this collection.

The slaves collection holds a set of documents describing each slave that is connected to the master. The document contains the name of the second special collection, local.oplog.$main (the master’s oplog), as well as a timestamp and a position indicator that shows the point in the oplog that the slave has read up to.

Examining the Slave Now let’s take a moment to poke around in the slave and see what information it contains: $mongo --port 27018 > show dbs admin

246

CHAPTER 11 ■ REPLICATION

local > use local switched to db local > show collections me pair.sync sources system.indexes Like the master, the slave has a local database that contains all the collections it uses to maintain its replication state. Let’s take a couple minutes to examine some of these collections. The first collection of interest is the me collection, which is shown here: > db.me.find(); { "_id" : ObjectId("4c332bbae1edee6634b251e5") }

■ Note If you do not see a me collection, then check the version number of your MongoDB system. This collection was also added as part of the general migration to Replication Sets in Version 1.6; versions prior to this may not show this collection.

The me collection contains a single document with no fields except for ObjectId. Eagle-eyed readers will have noticed that this ID is identical to the ID shown in the corresponding slaves collection document that describes this slave on the master machine. This ID is the identity record for the slave, and it is used to find the slave’s replication status on the master server: > db.sources.find(); { "_id" : ObjectId("4c332bbae1edee6634b251e4"), "host" : "localhost:27017", "source" : "main", "syncedTo" : { "t" : 1278423076000, "i" : 1 }, "localLogTs" : { "t" : 0, "i" : 0 } } The preceding example shows the sources collection maintained by the slave. This collection contains a set of documents that describes each of the masters the slave is connected to. Notice the synchedTo: element is the same as the corresponding slave document element of the same name in the master database’s slaves collection. In our example, because there is only one master, there is only one document in the collection. It is possible to specify the master that a slave is attached to by directly inserting the master’s details into the sources collection, this is the method used to add multiple masters to a multiple master/slave configuration (you will learn more about such configurations later in this chapter). Finally, there is the pair.sync collection that contains the status of the relationship between the two machines. This collection’s initialsynccomplete element is set to 1, which indicates that the two machines have synchronized their initial contents. From now on, the slave will track changes on the master by periodically reading the master’s oplog: > db.pair.sync.find() { "_id" : ObjectId("4c332bbfe1edee6634b251e6"), "initialsynccomplete" : 1 } You can prove the system is working by changing some data on the master, as shown in the following example:

247

CHAPTER 11 ■ REPLICATION

$ mongo > use testdb switched to db testdb > db.testcollection.insert({name:'tim',surname:'hawkins'}) > db.testcollection.find() { "_id" : ObjectId("4c33344c55ca4b3534b1d930"), "name" : "tim", "surname" : "hawkins" } Now you can inspect the slave to see whether the change has been made there as well: $ mongo --port 27018 > show dbs admin local testdb > use testdb switched to db testdb > show collections system.indexes testcollection > db.testcollection.find() { "_id" : ObjectId("4c33344c55ca4b3534b1d930"), "name" : "tim", "surname" : "hawkins" } In this case, your change has been replicated from the master to the slave, indicating that you’ve set up a master/slave replication successfully—gold stars all around! As the preceding example illustrates, setting up simple replication is extremely easy on MongoDB. In its basic form, replication requires just a few commands to get it to work, which makes it considerably easier to implement in MongoDB than the relational database systems that MongoDB is often replacing.

Implementing Single Master/Multiple Slave Replication So far, you’ve learned how to set up a single master/single slave configuration; however, MongoDB also supports multiple slaves on a single master, and this configuration is just as easy to implement as the configuration you just learned about. To add another slave, repeat the process you used to add the first slave to the master, as in the following example: $sudo mkdir –p /db/slave2/data $sudo mongod --slave --source localhost:27017 --dbpath /db/slave2/data –port 27019 This preceding snippet will add a second slave machine to the master/slave topology you configured previously, making the second slave available on localhost:27019. Again, you can attach to the new slave and confirm that your testdb/testcollection has been replicated to the new slave.

Configuring a Master/Slave Replication System The simple replication set up shown previously will suffice for a large number of needs. However, MongoDB also includes a number of initialization options that allow you to tune the replication behavior to meet your needs more precisely (see Table 11–3). Before exploring some of the more complex replication topologies, let’s take a minute to examine some of the available configuration parameters, which are common to most setups.

248

CHAPTER 11 ■ REPLICATION

Like all mongod options, these options can be issued on the mongod startup command line or placed in a configuration file that is used when the server is initialized. Table 11–3. Command-Line Options that Affect Master/Slave Replication

Option

Description

--only arg

(Optional) Slave: By default, the replication system will replicate all databases on the master to the slave. If you want only one database to be replicated, then use the –only option to specify the single database you want replicated; all other databases on the master server will be ignored.

--slavedelay arg

(Optional) Slave: In any replication system, there is always a certain amount of delay between an update operation occurring on the master and its effect propagating down to the slave. In most cases, you would strive to minimize this delay, so that the slave reflects as accurate a replica of the master as possible at any point in time. However, there are some cases where you will want to increase the size of this delay. For example, you might want to create a delayed backup that can be used if the data in the master becomes corrupted on a realtime system. By setting this option to a delay time in seconds, you can ensure the slave is always at minimum, the specified number of seconds behind the master.

--fastsync

(Optional) Slave: In the preceding example in this chapter, you started with an empty database, so the time to perform the initial synchronization of the slave to the master was extremely small. If you try adding an empty slave to a master that has a lot of data already in its databases, then it can take a long time to initially synchronize the two machines. Often, it is much faster to copy the datafiles from the master server’s dbpath to the slave server’s dbpath using your operating system’s filesystem copy commands. If you use this method to perform the initial synchronization of the two servers, then adding --fastsync to the slave server’s startup routine will force it to add some required extra information to the copied database files. This option enables the slave to accept these files as its initial sync.

--oplogSize arg

(Optional) Master: You have already seen that the oplog is used to record changes to the contents of the master database. The size of the oplog determines how long a slave can stay offline before it is no longer capable of resyncing its data without copying the entire master dataset again.

Resynchronizing a Master/Slave Replication System If you have taken a slave offline for longer than the time window the master server’s oplog can support, then you will find that the slave will fail to restart replication. Instead, it will need to be resynchronized (resynced) with the master. The following sections explain the three main methods you can use to resync a slave with its master.

249

CHAPTER 11 ■ REPLICATION

Issuing a Manual Resync Command to the Slave The easiest way to force a resync is to use the mongo console tool to connect to the slave and request a resync. In the preceding case, the slave is running on port 27018 of your local machine. This following command forces the slave to resynchronize with the master server: $mongo localhost:27018 >use admin >db.runCommand({resync: 1})

Resyncing by Deleting the Slaves Datafiles Let’s assume that you stop the slave server and delete its datafiles from the dbpath. When you restart the slave server (be careful to ensure it is started with the same startup parameters), it will detect this condition and resynchronize its dataset with its master automatically.

Resyncing a Slave with the --fastsync Option Both of the resynchronization methods described thus far work fine for small databases. However, it can take a considerable amount of time if you use those approaches to resynchronize a multi-gigabyte database. For large databases, it may be better to use a method introduced earlier, where you used the --fastsync option after moving a copy of the master’s datafiles to the slave’s dbpath. You can perform a manual resync by copying the datafiles stored in the dbpath from the master to the slave, and then restarting the slaves. To do this without shutting down the master, you must freeze the master for writes by connecting to your master and issuing the following commands: $mongo >use admin >db.runCommand({fsync:1,lock:1}) { "info" : "now locked against writes", "ok" : 1 } The preceding snippet causes the master to flush all outstanding writes to the disk and to block any updates from being performed; however, read queries will continue to be serviced without any blocks applied. Now stop the slaves that you wish to update. Next, copy the contents of the directories specified in the --dbpath option (e.g., /db/master/data) to the slave servers, placing the files in the directory specified by the slave’s --dbpath option (e.g., /db/slave/data). After you do this, you can run the following commands to release the lock on the master server: $mongo >use admin >db.$cmd.sys.unlock.findOne(); { "ok" : 1, "info" : "unlock requested" } Next, restart the slave with the --fastsync command-line option; this will tell the slave to use the newly copied datafiles as its own version of the database. The slave will check back in with the master, and then propagate any writes that the master stored in the oplog when the slave was being restarted.

250

CHAPTER 11 ■ REPLICATION

Implementing Multiple Master/Single Slave Replication In some circumstances, it may be advantageous to have a single slave track the contents of multiple master servers. For example, you might want to adopt such an approach if you are using master/slave replication for backup and reporting, but have multiple applications with different master database servers (one for each application), as shown in Figure 11–2. You can reduce your costs by sharing a common slave between masters. This is especially effective if the slave is not heavily loaded all of the time, and the backups of each application can be staggered. The characteristics of this model are listed in Table 11–4. Table 11–4. Characteristics of Multiple Master/Single Slave Replication

Property

Description

Number of masters

Multiple, fixed, role does not change

Number of slaves

Single, fixed, role does not change

Failover Mode

Manual

Figure 11–2 shows a typical multiple master/single slave replication setup.

Figure 11–2. Setting up multiple master/single slave replication

Setting up a Multiple Master/Slave Replication Configuration The essentials of setting up a multiple master/slave configuration are the same as for setting up a single master/slave configuration. In the next example, you will set up a multiple master/slave configuration with the characteristics shown in Table 11–5.

251

CHAPTER 11 ■ REPLICATION

Table 11–5. MongoDB Instances Used in a Multiple Master/Single Slave Example

Service

Daemon

Address

Dbpath

Master 1: Holds database foo

mongod

localhost:27021

/db/master1/data

Master 2: Holds database bar

mongod

localhost:27022

/db/master2/data

Slave 1: Holds a replica of the databases foo and bar

mongod

localhost:27023

/db/slave1/data

As in the earlier example, your next step is to create a small example of this configuration running on a single machine (localhost). Begin by creating the first master (master1); as before, you type these commands into a fresh terminal window: $sudo mkdir –p /db/master1/data $sudo mongod --master --dbpath /db/master1/data --port 27021 Now open another terminal window to attach to this server and create a simple database named foo: $mongo localhost:27021 >use foo >db.foocollection.insert({foodata:’our first foo document’}) >quit() $ At this point, you can set up the second master and its corresponding database by typing the following into another fresh terminal window: $sudo mkdir –p /db/master2/data $sudo mongod --master --dbpath /db/master2/data --port 27022 From your console window, enter the following code to launch the second instance of a MongoDB master database: $mongo localhost:27022 >use bar >db.barcollection.insert({bardata:’our first bar document’}) >quit() $ At this point, you have two separate instances of MongoDB running in master mode, with each hosting a different database. The following code sets up the slave and attaches it to both masters: $sudo mkdir –p /db/slave1/data $sudo mongod --slave --dbpath /db/slave1/data –port 27023 This preceding snippet creates the slave. Notice that you haven’t specified a source yet. This is because you will use the shell to add the two masters to the slave. In the window you are using for your MongoDB shell, enter this code to attach the slave to the two masters and start the initial sync of the databases for each master:

252 u

CHAPTER 11 ■ REPLICATION

$mongo localhost:27023 >use local > switched to db local > db.sources.insert({host:'localhost:27021'}) > db.sources.insert({host:'localhost:27022'}) > db.sources.find() { "_id" : ObjectId("4c52347d6a26e5025bf5d6d5"), "host" : "localhost:27021", "source" : "main", "syncedTo" : { "t" : 1280455835000, "i" : 1 }, "localLogTs" : { "t" : 0, "i" : 0 } } { "_id" : ObjectId("4c52348d6a26e5025bf5d6d6"), "host" : "localhost:27022", "source" : "main", "syncedTo" : { "t" : 1280455825000, "i" : 1 }, "localLogTs" : { "t" : 0, "i" : 0 } } After a short while, you should see the following when you look at your slave: $ mongo localhost:27023 > show dbs admin bar foo local > use foo switched to db foo > db.foocollection.find() { "_id" : ObjectId("4c52351575fdd94439dc9051"), "foodata" : "our first foo document" } > use bar switched to db bar > db.barcollection.find() { "_id" : ObjectId("4c523551fce873451e9aa261"), "bardata" : "our first bar document" } Don’t be concerned if the replication of these tiny databases seems to take a long time. It takes a while because, as each slave source is initialized, it must create an empty database file to hold the contents of the initial sync. By default, this creates an empty 2 GB file on the disk. This is a one-time operation, and it will not happen again during normal operation. If you look at the log output in the slave’s terminal window, you can see this process happening. So far you have successfully configured a master-master/slave setup, and we can see from output that the slave has both databases replicated to it. In the preceding example, you are replicating all of the databases from each master to the slave. Each master had only one database, and the names of those databases were unique; therefore, you encountered no problems with collisions on the slave. However, you can face potential issues if, for example, there is a common temporary database on each master that you don’t want to replicate because it is only used for temporary operations. In this scenario, you can restrict the replication to a single database by adding the only: field to the specification of the source master when configuring the slave, as shown in the following example: >db.sources.insert({ host:’localhost:27021’, only:’foo’ }) >db.sources.insert({ host:’localhost:27022’, only:’bar’ })

■ Caution You should not replicate the same database from two different masters; doing so is not supported, and it will lead to unpredictable results.

253

CHAPTER 11 ■ REPLICATION

Exploring Various Replication Scenarios The master/slave and multiple master/slave topologies are the only “officially” supported approaches for simple mode replication; however, there are a number of variants that also work to varying degrees. In the previous examples, you saw cases where a server ran in either master or slave mode. However, a MongoDB instance is capable of taking on both roles at the same time. You can place an instance into this schizophrenic mode by specifying both the master and the slave options when the instance starts up. The approach permits three additional topologies: cascade replication, master/master replication, and interleaved replication. You’ll learn about each of these approaches in the following sections.

Implementing Cascade Replication A cascade replication model requires a minimum of three servers: one operates as a pure master, one operates as a slave/master, and one operates as a slave (see Figure 11–3). These servers are effectively chained or cascaded together. Thus, any changes applied to the first master will be passed to the intermediate slave/master, and then to the final slave, which is the last machine in the chain. Theoretically, you can apply this chaining to any number of levels; however, at some point the overall replication lag time will become excessive and render the downstream slaves unusable.

Figure 11–3. A cascade replication system This approach may provide an architectural benefit in cases where the final slave has a --slavedelay option applied to it during startup; this would offer the system integrator the opportunity to have both a near realtime slave and a time-delayed slave, so you get the best of both worlds.

Implementing Master/Master Replication The master/master replication topology is popular in relational database circles. This topology allows you to maintain a cluster where reads and writes can be made from Client A or Client B to either Master A or Master B, and the cluster will remain synchronized (see Figure 11–4).

254

CHAPTER 11 ■ REPLICATION

Figure 11–4. Master/master replication However, you do need to be aware of a certain number of edge cases with MongoDB operating in master/master mode. For example, some database operations (e.g., $inc and upsert) may not operate correctly if there is appreciable lag between the two halves of the cluster, or if there are contention issues on documents. The behavior of the cluster in these cases is not predictable. If your application does not use any of the potentially problematic operations, however, this may be a viable solution for you. One point in favor of this topology is that, unlike when using MySQL and other relational database systems, the IDs for MongoDB documents are unique across an entire cluster. Relational database systems tend to use constructs such as auto-incrementing integers or sequences to create unique record IDs. If proper care and attention are not taken, then collisions can occur with insertions on different servers. MongoDB uses document ID values called _id. These values are safe from this problem because each ID generated on each server has as a component part of the Ethernet address of the server’s network card; hence, MongoDB IDs are unique across both machines, and the server does not have to resolve collisions in master/master mode.

Implementing Interleaved Replication Interleaved replication is a simple variation on the master/slave replication approach. This approach is useful if you have multiple applications and multiple database servers. You can reduce the cost of your installations by allowing the master server for one application to act as the slave server for another, and vice versa. For example, if you have two applications, you can use two MongoDB servers that are configured as masters for one app and slaves for the other (see Figure 11–5).

255

CHAPTER 11 ■ REPLICATION

Figure 11–5. Interleaved replication You can use the --only option to set which database is replicated on a master/slave pair. This enables you to determine which database each server is operating as a slave for.

Using Replica Pairs Replica pairs are a legacy MongoDB approach for implementing clusters of database servers. In this approach, each server in the pair is initially given master or slave status, which lets that server operate in a fashion similar to a simple master/slave configuration.

■ Note Replica pairs are available in MongoDB 1.4.x and earlier. Officially, these versions of MongoDB support only two servers per cluster. This mechanism will be deprecated in later versions of MongoDB in favor of replica sets, which provide a more flexible topology that allows more than two servers in a cluster.

The feature that differentiates replica pairs from the simple master/slave replication topologies is what happens if the machine that performs the role of master fails (or otherwise becomes unavailable). That failure could be a software fault in the master server or its operating system, a physical hardware failure of the server itself, or a network break between the server and the application. To resolve this, machines are able to change their respective roles automatically. For example, a machine that had been operating as a slave can promote itself to the role of master when it finds it is no longer able to verify that the master is fully functional.

256

CHAPTER 11 ■ REPLICATION

The interesting part occurs when the original master becomes available again (e.g., it reboots or a network cable is plugged back in). The original master can detect that it has been usurped in its role as master, assume the new role of slave, and then resynchronize itself with the new master. The two machines will remain in this reversed state until they are restarted and the original roles are reasserted, or another failure forces them to flip back. The characteristics of this model are listed in Table 11–6. Table 11–6. Characteristics of a Replica Pair

Property

Description

Number of masters

Single, dynamic, role will switch on failure

Number of slaves

Single, dynamic, role will switch on failure

Failover Mode

Automatic

Setting up a Replica Pair Next, you will learn how to set up a replica pair using two separate instances of MongoDB running on the same machine. This procedure is similar to the process for setting up a master/slave configuration, but it uses a slightly different syntax to initialize the two halves of the pair. Let’s begin by setting up the first half of the replica pair. Open a new terminal window and type the following code: $sudo mkdir –p /db/server1/data $sudo mongod --pairwith localhost:27018 --dbpath /db/server/data The server will show a lot of fascinating but irrelevant messages about creating an oplog that is approximately 5% of available disk space, etc. Finally, the server will announce that it’s ready to accept replication requests with the following notification: Wed Wed Wed Wed

Jul Jul Jul Jul

7 7 7 7

10:14:46 10:14:46 10:14:46 10:14:49

repl: from host:localhost:27018 repl: couldn't connect to server localhost:27018 localhost:27018 repl: sleep 3sec before next pass repl: from host:localhost:27018

The server will repeat this phrase, showing that it is waiting for the other half of the replica pair to become available. You will see the following notification once the second machine starts up: Wed Jul Wed Jul orderby: Wed Jul Wed Jul Wed Jul Wed Jul Wed Jul Wed Jul { Wed Jul Wed Jul Wed Jul

7 10:19:04 connection accepted from localhost:35772 #1 7 10:19:04 query local.oplog.$main ntoreturn:1 reslen:80 nscanned:1 { query: {}, { $natural: -1 } } nreturned:1 138ms 7 10:19:04 repl: from host:localhost:27018 7 10:19:04 pair: setting master=0 was -1 7 10:19:04 repl: applied 1 operations 7 10:19:04 repl: end sync_pullOpLog syncedTo: Wed Jul 7 10:19:03 2010 4c33e417:1 7 10:19:04 building new index on { _id: 1 } for local.pair.sync 7 10:19:04 Buildindex local.pair.sync idxNo:0 name: "_id_", ns: "local.pair.sync", key: { _id: 1 } } 7 10:19:04 done for 0 records 0.029secs 7 10:19:15 repl: sleep 2sec before next pass 7 10:19:17 repl: from host:localhost:27018

257

CHAPTER 11 ■ REPLICATION

Wed Jul Wed Jul Wed Jul

7 10:19:17 repl: applied 1 operations 7 10:19:17 repl: end sync_pullOpLog syncedTo: Wed Jul 7 10:19:17 repl: sleep 1sec before next pass

7 10:19:17 2010 4c33e425:1

The preceding shows a copy of MongoDB running on the default port (27017); this instance uses the directory /db/server1/data to store its databases and collections. Next, you need to set up the other half of the replica pair. Do this by opening up another terminal, and typing the following code: $sudo mkdir –p /db/server2/data $sudo mongod --pairwith localhost:27017 --dbpath /db/server2/data –port 27018 Again, you will see a lot of information about the slave machine starting up; eventually, you should see a notification like the following, which indicates that the slave has managed to connect to the master and start the replication process: Wed Wed Wed Wed Wed

Jul Jul Jul Jul Jul

Wed Jul Wed Jul Wed Jul Wed Wed Wed Wed Wed Wed Wed

Jul Jul Jul Jul Jul Jul Jul

7 7 7 7 7 { 7 7 7

10:19:04 connection accepted from localhost:52114 #1 10:19:04 pair: setting master=1 was -1 10:19:05 repl: from host:localhost 10:19:05 building new index on { ns: 1, id: 1 } for local.temp.replIds 10:19:05 Buildindex local.temp.replIds idxNo:0 name: "setIdx", ns: "local.temp.replIds", key: { ns: 1, id: 1 }, unique: true } 10:19:05 done for 0 records 0.019secs 10:19:05 building new index on { ns: 1, id: 1 } for local.temp.replModIds 10:19:05 Buildindex local.temp.replModIds idxNo:0 { name: "setIdx", ns: "local.temp.replModIds", key: { ns: 1, id: 1 }, unique: true } 7 10:19:05 done for 0 records 0secs 7 10:19:05 repl: sleep 2sec before next pass 7 10:19:15 repl: from host:localhost 7 10:19:15 repl: applied 1 operations 7 10:19:15 repl: end sync_pullOpLog syncedTo: Wed Jul 7 10:19:14 2010 4c33e422:1 7 10:19:15 repl: sleep 1sec before next pass 7 10:19:16 repl: from host:localhost

If you don’t see something like the preceding output, then you need to double-check your configuration. If all is well, you will have two instances of MongoDB running as a replica pair with server1 on port 27017 and server2 on port 27018. You can perform the same tests that you did with the master/slave replication configuration; however, first you need to identify which server will be the master. You need to find and connect to the master server. To determine which server is the master, you can check the replica status on both machines. The first server you started up is the instance on port 27017; the following output shows clearly this instance is not the master: $ mongo > db.$cmd.findOne({ismaster:1}) { "ismaster" : 0, "remote" : "localhost:27018", "info" : "direct negotiation", "ok" : 1 } Connecting to the other server on port 27018 shows that this instance is the master, so the last server you started up has taken over the role of master server:

258

CHAPTER 11 ■ REPLICATION

$mongo localhost:27018 > db.$cmd.findOne({ismaster:1}) { "ismaster" : 1, "remote" : "localhost", "info" : "CmdNegotiateMaster::run()", "ok" : 1 } You can now run the same tests that you ran against the master/slave configuration to show that writes to the master server are replicated to the slave server.

Coping with Failure One of the key features of a replica pair is its ability to flip the roles of the two servers in response to a server failure. You can simulate a failure by shutting down the server that you started up on port 27018. You can do this by selecting the open terminal that you started the second server (server2) in, and then typing ^C to send a termination request to the server. Remember that this server has been operating as the master server. This should cause the server to shut down; if you are watching the terminal window of the first server, you should see a stream of messages like the following printed to the console: Wed Jul Wed Jul Wed Jul

7 10:34:36 repl: couldn't connect to server localhost:27018 localhost:27018 7 10:34:36 pair: setting master=1 was 0 7 10:34:36 repl: sleep 3sec before next pass

The preceding messages tell you that the first server (server1) has changed its role and is now operating as the master server of the pair. If you restart server2 again, you should see a message stream in the console output similar to the following once the server finishes starting up: Wed Jul Wed Jul Wed Jul

7 10:56:09 connection accepted from localhost:48966 #1 7 10:56:09 pair: setting master=0 was -1 7 10:56:11 repl: from host:localhost

The preceding console output indicates that the newly restored server has started up and has assumed the role of the slave server. It adopted the slave role because it found an already active master server (server1). Your application will automatically stay attached to the active master, so bringing this server back online will have no effect on which server your application is using (see Figure 11–6). This is by design. A newly restored server will probably be busy immediately after being inserted back into the pair as it synchronizes itself with the current state of its peer. If the system were to restore itself to its prefailover condition, then your application performance would be impacted by this synchronization activity. This example illustrates how, in a replica pair, neither of the instances is naturally considered to be the master or the slave; which role a given server assumes will change based on the needs of the moment.

259

CHAPTER 11 ■ REPLICATION

Figure 11–6. The effect of a failover in a replica pair

Connecting Your Application to a Replica Pair Failing over your replica pair servers is only one problem you must solve when creating a high availability cluster. To make this approach useful, your application must be able to switch seamlessly between the two when the flip occurs. Your application also needs to know which of the two servers is the active master server at all times, so it can direct all of its updates to that instance. All of the MongoDB language drivers include the ability to make connections to a replica pair, as well as the ability to flip connections back and forth to the correct server. The following snippet shows how to set up a replica pair connection using PHP; the principles are the same in all of the other supported languages: A connection that you establish in this fashion, with the URLs of both replica pair members, will track the status of the pair and ensure that commands and updates are sent to the correct server.

260

CHAPTER 11 ■ REPLICATION

Resolving Server Disputes with an Arbiter A replica pair normally consists of only two servers, the two halves of the pair. However, some circumstances may require that you add a third server to the mix to resolve disputes between the two servers about which one should be the master and which should be the slave. Assume that you have a large installation, or one that is physically distributed over a wide area, where each half of the pair is hosted in a different location, maybe one on the east coast and the other on the west coast. It is possible for the network path between the two servers to become disrupted or broken, so that the machines can no longer see each other. In this case, each server will assume that the other has failed, and both will attempt to assume the role of master. If your application can still see both machines, then it will be presented with the confusing situation where both halves of the replica pair claim to be the master server. In this case, you can place a third server at the same location at which your application is hosted. This server will have the same ability to see the two servers that your app does, and will arbitrate these disputes to ensure that order is preserved. This type of machine is called an arbiter. A good place to install arbiters may be on the application’s web servers, as they will have the same viewpoint of the two halves of the replica pair that your application has. It is just a small mongod instance running on the same servers, and it will consume virtually no memory, disk space, or processing capacity, and will only be needed when a negotiation occurs. No special configuration of the arbiter is required. Configuring an arbiter is as simple as using mongod’s --arbiter option: --arbiter Placing this option in the startup command line or configuration file of each of the servers in your replica pair enables you to instruct them to defer to the arbiter machine to cast the deciding vote on which machine should play which role. The arbiter server does not need to be running a full dataset, nor is it involved in any part of the replication process, other than to act as an unbiased observer to help iron out disputes between the two replica pair servers. Figure 11–7 illustrates a scenario where replication has been enacted across a private link between two machines and this connection which has been disrupted, or partitioned in network speak. The arbiter server can still see both servers and maintains order.

261

CHAPTER 11 ■ REPLICATION

Figure 11–7. Using an arbiter

Implementing Advanced Clustering with Replica Sets Replica sets are a far more sophisticated and flexible version of the replica pair concept. Whereas replica pairs are limited to only two participants in the cluster, replica sets can have any number of participants.

■ Note Replica sets will be available in MongoDB 1.6.x and later. This feature will replace replica pairs as the primary clustering technology in MongoDB. Replica sets extend considerably the range of topologies that MongoDB can support, as well as the number of servers that can be aggregated together.

In this section, you will learn how to set up a simple replica set configuration. You will also learn how to add and remove members from the cluster. Like replica pairs, replica sets feature the concept of a single master server. In replica sets, this master server is called the primary, while the multiple slave servers are called secondary servers (see Figure 11–8).

262

CHAPTER 11 ■ REPLICATION

Figure 11–8. A cluster implemented with a replica set The characteristics of this model are detailed in Table 11–7. Table 11–7. The Characteristics of a Replica Set

Property

Description

Number of masters

Single, dynamic, role will switch on failure

Number of slaves

Multiple, dynamic, role will switch on failure, supports mixed fixed and dynamic slaves

Failover Mode

Automatic

Replica sets also feature the concept of active and passive members. Passive secondary servers don’t participate in elections for a new primary when the current primary becomes unavailable; instead, they serve the same purpose as fixed slaves, and they can be used as reporting or backup datasets. Member servers of replica sets do not need to be designated as set members on startup. Instead, configuration is done through server-level commands that are sent through the normal server interface.

263

CHAPTER 11 ■ REPLICATION

This makes it easier to create configuration management tools that allow dynamic configuration and management of clusters of machines. Note that the vocabulary for describing servers changes with replica sets; instead of master/slave servers, you now have primary/secondary servers, and the secondary servers can have either active or passive status. This change of vocabulary was necessary to reflect the more dynamic nature of replica sets; however, you will still find some vestiges of the master/slave vocabulary creeping back in with references to isMaster() tests, and master status. In the upcoming sections, you will learn how to accomplish the following: 1.

Create a replica set.

2.

Create a working example of a replica set.

3.

Add a server to a replica set.

4.

Inspect and perform operations on a replica set.

5.

Configure individual members of a replica set.

6.

Connect to a replica set from your application.

7.

Use the web interface to inspect the status of your replica set.

Creating a Replica Set The best way to learn how to create a replica set is by looking at an example. In the example that follows, you will create a replica set called testset. This set will have three members (two active and one passive). Table 11–8 lists the members of this set. Table 11–8. Configuring the Replica Set

Service

Daemon

Address

Dbpath

Active Member 1

mongod

[hostname]:27021

/db/active1/data

Active Member 2

mongod

[hostname]:27022

/db/active2/data

Passive Member 1

mongod

[hostname]:27023

/db/passive1/data

Unfortunately, the replica set system won’t allow you to use localhost or port 127.0.0.1, as you have in this chapter’s previous examples. This locking out of local addresses is intended to stop people from misconfiguring clusters that run on multiple machines. All machines in a replica set cluster must be accessible to each other for the replication to work properly, and using a local address could result in inaccessible hosts, unless all components are running on the same host. To get around this problem, you have to use the hostname of your machine; you can find this by using the hostname command, as in the following example: $hostname testlaptop.local

264

CHAPTER 11 ■ REPLICATION

Getting a Replica Set Member Up and Running In the examples that follow, substitute the term “[hostname]” with whatever value is returned by running the hostname command on your own system. The first step is to get the first active member up and running. To do so, open a terminal window and type the following: $sudo mkdir -p /db/active1/data $sudo mongod --dbpath /db/active1/data --port 27021 --replSet testset/[hostname]:27022 --rest The --replSet option tells the instance the name of the replica set it is joining, as well as the name of at least one other member of the set. This is the first member of the replica set, so you can give it the address of any other member, even if that member has not been started up yet. Only one member address is required, but you can also provide the names of other members by separating their addresses with commas, as shown in the following example: $sudo mongod --dbpath /db/active1/data --port 27021 --replSet testset/[hostname]:27022,[hostname]:27023 --rest To keep things simple, this example will rely on only one address. The next step is to get the other members up and running. Do so by opening two more terminal windows, and then type the following into the first window to get the second member up and running: $sudo mkdir -p /db/active2/data $sudo mongod --dbpath /db/active2/data --port 27022 --replSet testset/[hostname]:27021 --rest Next, type this into the second window to get the final (passive) member up and running: $sudo mkdir -p /db/passive1/data $sudo mongod --dbpath /db/passive1/data --port 27023 --replSet testset/[hostname]:27021 --rest The --rest option in the preceding examples activates a rest interface on port+1000 (i.e., for port 27021, the rest interface is available on port 28021). This approach enables you to use the web interface to inspect the status of your replica set, as you’ll see later in this chapter. At this point, you have three server instances up and running and communicating with each other; however, you do not quite have your replica set running because you haven’t yet initialized the replica set and instructed each member about its role and responsibilities. To do that, you need to attach to one of the servers and initialize the replica set. The following code selects the first server to attach to: $mongo [hostname]:27021 >use admin Next, you need to set up a configuration data structure that lists all the servers and their respective roles: > cfg = { _id: 'testset', members: [ { _id: 0, host: ‘[hostname]:27021' }, { _id: 1, host: '[hostname]:27022' }, { _id: 2, host: ‘[hostname]:27023', priority: 0 } ] }

265

CHAPTER 11 ■ REPLICATION

You’ve configured the structure of the replica set; notice the use of the “priority 0” in the configuration structure to specify that the passive member is not to be considered as a possible candidate for promotion to the primary role: next, you need to issue the command to initialize the replica set: > rs.initiate(cfg) { "info" : "Config now saved locally. "ok" : 1 }

Should come online in about a minute.",

Finally, you should check the status of the replica set to determine whether it has been set up correctly: >rs.status() { "set" : "testset", "date" : "Mon Jul 26 2010 01:57:06 GMT+0800 (PHT)", "myState" : 1, "members" : [ { "name" : "[hostname]:27021", "self" : true, "errmsg" : "" }, { "name" : "[hostname]:27023", "health" : 1, "uptime" : 1562, "lastHeartbeat" : "Mon Jul 26 2010 01:57:04 GMT+0800 (PHT)", "errmsg" : "initial sync done" }, { "name" : "[hostname]:27022", "health" : 1, "uptime" : 1562, "lastHeartbeat" : "Mon Jul 26 2010 01:57:04 GMT+0800 (PHT)", "errmsg" : "initial sync done" } ], "ok" : 1 } The output in the preceding example indicates that all is OK: you’ve successfully configured and set up your replica set. Remember that you should use the name of your own machine in place of [hostname] because neither “localhost” nor “127.0.0.1” will work.

Adding a Server to a Replica Set As your site grows in size, you may want to increase the number of MongoDB instances in your existing replica sets. Fortunately, this is easy to do. The next example will show you how to add a third active set member to replica set. Begin by creating the MongoDB instance:

266

CHAPTER 11 ■ REPLICATION

$sudo mkdir -p /db/active3/data $sudo mongod --port 27024 --dbpath /db/active3/data --replSet testset/[hostname]:27021 --rest Moving on, you need to add the new server to the replica set. This requires that you connect to the primary server instance with the mongo console, as in the following example: $mongo [hostname]:27021 >use admin >rs.add(“[hostname]:27024”) { “ok”: 1}

Managing Replica Sets MongoDB provides a number of commands for managing the configuration and status of a replica set. Table 11–9 shows the available commands that you can use to create, manipulate, and inspect the state of a cluster in a replica set. Table 11–9. Commands for Manipulating and Inspecting Replica Sets

Command

Description

rs.help()

Returns the list of the commands in this table.

rs.status()

Returns information about the current state of the replica set. This command lists each member server, along with information about its status, including the last time it was contacted. This call can be used to provide a simple health check of the entire cluster.

rs.initiate()

Initializes a replica set using default parameters. This call is useful if you want to use the rs.add(..) command to add all the members individually, rather than supplying the entire configuration set all at once with a configuration description.

rs.initiate(replSetcfg)

Initializes a replica set using a configuration description.

rs.add(“host:port”)

Adds a member server to the replica set with a simple string that provides hostname and (optionally) a specific port.

rs.add(membercfg)

Adds a member server to the replica set using a configuration description. You must use this method if you want to specify specific attributes (e.g., a priority for the new member server).

rs.addArbiter(“host:port”)

Adds a new member server to operate as an arbiter. The member does not need to have been started with a --replSet option; any mongod instance running on any reachable machine can perform this task. Take care that this server is reachable by all members of the replica set.

rs.stepDown()

Makes the primary server relinquish its role and forces the election of a new primary server in the cluster when you run this command against the primary member of a replica set. Note that only active secondary servers are available as candidates for becoming the new primary server.

267

CHAPTER 11 ■ REPLICATION

Command

Description

rs.conf()

Redisplays the configuration structure of the current replica set. This command is useful for obtaining a configuration structure for a replica set. This configuration structure can be modified and then supplied to rs.initiate() again to change the structure’s configuration. This technique provides the only supported way to remove a member server from a replica set; there is currently no direct method available for doing that.

db.isMaster()

This function is not specific to replica sets; rather, it is a general replication support function that allows an application or driver to determine whether a particular connected instance is the master/primary server in a replication topology.

The following sections will take a closer look at some of the more commonly used commands listed in Table 11–9, providing additional details about what they do and how you use them.

Inspecting an Instance’s Status with rs.status() The rs.status() command is probably the most common command that you will use when working with replica sets. It allows you to inspect the status of the instance you are currently attached to, including its role in the replica set: >rs.status() { "set" : "testset", "date" : "Sat Jul 31 2010 12:17:46 GMT+0800 (PHT)", "myState" : 1, "members" : [ { "name" : "[hostname]:27021", "self" : true, "errmsg" : "" }, { "name" : "[hostname]:27023", "health" : 1, "uptime" : 78, "lastHeartbeat" : "Sat Jul 31 2010 12:17:44 GMT+0800 (PHT)", "errmsg" : "initial sync done" }, { "name" : "[hostname]:27022", "health" : 1, "uptime" : 82, "lastHeartbeat" : "Sat Jul 31 2010 12:17:44 GMT+0800 (PHT)", "errmsg" : "initial sync done" } ], "ok" : 1 }

268

CHAPTER 11 ■ REPLICATION

The myState field shown in the preceding example has the values shown in Table 11–10. These values indicate the status of any member you run the rs.status() command against. Table 11–10. Values for the myState Field

myState

Description

0

Member is starting up and is in phase 1.

1

Member is operating as a primary (master) server.

2

Member is operating as a secondary server.

3

Member is recovering; the sysadmin has restarted the member server in recovery mode after a possible crash or other data issue.

4

Member has encountered a Fatal Error; the errMsg field in the members array for this server should show more details about the problem.

5

Member is starting up and has reached phase 2.

6

Member is in an unknown state; this could indicate a misconfigured replica set, where some servers are not reachable by all other members.

7

Member is operating as an arbiter.

8

Member is down or otherwise unreachable. The lastheartbeat timestamp in the member array associated with this server should provide the date/time that the server was last seen alive.

In the preceding example, the rs.status() command is run against the primary server member. The information returned for this command shows that the primary server is operating with a myState value of 1; in other words, the “Member is operating as a primary (master).”

Forcing a New Election with rs.stepDown() You can use the rs.stepDown() command to force a primary server to stand down; the command also forces the election of a new primary server. This command is useful in the following situations: •

You would like to take the server hosting the primary instance offline, whether to investigate the server or to implement hardware upgrades or maintenance.

•

You would like to run a diagnostic process against the data structures.

•

You would like to simulate the effect of a primary failure and force your cluster to fail over, enabling you to test how your application responds to such an event.

The following example shows the output returned if you run the rs.stepDown() command against the testset replica set:

269

CHAPTER 11 ■ REPLICATION

> rs.stepDown() { "ok" : 1 } > rs.status() { "set" : "testset", "date" : "Sat Jul 31 2010 12:57:14 GMT+0800 (PHT)", "myState" : 2, "members" : [ { "name" : "[hostname]:27021", "self" : true, "errmsg" : "" }, { "name" : "[hostname]:27023", "health" : 1, "uptime" : 2446, "lastHeartbeat" : "Sat Jul 31 2010 12:57:13 GMT+0800 (PHT)", "errmsg" : "initial sync done" }, { "name" : "[hostname]:27022", "health" : 1, "uptime" : 2450, "lastHeartbeat" : "Sat Jul 31 2010 12:57:13 GMT+0800 (PHT)", "errmsg" : "" } ], "ok" : 1 } In the preceding example, you run the rs.stepDown() command against the primary server. The output of the rs.status() command shows that the server now has a myState value of 2: “Member is operating as secondary.”

Determining If a Member is the Primary Server The db.isMaster() command isn’t strictly a replica set command. Nevertheless, this command is extremely useful because it allows an application to test whether it is connected to a master/primary server: >db.isMaster() { "ismaster" : false, "secondary" : true, "msg" : "", "hosts" : [ "[hostname]:27021", "[hostname]:27022" ], "passives" : [ "[hostname]:27023" ],

270

CHAPTER 11 ■ REPLICATION

"primary" : "[hostname]", "ok" : 1 } If you run isMaster() against your testset replica set cluster at this point, it shows that the server you have run it against is not a master/primary server (“ismaster” == false). If the server instance you run this command against is a member of a replica set, the command will also return a map of the known server instances in the set, including the roles of the individual servers in that set.

Configuring the Options for Replica Set Members The replica set functionality ships with a number of options you can use to control the behavior of a replica set’s members. When you run the rs.initiate(replSetcfg) or rs.add(membercfg) options, you have to supply a configuration structure that describes the characteristics of a replica set’s members: { _id : <setname>, members: [ { _id : , host : , [, priority: <priority>] [, arbiterOnly : true] [, votes : ] } , ... ], settings: { [heartbeatSleep : <seconds>] [, heartbeatTimeout : <seconds>] [, heartbeatConnRetries : ] [, getLastErrorDefaults: ] } } For rs.initiate(), you should supply the full configuration structure, as shown in the preceding example. The topmost level of the configuration structure itself includes three levels: _id, members, and settings. The _id is the name of the replica set, as supplied with the --replSet command-line option when you create the replica set members. The members array consists of a set of structures that describe each member of the set; this is the member structure that you supply to the rs.add() command when adding an individual server to the set. Finally, the settings array contains options that apply to the entire replica set.

Organization of the Members Structure The members structure contains all the entries required to configure each of the member instances of the replica set; you can see all of these entries listed in Table 11–11.

271

CHAPTER 11 ■ REPLICATION

Table 11–11. Configuring Member Server Properties

Option

Description

members.$._id

(Mandatory) Integer: This element specifies the ordinal position of the member structure in the member array. Possible values for this element include integers greater than or equal to 0. This value enables you to address specific member structures, so you can perform add, remove, and overwrite operations.

members.$.host

(Mandatory) String: This element specifies the name of the server in the form host:port; note that the host portion cannot be localhost or 127.0.0.1.

members.$.priority

(Optional) Float: The element represents the weight assigned to the server when elections for a new primary server are conducted. If the primary server becomes unavailable, then a secondary server will be promoted based on this value. Any secondary server with a non-zero value is considered to be active and eligible to become a primary server. Thus, setting this value to zero forces the secondary to become passive. If multiple secondary servers share equal priority, then a vote will be taken, and an arbiter (if configured) may be called upon to resolve any deadlocks. The default value for this element is 1.0, unless you specify otherwise.

members.$.arbiterOnly

(Optional) Boolean: This member operates as an arbiter for electing new primary servers. It is not involved in any other function of the replica set, and it does not need to have been started with a --replSet command-line option. Any running mongod process in your system can perform this task. The default value of this element is false.

members.$.votes

(Optional) Integer: This element specifies the number of votes that this instance can cast to elect other instances as a primary server; the default value of this element is 1.

Exploring the Options Available in the Settings Structure Table 11–12 lists the replica set properties available in the Settings structure. These settings are applied globally to the entire replica set; you use these properties to configure how replica set members communicate with each other.

272

CHAPTER 11 ■ REPLICATION

Table 11–12. Inter-server Communication Properties for the Settings Structure

Option

Description

settings.heartbeatSleep

(Optional) Integer: This element specifies how often the members of the replica set should announce themselves to each other, and its value is expressed in milliseconds. If not specified, this element has a default value of 2000 (2 seconds).

settings.heartbeatTimeout

(Optional) Integer: This element specifies the amount of time that the members of a replica set should wait after not hearing from a specific member before assuming it is unavailable. This value is expressed in milliseconds; if not specified, this element has a default value of 10000 (10 seconds).

Settings.heartbeatConnRetries

(Optional) Integer: The element specifies the number of attempts a member should make to reach another member before assuming it is down. If not specified, this element has a default value of 3.

Determining the Status of Replica Sets Replica sets are only becoming available as a stable implementation in the very latest version of MongoDB (1.6.x) at the time of writing. 10gen (the company that produces MongoDB) is very proactive in rolling out enhancements, so it is a good idea to peruse the online documentation to see what has changed since this book’s publication. For up-to-the-minute information on replica sets, visit this page on the MongoDB site: www.mongodb.org/display/DOCS/Replica+Sets. You should keep in mind the fact that, from version 1.6.x onwards, replica sets will become the preferred mechanism for setting up replicated clusters of machines, and the simpler “Replica Pairs” will become deprecated.

Connecting to a Replica Set from Your Application Connecting to a replica set from PHP is similar to connecting to a single MongoDB instance. The only difference is that it can provide either a single replica set instance address or a list of replica set members; the connection library will work out which server is the primary server and direct queries to that machine, even if the primary server is not one of the members that you provide. The following example shows how to connect to a replica set from a PHP application: true)); ... ?>

273

CHAPTER 11 ■ REPLICATION

Viewing Replica Set Status with the Web Interface MongoDB maintains a web-based console for viewing the status of your system. For our previous example, you can access this console by opening the URL http://localhost:28021 with your web browser. The port number of the web interface is set by default to port n+1000, where n is the port number of your instance. So, assuming your primary instance is on port 27021, as in this chapter’s example, its web interface can be found on port 28021. If you open this interface in your web browser, you will see a link to the status of the replica set at the top of the page (see Figure 11–9).

Figure 11–9. Viewing the status of the replica set in a browser Clicking the Replica set status link will take you to the Replica Set dashboard shown in Figure 11–10.

Figure 11–10. The replica set dashboard in the MongoDB web interface

274

CHAPTER 11 ■ REPLICATION

Summary MongoDB provides a rich set of tools for implementing a wide range of clustering and replication topologies. In this chapter, you learned about many of these tools, including some of the reasons and motivations for using them. You also learned how to set up a number of different replication topologies, from the simplest replication configuration all the way through to the latest, most advanced replica set capability introduced in the most recent version of MongoDB. Additionally, you learned how to inspect the status of replication systems using both the command-line tools and the built-in web interface. Please take the time required to evaluate each of the topologies described in this chapter to make sure you choose the one best suited to your particular needs before attempting to use any of them in a production environment. It is incredibly easy to use MongoDB to create test beds on a single machine; therefore, you are strongly encouraged to experiment with each method to make sure that you fully understand the benefits and limitations of each approach, including how it will perform with your particular data and application.

275

C H A P T E R 12 ■■■

Sharding Whether you’re building the next Facebook or just a simple database application, you will probably need to scale your app up at some point if it’s successful. If you don’t want to be continually replacing your hardware, then you will want to use a technique that allows you to add capacity incrementally to your system, as you need it. Sharding is a technique that allows you to spread your data across multiple machines, yet does so in a way that mimics an app hitting a single database. Ideally suited for the new generation of cloud-based computing platforms, sharding as implemented by MongoDB is perfect for dynamic, load-sensitive automatic scaling, where you ramp up your capacity as you need it and turn it down when you don’t. This chapter will walk you through implementing sharding in MongoDB.

Exploring the Need for Sharding When the World Wide Web was just getting under way, the number of sites, users, and the amount of information available online was low. The Web consisted of a few thousand sites and a population of only tens or perhaps hundreds of thousands of users predominantly centered on the academic and research communities. In those early days, data tended to be simple: hand-maintained HTML documents connected together by hyperlinks. The original design objective of the protocols that make up the Web was to provide a means of creating navigable references to documents stored on different servers around the Internet. Even current big brand names such as Yahoo had only a minuscule presence on the Web compared to its offerings today. The Yahoo directory that comprised the original product around which the company was formed was little more than a network of hand-edited links to popular sites. These links were maintained by a small but enthusiastic band of people called the surfers. Each page in the Yahoo directory was a simple HTML document stored in a tree of filesystem directories and maintained using a simple text editor. But as the size of the net started to explode—and the number of sites and visitors started its nearvertical climb upwards—the sheer volume of resources available forced the early Web pioneers to move away from simple documents to more complex dynamic page generation from separate data stores. Search engines started to spider the Web and pull together databases of links that today number in the hundreds of billions of links and tens of billions of stored pages. These developments prompted the movement to datasets managed and maintained by evolving content management systems that were stored mainly in databases for easier access. At the same time, new kinds of services evolved that stored more than just documents and link sets. For example, audio, video, events, and all kinds of other data started to make its way into these huge datastores. This is often described as the “industrialization of data”—and in many ways it shares parallels with the evolution of the industrial revolution centered on manufacturing during the 19th century. Eventually, every successful company on the Web faces the problem of how to access the data stored in these mammoth databases. They find that there are only so many queries per second that can

277

CHAPTER 12 ■ SHARDING

be handled with a single database server, and network interfaces and disk drives can only transfer so many megabytes per second to and from the web servers. Companies that provide web-based services can quickly find themselves exceeding the performance of a single server, network, or drive array. In such cases, they are compelled to divide and distribute their massive collections of data. The usual solution is to partition these mammoth chunks of data into smaller pieces that can be managed more reliably and quickly. At the same time, these companies need to maintain the ability to perform operations across the entire breadth of the data held in their large clusters of machines. Replication, which you learned about in some detail in Chapter 11, can be an effective tool for overcoming some of these scaling issues, enabling you to create multiple copies of your data in multiple servers. This enables you to spread out your server load across more machines. Before long, however, you run headlong into another problem, where the size of the individual tables or collections that make up your data set grow so large that they exceed the capacity of a single database system to manage them effectively. For example, Flickr announced that on October 12th 2009 it had received its 4 billionth photo, and the site is now well on its way to crossing the 10 billion photos mark. Attempting to store the details of 10 billion photos in one table is not feasible, so Flickr looked at ways of distributing that set of records across a large number of database servers. The solution adopted by Flickr serves as one of the better-documented (and publicized) implementations of sharding in the real world.

Partitioning Horizontal and Vertical Data Data partitioning is the mechanism of splitting data across multiple independent datastores. Those datastores can be co-resident (on the same system) or remote (on separate systems). The motivation for co-resident partitioning is to reduce the size of individual indices and reduce the amount of I/O that is needed to update records. The motivation for remote partitioning is to increase the bandwidth of access to data, by having more network interfaces and disc data I/O channels available.

Partitioning Data Vertically In the traditional view of databases, data is stored in rows and columns. Vertical partitioning consists of breaking up a record on column boundaries and storing the parts in separate tables or collections. It can be argued that a relational database design that uses joined tables with a one-to-one relationship is a form of co-resident vertical data partitioning. MongoDB, however, does not lend itself to this form of partitioning because the structure of its records (documents) does not fit the nice and tidy row and column model. Therefore, there are few opportunities to cleanly separate a row based on its column boundaries. MongoDB also promotes the use of embedded documents, and it does not directly support the ability to join associated collections together.

Partitioning Data Horizontally Horizontal partitioning is where all the action is when using MongoDB, and sharding is the common term for a popular form of horizontal partitioning. Sharding allows you to split a collection across multiple servers to improve performance in a collection that has a large number of documents in it. A simple example of sharding occurs when a collection of user records is divided across a set of servers, so that all the records for people with last names that begin with the letters A–G are on one server, H–M are on another, and so on. The rule that splits the data is known as the sharding key function, or the data hashing function. In simple terms, sharding allows you to treat the cloud of shards as through it were a single collection, and an application does not need to be aware that the data is distributed across multiple

278

CHAPTER 12 ■ SHARDING

machines. Traditional sharding implementations require the application to be actively involved in determining which server a particular document is stored on, so it can route its requests properly. Traditionally, there is a library bound to the application, and this library is responsible for storing and querying data in sharded data sets. MongoDB is virtually unique in its support for auto-sharding, where the database server manages the splitting of the data and the routing of requests to the required shard server. If a query requires data from multiple shards, then MongoDB will manage the process of merging the data obtained from each shard back into a single cursor. This feature, more than any other, is what earns MongoDB its stripes as a cloud or web-oriented database.

Analyzing a Simple Sharding Scenario Let’s assume you want to implement a simple sharding solution for a fictitious Gaelic social network. Figure 12–1 shows a simplified representation of how this application could be sharded.

Figure 12–1. Simple sharding of a User collection There are a number of problems with this simplified view of our application. Let’s look at the most obvious ones. First, if your Gaelic network is targeted at the Irish and Scottish communities around the world, then the database will have a large number of names that start with Mac and Mc (e.g., MacDonald, McDougal, and so on) for the Scottish population and O’ (e.g., O’Reilly, O’Conner, and so on) for the Irish population. Thus, using the simple sharding key function based on the first letter of the last name will

279

CHAPTER 12 ■ SHARDING

place an undue number of user records on the shard that supports the letter range “M–O.” Similarly, the shard that supports the letter range “X–Z” will perform very little work at all. An important characteristic of a sharding system is that it must ensure that the data is spread evenly across the available set of shard servers. This prevents hotspots from developing that can affect the overall performance of the cluster. Let’s call this Requirement 1: The ability to distribute data evenly across all shards. Another thing to keep in mind: when you split your dataset across multiple servers, you effectively increase your dataset’s vulnerability to hardware failure. That is, you increase the chance that a single server failure will affect the availability of your data as you add servers. Again, an important characteristic of a reliable sharding system is that—like a RAID system commonly used with disk drives—it stores each piece of data on more than one server, and it can tolerate individual shard servers becoming unavailable. Let’s call this Requirement 2: The ability to store shard data in a fault-tolerant fashion. Finally, you want to make sure that you can add or remove servers from the set of shards without having to back up and restore the data and redistribute it across a smaller or larger set of shards. Further, you need to be able to do this without causing any downtime on the cluster. Let’s call this Requirement 3: The ability to add or remove shards while the system is running. The upcoming sections will cover how to address these requirements.

Implementing Sharding with MongoDB MongoDB uses a proxy mechanism to support sharding (see Figure 12–2); the provided mongos daemon acts as a controller for multiple mongod-based shard servers. Your application attaches to the mongos daemon as though it were a single MongoDB database server; thereafter, your application sends all of its commands (e.g., updates, queries, and deletes) to that mongos daemon.

Figure 12–2. A simple sharding setup without redundancy

280

CHAPTER 12 ■ SHARDING

The mongos daemon is responsible for managing which MongoDB server is sent the commands from your application, and this daemon will reissue queries that cross multiple shards to multiple servers and aggregate the results together. MongoDB implements sharding at the collection level, not the database level. In many systems, only one or two collections may grow to the point where sharding is required. Thus, sharding should be used judiciously; you don’t want to impose the overhead of managing the distribution of data for smaller collections if you don’t need to. Let’s return to the fictitious Gaelic social network example. In this application, the user collection contains details about its users and their profiles. This collection is likely to grow to the point where it needs to be sharded. However, other collections such as events, countries, and states are unlikely to ever become so large that sharding would provide any benefit. The sharding system uses a sharding key function to map data into chunks, which are blocks of storage containing documents (see Chapter 5 for more information on chunks). Each chunk stores documents with a particular continuous range of sharding key values; these values enable the mongos controller to quickly find a chunk that contains a document it needs to work on. MongoDB’s sharding system then stores this chunk on an available shard store; the config servers keep track of which chunk is stored on which shard server. This is an important feature of the implementation because it allows you to add and remove shards from a cluster without having to back up and restore the data. When you add a new shard to the cluster, the system will redistribute its chunks across the new set of servers in order to improve performance. Similarly, when you remove a shard, the sharding controller will drain the chunks out of the shard being taken offline and redistribute them to the remaining shard servers. A sharding setup for MongoDB also needs a place to store the configuration of its shards, as well as a place to store information about each shard server in the cluster. To support this, a MongoDB server called a config server is required; this server instance is a normal mongod server running in a special role. As explained above, the config servers also act as directories that allow the location of each chunk to be determined At first glance, it appears that implementing a solution that relies on sharding requires a lot of servers! However, similar to what you saw in Chapter 11’s coverage of replication, you can co-host multiple instances of each of the different services required to create a sharding setup on a relatively small number of physical servers. Figure 12–3 shows a fully redundant sharding system that uses replica sets for the shard storage and the config servers, as well as a set of mongos daemons to manage the cluster. It also shows how those services can be condensed to run on just three physical servers. Carefully placing the shard storage instances so that they are correctly distributed among the physical servers enables you to ensure that your system can tolerate the failure of one or more servers in your cluster. This mirrors the approach used by RAID disk controllers to distribute data across multiple drives in stripes, enabling RAID configurations to recover from a failed drive.

■ Note If you don’t supply a sharding key, then MongoDB will automatically use the _id field to distribute documents among your database’s shards; however, this field may not provide the optimal data value to use to shard your data.

281

CHAPTER 12 ■ SHARDING

Figure 12–3. A redundant sharding configuration

Setting Up a Sharding Configuration To use sharding effectively, it’s important that you understand how it works. The next example will walk you through how to set up a test configuration on a single machine. You will configure this example like the simple sharding system shown in Figure 12–2, with one difference: this example will keep things simple by using only two shards. Finally, you will learn how to create a sharded collection and a simple PHP test program that demonstrates how to use this collection. In this test configuration, you will use the services listed in Table 12–1. Table 12–1. Server Instances in the Test Configuration

282

Service

Daemon

Port

Dbpath

Shard Controller

mongos

27021

N/A

Config Server

mongod

27022

/db/config/data

Shard0

mongod

27023

/db/shard1/data

Shard1

mongod

27024

/db/shard2/data

CHAPTER 12 ■ SHARDING

Let’s begin by setting up the configuration server. Do so by opening a new terminal window and typing the following code. $sudo mkdir -p /db/config/data $sudo mongod --port 27022 --dbpath /db/config/data --configsvr Be sure to leave your terminal window open once you have the config server up and running. Next, you need to set up the shard controller (mongos). To do so, open a new terminal window and type the following: $sudo mongos --configdb localhost:27022 --port 27021 --chunkSize 1 This brings up the shard controller, which should announce that it’s listening on port 27021. If you look at the terminal window for the config server, you should see that the shard server has connected to its config server and registered itself with it. In this example, you set the chunk size to its smallest possible size of 1 MB. This is not a practical value for real-world systems because it means that the chunk storage is smaller than the maximum size of a document (4 MB). However, this is just a demonstration, and the small chunk size allows you to create a lot of chunks to exercise the sharding setup without also having to load a lot of data. By default, chunkSize is set to 128 MB unless otherwise specified. Finally, you’re ready to bring up the two shard servers. To do so, you will need two fresh terminal windows, one for each server. Type the following into one window to bring up the first server: $sudo mkdir -p /db/shard0/data $sudo mongod --port 27023 --dbpath /db/shard0/data --shardsvr And type the following into the second window to bring up the second server: $sudo mkdir -p /db/shard1/data $sudo mongod --port 27024 --dbpath /db/shard1/data --shardsvr You have your servers up and running. Next, you need to tell the sharding system where the shard servers are located. To do this, you need to connect to your shard controller (mongos). It’s important to remember that, even though mongos is not a full MongoDB instance, it appears to be a full instance to your application. Therefore, you can just use the mongo command shell to attach to the shard controller and add your two shards, as shown in the following example: $ mongo localhost:27021 > use admin switched to db admin > db.runCommand( { addshard : "localhost:27023", allowLocal : true } ) { "added" : "localhost:27023", "ok" : 1 } > db.runCommand( { addshard : "localhost:27024", allowLocal : true } ) { "added" : "localhost:27024", "ok" : 1 } Your two shard servers are now activated; next, you need to check the shards using the listshards command: > db.runCommand({listshards:1}) { "shards" : [ { "_id" : "shard0", "host" : "localhost:27023" }, {

283

CHAPTER 12 ■ SHARDING

"_id" : "shard1", "host" : "localhost:27024" } ], "ok" : 1 } You now have a working sharded server; next, you will create a new database called testdb, and then activate a collection called testcollection inside this database. You will shard this collection, so you will give this collection an entry called testkey that you will use as the sharding function: > testdb = db.getSisterDB("testdb") testdb > db.runCommand({ enablesharding: "testdb"}) { "ok" : 1 } > db.runCommand({ shardcollection : "testdb.testcollection", key : {testkey : 1}}) { "collectionsharded" : "testdb.testcollection", "ok" : 1 } Thus far, you have created a sharded cluster with two shard storage servers. You have also created a database on it with a sharded collection. A server without any data in it is of no use to anybody, so it’s time to get some data into this collection, so you can see how the shards are distributed. To do this, you will use a small PHP program to load the sharded collection with some data. The data you will load consists of a single field called testkey. This field contains a random number and a second field with a fixed chunk of text inside it (the purpose of this second field is to make sure you can create a reasonable number of chunks to shard). This collection serves as the main data table for a fictitious website called TextAndARandomNumber.com. The following code creates a PHP program that inserts data into your sharded server: selectDB('testdb'); // Select the TestIndex collection $collection = $db->testcollection; for($i=0; $i < 100000 ; $i++){ $data=array(); $data['testkey'] = rand(1,100000); $data['testtext'] = "Because of the nature of MongoDB, many of the more " . "traditional functions that a DB Administrator " . "would perform are not required. Creating new databases, " . "collections and new fields on the server are no longer necessary, " . "as MongoDB will create these elements on-the-fly as you access them." . "Therefore, for the vast majority of cases managing databases and " . "schemas is not required."; $collection->insert($data); } This small program will connect to the shard controller (mongos) and insert 100,000 records with random testkeys and some testtext to pad out the documents. As mentioned previously, this sample

284

CHAPTER 12 ■ SHARDING

text causes these documents to occupy a sufficient number of chunks to make using the Sharding mechanism feasible. The following command runs the test program: $php testshard.php Once the program has finished running, you can connect to the mongos instance with the command shell and verify that the data has been stored: $mongo localhost:27021 >use testdb >db.testcollection.count() 100000 At this point, you can see that your server has stored 100,000 records. Now you need to connect to each shard and see how many items have been stored in testdb.testcollection for each shard. The following code enables you to connect to the first shard and see how many records are stored in it from the testcollection collection: $mongo localhost:27023 >use testdb >db.testcollection.count() 48875 And this code enables you to connect to the second shard and see how many records are stored in it from the testcollection collection: $mongo localhost:27024 >use testdb >db.testcollection.count() 51125

■ Note You may see different values for the number of documents in each shard, depending on when exactly you look at the individual shards. The mongos instance may initially place all the chunks on one shard, but over time it will rebalance the shard set to evenly distribute data among all the shards by moving chunks around. Thus, the number of records stored in a given shard may change from moment to moment. This satisfies “Requirement 1: The ability to distribute data evenly across all shards.”

Adding a New Shard to the Cluster Let’s assume business is really jumping at TextAndARandomNumber.com. To keep up with the demand, you decide to add a new shard server to the cluster to spread out the load a little more. Adding a new shard is easy; all it requires is that you repeat the steps described previously. Begin by creating the new shard storage server and place it on port 27025, so it does not clash with your existing servers: $ sudo mkdir -p /db/shard2/data $ sudo mongod --port 27025 --dbpath /db/shard2/data --shardsvr

285

CHAPTER 12 ■ SHARDING

Next, you need to add the new shard server to the cluster. You do this by logging into the sharding controller (mongos), and then using the admin addshard command: $mongo localhost:27021 >use admin admin >db.runCommand( { addshard : "localhost:27025", allowLocal : true } ) At this point, you can run the listshards command to verify that the shard has been added to the cluster. Doing so reveals that a new shard server (shard2) is now present in the shards array: >db.runCommand({listshards:1}) { "shards" : [ { "_id" : "shard0", "host" : "localhost:27023" }, { "_id" : "shard1", "host" : "localhost:27024" }, { "_id" : "shard2", "host" : "localhost:27025" } ], "ok" : 1 } If you log in to the new shard storage server you have created on port 27025 and look at testcollection, you will see something interesting: $mongo localhost:27025 > use testdb switched to db testdb > show collections system.indexes testcollection > db.testcollection.count() 4657 > db.testcollection.count() 4758 > db.testcollection.count() 6268 This shows that the number of items in the testcollection on your new shard2 storage server is slowly going up. What you are seeing is proof that the sharding system is rebalancing the data across the expanded cluster. Over time, the sharding system will migrate chunks from the shard0 and shard1 storage servers to create an even distribution of data across the three servers that make up the cluster. This process is automatic, and it will happen even if there is no new data being inserted into the testcollection collection. In this case, the mongos shard controller is moving chunks to the new server, and then registering them with the config server.

286

CHAPTER 12 ■ SHARDING

This is one of the factors to consider when choosing a chunk size. If your chunkSize value is very large, then you will get a less even distribution of data across your shards; conversely, the smaller your chunkSize value, the more even the distribution of your data will be.

Removing a Shard from the Cluster It was great while it lasted, but now assume that TextAndARandomNumber.com was a flash in the pan and its sizzle fizzled. After a few weeks of frenzied activity, the site’s traffic started to fall off, so you had to start to look for ways to cut your running costs—in other words, that new shard server had to go! In the next example, you will remove the shard server you added previously. To initiate this process, log in to the shard controller (mongos) and issue the removeShard command: $ mongo localhost:27021 > use admin switched to db admin > db.runCommand({removeShard : "localhost:27025"}) { "msg" : "draining started successfully", "state" : "started", "shard" : "shard2", "ok" : 1 } The removeShard command responds with a message indicating that the removal process has started. It also indicates that the shard controller (mongos) has begun relocating the chunks on the target shard server to the other shard servers in the cluster. This process is known as draining the shard. You can check the progress of the draining process by reissuing the removeShard command. The response will tell you how many chunks and databases still need to be drained from the shard: > db.runCommand({removeShard : "localhost:27025"}) { "msg" : "draining ongoing", "state" : "ongoing", "remaining" : { "chunks" : NumberLong( 12 ), "dbs" : NumberLong( 0 ) }, "ok" : 1 } Finally, the removeShard process will terminate, and you will get a message indicating that the removal process is complete: > db.runCommand({removeShard : "localhost:27025"}) { "msg" : "removeshard completed successfully", "state" : "completed", "shard" : "shard2", "ok" : 1 } To verify that the removeShard command was successful, you can run listshards to confirm that the desired shard server has been removed from the cluster. For example, the following output shows that the shard2 server that you created previously is no longer listed in the shards array:

287

CHAPTER 12 ■ SHARDING

>db.runCommand({listshards:1}) { "shards" : [ { "_id" : "shard0", "host" : "localhost:27023" }, { "_id" : "shard1", "host" : "localhost:27024" } ], "ok" : 1 } At this point, you can terminate the Shard2 mongod process and delete its storage files because its data has been migrated back to the other servers.

■ Note The ability to add and remove shards to your cluster without having to take it offline is a critical component of MongoDB’s ability to support highly scalable, highly available, large-capacity datastores. This satisfies the final requirement: “Requirement 3: The ability to add or remove shards while the system is running.”

Determining How You’re Connected Your application can be connected either to a standard non-sharded database (mongod) or to a shard controller (mongos). MongoDB makes both of these processes; for all but a few use cases, the database and shard controller look and behave exactly the same way. However, sometimes it may be important to determine what type of system you are connected to. MongoDB provides the isdbgrid command, which you can use to interrogate the connected data system to determine whether it is sharded. The following snippet shows how to use this command, as well as what its output looks like: $mongo >use testdb >db.runCommand({ isdbgrid : 1}); { "isdbgrid" : 1, "hostname" : "localhost", "ok" : 1 } The response includes the isdbgrid:1 field, which tells you that the database you are connected to is enabled for sharding. A response of isdbgrid:0 would indicate that you are connected to a non-sharded database.

Listing the Status of a Sharded Cluster MongoDB also includes a simple command for dumping the status of a sharding cluster: printShardingStatus().

288

CHAPTER 12 ■ SHARDING

This command can give you a lot of insight into the internals of the sharding system. The following snippet shows how to invoke the printShardingStatus() command, but strips out some of the output returned to make it easier to read: $mongo localhost:27021 >use admin >db.printShardingStatus(); --- Sharding Status --sharding version: { "_id" : 1, "version" : 3 } shards: { "_id" : "shard0", "host" : "localhost:27023" } { "_id" : "shard1", "host" : "localhost:27024" } databases: { "_id" : "admin", "partitioned" : false, "primary" : "config" } { "_id" : "testdb", "partitioned" : true, "primary" : "shard0", "sharded" : { "testdb.testcollection" : { "key" : { "testkey" : } } testdb.testcollection chunks: { "testkey" : { $minKey : 1 } } -->> { "testkey" : 47 } on "i" : 2 } { "testkey" : 47 } -->> { "testkey" : 1702 } on : shard1 { { "testkey" : 1702 } -->> { "testkey" : 3356 } on : shard1 } { "testkey" : 3356 } -->> { "testkey" : 5020 } on : shard1 } { "testkey" : 5020 } -->> { "testkey" : 6679 } on : shard1 } { "testkey" : 6679 } -->> { "testkey" : 8137 } on : shard1 } { "testkey" : 8137 } -->> { "testkey" : 9637 } on : shard1 }

1 }, "unique" : false } : shard1 { "t" : 2000, "t" : 66000, "i" : 0 } { "t" : 66000, "i" : 1 { "t" : 66000, "i" : 2 { "t" : 66000, "i" : 3 { "t" : 66000, "i" : 4 { "t" : 66000, "i" : 5

... { "testkey" : 64878 } -->> { "testkey" : 66399 } on : shard0 { "t" : 66000, "i" : 152 } { "testkey" : 66399 } -->> { "testkey" : 67934 } on : shard0 { "t" : 66000, "i" : 148 } { "testkey" : 67934 } -->> { "testkey" : 69410 } on : shard0 { "t" : 66000, "i" : 43 } { "testkey" : 69410 } -->> { "testkey" : 70962 } on : shard0 { "t" : 66000, "i" : 44 } { "testkey" : 70962 } -->> { "testkey" : 72470 } on : shard0 { "t" : 66000, "i" : 45 } { "testkey" : 72470 } -->> { "testkey" : 73991 } on : shard0 { "t" : 66000, "i" : 46 } { "testkey" : 73991 } -->> { "testkey" : 75478 } on : shard0 { "t" : 66000, "i" : 47 } { "testkey" : 75478 } -->> { "testkey" : 77063 } on : shard0 { "t" : 66000, "i" : 48 } { "testkey" : 77063 } -->> { "testkey" : 78654 } on : shard0 { "t" : 66000, "i" : 49 } ...

289

CHAPTER 12 ■ SHARDING

{ "testkey" : 96954 } -->> { "testkey" : 98468 } on : shard0 { "t" : 66000, "i" : 62 } { "testkey" : 98468 } -->> { "testkey" : 99979 } on : shard0 { "t" : 66000, "i" : 145 } { "testkey" : 99979 } -->> { "testkey" : { $maxKey : 1 } } on : shard0 { "t" : 6000, "i" : 4 } > This output lists the shard servers, the configuration of each sharded database/collection, and each chunk in the sharded dataset. Because you used a small chunkSize value to simulate a larger sharding setup, this report lists a lot of chunks. An important piece of information that can be obtained from this listing is the range of sharding keys associated with each chunk. The preceding output also shows which shard server the specific chunks are stored on. You can use the output returned by this command as the basis for a tool to analyze the distribution of a shard server’s keys and chunks. For example, you might use this data to determine whether there is any clumping of data in the dataset.

Using Replica Sets to Implement Shards The examples you have seen so far rely on a single mongod instance to implement each shard. In Chapter 11, you learned how to create replica sets, which are clusters of mongod instances working together to provide redundant and fail-safe storage. When adding shards to the sharded cluster, you can provide the name of a replica set and the address of a member of that replica set, and that shard will be instanced on each of the replica set members. Mongos will track which instance is the primary server for the replica set; it will also make sure that all shard writes are made to that instance. Combining sharding and replica sets enables you to create high-performance, highly reliable clusters that can tolerate multi-machine failure. It also enables you to maximize the performance and availability of cheap, commodity-class hardware.

■ Note The ability to use replica sets as a storage mechanism for shards satisfies “Requirement 2: The ability to store shard data in a fault-tolerant fashion.”

Sharding to Improve Performance In the MongoDB versions available at the time of writing, several MongoDB operations are singlethreaded only, such as background indexing and aggregate functions (e.g., grouping and map/reduce). The JavaScript engine (spidermonkey) used in MongoDB also imposes this restriction. Modern CPUs support features such as multiple cores and hyperthreading. These features enable simultaneous code execution, so it is advantageous to break up long-running tasks into separate processes to take advantage of the multiple cores. If you have a four core CPU, it may be advantageous to create a four-shard database configuration inside a single physical machine because all of the aforementioned operations will be passed to the individual shard storage server instances for execution. Doing this enables each core to handle a separate operation at the same time.

290

CHAPTER 12 ■ SHARDING

■ Caution If you are planning to run multiple shards on the same machine, then it is advisable to dedicate a separate drive to each shard. With many different threads trying to access different shards at the same time, it can over-stress a single drive, by forcing it to perform rapid read/write head movements between shards. By confining each shard on its own drive, you can minimize the movement of the heads and improve the overall performance considerably.

Summary Sharding enables you to scale your datastores to handle extremely large datasets. It also enables you to grow the cluster to match the growth in your system. MongoDB provides a simple automatic sharding configuration that works well for most requirements. Even though this process is automated, you can still fine-tune its characteristics to support your specific needs. Auto-sharding is one of the key features of MongoDB that set it apart from other data-storage technologies. We hope this book has helped you see the many ways that MongoDB is designed to cope better with the rigorous demands of modern web-based applications than is possible using more traditional database tools. Topics you have learned about in this book include the following: •

How to install and configure MongoDB on a variety of platforms.

•

How to access MongoDB from various development languages.

•

How to connect with the community surrounding the product, including how to obtain help and advice.

•

How to design and build applications that take advantage of MongoDB’s unique strengths.

•

How to optimize, administer, and troubleshoot MongoDB-based datastores.

•

How to create scalable fault-tolerant installations that span multiple servers.

You are strongly encouraged to explore the many samples and examples provided in this book. Other PHP examples can be found in the PHP MongoDB driver documentation located at www.php.net/manual/en/book.mongo.php. MongoDB is an extremely approachable tool, and its ease of installation and operation encourage experimentation. So don’t hold back: crank it up and start playing with it! And remarkably soon you too will begin to appreciate all the possibilities that this intriguing product opens up for your applications.

291

Index ■■■

■ Symbols

$ character in key names, 49 in queries, 70 . character dot notation (queries), 51 in key names, 49

■A

ABA problem, 72 access restrictions, 208–12 active members, replica sets, 263 active/active clusters, 7 adding data to collections, 48–49 PHP driver for, 102–4 PyMongo for, 139–40 using batches, 239 adding files to database, 85, 93 with PHP driver, 133 adding indexes, 42. See also indexing documents addshard command, 286 $addToSet operator, 68, 123, 158 addUser() function, 209, 210, 211 admin database, 208 admin user, adding, 209 administration. See database administration aggregate data servers, 242 aggregation commands, 55–57

with PHP driver, 108–10 with PyMongo, 145–47 $all operator, 59, 115, 151 allPlans element, explain(), 229 appending values to fields, 68, 123, 157 aptitude (software), 21 --arbiter option (mongod), 261 arbiters, for replica pair disputes, 261 arg parameter, update() [PyMongo], 154 Array data type, 38 arrays $ for position in, 70 adding values to, 68, 123, 158 deleting values from, 69, 124–25, 159–60 indexes on embedded keys, 79 matching entire, 63 using in queries, 59, 114, 150, 151 ascending order, 51, 52, 78 asserts section, serverStatus() output, 216, 217 atomic operations, 71–73 with PHP driver, 121–25, 126–28 with PyMongo, 156, 161 atomic updates on keys, 66, 71, 121, 156 --auth startup option, 209, 214 authentication, 208–12 auto-sharding, 15, 279 automatic backups, 199–203 available databases and collections, viewing, 47

293

■ INDEX

■B

background indexing, 235, 236 background option, ensureIndex(), 235 backing up MongoDB server, 194–97 automatic backups, 199–203 customization of, 197–98 large databases, 203–5 backups, replication and, 243 batching inserts, 239 Big Endian, about, 41 bin directory, 22 binary data, 38 storing, 14, 84. See also files; GridFS blog application (example), 167–90 creating index pages, 180–81 document structure, 168–69 final code, 181–90 listing posts, 169–72 looking at single posts, 172–75 comment management, 174–75 managing posts, 176–80 adding posts, 177 deleting posts, 179 editing posts, 178 searching posts, 175–76, 182 Boolean data type, 38 $box shape, 44 BSON, 5, 11 matching results by BSON type, 62 bug tracking for MongoDB, 17

■C

--c option (mongodump), 197 --c option (mongorestore), 199 capped collections, 37, 53 cascade replication, 254 case sensitivity in naming, 193 $center shape, 44 chat with MongoDB developers, 16 checksums, 87 chunks (binary data), 14, 86, 281 chunk size, 86–87 chunks collection, 14, 84 content of, 88

294

$circle shape, 44 close() command, 29 close() function (Mongo), 102 cloud-based datastores, for backups, 202 clusters of servers. See replication collection argument, DBRef(), 164 collection parameter, create() [DBRef], 131 collections, 11 about, 36 accessing directly, 89 backing up single, 197 capped, 37, 53 counting documents in, 55 with PHP driver, 108 with PyMongo, 145 defined, 35 inserting data into, 48–49 with PHP driver, 102–4 with PyMongo, 139–40 using batches, 239 reindexing, 237 removing, 74, 129, 163 removing documents from, 74 renaming, 74 repairing datafiles, 220 repairing indexes, 220 repairing validation faults, 219 size of, determining, 54 validating single collection, 218 viewing available, 47 collision (cryptographic), 88 comma-delimited data, 6 commands, MongoDB shell, 24 comments in blog (example), 174–75 compatibility of MongoDB, 9 complex data structures, 6 composite indexes, 13 compound indexing, 79, 234 compound keys, 43 compound primary keys, 10 cond parameter (group), 57 conditional operators, 57–65 with PHP driver, 111–18 with PyMongo, 148–53, 155

■ INDEX

config server (sharding), 281 configuring servers, 213, 214 conn column (mongostat), 222 connect() command, 30 connecting to database, 92 with PHP driver, 101–2 with PyMongo, 138–39 with replica pairs, 260 connecting to PHP driver, 29 copies of database, multiple, 4 CouchDB, 12 count() function, 55 count(true), 55 invoked from PHP driver, 108 invoked from PyMongo, 145 counter field, 41 create() function (DBRef), 131 create_index() function (PyMongo), 147 createCollection() function, 37, 53 creating indexes, performance and, 42. See also indexing documents creation date (files), 87 credentials, user. See authentication criteria argument (update), 65 CSV data, importing, 206 CSV format, 6 cursor element, explain(), 228

■D

--d option (mongodump), 197 --d option (mongorestore), 199 data exporting into MongoDB, 207–8 importing into MongoDB, 206–7 querying for. See queries reading and writing, 84 securing, 208–12 validating and repairing, 217–20 repairing collection faults, 219 repairing datafiles, 220 repairing indexes, 220 repairing server for, 217 single collection, 218

data hashing function. See sharding key function data isolation, replication and, 243 data model, 35–46 building indexes, 41–42 designing database, 35–41 collections, about, 36 documents, using, 38–40 _id field, creating, 40–41 geospatial indexing, 42–45 querying geospatial information, 43–45 data partitioning, 278–79 data replication, 15 data structures, 6 data types, 38 matching results by, 62 data updates. updating data /data/db directory, 22 database administration, 193–223 backing up MongoDB server, 194–97 automatic backups, 199–203 customization of, 197–98 large databases, 203–5 exporting data into MongoDB, 207–8 importing data into MongoDB, 206–7 log files, using, 217 monitoring MongoDB, 221–22 securing data, 208–12 server management, 212–16 getting server status, 214 getting version number, 214 reconfiguring servers, 213 shutting down, 216 starting servers, 212 database administration (continued) tools for, 194 upgrading MongoDB, 221 validating and repairing data, 217–20 repairing collection faults, 219 repairing datafiles, 220 repairing indexes, 220 repairing server for, 217 single collection, 218 database argument, DBRef() function, 164

295

■ INDEX

database files. See files database parameter, create() [DBRef], 131 databases about, 11 administration. See database administration backing up single, 197. See also backing up MongoDB server connecting to, 92 with replica pairs, 260 designing, 35–41 collections, about, 36 documents, using, 38–40 _id field, creating, 40–41 files for. See files multiple copies of, 4 natural order, 53 navigating, 47–48 nonrelational, 7, 35 querying. See queries referencing, 75–78 with DBRef, 76–78, 130–32, 163 manually, 75–76 removing data from, 74–75 with PHP driver, 129–30, 179, 186 with PyMongo, 162 removing documents from, 74–75 with PHP driver, 129–30, 179, 186 with PyMongo, 162 removing entire, 75, 130, 163 replicating data and, 15 schemaless, 12, 35 storing files, 84 updating. See updating data viewing available, 47 datastores for backups local, 199 remove (cloud-based), 202 Date data type, 38 db command, 48 db directory, 22 --dbpath flag (mongod), 23, 250 dbpath configuration option, 214 DBRef, 130–32, 163

296

in blog application (example), 173, 184 DBRef() function (PyMongo), 164 delete command (mongofiles), 90, 94 delete() function (MongoGridFS), 135 deleting field values, 67, 122, 157 files, 94 files from database, 90, 135 indexes, 42, 236 posts from blog application (example), 179 shards, from clusters, 287 slaves datafiles, for resync, 250 users (credentials), 211 values from arrays, 69, 124–25, 159–60 dereference() function (PyMongo), 165 descending order, 78 developers, MongoDB, 16 development releases, MongoDB, 19 development system, replication and, 243 dictionaries, Python, 137 --directoryperdb option (mongodump), 198 disconnecting from database with PHP driver, 101–2 with PyMongo, 138–39 disconnecting from PHP driver, 29 disk layout, 205 distinct() function, 55, 145 doc parameter, update() [PyMongo], 154 document size, 87 document-orientated storage, 11. See also BSON documents, 9 accessing directly, 89 atomic operations on, 71–73 with PHP driver, 121–25, 126–28 with PyMongo, 156, 161 collections. See collections counting, in collections, 55 with PHP driver, 108 with PyMongo, 145 creating links between, 130–32, 163 in blog application (example), 173, 184 defined, 35 embedded vs. referenced data, 39–40

■ INDEX

example (blog application), 168–69 how used, 38–40 indexing. See indexing documents natural order, 53 in PHP, 100 in Python, 137–38 removing, 74–75 with PHP driver, 129–30, 179, 186 with PyMongo, 162 skipping in queries, 52 blog application (example), 171 with PHP driver, 108 with PyMongo, 144 unique identifiers for, 10. See also _id identifier updating. See updating data dot notation, 51, 106, 142 Double data type, 38 --dpath option (mongodump), 198 draining shards, 287 drivers, MongoDB, 24–33 PHP driver, 25–30 Python driver, 30–33 drop() function (MongoDB), 74, 130 drop() function (PHP), 129 drop() function (PyMongo), 163 --drop option (mongorestore), 196, 199 --drop option (mongoimport), 207 drop_collection() function (PyMongo), 163 drop_database() function (PyMongo), 163 dropDatabase() function, 75 dropdups option, ensureIndex(), 236 dropIndex() function, 236 dropIndexes() function, 236 duplicates, disallowing. See uniqueness, index durability, replication and, 242 dynamic queries, 12

■E

easy_install command, 31 $elemMatch operator, 63 embedded documents, 7 data partitioning and, 278 indexes on, 13

embedding information in documents, 39–40 ensureIndex() function, 42, 78, 233 background option, 235 options for, 235 even integers, searching for, 61 $exists operator, 62, 117 explain() function, 226–28 exporting data into MongoDB, 207–8 Ext (Extensions) directory, 28

■F

--f option (mongoexport), 208 --fastsync option (mongod), 249, 250 features of MongoDB, 8, 11–16 field parameter, findandmodify(), 127 field values adding to arrays, 68, 123, 158 appending to fields, 68, 123, 157 deleting, 67, 122, 157 deleting from arrays, 69, 124–25, 159–60 editing, 67, 122, 156 fields length of field names, 239 querying with PHP driver, 106–7 with PyMongo, 142 fields parameter, findandmodify(), 161 Filename key, 86 files adding metadata to, 133 adding to database, 85, 93 with PHP driver, 133 deleting from database, 90, 94 with PHP driver, 135 hashing, 87 length of, determining, 86 managing with PyMongo, 93–94 memory-mapped, about, 225 repairing, 220 retrieving, 94 retrieving from database, 91, 134 storage of, 84 files collection, 14, 84, 87 filtering query results. See query results

297

■ INDEX

finalize parameter (group), 57 find_one() function (PyMongo), 140 find() function, 49–53. See also queries dot notation, 51, 106, 142 explain() function with, 228–32 with PHP driver, 104, 105 in blog example, 175–76, 182 find() function (PyMongo), 141–43 findandmodify() function, 73 invoked from PHP driver, 126–28 invoked from PyMongo, 161 finding slow queries, 227 findOne() function, 55, 76 in blog application (example), 173 with PHP driver, 104 forward natural order, 53 framesets, designing, 180 freezing master server for writes, 250 fsync operation (backups), 204 fsync option, insert() (PHP), 103 fsync option, remove() (PHP), 129 fsync option, update() (PHP), 119

■G

generating keys, 9 geoNear() function, 45. See also queries geospatial indexing, 13, 42–45 querying geospatial information, 43–45 get command (mongofiles), 91, 94 get() function (DBRef), 132 getBytes() function (MongoGridFSFile), 134 getFilename() function (MongoGridFS), 134 getlasterror method, 72 github website, 27 Google group on MongoDB, 17 greater than or equal parameter (find), 58, 113, 149 greater than parameter (find), 58, 112, 148 min() function vs., 81 GridFS, 14, 83–94 accessing from Python, 91–94 mongofiles command-line utility, 85–88 PHP driver and, 132–35 group() function, 56, 109

298

grouping query results, 56 with PHP driver, 109–10 with PyMongo, 146–47 $gt parameter, 58, 112, 148 min() function vs., 81 $gte parameter, 58, 113, 149

■H

hardware, optimizing for performance, 225–26 hashing files, 87 --headerline option (mongoimport), 207 help on MongoDB, 16–17 hint() function, 80, 238 invoked from PHP driver, 111 invoked from PyMongo, 147 horizontal partitioning, 278–79 hostname command, 264 hotspots (sharding system), 280 hyperthreading, 290

■I

i flag (MongoRegex), 118 id argument, DBRef() function, 164 _id identifier, 10 creating, 40–41 referencing data manually, 75–76, 130, 163 using with GridFS, 86 id parameter, create() [DBRef], 131 _id parameter, update(), 178 identify indexes, 233 %idx miss column (mongostat), 222 --ignoreblanks option (mongoimport), 207 import pymongo command, 33 import() function, invoked from PyMongo, 147 importing data into MongoDB, 206–7 $in operator, 59, 114, 150 in-place updating, 14. See also updating data $inc operator, 66, 71, 72, 121, 156 indexBounds element, explain(), 228 indexes, 232–38 creating compound, 234 creating simple, 233 how selected, 237–38. See also hint() listing, 233

■ INDEX

options for, 235–37 repairing, 220 requiring for certain queries, 80, 238 with PHP driver, 111 with PyMongo, 147 unique, creating, 236 indexes.find() function, 42 indexing documents, 13, 41–42, 78–81 in background, 235 deleting indexes, 236 enforcing uniqueness, 13 geospatial indexing, 13, 42–45 performance implications, 42, 79 info field, system.profile record), 227 initial parameter (group), 56 initialsynccomplete element (pair.sync), 247 in-place updating, 14. See also updating data insert() function, 48, 239 invoked from PHP driver, 102, 177 in blog application (example), 178 invoked from PyMongo, 139 inserting data into collections, 48–49 PHP driver for, 102–4 PyMongo for, 139–40 using batches, 239 installing MongoDB, 19–22 choosing version, 19–20 under Linux, 20–22 under Windows, 22 installing PHP driver, 25–30 automatically on UNIX platforms, 26 manually on UNIX platforms, 27 on Windows, 28 installing Python driver, 30–33 Integer data type, 38 interleaved replication, 255–56 isdbgrid command, 288 isolation, replication and, 243 issue tracking for MongoDB, 17

■J

JavaScript Code data type, 39 JavaScript query expressions, 65 JIRA tracking system, 17

journaling filesystem, for snapshots, 203 JSON, 5–7 BSON vs., 12 importing JSON data, 206 justOne option, remove() function (PHP), 129

■K

key parameter (group), 56 keyf parameter (group), 57 keys atomic updates on, 66, 71, 121, 156 constraining query matches to, 80 defined, 9 embedded in arrays, indexing, 79 generating (creating), 9 how to use, 10 _id field. See _id identifier names for, 49 killOp() function, 235

■L

l flag (MongoRegex), 118 large databases, backups of, 203–5 lazy writes, 14 length, file, 86 less than or equal parameter (find), 58, 113, 149 less than parameter (find), 58, 112, 148 min() function vs., 81 levels, profiling, 227 limit() function, 52 with count, 55 invoked from PHP driver, 108 invoked from PyMongo, 144 $slice operator vs., 60, 116 linebreaks in shell commands, 48 linking documents, 130–32 in blog application (example), 173, 184 links (navigation) for paging, 171 Linux, installing MongoDB under, 20–22 list command (mongofiles), 85, 90 listCollections() function (Mongo), 102 listDBs() function (Mongo), 102 listshards command, 283, 286, 287 Little Endian, about, 41

299

■ INDEX

local datastores, for backups, 199 localhost, 92 lock operation (backups), 204 % locked miss column (mongostat) locked column, 222 percent locked column, 222 locking master server for writes, 250 log files, using, 217 logappend configuration option, 214 logpath configuration option, 214, 217 $lt parameter, 58, 112, 148 max() function vs., 81 $lte parameter, 58, 113, 149

■M

m flag (MongoRegex), 118 managing servers, 212–16 getting server status, 214 getting version number, 214 reconfiguring servers, 213 shutting down, 216 starting servers, 212 manipulate argument, update() [PyMongo], 155 manual installation of MongoDB, 21 manual referencing with DBRef, 130, 163 manual sharding, 15 manually defined compound indexes, 235 map dictionary (PyMongo), 146 map function, 16 map_reduce() function (PyMongo), 146–47 Map/Reduce, 109–10 mapreduce parameter (group), 57 mapreduce() function (PHP), 109–10 master databases, 15 --master option (mongod), 245 master/master replication, 8, 254–55 master/slave replication configuring, 248–49 multiple master, single slave, 251–53 replica pairs, 256–74 connecting applications to, 260 coping with failure, 259 resolving disputes with arbiter, 261 replica sets, 262–74

300

adding servers to, 266–67 connecting to, from application, 273 creating, 264 determining status of, 273 implementing shards with, 290 master/slave replication (continued) replica sets (continued) launching member, 265–66 managing, 267–71 options for members, 271–73 resynchronizing, 249–50 scenarios for, 254–56 cascade replication, 254 interleaved replication, 255–56 master/master replication, 254–55 single master, multiple slave, 248 single master, single slave, 244–48 matches, query. See query results max: parameter (createCollection), 54 max() function, 80 maximum number of query results, 52 with PHP driver, 108 with PyMongo, 144 MaxKey data type, 38 MD5 hashing algorithm, 87 me collection, 247 members structure (replica sets), 271 memcached application, 8 memory, how used, 225 memory-mapped files, about, 225 metadata adding to files, 133 Microsoft Windows installing MongoDB under, 22 installing PHP driver on, 28 installing PyMongo under, 31 milis field, system.profile record), 227 millis element, explain(), 229 min() function, 80 MinKey data type, 38 $mod operator, 61 modifier operations. See save() command; update() function; updating data modules, PyMongo, 138

■ INDEX

mongo application, 23 Mongo class, 29, 100–102 mongo console, 194 authenticating in, 209 MongoCode class, 109 MongoCollection class, 101, 103 MongoCursor class, 101, 107 mongod application, 23, 223 MongoDB, installing, 19–22 choosing version, 19–20 under Linux, 20–22 under Windows, 22 MongoDB, running, 22–24 MongoDB drivers, 24–33 PHP driver, 25–30 Python driver, 30–33 #MongoDB channel, 16 MongoDB class, 101, 130 MongoDB philosophy, 3–9 MongoDB profiler, 226–28, 229–32 MongoDB shell, 23 mongodb.conf file, 213 mongodb-user group, 17 mongodump utility, 195, 196 mongoexport utility, 207–8 mongofiles utility, 85–88 MongoGridFS class, 133 MongoGridFSCursor class, 133 MongoGridFSFile class, 133 mongoimport utility, 206–7 MongoRegex class, 118–19 mongorestore utility, 195, 198 mongos daemon, 280 mongostat utility, 221–22 monitoring MongoDB, 221–22 mounting filesystems, 205 multi argument, update(), 65, 155 multi-key indexes, 234 multiple argument, update() (PHP), 119 multiple copies of database, 4 multiple expressions in documents, 60, 115, 151 multiple keys (multi keys), 79 multiple-master, single-slave replication, 251–53

multiple-slave, single-master replication, 248. See also replication

■N

n element, explain(), 229 names case sensitivity, 193 for collections, 37, 74 of fields, length of, 239 for keys, 49 namespaces, limit on, 38 natural order, 53 $natural parameter, 53 navigating databases, 47–48 navigation links for paging, 171 $ne parameter, 59, 114, 150 $near operator, 44 new parameter, findandmodify(), 127, 161 $nin operator, 59, 151 nonrelational databases, 7, 35 not equals parameter (find), 59, 114, 150 $not meta-operator, 64 nScanned element, explain(), 228 nScannedObjects element, explain(), 229 nssize parameter, 38 Null data type, 38

■O

--o (out) option (mongodump), 198 --objcheck option (mongorestore), 199 Object data type, 38 Object ID data type, 38 objNew argument (update), 65 odd integers, searching for, 61 $offset parameter, for paging, 171 :1 and :-1 parameters (ensureIndex), 78 --only option (mongod), 249, 256 opcounter section, serverStatus() output, 216 oplog, 243–44 oplog.$main collection, 246 optimization, 225–39 index management, 232–38 creating compound indexes, 234 creating simple indexes, 233

301

■ INDEX

index selection, 237–38 listing indexes, 233 specifying index options, 235–37 query performance, 226–38 evaluating with explain(), 228–32 evaluating with MongoDB profiler, 226–28, 229–32 server hardware, 225–26 sharding for, 290 storage of small objects, 238–39 options parameter, update(), 119 --oplogSize option (mongod), 243, 249 $or operator, 60, 115, 151 ordering documents ascending order, defined, 52 capped collections. See capped collections natural order, 53 in results lists, 52 with PHP driver, 107 with PyMongo, 143 ordering index elements, 78

■P

paging, 60, 116, 145 blog application example, 171 pair.sync collection, 247 partitioning data, 278–79 passive members, replica sets, 263 password, changing, 210 Pastie website, 17 PECL repository, 26 performance, 8 indexes and, 42, 79 query results and, 44 replication and, 242 performance optimization, 225–39 index management, 232–38 creating compound indexes, 234 creating simple indexes, 233 index selection, 237–38 listing indexes, 233 specifying index options, 235–37 query performance, 226–38 evaluating with explain(), 228–32

302

evaluating with MongoDB profiler, 226–28, 229–32 server hardware, 225–26 sharding for, 290 storage of small objects, 238–39 permissions, changing, 210 PHP, authentication with, 212 PHP, documents in, 99–100 PHP driver, 99–135. See also blog application connecting to database, 101–2 core MongoDB classes, 100–101 DBRef with, 130–32 in blog application (example), 173, 184 deleting data, 129–30 GridFS and, 132–35 inserting data, 102–4 installing, 25–30 automatically on UNIX platforms, 26 manually on UNIX platforms, 27 on Windows, 28 modifying data, 119–28 atomically, 126–28 using modifier operators, 121–25 using save(), 125–26 using update(), 119–21, 178 querying for data, 104–19 in blog example, 175–76, 182 couting results, 108 filtering for specific information, 106–7 finding all documents, 105 grouping with Map/Reduce, 109–10 querying for specific information, 106 regular expressions, 118–19 returning single document, 104 sorting, limiting, and skipping, 107–8 specifying index, 111 using conditional operators, 111–18 php.ini file, extensions section, 26 phpinfo() command, 28 $pop operator, 69, 124, 159 prerequisites to running MongoDB, 22 previous releases, MongoDB, 19 primary keys, 10 primary server, 262

■ INDEX

print_r() function (PHP), 102 printReplicationInfo() method, 243 printShardingStatus() command, 288 production releases, MongoDB, 19 profiling levels, 227 profiling queries, 14 profiling query performance, 226–28, 229–32 $pull operator, 70, 72, 125, 159 $pullAll operator, 70, 72, 125, 160 $push operator, 68, 72, 123, 157 $pushAll operator, 68, 72, 123, 158 put command (mongofiles), 85, 86, 90, 93 PyMongo driver, 92 connecting to database, 92, 138–39 DBRef with, 163 inserting data, 139–40 modifying data, 154 atomically, 161 using modifier operators, 156 using save(), 160 using update(), 154 modules, 138 querying for data, 140–54 couting results, 145 filtering for specific information, 142 finding all documents, 141 grouping with Map/Reduce, 146–47 querying for specific information, 142 regular expressions, 153–54 returning single document, 140 sorting, limiting and skipping, 143 PyMongo driver (continued) querying for data (continued) specifying index, 147 using conditional operators, 148–53, 155 PyMongo package, 30–33 Python, 137–66 accessing GridFS from, 91–94 documents in, about, 137 using PyMongo modules, 138 working with documents, 137–38 Python driver, installing, 30–33

■Q

--q option (mongoexport), 208 queries, 49–65. See also dynamic queries; find() function; geoNear() function $ character in, 70 aggregation commands, 55–57 with PHP driver, 108–10 with PyMongo, 145–47 conditional operators, 57–65 with PHP driver, 111–18 with PyMongo, 148–53, 155 dot notation, 51, 106, 142 of geospatial information, 43–45 JavaScript expressions in, 65 PHP driver for, 104–19 in blog application (example), 175–76, 182 filtering for specific information, 106–7 profiling, 14 PyMongo for, 140–54 filtering for specific information, 142 regular expressions in, 65 with PHP driver, 118–19 with PyMongo, 153–54 requiring specific indexes for, 80, 238 with PHP driver, 111 with PyMongo, 147 for single documents, 55 with PHP driver, 104 with PyMongo, 140 sort, limit, and skip functions, 52 with PHP driver, 107–8 with PyMongo, 143 query analyzer component, 238 query parameter, findandmodify(), 126, 161 query performance, 226–38 evaluating with explain(), 228–32 evaluating with MongoDB profiler, 226–28, 229–32 index management, 232–38 creating compound indexes, 234 creating simple indexes, 233 index selection, 237–38 listing indexes, 233

303

■ INDEX

specifying index options, 235–37 query plan, 237 query results arrays of matches, 59, 114, 150 based on BSON type, 62 constraining to specific index keys, 80 ensuring unique values, 55, 145 entire arrays, 63 excluding documents from, 59, 114, 150 filtering by size, 61 grouping, 56 with PHP driver, 109–10 with PyMongo, 146–47 maximum number of, setting, 52 with PHP driver, 108 with PyMongo, 144 sorting, 52 with PHP driver, 107 with PyMongo, 143

■R

RAC architecture, 7 re module (Python), 153–54 reading data, as hard, 84 read-only permissions, 211 Real Application Clusters (RAC), 7 rebalancing shards, 285 rebuilding indexes, 42. See also indexing documents reconfiguring servers, 213 reduce dictionary (PyMongo), 146 reduce function, 16 reduce parameter (group), 56 redundancy, replication and, 242 referencing databases, 75–78 with DBRef, 76–78, 130–32, 163 in blog application (example), 173, 184 manually, 75–76 referencing information in documents, 39–40 regular expressions, 39, 65 with PyMongo, 153–54 with PHP driver, 118–19 reIndex() function, 220, 237 reindexing collections, 237

304

reliability, replication and, 242 remote datastores, for backups, 202 remote partitioning. See partitioning data remove parameter, findandmodify(), 127, 128, 161 remove() function, 74, 211 invoked from PHP driver, 129 in blog application (example), 179, 186 invoked from PyMongo, 162 removeShard command, 287 removing data, 74–75 with PHP driver, 129–30 in blog application (example), 179, 186 with PyMongo, 162 renameCollection() function, 74 renaming collections, 74 --repair option (mongod), 217, 220 repairDatabase() function, 220 repairing data, 217–20 collection datafiles, 220 collection faults, 219 collection indexes, 220 repairing server for, 217 --repairpath option (mongod), 218 replica pairs, 15 replicating data, 15 replication, 8, 241–74, 278 configuring, 248–49 goals of, 242–43 hardware for, 226 multiple-master, single-slave, 251–53 oplog, about, 243–44 replica pairs, 256–74 connecting applications to, 260 coping with failure, 259 resolving disputes with abriters, 261 replica sets, 262–74 adding servers to, 266–67 connecting to, from application, 273 creating, 264 determining status of, 273 implementing shards with, 290 launching member, 265–66 managing, 267–71 options for members, 271–73

■ INDEX

resynchronizing, 249–50 scenarios for, 254–56 cascade replication, 254 interleaved replication, 255–56 master/master replication, 254–55 single-master, multiple-slave, 248 single-master, single-slave, 244–48 --replSet option (mongod), 265 repositories, installing MongoDB through, 21 requirements for running MongoDB, 22 --rest option (mongod), 214, 265 restricting server access, 208–12 results, query. See also queries arrays of matches, 59, 114, 150 based on BSON type, 62 constraining to specific index keys, 80 ensuring unique values, 55, 145 entire arrays, 63 excluding documents from, 59, 114, 150 filtering by size, 61 results, query (continued) grouping, 56 with PHP driver, 109–10 with PyMongo, 146–47 maximum number of, setting, 52 with PHP driver, 108 with PyMongo, 144 sorting, 52 with PHP driver, 107 with PyMongo, 143 resynchronizing master-slave replication, 249–50 retrieving files, 91, 94, 134 right tool for the right job, 3 rights, changing, 210 rs.add() method, 267, 271 rs.addArbiter() method, 267 rs.conf() method, 268 rs.help() method, 267 rs.initiate() method, 266, 267, 271 rs.isMaster() method, 268, 270 rs.status() method, 266, 267, 268 rs.stepDown() method, 267, 269 running MongoDB, 22–24

■S

s flag (MongoRegex), 118 s3cmd utility, 202 safe argument, update() (PHP), 119 safe argument, update() (PyMongo), 155 safe option, insert() (PHP), 103 safe option, remove() (PHP), 129 save() function, 66 invoked from PHP driver, 125–26 invoked from PyMongo, 160 scalability, replication and, 242 schema design, 7, 36 schemaless databases, 12, 35 search command (mongofiles), 90 secondary keys, 43 secondary servers, 262 securing data, 208–12 selectCollection() function (Mongo), 101 selectDB() function (Mongo), 101 server backing up, 194–97 automatic backups, 199–203 customization of, 197–98 large databases, 203–5 hardware, optimizing, 225–26 management of, 212–16 reconfiguring, 213 repairing, 217 replication. See replication restricting access to, 208–12 shutting down, 216 starting, 212 status of, getting, 214 version of, getting, 214 serverStatus() command, 214, 222 $set operator, 67, 72, 122, 156 $set parameter, update(), 178 setProfilingLevel() function, 227 settings structure (replica sets), 272 SHA encryption, 88 shapes, for queries, 44 sharding, 15, 242, 277–91 auto-sharding, 15, 279 configuration for, 282–88

305

■ INDEX

adding shards to cluster, 285 removing shards from cluster, 287 data partitioning, 278–79 determining how connected, 288 getting status of shared clusters, 288–90 implementing, 280–82, 290 improving performance with, 290 need for, 277–78 scenario (example), 279–80 sharding key function, 278 sharding keys, 290 show collections command, 24, 48 show dbs command, 24, 47 show users command, 24 shutdownServer() command, 216 shutting down servers, 216 SIG_KILL(-9) signal, 216 single-document queries, 55 with PHP driver, 104 with PyMongo, 140 single-master, multiple-slave replication, 248 single-master, single-slave replication, 244–48 single-slave, multiple-master replication, 251–53 single-threaded operations, 290 size, chunks, 86–87 size, document, 87 size, file, 86 size, oplog, 243 $size operator, 61 skip() function, 52 $slice operator vs., 60, 116 in blog application (example), 171 with count, 55 invoked from PHP driver, 108 invoked from PyMongo, 144 skipping documents in queries, 52 blog application (example), 171 with PHP driver, 108 with PyMongo, 144 slave databases, 15 --slave option (mongod), 245 slave servers for backups, 203 --slavedelay option (mongod), 249

306

slaves collection, 246 $slice operator, 60, 116, 152 in blog application (example), 173 slow queries, finding, 227 small objects, storage of, 238–39 snapshots, 203 sort parameter, findandmodify(), 126, 128, 161 sort() function, 52 invoked from PHP driver, 107 invoked from PyMongo, 143 on unindexed fields, 232 sorting index elements, 78 sorting query results, 52 with PHP driver, 107 with PyMongo, 143 --source option (mongod), 246 sources collection, 247 stable releases, MongoDB, 20 staging system, replication and, 243 starting servers, 212 status, server, 214 storage of small objects, 238–39 storeUpload() function (MongoGridFS), 133 storing binary data, 14, 84. See also files; GridFS storing files, 84. See also files String data type, 38 subdocument compound indexes, 234 support for MongoDB, 16–17 symbol data type, 38 syncedTo element (sources), 247 --sysinfo flag, 23 system.indexes collection, 42, 48, 233 system.profile collection, 226, 227 system.users collection, 208

■T

tagcloud functions, 56 third-party administration tools, 194 Timestamp data type, 38 timestamp field, 41 transactions, lack of innate support for, 5 ts field, system.profile record), 227 TSV data, importing, 206 $type operator, 62

■ INDEX

■U

u flag (MongoRegex), 118 unique option, ensureIndex(), 236 uniqueness of document identifiers, 10. See also _id identifier of indexes, 13, 236 of query results, ensuring, 55, 145 Unix platforms, installing PHP driver on, 26–27 $unset operator, 67, 72, 122, 157 unstable releases, MongoDB, 20 Update if current method, 72 update parameter, findandmodify(), 127, 161 update() function, 65 invoked from PHP driver, 119 in blog application (example), 178 invoked from PyMongo, 154 update() function (MongoGridFS), 133 in blog application (example), 175 updating data, 65–73 atomic operations, 71–73 with PHP driver, 121–25, 126–28 with PyMongo, 156, 161 automatically, 66–70 in capped collections, 54 in-place updates, 14 with PHP driver, 119–21 in blog application (example), 178 with PyMongo, 154 upserts. See upserts --upgrade option (mongod), 221 upgrading MongoDB, 221 uploadDate key, 87 upsert argument, update(), 65, 119, 154 upsert parameter, findandmodify(), 161 upserts with save() function, 66, 125–26, 160 with update() function, 65

use command, 24, 43, 47 user authentication. See authentication

■V

validate() function, 54, 218 validating data, 217–20 repairing collection faults, 219 repairing datafiles, 220 repairing indexes, 220 repairing server for, 217 single collection, 218 values for keys, 10. See also keys version, MongoDB, 19–20 version, server, 214 version() command, 214 version numbers, MongoDB, 20 vertical partitioning, 278 viewing available databases and collections, 47 volume managers, 205

■W

website for MongoDB, 16 while() function (PHP), 105 Windows installing MongoDB under, 22 installing PHP driver on, 28 installing PyMongo under, 31 $within operator, 44 writing data, as hard, 84

■X

x flag (MongoRegex), 118 XML for data structures, 6

307