[redis.git] / doc / FAQ.html


<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN">
<html>
    <head>
        <link type="text/css" rel="stylesheet" href="style.css" />
    </head>
    <body>
        <div id="page">
        
            <div id='header'>
            <a href="index.html">
            <img style="border:none" alt="Redis Documentation" src="redis.png">
            </a>
            </div>
        
            <div id="pagecontent">
                <div class="index">
<!-- This is a (PRE) block.  Make sure it's left aligned or your toc title will be off. -->
<b>FAQ: Contents</b><br>&nbsp;&nbsp;<a href="#Why I need Redis if there is already memcachedb, Tokyo Cabinet, ...?">Why I need Redis if there is already memcachedb, Tokyo Cabinet, ...?</a><br>&nbsp;&nbsp;<a href="#Isn't this key-value thing just hype?">Isn't this key-value thing just hype?</a><br>&nbsp;&nbsp;<a href="#Can I backup a Redis DB while the server is working?">Can I backup a Redis DB while the server is working?</a><br>&nbsp;&nbsp;<a href="#What's the Redis memory footprint?">What's the Redis memory footprint?</a><br>&nbsp;&nbsp;<a href="#I like Redis high level operations and features, but I don't like it takes everything in memory and I can't have a dataset larger the memory. Plans to change this?">I like Redis high level operations and features, but I don't like it takes everything in memory and I can't have a dataset larger the memory. Plans to change this?</a><br>&nbsp;&nbsp;<a href="#Ok but I absolutely need to have a DB larger than memory, still I need the Redis features">Ok but I absolutely need to have a DB larger than memory, still I need the Redis features</a><br>&nbsp;&nbsp;<a href="#I have an empty Redis server but INFO and logs are reporting megabytes of memory in use!">I have an empty Redis server but INFO and logs are reporting megabytes of memory in use!</a><br>&nbsp;&nbsp;<a href="#What happens if Redis runs out of memory?">What happens if Redis runs out of memory?</a><br>&nbsp;&nbsp;<a href="#How much time it takes to load a big database at server startup?">How much time it takes to load a big database at server startup?</a><br>&nbsp;&nbsp;<a href="#What Redis means actually?">What Redis means actually?</a><br>&nbsp;&nbsp;<a href="#Why did you started the Redis project?">Why did you started the Redis project?</a>
                </div>
                
                <h1 class="wikiname">FAQ</h1>

                <div class="summary">
                    
                </div>

                <div class="narrow">
                    <h1><a name="Why I need Redis if there is already memcachedb, Tokyo Cabinet, ...?">Why I need Redis if there is already memcachedb, Tokyo Cabinet, ...?</a></h1>Memcachedb is basically memcached done persistent. Redis is a different evolution
path in the key-value DBs, the idea is that the main advantages of key-value DBs
are retained even without a so severe loss of comfort of plain key-value DBs.
So Redis offers more features:<br/><br/><ul><li> Keys can store different data types, not just strings. Notably Lists and Sets. For example if you want to use Redis as a log storage system for different computers every computer can just <code name="code" class="python">RPUSH data to the computer_ID key</code>. Don't want to save more than 1000 log lines per computer? Just issue a <code name="code" class="python">LTRIM computer_ID 0 999</code> command to trim the list after every push.</li></ul>
<ul><li> Another example is about Sets. Imagine to build a social news site like <a href="http://reddit.com" target="_blank">Reddit</a>. Every time a user upvote a given news you can just add to the news_ID_upmods key holding a value of type SET the id of the user that did the upmodding. Sets can also be used to index things. Every key can be a tag holding a SET with the IDs of all the objects associated to this tag. Using Redis set intersection you obtain the list of IDs having all this tags at the same time.</li></ul>
<ul><li> We wrote a <a href="http://retwis.antirez.com" target="_blank">simple Twitter Clone</a> using just Redis as database. Download the source code from the download section and imagine to write it with a plain key-value DB without support for lists and sets... it's <b>much</b> harder.</li></ul>
<ul><li> Multiple DBs. Using the SELECT command the client can select different datasets. This is useful because Redis provides a MOVE atomic primitive that moves a key form a DB to another one, if the target DB already contains such a key it returns an error: this basically means a way to perform locking in distributed processing.</li></ul>
<ul><li> <b>So what is Redis really about?</b> The User interface with the programmer. Redis aims to export to the programmer the right tools to model a wide range of problems. <b>Sets, Lists with O(1) push operation, lrange and ltrim, server-side fast intersection between sets, are primitives that allow to model complex problems with a key value database</b>.</li></ul>
<h1><a name="Isn't this key-value thing just hype?">Isn't this key-value thing just hype?</a></h1>I imagine key-value DBs, in the short term future, to be used like you use memory in a program, with lists, hashes, and so on. With Redis it's like this, but this special kind of memory containing your data structures is shared, atomic, persistent.<br/><br/>When we write code it is obvious, when we take data in memory, to use the most sensible data structure for the work, right? Incredibly when data is put inside a relational DB this is no longer true, and we create an absurd data model even if our need is to put data and get this data back in the same order we put it inside (an ORDER BY is required when the data should be already sorted. Strange, dont' you think?).<br/><br/>Key-value DBs bring this back at home, to create sensible data models and use the right data structures for the problem we are trying to solve.<h1><a name="Can I backup a Redis DB while the server is working?">Can I backup a Redis DB while the server is working?</a></h1>Yes you can. When Redis saves the DB it actually creates a temp file, then rename(2) that temp file name to the destination file name. So even while the server is working it is safe to save the database file just with the <i>cp</i> unix command. Note that you can use master-slave replication in order to have redundancy of data, but if all you need is backups, cp or scp will do the work pretty well.<h1><a name="What's the Redis memory footprint?">What's the Redis memory footprint?</a></h1>Worst case scenario: 1 Million keys with the key being the natural numbers from 0 to 999999 and the string &quot;Hello World&quot; as value use 100MB on my Intel macbook (32bit). Note that the same data stored linearly in an unique string takes something like 16MB, this is the norm because with small keys and values there is a lot of overhead. Memcached will perform similarly.<br/><br/>With large keys/values the ratio is much better of course.<br/><br/>64 bit systems will use much more memory than 32 bit systems to store the same keys, especially if the keys and values are small, this is because pointers takes 8 bytes in 64 bit systems. But of course the advantage is that you can have a lot of memory in 64 bit systems, so to run large Redis servers a 64 bit system is more or less required.<h1><a name="I like Redis high level operations and features, but I don't like it takes everything in memory and I can't have a dataset larger the memory. Plans to change this?">I like Redis high level operations and features, but I don't like it takes everything in memory and I can't have a dataset larger the memory. Plans to change this?</a></h1>The whole key-value hype started for a reason: performances. Redis takes the whole dataset in memory and writes asynchronously on disk in order to be very fast, you have the best of both worlds: hyper-speed and persistence of data, but the price to pay is exactly this, that the dataset must fit on your computers RAM.<br/><br/>If the data is larger then memory, and this data is stored on disk, what happens is that the bottleneck of the disk I/O speed will start to ruin the performances. Maybe not in benchmarks, but once you have real load from multiple clients with distributed key accesses the data must come from disk, and the disk is damn slow. Not only, but Redis supports higher level data structures than the plain values. To implement this things on disk is even slower.<br/><br/>Redis will always continue to hold the whole dataset in memory because this days scalability requires to use RAM as storage media, and RAM is getting cheaper and cheaper. Today it is common for an entry level server to have 16 GB of RAM! And in the 64-bit era there are no longer limits to the amount of RAM you can have in theory.<h1><a name="Ok but I absolutely need to have a DB larger than memory, still I need the Redis features">Ok but I absolutely need to have a DB larger than memory, still I need the Redis features</a></h1>You may try to load a dataset larger than your memory in Redis and see what happens, basically if you are using a modern Operating System, and you have a lot of data in the DB that is rarely accessed, the OS's virtual memory implementation will try to swap rarely used pages of memory on the disk, to only recall this pages when they are needed. If you have many large values rarely used this will work. If your DB is big because you have tons of little values accessed at random without a specific pattern this will not work (at low level a page is usually 4096 bytes, and you can have different keys/values stored at a single page. The OS can't swap this page on disk if there are even few keys used frequently).<br/><br/>Another possible solution is to use both MySQL and Redis at the same time, basically take the state on Redis, and all the things that get accessed very frequently: user auth tokens, Redis Lists with chronologically ordered IDs of the last N-comments, N-posts, and so on. Then use MySQL as a simple storage engine for larger data, that is just create a table with an auto-incrementing ID as primary key and a large BLOB field as data field. Access MySQL data only by primary key (the ID). The application will run the high traffic queries against Redis but when there is to take the big data will ask MySQL for specific resources IDs.<h1><a name="I have an empty Redis server but INFO and logs are reporting megabytes of memory in use!">I have an empty Redis server but INFO and logs are reporting megabytes of memory in use!</a></h1>This may happen and it's prefectly ok. Redis objects are small C structures allocated and freed a lot of times. This costs a lot of CPU so instead of being freed, released objects are taken into a free list and reused when needed. This memory is taken exactly by this free objects ready to be reused.<h1><a name="What happens if Redis runs out of memory?">What happens if Redis runs out of memory?</a></h1>With modern operating systems malloc() returning NULL is not common, usually the server will start swapping and Redis performances will be disastrous so you'll know it's time to use more Redis servers or get more RAM.<br/><br/>However it is planned to add a configuration directive to tell Redis to stop accepting queries but instead to SAVE the latest data and quit if it is using more than a given amount of memory. Also the new INFO command (work in progress in this days) will report the amount of memory Redis is using so you can write scripts that monitor your Redis servers checking for critical conditions.<br/><br/>Update: redis SVN is able to know how much memory it is using and report it via the <a href="InfoCommand.html">INFO</a> command.<h1><a name="How much time it takes to load a big database at server startup?">How much time it takes to load a big database at server startup?</a></h1>Just an example on normal hardware: It takes about 45 seconds to restore a 2 GB database on a fairly standard system, no RAID. This can give you some kind of feeling about the order of magnitude of the time needed to load data when you restart the server.<h1><a name="What Redis means actually?">What Redis means actually?</a></h1>Redis means two things:
<ul><li> it's a joke on the word Redistribute (instead to use just a Relational DB redistribute your workload among Redis servers)</li><li> it means REmote DIctionary Server</li></ul>
<h1><a name="Why did you started the Redis project?">Why did you started the Redis project?</a></h1>In order to scale <a href="http://lloogg.com" target="_blank">LLOOGG</a>. But after I got the basic server working I liked the idea to share the work with other guys, and Redis was turned into an open source project.

                </div>
        
            </div>
        </div>
    </body>
</html>
Commit	Line	Data
ed9b544e	1
	2	<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN">
	3	<html>
	4	<head>
	5	<link type="text/css" rel="stylesheet" href="style.css" />
	6	</head>
	7	<body>
	8	<div id="page">
	9
	10	<div id='header'>
	11	<a href="index.html">
	12	<img style="border:none" alt="Redis Documentation" src="redis.png">
	13	</a>
	14	</div>
	15
	16	<div id="pagecontent">
	17	<div class="index">
	18	<!-- This is a (PRE) block. Make sure it's left aligned or your toc title will be off. -->
02fdd5ab	19	<b>FAQ: Contents</b><br>  <a href="#Why I need Redis if there is already memcachedb, Tokyo Cabinet, ...?">Why I need Redis if there is already memcachedb, Tokyo Cabinet, ...?</a><br>  <a href="#Isn't this key-value thing just hype?">Isn't this key-value thing just hype?</a><br>  <a href="#Can I backup a Redis DB while the server is working?">Can I backup a Redis DB while the server is working?</a><br>  <a href="#What's the Redis memory footprint?">What's the Redis memory footprint?</a><br>  <a href="#I like Redis high level operations and features, but I don't like it takes everything in memory and I can't have a dataset larger the memory. Plans to change this?">I like Redis high level operations and features, but I don't like it takes everything in memory and I can't have a dataset larger the memory. Plans to change this?</a><br>  <a href="#Ok but I absolutely need to have a DB larger than memory, still I need the Redis features">Ok but I absolutely need to have a DB larger than memory, still I need the Redis features</a><br>  <a href="#I have an empty Redis server but INFO and logs are reporting megabytes of memory in use!">I have an empty Redis server but INFO and logs are reporting megabytes of memory in use!</a><br>  <a href="#What happens if Redis runs out of memory?">What happens if Redis runs out of memory?</a><br>  <a href="#How much time it takes to load a big database at server startup?">How much time it takes to load a big database at server startup?</a><br>  <a href="#What Redis means actually?">What Redis means actually?</a><br>  <a href="#Why did you started the Redis project?">Why did you started the Redis project?</a>
ed9b544e	20	</div>
	21
	22	<h1 class="wikiname">FAQ</h1>
	23
	24	<div class="summary">
	25
	26	</div>
	27
	28	<div class="narrow">
	29	<h1><a name="Why I need Redis if there is already memcachedb, Tokyo Cabinet, ...?">Why I need Redis if there is already memcachedb, Tokyo Cabinet, ...?</a></h1>Memcachedb is basically memcached done persistent. Redis is a different evolution
	30	path in the key-value DBs, the idea is that the main advantages of key-value DBs
	31	are retained even without a so severe loss of comfort of plain key-value DBs.
	32	So Redis offers more features:<br/><br/><ul><li> Keys can store different data types, not just strings. Notably Lists and Sets. For example if you want to use Redis as a log storage system for different computers every computer can just <code name="code" class="python">RPUSH data to the computer_ID key</code>. Don't want to save more than 1000 log lines per computer? Just issue a <code name="code" class="python">LTRIM computer_ID 0 999</code> command to trim the list after every push.</li></ul>
	33	<ul><li> Another example is about Sets. Imagine to build a social news site like <a href="http://reddit.com" target="_blank">Reddit</a>. Every time a user upvote a given news you can just add to the news_ID_upmods key holding a value of type SET the id of the user that did the upmodding. Sets can also be used to index things. Every key can be a tag holding a SET with the IDs of all the objects associated to this tag. Using Redis set intersection you obtain the list of IDs having all this tags at the same time.</li></ul>
	34	<ul><li> We wrote a <a href="http://retwis.antirez.com" target="_blank">simple Twitter Clone</a> using just Redis as database. Download the source code from the download section and imagine to write it with a plain key-value DB without support for lists and sets... it's <b>much</b> harder.</li></ul>
	35	<ul><li> Multiple DBs. Using the SELECT command the client can select different datasets. This is useful because Redis provides a MOVE atomic primitive that moves a key form a DB to another one, if the target DB already contains such a key it returns an error: this basically means a way to perform locking in distributed processing.</li></ul>
	36	<ul><li> <b>So what is Redis really about?</b> The User interface with the programmer. Redis aims to export to the programmer the right tools to model a wide range of problems. <b>Sets, Lists with O(1) push operation, lrange and ltrim, server-side fast intersection between sets, are primitives that allow to model complex problems with a key value database</b>.</li></ul>
02fdd5ab	37	<h1><a name="Isn't this key-value thing just hype?">Isn't this key-value thing just hype?</a></h1>I imagine key-value DBs, in the short term future, to be used like you use memory in a program, with lists, hashes, and so on. With Redis it's like this, but this special kind of memory containing your data structures is shared, atomic, persistent.<br/><br/>When we write code it is obvious, when we take data in memory, to use the most sensible data structure for the work, right? Incredibly when data is put inside a relational DB this is no longer true, and we create an absurd data model even if our need is to put data and get this data back in the same order we put it inside (an ORDER BY is required when the data should be already sorted. Strange, dont' you think?).<br/><br/>Key-value DBs bring this back at home, to create sensible data models and use the right data structures for the problem we are trying to solve.<h1><a name="Can I backup a Redis DB while the server is working?">Can I backup a Redis DB while the server is working?</a></h1>Yes you can. When Redis saves the DB it actually creates a temp file, then rename(2) that temp file name to the destination file name. So even while the server is working it is safe to save the database file just with the <i>cp</i> unix command. Note that you can use master-slave replication in order to have redundancy of data, but if all you need is backups, cp or scp will do the work pretty well.<h1><a name="What's the Redis memory footprint?">What's the Redis memory footprint?</a></h1>Worst case scenario: 1 Million keys with the key being the natural numbers from 0 to 999999 and the string "Hello World" as value use 100MB on my Intel macbook (32bit). Note that the same data stored linearly in an unique string takes something like 16MB, this is the norm because with small keys and values there is a lot of overhead. Memcached will perform similarly.<br/><br/>With large keys/values the ratio is much better of course.<br/><br/>64 bit systems will use much more memory than 32 bit systems to store the same keys, especially if the keys and values are small, this is because pointers takes 8 bytes in 64 bit systems. But of course the advantage is that you can have a lot of memory in 64 bit systems, so to run large Redis servers a 64 bit system is more or less required.<h1><a name="I like Redis high level operations and features, but I don't like it takes everything in memory and I can't have a dataset larger the memory. Plans to change this?">I like Redis high level operations and features, but I don't like it takes everything in memory and I can't have a dataset larger the memory. Plans to change this?</a></h1>The whole key-value hype started for a reason: performances. Redis takes the whole dataset in memory and writes asynchronously on disk in order to be very fast, you have the best of both worlds: hyper-speed and persistence of data, but the price to pay is exactly this, that the dataset must fit on your computers RAM.<br/><br/>If the data is larger then memory, and this data is stored on disk, what happens is that the bottleneck of the disk I/O speed will start to ruin the performances. Maybe not in benchmarks, but once you have real load from multiple clients with distributed key accesses the data must come from disk, and the disk is damn slow. Not only, but Redis supports higher level data structures than the plain values. To implement this things on disk is even slower.<br/><br/>Redis will always continue to hold the whole dataset in memory because this days scalability requires to use RAM as storage media, and RAM is getting cheaper and cheaper. Today it is common for an entry level server to have 16 GB of RAM! And in the 64-bit era there are no longer limits to the amount of RAM you can have in theory.<h1><a name="Ok but I absolutely need to have a DB larger than memory, still I need the Redis features">Ok but I absolutely need to have a DB larger than memory, still I need the Redis features</a></h1>You may try to load a dataset larger than your memory in Redis and see what happens, basically if you are using a modern Operating System, and you have a lot of data in the DB that is rarely accessed, the OS's virtual memory implementation will try to swap rarely used pages of memory on the disk, to only recall this pages when they are needed. If you have many large values rarely used this will work. If your DB is big because you have tons of little values accessed at random without a specific pattern this will not work (at low level a page is usually 4096 bytes, and you can have different keys/values stored at a single page. The OS can't swap this page on disk if there are even few keys used frequently).<br/><br/>Another possible solution is to use both MySQL and Redis at the same time, basically take the state on Redis, and all the things that get accessed very frequently: user auth tokens, Redis Lists with chronologically ordered IDs of the last N-comments, N-posts, and so on. Then use MySQL as a simple storage engine for larger data, that is just create a table with an auto-incrementing ID as primary key and a large BLOB field as data field. Access MySQL data only by primary key (the ID). The application will run the high traffic queries against Redis but when there is to take the big data will ask MySQL for specific resources IDs.<h1><a name="I have an empty Redis server but INFO and logs are reporting megabytes of memory in use!">I have an empty Redis server but INFO and logs are reporting megabytes of memory in use!</a></h1>This may happen and it's prefectly ok. Redis objects are small C structures allocated and freed a lot of times. This costs a lot of CPU so instead of being freed, released objects are taken into a free list and reused when needed. This memory is taken exactly by this free objects ready to be reused.<h1><a name="What happens if Redis runs out of memory?">What happens if Redis runs out of memory?</a></h1>With modern operating systems malloc() returning NULL is not common, usually the server will start swapping and Redis performances will be disastrous so you'll know it's time to use more Redis servers or get more RAM.<br/><br/>However it is planned to add a configuration directive to tell Redis to stop accepting queries but instead to SAVE the latest data and quit if it is using more than a given amount of memory. Also the new INFO command (work in progress in this days) will report the amount of memory Redis is using so you can write scripts that monitor your Redis servers checking for critical conditions.<br/><br/>Update: redis SVN is able to know how much memory it is using and report it via the <a href="InfoCommand.html">INFO</a> command.<h1><a name="How much time it takes to load a big database at server startup?">How much time it takes to load a big database at server startup?</a></h1>Just an example on normal hardware: It takes about 45 seconds to restore a 2 GB database on a fairly standard system, no RAID. This can give you some kind of feeling about the order of magnitude of the time needed to load data when you restart the server.<h1><a name="What Redis means actually?">What Redis means actually?</a></h1>Redis means two things:
ed9b544e	38	<ul><li> it's a joke on the word Redistribute (instead to use just a Relational DB redistribute your workload among Redis servers)</li><li> it means REmote DIctionary Server</li></ul>
	39	<h1><a name="Why did you started the Redis project?">Why did you started the Redis project?</a></h1>In order to scale <a href="http://lloogg.com" target="_blank">LLOOGG</a>. But after I got the basic server working I liked the idea to share the work with other guys, and Redis was turned into an open source project.
	40
	41	</div>
	42
	43	</div>
	44	</div>
	45	</body>
	46	</html>
	47