I'm going to echo Jason and ask: why do Tokyo Cabinet tables suck?
I have a particular problem in an app I'm working on and tokyo cabinet tables seem to address it perfectly. I have a server that provides a service that basically runs func(ItemA, ItemB, ItemC) and that takes a while so I'd like to cache the results. It's easy to stick them in a KV store and just get them out with a simple key (ItemA.id, ItemB.id, ItemC.id), and TC is wicked fast for that. The issue is, when some given ItemA changes, how do I invalidate the cached results for that? I need to make a simple query for whatever items have modified_a.id as the first part of their key so I can delete them. TC tables seem *perfect* for that.
I don't want to use a relational DB because that's *way* to heavyweight. I don't need joins. All I've got is a single table and I want to do some really simple querying on it.
Adam
· 4 months ago
+1 Also very interested in issues with TC tables
irafromstpetersburg
· 2 months ago
I agree with your posting.
Very good informations.
Greetings and all good wishes from Russia
Jason Seifer
· 5 months ago
Can you go in to more detail or provide some links about why table mode sucks with Tokyo Cabinet? I'm looking in to this for a project and it would be very helpful to know any drawbacks before it goes in to production.
Ville Laurikari
· 5 months ago
Eric, excellent summary. One aspect of NoSQL is overcoming the impedance mismatch. Things like LINQ and Hibernate have been very helpful, but there are still classes of applications which would benefit from a system which allows data to be disk-backed more casually.
It's true that 99.8% sites are fine with SQL. SQL is tried and true technology and if it works for you, why look elsewhere? However, I believe there are a lot of potential applications out there that nobody gets done because they don't fit well in the standard databases.
Jan
· 5 months ago
Eric, an excellent post. Thanks! :)
Regarding the implied SQL-hate of "the nosql movement":
I have just finished reading C-Store paper a couple days ago. I have been thinking that it would be nice to have a powerful, widely used column oriented DBMS, since still most web applications are read-mostly.
Now I wonder if non-relational databases really offer performance advantages on read/write and write mostly workloads? Or will we end up implementing ad-hoc relational abstraction layers on top of them?
I only used CouchDB briefly and my first reaction was to exactly that.
Chris Bird
· 3 months ago
There is no inherent clash between a column store and SQL. Column store is just what it says - a way of organizing the physical data by columns and not by rows. That does not mean that you cannot access rows or projections (a subset of columns) using SQL.
We shouldn't confuse access layer activities with data access language semantics
Jamie Thomson
· 5 months ago
That's the most sensible discussion I've seen of this field yet. I particularly like "For the vast majority of sites out there (we're talking 99.9% of the sites out there, folks) [SQL] suits their needs very well, providing a good balance of ease of use and performance."
I'd add another contender into the mix even if its only in beta. MSFT's Live Mesh is a consumer app built on top of a hierarchical sync'd entity data store. It doesn't really have a name per se but they have a RESTful API in beta that will soon be available for anyone to party on.
-Jamie
Richard Reddy
· 5 months ago
Great post! I'm always amazed when developers tell me they don't like SQL/database design because they find it too hard to manage.
I think you hit the nail on the head with your comment "Most people ... relying less on the features provided by traditional relational databases and engineering more database logic in their application code."
For the time being, I don't mind SQL and can get it to do what I want so I can wait for these new technologies to mature.
Rich
Michael Greene
· 5 months ago
Great write-up, Eric. Personally, I've never encountered anyone who is anti-SQL and it's unfortunate that NoSQL has that association. But, it also has the association of being fact & scalable and I guess you can't win them all.
Terry Jones (@terrycojones)
· 5 months ago
Hi Eric - thanks for the write up. We're planning a private alpha release of FluidDB on Aug 17th. We've also been thinking about Cassandra for backing storage. It's especially attractive to us as we already use txThrift, a Twisted compatible extension to Thrift that we open sourced late last year. Let me know if you want an early look at the FluidDB docs.
Terry
Tim
· 5 months ago
"I love my precious ORM" ... "For the vast majority of sites out there (we're talking 99.9% of the sites out there, folks) it suits their needs very well, providing a good balance of ease of use and performance"
Can you tell me which ORM you're using? I've used several (ActiveRecord, NHibernate, etc.), and found them all to be pretty lousy at ease-of-use compared to even a crappy OODB.
It's like C. Most people writing OO code today don't write it in C. You certainly can, but it never feels like more than a pale imitation of the real thing. ORMs feel the same way to me: you're pretending your RDBMS records are objects, with an abstraction layer so flimsy you can almost never forget about what's just on the other side.
It sounds like you're making the classic mistake of assuming the only types of databases are ancient RDBMSs and brand-new distributed key-value stores. You don't mention any OODBs, even in your list in the conclusion. Object databases are doing just fine, thanks! :-)
Eric Florenzano
· 5 months ago
Jan: You're right, I agree that a lot of people think that data persistence == SQL, which is something that will change as these alternative systems mature, I think. I certainly don't think SQL is inherently bad, and from some of the comments here, others don't think SQL is inherently bad, so I guess I was wrong about that in my post :D
Atamert: Trust me when I say that I want a good, open source, column store as well. At work we evaluated lots of options and ended up buying a license of Vertica. Vertica is pretty awesome, but being a fan of open source it would have been great to have found an open source product. To my knowledge one doesn't exist still.
Michael: You're probably right, I guess I was wrong about people hating SQL. It must have just been a few of the people that I talked to in person or something.
Ville: I think there's an interesting case to be made there. Do we change our way of thinking simply because we don't have tools readily available to do things in other ways? I think that yes, we do.
Terry: I'm super excited about FluidDB, and I'd love an early peek at the docs :)
Tim: I've always been partial to Django's ORM, but I also like SQLAlchemy as well. I've never found the impedence mismatch to be a real problem in practice.
terrycojones
· 4 months ago
Hi Eric
I'm not sure if you received my dm in Twitter (perhaps not the most reliable comms medium right now). Anyway, if you send me a mail I'll send you some details.
I'm terry <at< fluidinfo >dot> com
James
· 5 months ago
Jan:"The idea is that people think data-persistence == SQL which is not the case. #nosql is not saying SQL is bad."
Try subscribing to the Planet CouchDB feed sometime, they, at least, do say SQL is bad.
Bradford Stephens
· 5 months ago
I think SQL and DBs have been treated as a "Swiss Army Knife" for too long. The main problem they have is scalability -- at a certain point, you're spending exponentially amounts of money for decreasing returns.
Most companies *don't* need to get rid of their SQL boxes. But if you're in the growing Social Media or BI fields, where you have to crunch hundreds of TBs of data, then you have certain problems that you can't just toss at SQL. You have to engineer your data from the ground up, and make sure you make the correct sacrifices (Transactions, Denormalization, etc.)
"It was written in such a way that nodes which are disconnected for long periods of time can reconnect, sync with each other, and reconcile their differences in a way that no other database (since Lotus Notes?) could do."
The FLAIM engine/db for Novells eDirectory does this as well.
Jan
· 5 months ago
@James: A lot of people don't like SQL for right and wrong reasons (I'm on the "right tool for the job" front). If you just care about object persistence dealing with a 20k+LOC ORM (ActiveRecord e.g.) is not something you want to chug along. Plus ORMs are not one-size-fits-all and leaky abstractions at best. Assuming 80:20, these 20% of devs are looking for alternatives.
PS: I'm the admin of Planet CouchDB :) It is a collection of personal opinions. It is not an official statement of the CouchDB project or its developers.
David
· 5 months ago
Thanks for this article!
In case some readers are interested, Prevayler (http://www.prevayler.org/) provides a more classic object persistence approach as an alternative to SQL databases.
Michael Galpin
· 5 months ago
Best tool for the job always sounds good in practice :-)
I do think a lot of people blame scalability issues on SQL and RDBMS's in general. I don't know how many times I've heard "joins don't scale." Unfortunately some of this ignorant crowd has been drawn to NoSQL. Coincidence? Self selection? Who knows.
What's even worse are the folks who stick memcached in front of an RDBMS.
Sandy B
· 5 months ago
You talk much while saying little.
Mike Dirolf
· 5 months ago
Interesting read - thanks!
I'm a developer on the MongoDB (http://www.mongodb.org) project. The goal of the project is to bridge the functionality gap between key/value stores and RDBMSes, while maintaining the high performance and scalability gains you get with K/V systems. If anybody has any questions check out the mailing list.
artsrc
· 5 months ago
Why is:
insert into My_Table (My_Column_1, My_Column_2) values ('My_Value_1', 'My_Value_2')
Better than:
insert into My_Table My_Column_1 = 'My_Value_1' My_Column_2 = 'My_Value_2'
Every ORM tool I have used (Hibernate etc.) has serious flaws and limitations. Just yesterday I found I could not have a unidirectional link to a 'table per concrete class' hierarchy.
I can work around the limitations, and issues of the SQL technologies, and create great software, but I still know SQL sucks in so many ways.
ryan
· 5 months ago
Well I think you wrote off mongoDB without even trying it? Is it because it is written in C++? It is very fast and not bad at all.
Sho
· 5 months ago
One aspect of high-speed read/write you didn't mention is session management. I think for many sites, this is the "gateway drug" into so-called "NoSQL" databases, as that's the one thing they may need to read and write *every access*.
Take a look at mainstream frameworks like Rails et al, who wave away the problem by storing sessions in cookies - a very suboptimal and inefficient solution, IMO. I think many sites who would not otherwise need to look outside RDBMSs for their *content* are interested in TC et al for their session management.
Also, a little harsh on MongoDB I think? It might be "jack of all trades but master of none" but if it gives you 90% of what you need in 3 different areas, for most people that's a lot better than running 3 different programs.
One more thing - Redis is fantastic for many uses, but its memory usage is astonishingly high. I don't think it should even be called a DB because you'd be nuts to store data in it. I would think of it more as a smart cache.
artsrc
· 5 months ago
@Michael
SQL databases are sophisticated and include many features (query optimizers, declarative referential integrity, security, etc.).
Sophistication is not free. Creating an SQL database with scalability and performance is harder than creating a database with a simpler model with equivalent performance and scalability.
Therefore given constant time and budget, a system with joins is less scalable than a system with no joins. Compared to no joins, joins do not scale.
keong
· 5 months ago
@artsrc:
In case you didn't know, but 'insert into' has more advanced forms which are way more powerful and useful than your trivial example. See also http://en.wikipedia.org/wiki/Insert_(SQL)
Also, regarding "unidirectional link to a 'table per concrete class' hierarchy", we do this all the time... you just need to know where to put your foreign keys. Also, FYI there is no such thing as 'unidirectional link' in SQL...that's only a concept in your object model not in your database schema.
ChanaChevhu
· 5 months ago
var DataPersistence = SQL ?? nothing; simple as that, sql is data persistence and data persistence is sql, if you can't have both then you have nothing. permanently persisting data in xml and the like is for playground purposes, its NOT for business...
sopyer
· 5 months ago
SQL is just a language for data queries. It is nothing bad in idea of querying data using some language. The problems lies in storage. Traditional RDBMS are not scalable. I think this is the point of NoSQL movement. For me the ideal architecture for DB is that of SQLite: query language(describes query algorithm), VM to execute query, storage. I think at the moment storage mechanism is at question.
Andrew Ingram
· 5 months ago
When I was taught about database design and relational algebra at university, by Hugh Darwen (one of the authors of The Third Manifest, and a big critic of SQL), we were given plenty of reasons why SQL isn't very good. The major reason given was that it's not a fully relational language (more advanced languages like D).
In the world of web development we inevitably end up denormalising a lot of data (especially if we depend on ORMs rather than writing the queries oursevles). You can basically make the argument that the longer a codebase is around, the further away from taking advantage of relational features it gets. Eventually you reach a point that you've shifted so much of the data management into code that you may as well switch to a key/value store.
Simon Gardner
· 5 months ago
Interesting article.
Personally I think the problem isn't with relational databases, it's with the SQL programming language. SQL is great for ad-hoc queries and manipulating data but as a programming language, it utterly sucks.
The reason applications need to have the significant overhead of a seperate business logic layer is that SQL is such a feeble language for expressing business rules.
Daniele
· 5 months ago
I think ZODB is a very nice non-relational database that has been around much longer than other contenders and that can be useful also outside of the Zope world. I think it is worth to take to look to it.
Goulwen
· 5 months ago
For me, the main feature of CouchDB is that it is document-oriented. I see a good way of storing document alongside with its metadata. CouchDB also provide versionning by default. All of this is doable with a RDBMS, even it's not so easy to do.
softprops
· 5 months ago
I think both sql and nonsql db both have a place for different types of applications. I think the general trend is moving away from monolithic-type applications to the idea smaller individual applications that are responsible for maintaining their own data and provide data via apis verses having one giant application having all data being stored under one giant schema. I think this makes things a little easier to manage and makes the software generally more reusable as components across different systems.
KV db's really shine when it comes to serving up the standard 'find this resource for that identifier' but maybe are not that great when it comes to generating reports across many different types of related data. CouchDB and its map-reduce facility may have solved that issue. It would be nice to see some standardization on how this could be accomplished across other schemaless db's (TokyoCabinet's Table store for example). Perhaps a UQL (unstructured query language) is in order.
j_king
· 5 months ago
Wasn't it Donald Knuth or some prominent Lisper who admonished the default idea that "SQL == persistence" by claiming that the model should fit the solution?
ie: If it doesn't make sense to use SQL as a data store, why use it?
That this common sense thinking has become typified and meme'd as a "movement," amazes me.
James
· 5 months ago
Second Daniele on the ZODB thing. It's definitely worth evaluating among NoDB options and has been around longer than any of the other things on here.
Also, it's not an either/or situation here: web apps can use both relational and non-relational data stores as appropriate. Sho gets at this a couple of comments up by talking about managing sessions in a key/value store.
Rodrigo Asensio
· 5 months ago
Now, here is a stupid question. How nonsql db stands with queries ? search by no key fields, etc ?
Jochen Kupperschmidt
· 5 months ago
As mentioned before, TC/TT can append to values already. But what Redis has besides that are mutable lists and sets, which I really appreciate as they come in very handy when trying to convert RDBMS models to a key/value structure. Other noSQL databases should add those, too, and I'd happily be checking them out in greater depth.
@Jochen The MongoDB data model allows lists and many other types, including embedded documents. Can also do atomic append to list, update embedded document, etc. Might be worth a look.
ryan
· 5 months ago
I don't understand how bigtable has a single point of failure... all so-called "spof" in the system are distributed, the bt master, and the gfs masters exist in auto-election multi-master mode. Furthermore, the bt master isn't in the read/write path anyways.
Note: cassandra hasn't been demonstrated in a multithousand node production config yet.
Jerry Silver
· 5 months ago
Interesting that almost no one has mentioned XML databases, like eXist, MarkLogic, or Documentum xDB. Since so much data now begins (and ends) life as XML, a native XML database is a natural fit for all kinds of applications. If your data is already defined by an XML schema (or even if it's just well-formed) you don't have to do any database design - just load and go, and schema changes don't require a database reload. Plus these databases implement standard XML processing languages like XQuery and XPath, and the commercial products are designed to scale to very large volumes and heavy loads with high availability. See also https://community.emc.com/docs/DOC-2999 for a comparison between a native XML database and the mainstream hybrid relational products.
Jan
· 5 months ago
@Goulwen CouchDB does *not* provide versioning. CouchDB revisions are MVCC access tokens. There's no guarantee of older versions staying around. That said, storing history in CouchDB is pretty easy, just store revisions in separate documents and use a view to provide history.
Jonathan Ellis
· 5 months ago
> I don't understand how bigtable has a single point of failure
Just because you have failover, doesn't mean it's not a single point of failure. Failover can break just like any other moving part.
artsrc
· 5 months ago
After reading this, and some other resources, I gave CouchDB a try. I was really impressed with how modern, open and simple the model is.
@keong
SQL insert has many forms. Complexity like this in SQL results from SQL not being the modern, powerful, general tool we should create. Also many examples on the link you gave are non-portable, the other issue I discussed.
Some people do no perceive link in SQL as directional. Links in ORM are understood to be directional:
This is the name "Table" that may not be appropriate here, I don't see Tokyo Cabinet Table as a replacement of a traditional RDBMs table, but more as an extension of the Hash database. Call it SuperHash, you'll think it's great ;-)
drowsy
· 5 months ago
@Jerry
I don't dispute that a lot of the world runs on XML. But I'm excited about noSQL projects that make the transport of JSON more popular. I think it is much easier for folks to share/merge broken XML than JSON, but time will tell.
@sho 100% right on.
Wrought Iron Furniture
· 4 months ago
SQL is the best and will always rule!
Dentists Halifax
· 4 months ago
Thanks for your useful info, I think it’s a good topic. I would like to read more about your articles.
ruslizaenal
· 4 months ago
MMM, I can't to say anythink
ruslizaenal
· 4 months ago
mmm, .....
eiei
· 4 months ago
I like very much the writings and pictures and explanations in your adress so I look forward to see your next writings.
gracewei
· 4 months ago
Good thoughts.Very Useful site for your reference.Pls clutch the opportunity:.Nike Dunks Nike SB online Nikeand check the detailed information. Once you have any question, you can connect our customer service at any time Nike DunksAir Max 90 .
sugeng
· 4 months ago
great post
Watch Movies Online
· 3 months ago
Great post! I'm always amazed when developers tell me they don't like SQL/database design because they find it too hard to manage.
I think you hit the nail on the head with your comment "Most people ... relying less on the features provided by traditional relational databases and engineering more database logic in their application code."
For the time being, I don't mind SQL and can get it to do what I want so I can wait for these new technologies to mature.
facebook-519910573
· 3 months ago
I created a "No SQL" t-shirt and mug on CafePress with 0% markup. You can find them here: www.cafepress.com/nosql
Ryan
pppoll37
· 3 months ago
Wow! Thank you! I always wanted to write in my site something like that. Can I take part of your post to my blog? Please come visit my site Columbus Business Directory when you got time.
Lots of comments here but adding a bit more. I have written a Squeak library for TC/TT, a view server for CouchDB and also Divan - the best (IMO so far!) library in C# for using CouchDB. I am also an OODB fan since many many years.
To sum up my feelings: TC/TT is awefully fast and if you really need a very fast non-sharded simple key/value store - it fits that bill. It is on the other hand a one-man show. The table extensions etc are interesting.
CouchDB is cool indeed. It is not yet really "distributed" though - but hopefully this will come more and more. It has AFAICT from having worked with it quite a bit one major problem - it handles dynamic queries quite poorly. Also, your description was slightly inaccurate regarding the views, you don't really "call" those functions. Also - people often miss one of the most important facts - not only the values in CouchDB are JSON - but the keys are too.
MongoDB is my next project - I think it does a lot of what Couch does BUT also supports dynamic queries and thus covers a different (although overlapping) area of applications.
deace08
· 3 months ago
Pre-order some aion cd key is crucial if you wanna be a gamer of Aion. And there is another item, aion time card is as important as the cd key in the game.
anastasiafrommoscow
· 3 months ago
Hello, my name is Anastasia, I am a russian girl from Moscow and I like this site very much. Hallo mein Name ist Anastasia, ich bin eine Russin aus Moskau und finde diese Seite super geil.
I can see that you are putting a lot of time and effort into your blog and detailed articles! I am deeply in love with every single piece of information you post here. Will be back often to read more updates! Please come visit my site Phone Directory Of Charlotte when you got time.
Hello, my name is Lena. I am a hot german girl from Bergheim. I like this site because it is very interesting. Thx for the good informations and a lot of good wishes from HotLena
rr8004
· 2 months ago
I enjoyed reading your work! GREAT post! I looked around for this… but I found you! Anyway, would you mind if I threw up a backlink from my site? Please come visit my site Local Business Directory Of Cincinnati U.S.A. when you got time. Thanks.
After spending a bit of time with Tokyo Cabinet both in the hash database and the table database, I wouldn't say that TC table database sucks. But it is substantially slower than the hash database (which is to be expected). In order to compensate for that, I modified the way in which the data was stored (basically serializing the data and dumping it into the value column). The obvious decision here is whether or not you want to add that extra requirement of serialization to your application. I found that it really wasn't much of a performance impact overall to add the serialization step, but my application doesn't require real time process or anything close to it.
My next step is to try a similar thing in Cassandra (without data serialization) and see how it fares. The issue that I think you should have mentioned is that although Cassandra uses the Perl Thrift API, the APIs in general into Cassandra are weak. The Ruby API (through the cassandra gem) is very straight forward though. TC has fairly well documented and straightforward Perl and Ruby APIs. -- http://eric.lubow.org/2009/databases/tokyo-tyra...
pppoll37
· 2 months ago
mind that a lot of people are in. The sense of wanting to help, but not knowing how or where, is something a lot of us are going through. Please come visit my site San Diego Web Link when you got time.
pppoll37
· 2 months ago
mind that a lot of people are in. The sense of wanting to help, but not knowing how or where, is something a lot of us are going through. Please come visit my site San Diego Web Link when you got time.
pppoll37
· 2 months ago
mind that a lot of people are in. The sense of wanting to help, but not knowing how or where, is something a lot of us are going through. Please come visit my site San Diego Business Directory Forum Blog Classifieds when you got time.
pppoll37
· 2 months ago
mind that a lot of people are in. The sense of wanting to help, but not knowing how or where, is something a lot of us are going through. Please come visit my site Directory San Diego City when you got time.
pppoll37
· 2 months ago
mind that a lot of people are in. The sense of wanting to help, but not knowing how or where, is something a lot of us are going through. Please come visit my site San Diego Web Link when you got time.
pppoll37
· 2 months ago
mind that a lot of people are in. The sense of wanting to help, but not knowing how or where, is something a lot of us are going through. Please come visit my site San Diego City Business Listings when you got time.
itjob123
· 3 weeks ago
Now I wonder if non-relational databases really offer performance advantages on read/write and write mostly workloads? Or will we end up implementing ad-hoc relational abstraction layers on top of them?
natis
· 1 week ago
Eric Great and very useful summary.
There are various reports that was gathered from data center statistics that analyze disk failure statistics: * Actual disk failure/year is 3% (vs. estimates of 0.5 - 0.9%) – this is a 600% difference on reported vs. actual disk failure. * There is NO correlation between failure rate and disk type – whether it is SCSI, SATA, or fiber channel. * * There is NO correlation between high disk temperature and failure rates
Those analysis shows that the approach of relying on a shared storage for reliability as with most RAC clusters is broken. Instead NOSSQL approach assumes that failure are inevitable and where designed to deal with those failure even under extreme scenarios.
This is a great post. i have been looking to get into Django and am strguggling with some of the conepts and methods you are describing here. Are there any other programming tools or tricks that are useful?
All Comments
I have a particular problem in an app I'm working on and tokyo cabinet tables seem to address it perfectly. I have a server that provides a service that basically runs func(ItemA, ItemB, ItemC) and that takes a while so I'd like to cache the results. It's easy to stick them in a KV store and just get them out with a simple key (ItemA.id, ItemB.id, ItemC.id), and TC is wicked fast for that. The issue is, when some given ItemA changes, how do I invalidate the cached results for that? I need to make a simple query for whatever items have modified_a.id as the first part of their key so I can delete them. TC tables seem *perfect* for that.
I don't want to use a relational DB because that's *way* to heavyweight. I don't need joins. All I've got is a single table and I want to do some really simple querying on it.
Very good informations.
Greetings and all good wishes from Russia
It's true that 99.8% sites are fine with SQL. SQL is tried and true technology and if it works for you, why look elsewhere? However, I believe there are a lot of potential applications out there that nobody gets done because they don't fit well in the standard databases.
Regarding the implied SQL-hate of "the nosql movement":
"The idea is that people think data-persistence == SQL which is not the case. #nosql is not saying SQL is bad." — http://twitter.com/janl/status/2742018744
Cheers
Jan
--
Now I wonder if non-relational databases really offer performance advantages on read/write and write mostly workloads? Or will we end up implementing ad-hoc relational abstraction layers on top of them?
I only used CouchDB briefly and my first reaction was to exactly that.
We shouldn't confuse access layer activities with data access language semantics
I'd add another contender into the mix even if its only in beta. MSFT's Live Mesh is a consumer app built on top of a hierarchical sync'd entity data store. It doesn't really have a name per se but they have a RESTful API in beta that will soon be available for anyone to party on.
-Jamie
I think you hit the nail on the head with your comment "Most people ... relying less on the features provided by traditional relational databases and engineering more database logic in their application code."
For the time being, I don't mind SQL and can get it to do what I want so I can wait for these new technologies to mature.
Rich
Terry
Can you tell me which ORM you're using? I've used several (ActiveRecord, NHibernate, etc.), and found them all to be pretty lousy at ease-of-use compared to even a crappy OODB.
It's like C. Most people writing OO code today don't write it in C. You certainly can, but it never feels like more than a pale imitation of the real thing. ORMs feel the same way to me: you're pretending your RDBMS records are objects, with an abstraction layer so flimsy you can almost never forget about what's just on the other side.
It sounds like you're making the classic mistake of assuming the only types of databases are ancient RDBMSs and brand-new distributed key-value stores. You don't mention any OODBs, even in your list in the conclusion. Object databases are doing just fine, thanks! :-)
Atamert: Trust me when I say that I want a good, open source, column store as well. At work we evaluated lots of options and ended up buying a license of Vertica. Vertica is pretty awesome, but being a fan of open source it would have been great to have found an open source product. To my knowledge one doesn't exist still.
Michael: You're probably right, I guess I was wrong about people hating SQL. It must have just been a few of the people that I talked to in person or something.
Ville: I think there's an interesting case to be made there. Do we change our way of thinking simply because we don't have tools readily available to do things in other ways? I think that yes, we do.
Terry: I'm super excited about FluidDB, and I'd love an early peek at the docs :)
Tim: I've always been partial to Django's ORM, but I also like SQLAlchemy as well. I've never found the impedence mismatch to be a real problem in practice.
I'm not sure if you received my dm in Twitter (perhaps not the most reliable comms medium right now). Anyway, if you send me a mail I'll send you some details.
I'm terry <at< fluidinfo >dot> com
Try subscribing to the Planet CouchDB feed sometime, they, at least, do say SQL is bad.
Most companies *don't* need to get rid of their SQL boxes. But if you're in the growing Social Media or BI fields, where you have to crunch hundreds of TBs of data, then you have certain problems that you can't just toss at SQL. You have to engineer your data from the ground up, and make sure you make the correct sacrifices (Transactions, Denormalization, etc.)
I wrote a blog article on this topic called "Social Media Kills the RDBMS": http://www.roadtofailure.com/2009/06/19/social-...
The FLAIM engine/db for Novells eDirectory does this as well.
PS: I'm the admin of Planet CouchDB :) It is a collection of personal opinions. It is not an official statement of the CouchDB project or its developers.
In case some readers are interested, Prevayler (http://www.prevayler.org/) provides a more classic object persistence approach as an alternative to SQL databases.
I do think a lot of people blame scalability issues on SQL and RDBMS's in general. I don't know how many times I've heard "joins don't scale." Unfortunately some of this ignorant crowd has been drawn to NoSQL. Coincidence? Self selection? Who knows.
What's even worse are the folks who stick memcached in front of an RDBMS.
I'm a developer on the MongoDB (http://www.mongodb.org) project. The goal of the project is to bridge the functionality gap between key/value stores and RDBMSes, while maintaining the high performance and scalability gains you get with K/V systems. If anybody has any questions check out the mailing list.
insert into My_Table
(My_Column_1, My_Column_2)
values
('My_Value_1', 'My_Value_2')
Better than:
insert into My_Table
My_Column_1 = 'My_Value_1'
My_Column_2 = 'My_Value_2'
SQL is a suboptimal language. It is also lacks a convenient portable subset which is sufficient for most applications (e.g. http://www.reddit.com/r/programming/comments/92...).
Every ORM tool I have used (Hibernate etc.) has serious flaws and limitations. Just yesterday I found I could not have a unidirectional link to a 'table per concrete class' hierarchy.
I can work around the limitations, and issues of the SQL technologies, and create great software, but I still know SQL sucks in so many ways.
Take a look at mainstream frameworks like Rails et al, who wave away the problem by storing sessions in cookies - a very suboptimal and inefficient solution, IMO. I think many sites who would not otherwise need to look outside RDBMSs for their *content* are interested in TC et al for their session management.
Also, a little harsh on MongoDB I think? It might be "jack of all trades but master of none" but if it gives you 90% of what you need in 3 different areas, for most people that's a lot better than running 3 different programs.
One more thing - Redis is fantastic for many uses, but its memory usage is astonishingly high. I don't think it should even be called a DB because you'd be nuts to store data in it. I would think of it more as a smart cache.
SQL databases are sophisticated and include many features (query optimizers, declarative referential integrity, security, etc.).
Sophistication is not free. Creating an SQL database with scalability and performance is harder than creating a database with a simpler model with equivalent performance and scalability.
Therefore given constant time and budget, a system with joins is less scalable than a system with no joins. Compared to no joins, joins do not scale.
In case you didn't know, but 'insert into' has more advanced forms which are way more powerful and useful than your trivial example. See also http://en.wikipedia.org/wiki/Insert_(SQL)
Also, regarding "unidirectional link to a 'table per concrete class' hierarchy", we do this all the time... you just need to know where to put your foreign keys. Also, FYI there is no such thing as 'unidirectional link' in SQL...that's only a concept in your object model not in your database schema.
simple as that, sql is data persistence and data persistence is sql, if you can't have both then you have nothing.
permanently persisting data in xml and the like is for playground purposes, its NOT for business...
In the world of web development we inevitably end up denormalising a lot of data (especially if we depend on ORMs rather than writing the queries oursevles). You can basically make the argument that the longer a codebase is around, the further away from taking advantage of relational features it gets. Eventually you reach a point that you've shifted so much of the data management into code that you may as well switch to a key/value store.
Personally I think the problem isn't with relational databases, it's with the SQL programming language. SQL is great for ad-hoc queries and manipulating data but as a programming language, it utterly sucks.
The reason applications need to have the significant overhead of a seperate business logic layer is that SQL is such a feeble language for expressing business rules.
KV db's really shine when it comes to serving up the standard 'find this resource for that identifier' but maybe are not that great when it comes to generating reports across many different types of related data. CouchDB and its map-reduce facility may have solved that issue. It would be nice to see some standardization on how this could be accomplished across other schemaless db's (TokyoCabinet's Table store for example). Perhaps a UQL (unstructured query language) is in order.
ie: If it doesn't make sense to use SQL as a data store, why use it?
That this common sense thinking has become typified and meme'd as a "movement," amazes me.
Also, it's not an either/or situation here: web apps can use both relational and non-relational data stores as appropriate. Sho gets at this a couple of comments up by talking about managing sessions in a key/value store.
MongoDB supports some cool advanced queries out of the box - see here:
http://www.mongodb.org/display/DOCS/Advanced+Qu...
@Jochen
The MongoDB data model allows lists and many other types, including embedded documents. Can also do atomic append to list, update embedded document, etc. Might be worth a look.
Note: cassandra hasn't been demonstrated in a multithousand node production config yet.
Really?
I refer you to Google's explanation of the recent App Engine outage: http://groups.google.com/group/google-appengine...
Just because you have failover, doesn't mean it's not a single point of failure. Failover can break just like any other moving part.
@keong
SQL insert has many forms. Complexity like this in SQL results from SQL not being the modern, powerful, general tool we should create. Also many examples on the link you gave are non-portable, the other issue I discussed.
Some people do no perceive link in SQL as directional. Links in ORM are understood to be directional:
http://docs.jboss.org/hibernate/stable/annotati...
This is the name "Table" that may not be appropriate here, I don't see Tokyo Cabinet Table as a replacement of a traditional RDBMs table, but more as an extension of the Hash database. Call it SuperHash, you'll think it's great ;-)
I don't dispute that a lot of the world runs on XML. But I'm excited about noSQL projects that make the transport of JSON more popular. I think it is much easier for folks to share/merge broken XML than JSON, but time will tell.
@sho
100% right on.
I think you hit the nail on the head with your comment "Most people ... relying less on the features provided by traditional relational databases and engineering more database logic in their application code."
For the time being, I don't mind SQL and can get it to do what I want so I can wait for these new technologies to mature.
You can find them here:
www.cafepress.com/nosql
Ryan
Greetings from Germany
HotLena
To sum up my feelings: TC/TT is awefully fast and if you really need a very fast non-sharded simple key/value store - it fits that bill. It is on the other hand a one-man show. The table extensions etc are interesting.
CouchDB is cool indeed. It is not yet really "distributed" though - but hopefully this will come more and more. It has AFAICT from having worked with it quite a bit one major problem - it handles dynamic queries quite poorly. Also, your description was slightly inaccurate regarding the views, you don't really "call" those functions. Also - people often miss one of the most important facts - not only the values in CouchDB are JSON - but the keys are too.
MongoDB is my next project - I think it does a lot of what Couch does BUT also supports dynamic queries and thus covers a different (although overlapping) area of applications.
Hallo mein Name ist Anastasia, ich bin eine Russin aus Moskau und finde diese Seite super geil.
Liebe Grüße Anastasia
Greetings from Anastasia
anastasia from moscow in russia
Greetings from Germany
Julia
www.sexyjulia.sex-privat.tv
Please come visit my site Local Business Directory Of Cincinnati U.S.A. when you got time. Thanks.
My next step is to try a similar thing in Cassandra (without data serialization) and see how it fares. The issue that I think you should have mentioned is that although Cassandra uses the Perl Thrift API, the APIs in general into Cassandra are weak. The Ruby API (through the cassandra gem) is very straight forward though. TC has fairly well documented and straightforward Perl and Ruby APIs. -- http://eric.lubow.org/2009/databases/tokyo-tyra...
Great and very useful summary.
There are various reports that was gathered from data center statistics that analyze disk failure statistics:
* Actual disk failure/year is 3% (vs. estimates of 0.5 - 0.9%) – this is a 600% difference on reported vs. actual disk failure.
* There is NO correlation between failure rate and disk type – whether it is SCSI, SATA, or fiber channel. * * There is NO correlation between high disk temperature and failure rates
Those analysis shows that the approach of relying on a shared storage for reliability as with most RAC clusters is broken. Instead NOSSQL approach assumes that failure are inevitable and where designed to deal with those failure even under extreme scenarios.
I summarized that topic on one of my recent post: Why Existing Databases (RAC) are So Breakable!: http://natishalom.typepad.com/nati_shaloms_blog...
You may also want to consider another category of In-Memory-Data-Grid (http://en.wikipedia.org/wiki/Data_grid )
See more details on that regard on Todd Hoff (highscalability.com) write-up: Are Cloud Based Memory Architectures the Next Big Thing? http://highscalability.com/blog/2009/3/16/are-c...