Mysql to MongoDb, chapter 2: Diving in

[This is a second part in an ongoing series, part 1 is here)

So, after making the call to go mongo, doing a mongo 101 crash course, we've started working on two major fronts (Only two developers):
1. DB Layer rewrite - This was pretty much straight forward. We've had about 100 functions to rewrite, but a lot of them were simple CRUD function. We've decided to use the native mongo node drivers, as I don't like to use frameworks in my code. (There's mongoose, which is a nice ODM layer, but, as I've said, I'd rather use native stuff, unless there's a performance advantage there.)
Major points you need to consider when transforming your code:

- Mysql has auto count features for unique inserts in table. Mongo has a unique id per object in database. If you're not using one, single unique id (And you're not) for each record in your mysql database, you need to use some sort of an applicative counter solution for inserts in mongo. This is also very useful to return insert id for new entities.

- Type checks and conversion: Rather than using a framework, I've decided to implement a simple hash table for field names and types.

- Logging: Like printing mysql statements to log, write a function which logs your query objects in mongo native format. (Like db.users.find({name:"yuval"}). It makes it much easier to debug.

THIS ONE IS REALLY IMPORTANT:

Do not break mysql support in order to support mongo! Fix both the mysql and mongo db layers, not in your application layer! Make it work, and don't do irreversible things that would brake mysql support. Support switching form mysql to mongo in a single configuration flag, so you can compare performance.   

2. Data conversion - We used mongify, a neat ruby tool, which translates sql databases into mongo. Performance was a bit dodgy for huge tables, se we've contributed some code, which also upped the performance by ~20 times.

Important note for people using open source software - Don't just report bugs. You can fix stuff and contribute to the community. 

[Especially if you're using it for commercial purposes]

Some things we've encountered during our conversion process:
- Dry run your conversion process. Dump your mysql, reload into a vanilla server with mongo installed, and do the dry runs from that server. The operational conversion is something you only need to do once, so it's ok to leave things for manual tinkering later!

- You'll see your application is working slower. Don't worry about it. There's a lot of tuning to do.

- Indexing: You need to take good care of this. Use explain({verbose:1}) for your big queries in order to find out why. Indexing in mongo will solve a lot of your performance problems.

- Large sorts won't work, even with indexes. In our case, it was a sort for an set of more than 130000 records. Instead of implementing paging, we've moved the sorting to the application (Works really fast, thank you). We will need to implement paging eventually, because we've just postponed the inevitable...

- Uniqueness: Like mysql, mongo has an ensureUnique method on index creation. We decided to add indexes manually and not automatically.


The next chapter will deal with more sophisticated tuning methods post conversion. Stay tuned :)

From mysql to mongo in less than a week, chapter one

This week, we finally took the plunge at Lightapp, and migrated our database into mongo.

In the coming weeks I'll be writing a series of posts about the experience, along with some insights and tips, and (Of course) the end result, in numbers.

The first step was deciding we need mongo instead of mysql. This one was pretty simple. Our company's product reads data from numerous sources, then aligns them and analyses them.
Since we're using big EAV tables for storing all that data, mongodb is a far more suitable solution than mysql.
The stage was right, as our database only contains tens of millions of records, right before we migrate all our current customers (Which will scale it towards hundreds of millions). So, with the help of my good friend (And scale pro) Moshe Kaplan, we started the process.

To make things easier, we also decided on a '1 to 1' conversion. This means that we only switch our db layer, and do it without touching our application. Luckily we've designed our application just like that - Only 1 module was responsible for db connection and querying, thus it was simple to write a parallel mongo db module, and switch between the two seamlessly.

So the first major tip I recommend: 
Build your db layer as a 'replaceable' layer. This means no SQL queries or DB connections outside your layer. If you currently have a written application with lots of 'history', just search for all the queries, and replace them with function calls (And filter parameters) into one file.
You might even find that you can get rid of some duplicated code in the process :)

Second tip:
Fiddle with mongo a bit. You can use a tool like pentaho to pull data from your mysql db, and into mongo, and then login to mongo, do some queries - Learn the whole query and CRUD mechanism in mongo. It's pretty simple, and has lots of great documentation all over the web.

Third tip:
Don't expect miracles. As we only switched our query functionality at first stage, we were not expecting a huge performance gain. Optimisation takes time, and the big value we were expecting from mongo was by moving our processing engine (Data bucketing and aggregation) lower from our application layer into the db layer.

Next thing was to approach the conversion process itself, but this would be elaborated in the next post of the series.

Stay tuned for more :)

[edit:] continue to chapter 2.

My take on Logging Levels

A long time ago, while developing a realtime server for Verint, the system's architect devised a great document called 'logging policy'.

It explained how a log message should look like, which information they should contain, and elaborated the rules for each log message level: For example: "There is no such thing as a 'Good' warning. Do NOT put 'Server is now up' messages in warning level".

Back then, the common practice ranged more than five levels, from critical to verbose.

At least at the beginning of development, I found that, though it's nice to do so, right now the most effective number of logging levels is 2. Debug mode and production mode.
Production mode should contain high level log events (Flow events, processing metrics) and errors, debug - the rest.

Logging is one of the strongest maintainability tools for your server. However, keep it simple, or you'll drown in misleading information.

Way too lean.

Everyone's going lean these days. Work lean and cheap, go to market as quickly as possible, fail fast.
While this approach works great in our time and age, it causes a lot of products (Jelly, for example) to come half baked.
The philosophy behind it is correct. Test the market as fast as you can, to validate your idea and gain ground. If you fail enough times, you might even succeed once. And one good success is all you need, right? (Yes, I am being ironic.)

However, unless your service goes ballistic right after launch, you might never know if it works or not. Hell, even if it does - What's your measure of success?

99% of the apps do not pass the 10000 download barrier. So what does success mean? You might fail your product a bit too fast there. You might even not know why it failed.

My advice would be to be a bit more thorough about your product. Get real feedback from as many people as you can (2nd hand friends are even better, as they don't owe you anything. You can even pay a symbolic token like a t shirt or something).

Don't launch too early, don't fail too fast. Thinking twice might not align with the cranky, wham bam lean approach, but might save you a lot of time and heartache.

Deliberate Mistake, correct end result

I've spent my army service days as an infantry soldier.
A lot of my unit's field practice included night navigation - Where you learn a path (By heart), and sent to navigate few miles at night, and collect a few waypoints.

The most challenging part of these drills was not walking with 35 pounds of gear on, or even seeing what's ahead of you to avoid pitfalls. It was memorising the path, and walking through it - If you don't know everything by heart (Sometimes you only have 20 minutes to prepare your gear and learn the path), you're bound to get lost. If you get lost, you walk more. If you walk more, you'll be more tired. And so begins a downward spiral no one likes.

I found that the most effective way for me to study these paths was using the deliberate mistake method: Where you choose a point on the map that's easy to recognise and reach, and from there walk to the nearest waypoints. This would sometimes mean walking more than the 'direct' approach, but you'd be less likely to forget a turn or a hill, and less 'marks' to count when you're walking, at night, with your gear on.

Nowadays, sometimes when I look at decisions I need to make, I sometimes take the 'wrong' decision, to reach a larger goal sooner - Like coding some ugly patch - I know I can correct this mistake later, but I also know I would also reach my goals on time, with confidence.

You can stop developing your product now, thanks.

Once a startup's MVP has been released, there's always the question of what's next.
A lot of startups tend to use the time after sorting out the quirks and the bugs to develop more features and enrich the product.

But sometimes, adding more features to your product before even finding out what works will only distract your new users, who are trying to get to know your product.

A good example might be Google Plus vs. Twitter.

Twitter has remained essentially the same as it's been since its inception. Google Plus has overhauled completely more than once, and is crammed with tons of features. While Google stuffed its social network with more and more features (Share and video and an amazing image viewer and more menus and what not), and how to implement them different than Facebook, Twitter focused on refining its mobile and web experience.

So next time your R&D department got spare time, have them improve performance, clean up code and take care of automation and scale rather than add more buttons and features that might just make your product too rich and too redundant.

Your users will appreciate more if they wait 1 second instead of 5 for a response rather than another copycat feature.

DO NOT UPGRADE!

Too often, when an application fails on memory or performs poorly, the immediate solution would be upgrading the machine.
Add more cores, more memory - And think that this would clear the problem from under the rug.

This strategy actually causes even more damage - As you will have to deal with the real issue (YOUR CODE...) later, when the system has more data, more angry clients, and more code that breaks.

If your application consumes too much CPU, you should profile it and solve the problem.
If your application leaks memory, you should find out where.

Taking the lazy approach would cause you credibility issues once you will really need that upgrade.