Mysql to MongoDb, chapter 2: Diving in

[This is a second part in an ongoing series, part 1 is here)

So, after making the call to go mongo, doing a mongo 101 crash course, we've started working on two major fronts (Only two developers):
1. DB Layer rewrite - This was pretty much straight forward. We've had about 100 functions to rewrite, but a lot of them were simple CRUD function. We've decided to use the native mongo node drivers, as I don't like to use frameworks in my code. (There's mongoose, which is a nice ODM layer, but, as I've said, I'd rather use native stuff, unless there's a performance advantage there.)
Major points you need to consider when transforming your code:

- Mysql has auto count features for unique inserts in table. Mongo has a unique id per object in database. If you're not using one, single unique id (And you're not) for each record in your mysql database, you need to use some sort of an applicative counter solution for inserts in mongo. This is also very useful to return insert id for new entities.

- Type checks and conversion: Rather than using a framework, I've decided to implement a simple hash table for field names and types.

- Logging: Like printing mysql statements to log, write a function which logs your query objects in mongo native format. (Like db.users.find({name:"yuval"}). It makes it much easier to debug.

THIS ONE IS REALLY IMPORTANT:

Do not break mysql support in order to support mongo! Fix both the mysql and mongo db layers, not in your application layer! Make it work, and don't do irreversible things that would brake mysql support. Support switching form mysql to mongo in a single configuration flag, so you can compare performance.   

2. Data conversion - We used mongify, a neat ruby tool, which translates sql databases into mongo. Performance was a bit dodgy for huge tables, se we've contributed some code, which also upped the performance by ~20 times.

Important note for people using open source software - Don't just report bugs. You can fix stuff and contribute to the community. 

[Especially if you're using it for commercial purposes]

Some things we've encountered during our conversion process:
- Dry run your conversion process. Dump your mysql, reload into a vanilla server with mongo installed, and do the dry runs from that server. The operational conversion is something you only need to do once, so it's ok to leave things for manual tinkering later!

- You'll see your application is working slower. Don't worry about it. There's a lot of tuning to do.

- Indexing: You need to take good care of this. Use explain({verbose:1}) for your big queries in order to find out why. Indexing in mongo will solve a lot of your performance problems.

- Large sorts won't work, even with indexes. In our case, it was a sort for an set of more than 130000 records. Instead of implementing paging, we've moved the sorting to the application (Works really fast, thank you). We will need to implement paging eventually, because we've just postponed the inevitable...

- Uniqueness: Like mysql, mongo has an ensureUnique method on index creation. We decided to add indexes manually and not automatically.


The next chapter will deal with more sophisticated tuning methods post conversion. Stay tuned :)

From mysql to mongo in less than a week, chapter one

This week, we finally took the plunge at Lightapp, and migrated our database into mongo.

In the coming weeks I'll be writing a series of posts about the experience, along with some insights and tips, and (Of course) the end result, in numbers.

The first step was deciding we need mongo instead of mysql. This one was pretty simple. Our company's product reads data from numerous sources, then aligns them and analyses them.
Since we're using big EAV tables for storing all that data, mongodb is a far more suitable solution than mysql.
The stage was right, as our database only contains tens of millions of records, right before we migrate all our current customers (Which will scale it towards hundreds of millions). So, with the help of my good friend (And scale pro) Moshe Kaplan, we started the process.

To make things easier, we also decided on a '1 to 1' conversion. This means that we only switch our db layer, and do it without touching our application. Luckily we've designed our application just like that - Only 1 module was responsible for db connection and querying, thus it was simple to write a parallel mongo db module, and switch between the two seamlessly.

So the first major tip I recommend: 
Build your db layer as a 'replaceable' layer. This means no SQL queries or DB connections outside your layer. If you currently have a written application with lots of 'history', just search for all the queries, and replace them with function calls (And filter parameters) into one file.
You might even find that you can get rid of some duplicated code in the process :)

Second tip:
Fiddle with mongo a bit. You can use a tool like pentaho to pull data from your mysql db, and into mongo, and then login to mongo, do some queries - Learn the whole query and CRUD mechanism in mongo. It's pretty simple, and has lots of great documentation all over the web.

Third tip:
Don't expect miracles. As we only switched our query functionality at first stage, we were not expecting a huge performance gain. Optimisation takes time, and the big value we were expecting from mongo was by moving our processing engine (Data bucketing and aggregation) lower from our application layer into the db layer.

Next thing was to approach the conversion process itself, but this would be elaborated in the next post of the series.

Stay tuned for more :)

[edit:] continue to chapter 2.

New stack, full stack.

In the past month I've been busy rebuilding the LightApp stack from scratch.
LightApp is a very cool startup in the industrial energy management field.
The product's essence is collecting data from various sources (Sensors, meters, production databases) of huge factories and analysing it to find ways to improve their effectiveness and reduce their costs and carbon footprint.
Previous product was built over LAMP stack (Linux/Apache/PHP/MySql), using the webii PHP framework, and had a lot of limitations - One example was heavy relying on SQL to perform statistical analysis, which was costly and shown poor performance and response time.

We've decided to rebuild the client (Not the database) with node.js, and after less than a month (We're a team of two) we've managed to almost completely finish our rewrite.
For UI we've decided to use bootstrap and amcharts, and manage the data ourselves (Rather than use a framework like angular or backbone).
Over the server side, express was used for main routing, mysql for database connection, and nunjucks for server side HTML.

The application doesn't have too many end users (Expecting few hundreds during 2014), but each user is performing heavy statistical analysis operations (Like a distribution of hourly data per category over a period of 6 months) - Which should be done really fast, and it is. Querying large data sets and crunching them in node is amazingly fast, but in the near future we will move our data to mongo, which can perform this analysis even faster.

Developing the whole stack in Javascript is fun - Though the language has a lot of limitations, it's fun to use, easy to debug and test, and, for our use, proves to be cost effective. Node.js asynchronous programming mode takes some time getting used to, but if you've dealt with AJAX on the client side, you'll be quite familiar with it.

I would highly recommend every programmer to try writing a full stack JS application. It's amazing how Javascript has evolved in the recent years.

Testing the water with a new iOS app.

A couple of weeks ago, I was looking for a fun educational app for my kids.
I've found out that all related apps aren't free, and most are crap. Real crap.
Deciding that something which I can easily implement should not cost, I've started to fiddle with it, and built a simple web prototype in a couple of hours.
When it was nearly ready, I loaded the page on my phone, and gave it to my girl to fiddle with.
Looks like she liked it, I tested it with a couple of my friend's children, and decided to make it an app.

As it's a very simple idea, I've decided to keep it simple, with an entry screen for parents to brag in Facebook that their kids are playing with my app, and also decided to use phonegap (Just so that I'd be able to easily create an Android version, if I wanted to).
Phonegap installation and integration with xcode is very simple, and it also has a cute Facebook native extension, using either the native app, or the web interface for approving posts. (Though for iOS6 it won't be necessary, as Apple would have its Facebook integrated inside its SDK)

The next step was enrolling to Apple's iOS dev program.
Due to the fact that I live in Israel, and not registering myself as an american company, I have to fax (!!!) a purchase form to Apple, along with credit card info. I know, this seems ridiculous.

A lame start, but I'm determined to have my first app published!

I will post more soon, after I begin the app approval process with Apple.

Don't fall in love with frameworks

The javascript world is on fire!
You've got tens of frameworks, all made to enable making javascript and web development easy.
angular, backbone, less, _, GWT, wicket, the old and established extjs, prototype, yui, and jquery (To name a few) have become common names among developers.

Some frameworks are meant to make your life easier with dom traversing and manipulation and encapsulate javascript features (jQuery, prototype), some add UI functionality (extjs, yui), while others offer full application frameworks.

However, these frameworks come with a price.

Once you bind your application to a framework, it's very hard to get rid of it. It's very hard to change parts of your application, once your web application is fitted with the paradigm of a framework.
Binding UI to model is another problem - Some of these frameworks bind data to UI components.

While this makes your initial development easy, once your application surpasses some volume and features, programmers tend to break this model (Something which is quite easy with javascript).

Binding is also very hard to debug - Sometimes you'll get an impossible to understand and follow call stack, especially with frameworks which bind model to UI components via events.

Another reason is performance: If all you want is a 3 page application which loads json objects, parses them and presents them on screen, using a full fledged MVP framework is a little bit of an overhead.

... And now, for the worst part.
You've written a nice widget/ control/ viewer/ whatever.
Now someone wants that object in his legacy system.
It's very hard to integrate binded code.
You tell him you can't.
He asks you why did you use the framework, if your code isn't recyclable.
You say 'it is, but you need to rewrite your whole client'.
CTO approaches you with angry eyes.

The rest - You know.

So next time you consider using a flashy, 'easy' to use framework, please ask yourself these questions:
- Will you need to reuse this code somewhere else?
- How much overhead will the framework add? Is it really necessary?
- How intrusive is that framework, in case you want to replace it?
- How much time would it take your team to develop the features you need from the framework? Don't add a full fledged framework for the sake of a neat button!
- How cryptic is the code? (Angular.js is a good example - Great framework, but you need to go all the way with it in order for you to enjoy its advantages.)
- How easy is it to debug? I know some of them are really, really hard.
And, finally:
In 6 months from now, will it bring value to your system, or will you need a month of knowledge transfer just to get someone new to write features into your app.

Be responsible - Use new technology that brings value, not heartbreak...

Tip: The JSP surgery

So, you've invested a whole lotta love and money in your java based web server.
And it's fine. Really.
But now it's time for an overhaul. The UI is dated, and you want to bring some fireworks back.
The problem - Your team has changed, it's been a while, and all your team has done so far were some bug fixes and some features. And the web layer is written in good ol' jsp - No one really knows how to rewrite it. Recently I've encountered a similar project - All the team left, the design was terrible, but I couldn't make changes to the back-end, only the jsp files.

What do you do?
My suggestion is to divide and conquer!

Here's a quick recipe on how to transform your server script code into a web service.

First off, see which data you need the server to really generate. Usually, it would be user details, a data table, plus some attributes.
After that, have your jsp code generate JSONs instead of html or javascript code.
This shouldn't be that hard.
Analyze all your code, and prepare new jsp files for each type of data needed. Use the current jsp function calls - There's no need to rewrite the java layer: All these files should do is write a JSON response. You can use Arrays.toString(), or a JSONArray, or you can generate the JSON explicitly from object attributes.


Then you can begin writing your client from scratch, and use these jsp as a web service. You may later write some Java servlets to replace them, or (Better off) use jax-rs to completely get rid of the jsp.
Step by step, you'll see your web application transform into something a lot more comfortable to use and maintain. A real web service, without rewriting your back-end. And when you do get the time to update your server, you'll be able to do it with only minor client modifications.

Have a good development week, and stay tuned for more!
y.

Three nice tips for web developers.

Actually, two basic dev tips and one neat resource site.

First tip is regarding dynamic HTML. We've been encountering some rendering performance problems with huge tables, and after some investigation, here are some pointers:

• Do not use tables. They're bad for you, especially large ones, and nested tables. By the way, YUI creates lots of inner tables. You can read some more with this link, top answer has some great detailed arguments for you. I know it's a corny advice, but there are lots of frameworks out there that use tables, even gwt.
• Do not create each dom element separately - If you need to create a very large table/ chunk of html, it's better to create a huge html string and then insert it into dom for processing, rather than use the dom to create each element, and the append them. Think of it as using 1000 'native' calls from a VM rather than 1 long call. The overhead for each dom element is huge.
• Establish a maximum size for a list and browse through results. Google does that with search results, why don't you? You can cache results in javascript, or keep an index to the query and go back to the server to fetch the next n entries, but displaying one thousand items for a user is highly ineffective. 

Second tip is rather basic - Regarding server/ client processing and separation. As we're moving towards cpu consumption related billing rather than permanent hosting, we need to rethink what we do on the server and on the client side. The main principal would be having the server perform validation and data (model), and do the rest in the client:

• Your server code should not create html. It should either create status/ session/ json, but not html. This is just a waste of server cpu. It speeds up server response, and when written well, makes client experience better. It scales a whole lot better.
• You must think of your client and server modules as replaceable. This means that if you're writing a server, imagine that you need to write two completely different (Say html and iOS) clients for it.
• If you're afraid that competitors may reverse engineer your client html/ js/ css code, you can always minimize it, or get an obfuscator.

Final tip is this site. Microsoft is putting a lot of effort these days - It's great to see them as the underdog. You'll find Azure php sdk, nice html5 references, and lots of useful resources. Check it out. I'm considering Azure for a project I'm working on, the php sdk seems quite cool.

Have a great week!
y.