How Netflix cracked the formula for hit shows

I'm a sucker for comics. Specifically super hero comics. And, after seeing Netflix's Daredevil, I was taken by how accurate and well done it is. It's one of the best adaptations I've seen to a comic series. Fast paced, good action, just blunt and dark enough.

That got me thinking - How the hell did they do that? After "Orange is the new black" and "House of cards", Netflix seem to be churning out great TV, again and again.

The answer is simple. Netflix, which started as a mail rental service (Just because broadband infrastructure wasn't relevant back then) and moved on to online video rental, simply know exactly what we want as viewers.
They know which movies we like.
Which movies we watch until the end, and which movies we "break" in the middle.
They also know which scenes we run over and over again.

By tagging every scene in every content, they have the ability to create the perfect show for each one of their viewers. They can actually make script decisions (Exactly how the story will be built) according to data.

Furthermore - In the future, story would not be linear as we know it, but every subscriber will get his own, personal, customised version of the show. Yoni Bloch's Interlude might actually close this gap from the production end.

That's pretty amazing.

The Cloud Will Be Disrupted.

The cloud is where it's all at.. A centralised, managed, virtual place, where all the data and logic go. It's comfortable. It's better than in house IT. It's safer than keeping your docs on your PC.

For my 100th blog post, I would like to make a careful prediction about the future of applications.

Like traditional IT before, the cloud also has its drawbacks. The biggest one: It's not really free, and it's not really yours.

The docs you put on Google, mails you send via your free provider, images you post on Facebook, files you put on dropbox - Unless you start paying a monthly tax for these conglomerates, is theirs to use, and sometimes delete.
Some of the big companies built an eco system (Google, Facebook, but also airBNB, Uber and a lot of the shared economy companies), which gets a lot of the population to be dependent on them for their monthly pay check.
Some of these companies have enough cash and political power to bend governments.

The shared economy is best for the customer, however it puts a lot of power in the hands of too few companies. Imagine that all taxi drivers at the states belong to Uber.

Part of the fear of societies and companies from bitcoin is not the currency itself, but the democratic, P2P model of its application - A distributed application, in which logic, as well as data, could be shared by every user, not just by the service provider - Because every user is, essentially, a service provider.

Imagine an airBNB service hosted by every airBNB user, containing all the users in his country - Without the need to pay commission to the airBNB. Or a 'floating' Uber like service, where drivers and customers connect and bid directly.

This will get the prices even lower, and give more to the specific service provider, rather than cut a fat commission to an already fat company.

The infrastructure is already there - In my opinion, we will start see those applications during the coming years. Like P2P shook the media world, decentralised applications will shake and disrupt the cloud.

Your lovely graphs are useless.

Visualisation used to be everything. If you need to make a point, show it with a nice graph.
A lot of times, when we demo our system to decision makers, they are really impressed with our ability to analyse their data and display it in graphs.

However, while increasing credibility and showing our software's lovely abilities, graphs have a learning curve, and sometimes it takes an expert to analyse graphs and conclude the necessary conclusions. While we accompany our clients during the initial use of the system, not too many of them are keen to learn the delicate art of data analysis, even if it looks great.

This is why we're starting to move towards insights. Instead of showing a scatter chart with quadrants and let the customer deduct where each group is located, we can display them a summary, showing each group's grade - This group consumes too much energy in relation to its throughput. this one is more efficient. A link to the graph is attached, for credibility, however once the customer trust your system, he won't even need it.

Build trust with your customer, and then only show him the bottom line instead of confusing him with complicated visuals. He will feel better treated as a decision maker rather than one of your analysts.

From one hosting to another...

This weekend, we moved our cluster from Rackspace to Amazon.

It was a very tense process, as we need to transport GBytes of data seamlessly from one server to another, as well as shift our data writing server from the old database to the new one, while losing as little data as possible.
Eventually, it was done in less than two hours, most of which were exporting data from the old server and transporting it into the new one.

My main tips for performing such a move:
- Sit down before, and write the process step by step. Preparations for the new server, exporting and transporting the data, updating DNS services, testing. Discuss and review with your team.
- Review the process, and estimate risks and contingencies. What happens if it takes too long to transport the data? What happens if you have to roll back? What happens if the new server crashes?
- In case of DNS updates, it would be faster and more reliable to add the new domain ip to /etc/hosts instead of waiting for DNS to refresh, which might take time. Don't forget to jot down the IPs of your old servers, you might need them :)
- In our case, we also had some one premise sensors with no access to DNS, which means we had to use both the old and new stacks active, both working with the new database, until we're able to access those sites and change the destination IP.
- After you've got your plan laid down, do a dry run. There's nothing like a dry run to sort out bugs and add missing steps.
- Be 100% clear about who performs which step and when, but keep one person in charge of the whole process.
- Keep everyone in the loop - Upgrade was performed Friday night, so we decided to do it from home.. We were a team of three, using Google Hangout to communicate while everyone is online.

Eventually, process went quite well, Amazon SSD servers are fast and zippy, and we're ready for our next scale challenge, which would have us scaling to tables larger than 100m records while keeping our high performance standards.

Shifting weights... And Cluster Performance.

At the beginning, there was one machine.

One server ran our database and web application.

But as our database began to scale, we needed to separate them.
Separating the servers is easy - Just define an 'internal' network, and let the machines communicate via the internal leg.

However, this caused another problem - We began to experience long lags when transferring large datasets between the machines.
When dataset was larger than X items, performance began to degrade, and was substantially slower due to network lag. Below that, performance was actually better.

Part of it was due to node's mysql drivers. They're just slow. (Compared to native mysql drivers)
The other part was network lag.

So how do you solve it?
Part of my original design was using mysql as a data store - Only use simple sql, no arithmetical operations and aggregations. This allowed us to move from mysql to mongodb easily, and eased CPU load for the database.

But now it's time to shift the weight, because bandwidth is our new bottleneck.
So we're moving some of the heavier processing into the database so that a smaller dataset would transfer faster. Much faster.

Performance is a delicate art of balance. Always remember that paradigms change as scale changes. Don't be afraid to shift weight in order to keep your system balanced.

Clear off the table

[ taking some time off the ongoing mysql->mongo series ]

Been visualising data a lot lately.
Scatter charts, realtime indicators, good ol' line charts and, of course, tables.
It's really easy to fill the screen with loads of information, screaming colours and as many numbers floating around the screen. This makes orientation difficult for the user, especially non-technical users.

Some of our users simply need insights and recommendations, but need the graphs for credibility.

This is a great post from darkhorseanalytics explaining how a good table looks like. [hint - not your good ol' office 1998 excel highlight table]

http://darkhorseanalytics.com/blog/clear-off-the-table/

Mysql to MongoDb, chapter 2: Diving in

[This is a second part in an ongoing series, part 1 is here)

So, after making the call to go mongo, doing a mongo 101 crash course, we've started working on two major fronts (Only two developers):
1. DB Layer rewrite - This was pretty much straight forward. We've had about 100 functions to rewrite, but a lot of them were simple CRUD function. We've decided to use the native mongo node drivers, as I don't like to use frameworks in my code. (There's mongoose, which is a nice ODM layer, but, as I've said, I'd rather use native stuff, unless there's a performance advantage there.)
Major points you need to consider when transforming your code:

- Mysql has auto count features for unique inserts in table. Mongo has a unique id per object in database. If you're not using one, single unique id (And you're not) for each record in your mysql database, you need to use some sort of an applicative counter solution for inserts in mongo. This is also very useful to return insert id for new entities.

- Type checks and conversion: Rather than using a framework, I've decided to implement a simple hash table for field names and types.

- Logging: Like printing mysql statements to log, write a function which logs your query objects in mongo native format. (Like db.users.find({name:"yuval"}). It makes it much easier to debug.

THIS ONE IS REALLY IMPORTANT:

Do not break mysql support in order to support mongo! Fix both the mysql and mongo db layers, not in your application layer! Make it work, and don't do irreversible things that would brake mysql support. Support switching form mysql to mongo in a single configuration flag, so you can compare performance.   

2. Data conversion - We used mongify, a neat ruby tool, which translates sql databases into mongo. Performance was a bit dodgy for huge tables, se we've contributed some code, which also upped the performance by ~20 times.

Important note for people using open source software - Don't just report bugs. You can fix stuff and contribute to the community. 

[Especially if you're using it for commercial purposes]

Some things we've encountered during our conversion process:
- Dry run your conversion process. Dump your mysql, reload into a vanilla server with mongo installed, and do the dry runs from that server. The operational conversion is something you only need to do once, so it's ok to leave things for manual tinkering later!

- You'll see your application is working slower. Don't worry about it. There's a lot of tuning to do.

- Indexing: You need to take good care of this. Use explain({verbose:1}) for your big queries in order to find out why. Indexing in mongo will solve a lot of your performance problems.

- Large sorts won't work, even with indexes. In our case, it was a sort for an set of more than 130000 records. Instead of implementing paging, we've moved the sorting to the application (Works really fast, thank you). We will need to implement paging eventually, because we've just postponed the inevitable...

- Uniqueness: Like mysql, mongo has an ensureUnique method on index creation. We decided to add indexes manually and not automatically.


The next chapter will deal with more sophisticated tuning methods post conversion. Stay tuned :)

From mysql to mongo in less than a week, chapter one

This week, we finally took the plunge at Lightapp, and migrated our database into mongo.

In the coming weeks I'll be writing a series of posts about the experience, along with some insights and tips, and (Of course) the end result, in numbers.

The first step was deciding we need mongo instead of mysql. This one was pretty simple. Our company's product reads data from numerous sources, then aligns them and analyses them.
Since we're using big EAV tables for storing all that data, mongodb is a far more suitable solution than mysql.
The stage was right, as our database only contains tens of millions of records, right before we migrate all our current customers (Which will scale it towards hundreds of millions). So, with the help of my good friend (And scale pro) Moshe Kaplan, we started the process.

To make things easier, we also decided on a '1 to 1' conversion. This means that we only switch our db layer, and do it without touching our application. Luckily we've designed our application just like that - Only 1 module was responsible for db connection and querying, thus it was simple to write a parallel mongo db module, and switch between the two seamlessly.

So the first major tip I recommend: 
Build your db layer as a 'replaceable' layer. This means no SQL queries or DB connections outside your layer. If you currently have a written application with lots of 'history', just search for all the queries, and replace them with function calls (And filter parameters) into one file.
You might even find that you can get rid of some duplicated code in the process :)

Second tip:
Fiddle with mongo a bit. You can use a tool like pentaho to pull data from your mysql db, and into mongo, and then login to mongo, do some queries - Learn the whole query and CRUD mechanism in mongo. It's pretty simple, and has lots of great documentation all over the web.

Third tip:
Don't expect miracles. As we only switched our query functionality at first stage, we were not expecting a huge performance gain. Optimisation takes time, and the big value we were expecting from mongo was by moving our processing engine (Data bucketing and aggregation) lower from our application layer into the db layer.

Next thing was to approach the conversion process itself, but this would be elaborated in the next post of the series.

Stay tuned for more :)

[edit:] continue to chapter 2.

Keep your data aligned!

Many big data systems analyse large, periodical data streams.

These data streams are sometimes event based (i.e. - Add an entry whenever a user visits a page, performs an operation etc.), and sometimes 'sample' based (i.e. - Measure CPU level every 5 seconds).

Sometimes, your sampling can be unreliable - For example, when monitoring activity via WAN.
Then you get 'holes' in your data stream. These holes cause problems when analysing your data.

Several companies I know have been known to develop utility functions, which periodically go over the data streams and 'fix' these holes. These functions are usually costly, as finding such holes could get complicated and performance demanding in large datasets. Fixing them (Especially if it's an 'update' operation) is also costly.

My suggestion: Fix the problem before it rises. Keep your data aligned before you insert data into the database - Whenever there's a missed reading, fix it in the next reading by keeping track of your last reading's timing.
You might find it cheaper to hold a pre-input processing machine for this than requiring a huge server for your database because it needs to align data over night.