Keep your data aligned!

Many big data systems analyse large, periodical data streams.

These data streams are sometimes event based (i.e. - Add an entry whenever a user visits a page, performs an operation etc.), and sometimes 'sample' based (i.e. - Measure CPU level every 5 seconds).

Sometimes, your sampling can be unreliable - For example, when monitoring activity via WAN.
Then you get 'holes' in your data stream. These holes cause problems when analysing your data.

Several companies I know have been known to develop utility functions, which periodically go over the data streams and 'fix' these holes. These functions are usually costly, as finding such holes could get complicated and performance demanding in large datasets. Fixing them (Especially if it's an 'update' operation) is also costly.

My suggestion: Fix the problem before it rises. Keep your data aligned before you insert data into the database - Whenever there's a missed reading, fix it in the next reading by keeping track of your last reading's timing.
You might find it cheaper to hold a pre-input processing machine for this than requiring a huge server for your database because it needs to align data over night.

New stack, full stack.

In the past month I've been busy rebuilding the LightApp stack from scratch.
LightApp is a very cool startup in the industrial energy management field.
The product's essence is collecting data from various sources (Sensors, meters, production databases) of huge factories and analysing it to find ways to improve their effectiveness and reduce their costs and carbon footprint.
Previous product was built over LAMP stack (Linux/Apache/PHP/MySql), using the webii PHP framework, and had a lot of limitations - One example was heavy relying on SQL to perform statistical analysis, which was costly and shown poor performance and response time.

We've decided to rebuild the client (Not the database) with node.js, and after less than a month (We're a team of two) we've managed to almost completely finish our rewrite.
For UI we've decided to use bootstrap and amcharts, and manage the data ourselves (Rather than use a framework like angular or backbone).
Over the server side, express was used for main routing, mysql for database connection, and nunjucks for server side HTML.

The application doesn't have too many end users (Expecting few hundreds during 2014), but each user is performing heavy statistical analysis operations (Like a distribution of hourly data per category over a period of 6 months) - Which should be done really fast, and it is. Querying large data sets and crunching them in node is amazingly fast, but in the near future we will move our data to mongo, which can perform this analysis even faster.

Developing the whole stack in Javascript is fun - Though the language has a lot of limitations, it's fun to use, easy to debug and test, and, for our use, proves to be cost effective. Node.js asynchronous programming mode takes some time getting used to, but if you've dealt with AJAX on the client side, you'll be quite familiar with it.

I would highly recommend every programmer to try writing a full stack JS application. It's amazing how Javascript has evolved in the recent years.

Rewrite your Product, and live to tell about it

In every organisation life, there comes a time when its product is due to a big overhaul.
From dated GUI to dying Database, a Business Logic layer that's filled with patches, or even personal changes ('The only girl that knew that code has left...').
So now the company gives you, her beloved manager, a clean slate to rewrite the product.

Supposedly a dream come true, right?
In a past post, I've discussed the new system dilemma, a situation where you're already knee deep inside the rewrite, and pressure is rising.

This post should help you avoid that terrible situation.

The most important rule when writing a product from scratch is deciding what it's NOT going to do in phase one (See time frame in the second rule). Trying to match all the features the old system supports (Sometimes while it's still being developed by tier 4) will result in an endless effort.
Focus, and be transparent with what you'll deliver and what you won't. Put an emphasis on what your rewrite brings to the table that the old stack doesn't.

The second most important rule is DON'T LINGER. 3 months. Show something (Even a playable mockup) after 1 month. Show progress every week.
Three months is a long time, and mind that things can change, but it's a short enough time frame that things won't turn completely around.

When phase one is near the end, you can "Let the company in" and plan the next stages.

Focus and Agility bring results. Go for it!

Draw me a unicorn

The recent talk about unicorns - Startups which passed the 1 Billion dollar valuation - Have started a goose chase by many VCs. Series A VCs are no longer interested in companies which would give them 5x return (ie - be part of a 5 Million $ investment, and receive 25 Million as part of a company's exit), but they need more than that.
So VCs are no longer looking for teams which can build 50 Million businesses, but a billion dollar business.

While that might make sense financial wise, it may actually kill the whole VC business, which is already dying. Super angles and private equity investors are already taking their place not just in seed but also in series A.
Startups might just be better like that.

YULES.CO is up - Check it out!

Got a new self promotion website.
Currently quite generic, but much more coming soon:
A more visual timeline of my career, and blog might also move there (though I really like the advantages of the Blogger platform).

Come visit me at www.yules.co. More to come soon!

Tunnel vision

During the life cycle of a 'live' system, things tend to break.
Sometimes because you've missed something, sometimes because you didn't get to test that bit of code. But whenever a system turns live, after a while (Sometimes it happens pretty quick...) the system needs to handle something you haven't tested.
Sometimes even an unexpected failure. Sometimes network lagging causing timing issues.

Few of these times, you encounter what I would like to call a gremlin. That's this illusive bug which impacts your system, occurs enough times a day to be considered as a problem, but not enough so it's easy to reproduce.

The worst thing programmers love to do is mark it as 'irreproducible', throw it back at the QA, and do something else.

Advice for this stage: Don't. QA would probably not be able to reproduce it, and even if they would, it would be difficult for them to do it in a consistent manner so that you'll be able to gain from it. You need to help them. These things escalate fast, so don't put yourself in the position of the developer who tried to put the mess under the rug.

Start digging. And sometimes, the solution requires processing through endless logs (Or even add these logs). Sometimes a lot of code is involved. And after a few hours, if you still haven't found it, you find yourself repeating your actions, adding more logs, running sequences again. You've entered a stage of tunnel vision, where you can't look around.

Best advice for this stage: Stop. Go get coffee, go home, call someone, take a break. My best solutions came from getting some distance from the problem I've been working on, in order to clear my vision and be able to try a new approach. Another fresh set of eyes could work as well.

Happy hunting.

Who wants to be a hacker?

<rant>
One of the internet's greatest achievements and failures is the democratisation of skills.
The internet has lowered the barriers for everything:
Want to be a journalist? Open a blog.
Want to be a film maker? Get your cell out and upload your 15 second masterpiece to Instagram.
Want to be a programmer? Take an online course.

Want to be a hacker? No, it's too dangerous, now with the NSA and stuff. The original hacking was considered the cool and 'living on the edge' side of geekness, and can now put you in jail. But hacking has gone a serious downgrade, from pioneers and brilliant yet morally challenged techies, to just plain people inventing and integrating stuff.

So why is hacking all the rage now? Unfortunately, geek is out of fashion. It's not a strong enough word for the tech savvy. Ninja isn't anymore as well. So hack is the next 'sexy' term for what we use to call 'automation expert'.

Check out this headline: "The free tools that let you hack your whole life".
What is actually means is "tools to automate and integrate different facets of your physical and online life."
Don't get me wrong - I think that the fact that cheap hardware running linux and cheap controller boards help you implement great ideas and lower the barrier for innovators is great. I think that it's quite similar to the 3d print revolution. But this isn't hacking.

So it's all about marketing. If I see someone who calls himself a 'hacker' in his bio or profile, I would stay away - He might be a former 'ninja' or a 'wizard' or, god forbid, a self proclaimed 'geek'.
</rant>

Yours truly during a "hack" session