Photosynth is
probably the coolest thing I have seen in a while. I would love
to know how they gather all these information and construct a 3D
image from it. Just absolutely amazing. Too bad they got bought
by Microsoft.
For some reason some builds where failing for me as soon as I
edited files with TextMate. Turns out TextMate writes some meta
data into file attributes (or hidden files).
Searching the manual
I found that you can disable this feature with
defaults write com.macromates.textmate OakDocumentDisableFSMetaData 1
Soon the contract for my dedicated server will run out. I was sharing a server from server4you.de with a few people. But more and more people dropped out and since we could not find people to fill the spot we decided to quit and move somewhere else. Well, that and the incredibly awful service. (Too many stories to tell!) Just to give some examples: When we got the server we wanted to run Debian instead of SuSE. The question whether they could just pop in a CD that we prepare was just refused without giving a reason. So we ended up installing Debian from the swap partition. Yay! But a much worse story is that hey refused to reboot my server when I was down in Australia. I didn’t have the password to the server admin at hand (I left the records in Germany and my parents where on vacation). Due to a special number you cannot even reach the support hotline from abroad. Somehow I worked out a different number and finally talked to them on the phone. So imagine – my server was down. No access to console. No access to the server admin website. I was explaining them the situation. I was able to give ALL my details. Customer number, of course even bank details etc. They still refused to reboot the server. They asked me to fax them a written request for a new admin password. As it was Saturday that would not be processed before Monday. Then they wanted to send me new credentials to my home address. That would have been at least 5 days downtime. The best thing – due to the time difference I was calling 20 minutes before the end of their business day. You know what? They just hung up on me! “Sorry, Sir I cannot help you”. I was almost getting mad. After some “meditation” I worked out the password and was able to reboot the server myself. So the downtime was just half a day. My complaint to the management was left unanswered. After they even f..cked up the billing several times I am soooo happy I finished moving all my service to a different provider yesterday. If you consider server4you – just don’t. They might have a competitive pricing – but it’s just so not worth it.
While the travel to Seattle was horrible (missed my connection
and got stuck at Washington Dullas for one night. Mental note:
never ever Lufthansa again!) the conference and city after-all
was quite nice. Not that I had much time for exploring Seattle
– but it looked beautiful while I was hasting through my
too-see list in a few hours. (So green!) I’ve not even had
the time to meet up with some Apache folks over there. (Sorry,
Henry!) The conference itself was well organized (what else
would you expect from Google) and covered some quite
interesting topics. Unfortunately it was not as technical as one would have
hoped. Still there were some lessons learned that surely should
be kept in mind while designing large scale systems. The session
are supposed to be available on google video/youtube at some
stage. So far I could only find
one of them.
The sessions from Google itself where mostly introductions to MapReduce and BigTable. They were interesting but no big surprises. They did even talk briefly about hadoop as an open source implementation. Unfortunately there is no comparison between hadoop and the Google in-house implementation yet …though they agreed that it might be interesting to see the results. A little annoying was that every remotely technical question was answered with “Sorry, can’t talk about it”. What I found most amazing though is that according to them all Google developers have access to the production machines.
The session about the Lustre File System was quite interesting. Turns out it is one of the top 5 clustered file systems. It’s full posix compliant and supports cross-site deployments. People run it in highly critical installations. Throughput they achieve is quite impressive. Setting up lustre properly can take some time though. That’s where the company behind lustre provides consulting for.
A session about test selection talked about scaling with the number of test that gets accumulated over the releases. Running all test for a release will take longer and longer with the increasing number of tests. In order minimize risk for emergency releases they developed a technique to statistically map code changes to testcases and requirements. So based on testing history they can then select certain tests and still are quite confident about the QA status. Not sure I really buy into this though. ;)
For me the most interesting session was indeed the session from the Amazon guys about infinite data storage. They emphasized on that if something can go wrong – it will go wrong. And one should plan for that. According to them you only truly scale with more data centers across the world. And you gotta be prepared what happens if a data center just dies! Not just is unreachable – but dies. They had a couple of approaches until they finally arrived where they are now. From mainframes over distributed databases. It all just didn’t work. Especially they were bashing distributed databases. According to them it just happens that nodes run out of sync. For some bizarre reasons – but they do. And then it gets messy. Two-phase-commit was almost a “don’t say that word” for them. They came up with their own system as they believe there was (and still is) no product one can buy or open sources that fits their needs yet. Their contract is that every page (even with one data center down) should be delivered within less than 100ms. A machine or even a data center down should not even be noticeable to the user. While reads are easier to scale writes are not. Still they require their writes (e.g. to the shopping card) to return without any blocking. Their call was that versioning is the key to true scalability in this area. A quorum based process makes sure nodes get synchronized properly across the cluster. On application level they handle merge conflicts if there really are. They have nothing what one would call a master. Any machine can go down – at any time. Load is balanced and data partitioned via consistent hashing. They did not go down the low-end machine route like Google did – but no big investments into machines either. Again developers have access to the production environment and are responsible for the contracts (like the 100ms). Definitely worth a watch once it’s up on Google video.
Unfortunately I missed the session from the YouTube guys. So I hope the rest of the sessions will be online soon too. I’ll give an update here once I found out.
There are a couple of nice write-ups of the conference as mentioned here.
Update: Sylvain found them. Definitely worth watching the YouTube one. If you don’t know what MapReduce/BigTable is about also check out the Google talks.
Today I am heading off for the
Google Scalability Conference
in Seattle (Bellevue). It’s not really around the corner
but I am really looking forward to it. The
agenda of the conference
sounds quite interesting and might give some new perspectives on
the technical challenges we are facing at
Joost. But it’s also
interesting to visit this city from a far more personal point of
view. It’s incredible how many bands that influenced my
taste of music are/were from Seattle. Just one word –
SubPop.
If you have suggestion on what to see or want to meet up – let me know. I am flying back on Sunday.