Are Cheap Androids An Electric Dream?

16 May 2011 at 18:52

Dbug

Last Christmas was tough on my netbook. The travelling around, the trains and planes being cancelled or delayed, the packing and unpacking, the plugging and unplugging, and the occasional knocks finally killed my good old and trusty Asus eeePC 901.

More exactly, the screen decided to die ¹.

Came the time to look for a replacement, dictated by the usual set of parameters to consider: cost, form factor, autonomy, ruggedness, planned usage, look.

2011 is a particularly rich year in this regard, due to the crossbreeding between the netbook, laptops, mobile phones, pda and tablets. It’s getting harder and harder to try to put these various devices in distinct categories, I’m not even going to try to do that so please bear with me: As far as I’m concerned, all these devices are potential candidates.

I guess the best way to find out what I need, is to look at what I used to have, and find out from there what was good and what was not good.

Gericom Webgine 1400

Fat and slow, but it has served well.

My first laptop was a 14 inch Gericom heavyweight. At least it was in the heavyweight category from a weight point of view because performance was not its forte: You can’t really expect a super cheap laptop that use a desktop version of an AMD processor to be super efficient at anything, or could you? So basically, it was “transportable”, the autonomy was pathetic due to the crappy battery and the desktop based processor, and it was freezing due to overheating during the summer so I had to buy one of these cooler support tablets

There was at least one positive thing: It was possible to use the machine for real work, either document editing, programming, debugging.

Asus eeePC 901

It's still working.

The eeePC on the other hand was small, silent and sturdy (I had the SSD version), a very practical small machine to keep in your bag when travelling around, and the battery life was good enough to allow watching few full length films when travelling. That’s nice.

The negative things, well the machine was slow (mostly due to the Physon SSD ²), the keyboard too cramped (impossible to do long sessions of text writing, either using instant messaging, so forget about programming or writing a blog post), the Intel graphics being as usual unusable for anything other that displaying basic visual effects.

In the end, all I was using this machine for was browsing web pages, using IRC, talking with friend, and watching films.

Possible options

The market has now changed quite a lot. The traditional desktops and full-size laptops are still available and provide the same type of experience as before, the netbook category is well alive as well, but these days it’s the smartphones and the tablets that are on the front pages of most tech sites, and of course the multiplicity in terms of operating systems is larger than ever.

The easiest choice would have been to buy a new netbook. The prices have gone down, so basically for about two third of the original price I would have got a similar machine, but with a dual core Atom, twice the amount of memory, ample storage size… and still a shitty Intel graphic system.

On the next price range we find the basic full-size laptops, cheap but heavy as bricks.

Then come the tablets and smartphones. Light, good battery life, but can you really do anything useful with iOS and Android, on what is basically just an internet enable touch screen?

Finally comes the top of the range laptops from Apple, like the MacBook Pro and Air.

Difficult choices.

Tablets with keyboards?

As if we did not have enough options already, at the last CES the various hardware vendors decided to bring on the tables the (possibly) best of both worlds: Tablets with detachable keyboards.

Two models in particular seemed interesting, the Asus Transformer, and the Acer Iconia 500. The Asus had some pretty good reviews, very slick hardware, Android 3, super long battery life, flash support… and out of stock everywhere ! Damn’it !

Since these machines were not available, I searched a bit more about Android with keyboard, and I discovered this very intriguing machine from Toshiba, the AC100.

Yeah, it's slim

Android netbook

The AC100 is a very special machine. Basically it fits in the same category as a normal 10 inch netbook, except that it does not run on Windows or Linux, and it does not use an Intel compatible processor. That means you can forget playing any windows game or generally speaking run anything that expect x86 compatibility.

I admit that it’s a bad start.

If you search for reviews on the web, written when the machine was introduced late 2010, you will find out that the grand majority are giving less than the average and some even less than that (The Register gave it 10%).

That looks even worse, isn’t it?

The interesting thing about these reviews, is that they basically all agree that the hardware is great, the machine is very snappy, the Tegra 2 chipset allows for very smooth display, the screen is great, it’s possible to watch 1080p videos (that you can watch on an external screen thanks to the hdmi connector).

The problem is that this machine was released with Android 2.1, does not have the Android Market, the unusual screen size and lack of touchscreen makes many application go hay-wire, and it did not even support flash. And the system was buggy too. And the web-browser did not always work.

I can understand the bad reviews, no hard feelings, too bad for Toshiba, and this probably hurt the possibilities that other companies try to do similar Android netbooks.

I guess at this point you are wondering why I spent so much time describing such a failure of machine. The answer is simple: I bought one, and I’m very happy

After market

When this machine was introduced, it was sold as a full price machine, at roughly the same price as normal Atom powered netbooks. Some people noticed this pretty nice and small machine, bought it, tried to use it, screamed, and brought it back to the shop for a full refund.

Then comes me, looking for the prices of various machines in the shop next door, and found out they had a pile of these machines, already used, with a big “slashed prices/must go” sticker. And indeed slashed it was, I paid it nearly half of the price it had the month before ³.

Of course, a turd is a turd, even if you got it for free.

When I plugged the machine, I indeed quickly realized that the default setup was sub-optimal, and that yes the machine was not very usable, but I gave a try.

That’s when the magic happened! After configuring the wifi and performing the registration sequence I was able to run the Toshiba Updater system which quickly decided to update itself. After this update phase I had the pleasure to discover a whole list of new available upgrades, including Android 2.2 (Yeah!), Adobe Flash support (Yeah!!), Multiple Keyboard Layout support – including norwegian -, updated Media Player, additional Market place software, Office document viewer, etc…

Post-upgrade feeling

1080p on external HDMI

After this upgrade, what was true about the hardware is still true: It’s a very light machine that feels comfortable to handle, with a very good screen and a perfectly usable keyboard. The touch pad is reactive and can be disabled by pressing one of the built in keys ⁴. The battery life is quite good and charges fast as well.

Basically for less than the price of the cheapest netbook I could find, I have a machine that can comfortably browse webpages – even when they have flash -, supports Skype, IRC, Remote Desktop, can play Angry Birds and a bunch of other games at full frame rate, supports an external mouse and usb card readers (it also has a built in SD card reader), can play all the movies I had (including multiple gigabyte MKV and AVI files). Google maps also works fine, and you can use DosBOX as well. (Apparently it can also access samba shares, use ssh, can play Spotify, but I did not try).

What it cannot do, is basically allow me to program on the fly, but this was not doable on the eeePC either, so all things considered it’s a choice I would do again.

State of Android

Now clearly for me it showed that the state of Android today is a bit like at the DOS far-west period when you had to make sure that each and every application would actually run on your particular system. You had to have to good type of video card, a compatible sound card, else the applications would misbehave to simply refuse to run at all.

For this machine it’s the same thing: A number of applications expect the screen to rotate freely, they expect a touch screen, some expect a particular screen size or aspect ratio, some expect a gps or 3g module, etc… so a significant number of applications will not work on this particular machine. But the one that work and can exploit the machine are really flying: All the recent 3d games are running perfectly smoothly in landscape format.

Hopefully developers will learn to stop assuming things and write correct code instead. The future will tell!

Technically I guess we can call “showing the same scanline repeated all over the screen” as “not really usable anymore. ↩
If you have one of these machines, I can recommend the Renice mini PCie SSD cards, it’s like over-clocking the machine, you will not recognize it: Everything opens almost instantaneously. ↩
Which for the sake of comparison means I paid for it one tenth of the price of an Apple MacBook Air in the same shop. ↩
You also have keys to change the screen brightness, volume, bring the list of running applications, disable the wifi, play and pause medias… ↩

I’m in love with Kira

27 Mar 2011 at 16:54

Dbug

computers

3 Comments

No, Kira is not the name of my girlfriend, it is not the name of my cat either, and I’m not in love with any particular film celebrity bearing a similar name.

Kira¹ is only a small device you connect to your network, which in turn allows you to control anything that uses infra-red, kind of like an universal remote control.

Here it is how it looks like ²:

Kira 128 show-off

Home-automation reloaded

As I posted earlier this week, I’m currently toying with the whole Home-Automation concept.

The way I see it, I should be able to control everything from one single place, and in 2011 what makes sense is to make it web-based. The internet protocols and software stack has evolved to the point where it’s basically a no-brainer: Between browsers that can display rich content using CSS (and/or HTML/canvas, Silverlight, Flash, or whatever else to come in the future) plus PHP and CGI on the server side, you can really do whatever you want³. So basically as long as a device can access web pages and understand javascript, it can be used to control my system.

The last post was about showing information collected from the web, in particular how to show when the next metro arrives. It’s cool, but it’s not really automation. With Kira you can start to build the real thing.

Universal remote controls

Of course I could have chosen the easy way and buy something like a Logitech Harmony universal remote control. As far as I can tell, anyone I know who bought one only had positive things to say about it, so kudos to Logitech for designing apparently a very good product.

The reason why I did not do that, is that I wanted to be able to control everything from a single location, possibly make it react to external events (like mute the TV or stereo system if somebody calls), be able to access all that out from home (perhaps because I forgot to record a program I absolutely wanted to see), and generally speaking: Just because I can

So here is how it works:

You plug the device on your network.
You program up to 128 commands using the provided software.
You use the Kira’s integrated web server to select one of the 128 commands.

That’s it. Nothing really mysterious, but I admit, that’s pretty basic and far to be as practical as a remote control. Now the cool thing is that you can do much much more than that:

The device itself has an infra-red receiver you can use to record control codes from your existing remote control; you can of course pilot your PC this way.
You can pair two devices to create a long-distance infra-red relay system
There is a complete UDP based API wich can be used to send raw streams of infra-red data, update the pre-programmed codes, change the DHCP parameters, update the firmware…

If you are interested by this device, you can find it on Keene Eletronics’s site, here is the link to their infra-red related products. They even have a magazine showing how the various products can be used together, available in PDF here.

My control panel

The end objective is of course to write complex macros to automatically do operations like switching on the media box, switching off the digital cable TV box, set the TV input source and amplifiers to use the media box, etc…

I’m not there yet, but what I have so far is a complete web version of my media box remote control commands (using a more practical layout), which can be used from any mobile device with a touch screen.

The remote control web interface (WIP)

When the web page buttons are selected, the page sends the correct pre-recorded control code to the Kira device, which then send the infra-red stream to the device.

The actual code

Interacting with the Kira device from a web page was actually less simple that I expected, mostly because when I started this project I was a total AJAX n00b

The Kira own’s remote control page is very simple, it’s basically just a list of clickable buttons with a onclick event:

POWER
UP
DOWN
...
SHUT DOWN INTERNET

The send_code function is just a simple javascript function performing an AJAX operation⁴:

function send_code(code)
{
	var ajax=new XMLHttpRequest();
	if (code<100) code="0"+code;
	if (code<10)  code="0"+code ;
	ajax.open("GET","remote2.htm?button"+code,true);
	ajax.send(null);
}

So given a code between 1 and 128, the function creates an URL containing a string from remote2.htm?button001 to remote2.htm?button128.

The first idea I had was of course to try to just have my control panel link to these buttons (did not work), then to copy the ajax code to my own page (did not work either). The reason is because of some nasty security constraints related to something called cross-site scripting.

Basically in order to avoid hackers and phishers to exploit real sites and trick users by integrating some elements from a real site into another, there are restrictions on what can be used. An example of that is that you cannot make a script access data coming from another domain than the one where the script was loaded from. Bummer.

I needed some help.

Keen support

After few hours of head-scratching, duckduckgoing⁵ and stackoverflowing I decided to contact Keene's customer support. Of course I did that on Saturday morning, so I was planning to do something else during the week end... until surprise! I got an answer from the guy who wrote the firmware, complete with explanations on how to do things, some Wireshark packet dump showing the content of the data exchanged between the Kira device and the browser. Call me impressed.

It took some back-and-forth exchanges until we finally managed to find something that works well, so I'm going to share it here. Basically the idea is to not use the ajax method (due to the security constraints), but to instead use the UDP API.

Here is the result:

POWER
UP>
DOWN>

I admit the code is a bit violent, but it's working well ⁶.

The same code can be used to send any of the other cmdT commands described in Keene's API details document.

The next step

What remains to do in this part of the system is first to add the control codes for the remaining devices (the TV, the GET Digital Box, and the Thomson 5.1 system), then add support for sending multiple commands to multiple devices in one single request.

At this point my biggest hurdle is to get the page layout to nicely adapt to the native resolution and orientation of the device so it stays at a usable size on screen: A 2x2 pixel icon is not super practical to click on

I'm going to document all the parts of the system, and probably release the whole set of pages, scripts and css data so anyone can have fun of their own.

That's all for today

Keene IR Anywhere ↩
I had to add a colourful gradient and orange neon glow effect, because truth be told, the while square box with cryptic text on it is not that exciting to look at. ↩
At the pain of having to deal with a security model which really does not want you to mix and match stuff from various sources, but that can be dealt with. ↩
The actual code is doing error checking, I removed it for the sake of readability. ↩
Other people are Googling, these days instead I'm using DuckDuckGo ↩
You of course need to replace the hard coded IP by the one used by your device. ↩

Home-automation experiments

25 Mar 2011 at 22:33

Dbug

Uncategorized

1 Comment

Inspiration

For about two weeks now I’ve been working on a small home-automation project. The inspiration came from few cool YouTube video, and in particular that one:

I’m not particularly fan of Star Trek, but the idea of controlling all the audio and video system from a touch screen using some fancy interface kind of resonated with my inner-geek.

So, what’s this home-automation concept anyway?

The home-automation concept

Well, if you push it to the limits, it is about making your house “smarter”, which mostly means you can control the power consumption, detect water leaks in the utility room, automatically regulate the temperature depending of the time of the day and your planning, start the coffee machine so it is ready when you wakeup, call the local police station if an intruder is detected, etc, etc…

There’s a whole industry¹ out there ready to provide wireless cameras, power-grid based transmitters and controllers, security and temperature sensors, smart appliances of all kinds… basically if you can think about something in this domain, it probably exist.

So now, my idea was not to do a full home-automation, if only because it would be overkill for both my small apartment and my wallet. No, what I wanted to do was to have the possibility to replace all my remote controls by one central system connected to a server able to control the whole audio and video equipment, start and stop video streaming from the media center, and then build on that to add small things like gadgets to show the weather forecast, the departure time of the next metro, the latest news headlines, etc… perfectly doable, absolutely pointless, and so by definition something I absolutely needed to give a try at!

The motivation

Of course I have my reasons for trying that: I wanted to invest some time in improving my web-programming skills: Let’s face it, Defence Force was written in 1997 in good old HTML 3.2 transitional, entirely built as static HTML made from tables, and without any CSS, database back-end, javascript or php.

So this was the perfect project to motivate me: The control interface had to be practical and nice looking, had to work on computers as well as mobile devices, the real-time nature of the whole thing required some scripting as well.

I’m going to detail another day about how to control the electronic equipment, today I’m only going to show how to display when the next metro is arriving.

Json and the metronautes

I guess I started this project at a particularly awesome time.

Not one week had elapsed after the moment I decided I wanted to show on my screen the metro arrival time, that Trafikanten² decided to provide a web API to access free of charge all the data they use for their main website and stops! Not only that, but they decided in a way that is actually practical and very easy to use.

All the information is available on the Trafikanten Labs page, but since apparently some people have been struggling about how to use it, I’m going to explain you now how to do it. It’s very simple, an you can almost copy-paste the code

This concept is very simple.

The API is basically just an URL to call, and in return you get some data formatted in JSON format³.

If you add a special ?jsoncallback=xxxx parameter at the end of the URL, the data returned will instead be a variant called JSONP (padded JSON), which is just exactly looking like a javascript function call (using xxxx as the function name).

Given this information, all you have to do is to find in the documentation the parameters you need to pass to get the data you want, and write the correct javascript function to process the result.

Then all you have to do is to interpret the data, which is of course dependent on what you requested. In this particular example I requested the real time traffic data for the metro stop Grorud (id 3011940⁴).

The complete code

Here is the complete html code you can use directly to show the results

function convertDate(stringDate)
{
   var convertedDate=stringDate
   convertedDate=new Date(parseInt(stringDate.substr(6)));
   return convertedDate
}

function showEntry(entry)
{
  document.write("Line "+entry['PublishedLineName']+" to "+entry['DestinationName']+"")

  var aimedTime   =convertDate(entry['AimedArrivalTime'])
  var expectedTime=convertDate(entry['ExpectedArrivalTime'])
  document.write("Arrival time - Normal:"+aimedTime.toLocaleTimeString())
  if (expectedTime.toLocaleTimeString()!=aimedTime.toLocaleTimeString())
  {
    document.write(" - Probable: "+expectedTime.toLocaleTimeString())
  }
}

function parseResponse(array)
{
  for (var i in array)
  {
    showEntry(array[i])
  }
}

Please excuse the quality of the code, it’s the first time I write javascript, hopefully I will manage to write better code when I understand a bit better how the whole language works.

Hope this helped!

If you are interested, check out products by Z-Wave, X-10, Smarthome’s Insteon, Zigbee, Keene, Waveman, Atlona, … ↩
The company in charge of the metro, bus and tramway in the Oslo area. ↩
You can think of JSON as some easier to parse XML ↩
You can query the id for a particular stop using the API call /Place/FindMatches/grorud ↩

Communication protocol exploits

27 Feb 2011 at 13:12

Dbug

computers

2 Comments

If you found this page using a search engine with the hope of finding any relevant information about exploiting things on the Internet, you are going to be disappointed. What I’m going to write about now was relevant only in a now semi-distant past.

I have to start first with a disclaimer: What you will read is related to events that happened a long time ago, when we were young, stupid and lucky. If you try to do this kind of thing now, you may very well do some jail time and have heavy fines to pay as well.

Tools of the trade

Stamp reuse

Back in 1989 I was part of a demogroup called NeXT.

The founders were closely related to a cracking group called The Replicants and happened to use the same mail box address in Marseille ¹. Soon NeXT got famous² so we decided it would be a wise move to not use the same mail address as a software cracking group. From this moment we started to use my own home address for all NeXT related communication, which in theory should have been only related to swapping demo disks, source code, disk magazines, etc… only legal things.

In all our productions we clearly stated that we would answer to mail only if the sender provided a self-addressed and pre-stamped envelope, and if necessary blank floppy disks.

This worked fine for a while.

Then one day somebody mixed up the addresses and sent me a large bubble wrap envelope fully packed with cracked software. The problem was not the cracked software. The problem was that this person had the super smart idea to use the UHU Stick trick, and unfortunately the agents at the central post office noticed it.

If you never heard of it, the concept is super simple: Stamps are costly, so why not just reuse them! Of course the problem is to find out how to remove the stamping mark added by the post office.

Well, the solution is super simple: Put the stamps on the envelope and cover them by applying a thin layer of glue. The weapon of choice was the famous yellow UHU stick, just spread a bit of glue on the stamp, wait for it to dry. When stamped the ink will not impregnate the paper and just brushing the surface of the glue will move the ink out. All the correspondent has to do is to steam remove the stamps and reuse them again when sending back the mail ³.

One day the postman rang, he had a very serious face and was handling this bubbled wrap letter – covered with stamp marks and big scary red writings. He told me that this was a serious infraction to the Postal Service Code, and that the Postal Administration was considering filling a complain. I pointed out that it was obviously the sender who was in fault, not me… Fortunately I was able to show my good faith, because the postman had other letters for us, each of them had a self-addressed envelope in it, with a clearly written address and legit stamps on them.

He asked me if I knew the person, I answered that I did not know all the persons who were contacting us (because we were famous of course!), and that possibly they could track the culprit by starting investigations from the place where the letter was posted. Obviously he decided that I was being honest because I had no more problems.

Now well, I knew who the moron was, and I told him what I thought of him by phone the very same day.

Booby-trapped floppies

When they were are asked about the dangers of swapping floppies around, most people would think of the risk of being infected by a nasty virus. Sure, they were common threats, and I had my share of these on the Atari ST. The most common one was the Ghost Virus, which was pretty much innocuous by itself: All it did was to copy itself on any other non protected disk you inserted, and after a short while was swapping the vertical axis of the mouse so when you moved the mouse up the cursor would move down⁴.

So yes, Virus were a nuisance, but they were not as dangerous as booby-trapped floppies which would pretty much physically damage your computer. I know of two methods used to do that, and both involve killing the floppy disk drive.

The first method required some programming, and assumed that the recipient would actually run the program present on the floppy. The idea is to write a program that repeatedly tries to access a track out of the normal use range. The way a floppy drive is built is that the reading head is moving along an Archimedes screw, when the screw rotates in one direction the head moves slowly in one direction, when you reverse the movement it goes in the other direction. If you continue to rotate the head will eventually be stopped, but if you push again and again it will result in the head getting de-calibrated: the drive is never going to be able to read a standard floppy again, because these things need to be very accurately calibrated by special heavy equipment⁵.

The second method is a lot simpler: Slide open the metal protective door on the floppy disk and put some of your favourite syrup on both sides of the exposed magnetic surface. Then all you have to do is rotate the disk a bit so it’s does not look like anything is wrong, put a sticker with an impossible to resist appeal (like “Dungeon Master 1.2 final 101%” or “TCB leaked source codes”) and send to your target (generally a high-profile swapper you want to get out of business). If you are lucky enough the person will have read errors on the machine he first tries to use and will try on a second machine as well. Both machine will be unable to read any floppy until the glue-ish syrup is removed from the magnetic head, which will require the opening of the machine, removal of the drive, and very very very gentle care: Touching too much the heads will de-calibrate them.

Fun no?

I guess that post was long enough for today, next one will be about how to make money using the Minitel!

The famous 5 & 7 Place des marseillaises. ↩
First with The Charts and then with the Phaleon Gigademo. ↩
Reusing the envelope by covering the original address is pushing the luck a bit too much. ↩
I eventually had to do a pass of eradication of this virus because it was buggy and would destroy the file allocation table if the disk was using a non system standard format. ↩
That’s the reason why there is some shiny red varnish spots in some places, it’s to stop vibrations and keep the assembly in perfect stable position ↩

Simple C++ threading

20 Feb 2011 at 14:11

Dbug

computers

8 Comments

For some reason, people tend to think that creating multi-threaded applications in C++ is a very difficult thing to do¹.

It is in fact not that difficult, but there are basically two main issues to consider: How to make the code correct (it should not crash, deadlock, corrupt data, etc…) and how to make it efficiently use as many cores as possible².

The usual suspects

One of the things I noticed is that quite often the code of the application is relatively decent, but the poor performance is due to the usage of inefficient functions called by the application.

Modern applications are not working in a vacuum; they have to allocate memory, load or save files, interact with the user through a GUI and input peripherals, they may use network connectivity, log information, encrypt data, etc… Gone are the days when a programmer would write every single component used by his application. Today you use APIs provided by third parties, and if these API’s are not designed correctly your application is going to suffer.

We are now going to investigate the fancy world of Logging Systems. Let’s examine how the typical logging code is done and what can be done about it.

Logs, the basic concept

Typically when you want to find out what your program is really doing you start adding printf all over the place³. Soon you realise it is not very practical, and you wish you had nicely formatted text files showing the log entries with accurate time stamps. Your program would look then like this:

#include "log.h"

void main()
{
  Log("Starting application");
  Log("Calling something");
  Something();
  Log("Calling something else");
  SomethingElse();
  Log("Quitting application");
}

and the result like that:

2011-02-19 08:23:00 Starting application
2011-02-19 08:23:00 Calling something
2011-02-19 08:23:01 Calling something else
2011-02-19 08:23:03 Quitting application

This is a very basic system, but with that you have a pretty clear idea of what was called, when, and by extension – how long it took to run -. A slightly more advanced system would use millisecond timestamps, and would allow you to add categories (information, warning, error, …) as well as advanced formatting to handle variables ⁴.

The devil is in the implementation details

The performance of such a system may be good, or it may be terrible. It all depends of how the Log function was implemented.

The simplest way to implement it would be something like that:

void Log(const char* message)
{
  handle=FileOpenForWrite("LOG.TXT");
  FileWriteToEnd(handle,message,strlen(message));
  FileFlushAndClose(handle);
}

Please note that this is not valid C++, it is just an example of the whole idea of adding a new line of text at the end of a text file. In the real world you would check for error conditions, use whatever file handling function you wish to, would handle the carriage return at the end, the timestamp, etc…

The subtlety here, is that most people write the Log system, test it, measure the performance on some small examples, find it satisfactory, and then run ahead using it without second thought. When they start to experience performance issues later on they will generally not think about suspecting the Log system because it was apparently good enough on the small initial tests.

What happens in the real world, is that two phenomenons happen: You may have reached some contention at the operating system level in the file buffering code, and you may be victim of thread synchronisation when you use the code in a threaded application.

So, what is wrong in this code?

Thread safety

If you use naïve code that shares common resources (in this case the file called “LOG.TXT”) without any form of synchronisation, what will happen is that the instructions in the function will get interleaved. This will results in unpredictable results, which most probably would be different depending of the operating system in charge of the handling of the files, the performance of the processor, etc…

If you want to have a rough idea of what will happen, try to imagine you are having two threads trying to print “Hello world\r\n” on the screen at the same time.

Which result do you expect:

Hello world
Hello world

HHeelllloo wwoorrlldd

HeHelllloo worworlld
d

The truth is, you don’t know because the result is not defined. Just don’t do it. You have to protect the shared resources to avoid simultaneous access to it. Either make sure that only one thread is in charge of writing information to the file, or use some of the synchronisation primitives provided by your operating system like the critical sections and the mutex.

Performance and synchronization

Ideally you want to be able to use your logger without incurring heavy runtime performance hits. If you have to disable your logger in order to use your program then you are never going to use it.

So, how bad is the performance of our naïve Log function? Well, it depends of how often it is called, of how efficient your operating system is to write to your hard drive, but also of how much stress you have on your disk at the moment: If you are copying a multi-gigabyte file in the background it’s possible that your Log function stalls for a while until it can actually write to disk.

If you want to be thread safe, you also need to have the synchronization primitive to avoid two simultaneous calls to Log to be interfering with each other, which in practice tend to make your threads to synchronize when they try to write to the log file. This is bad.

A stupid example

Let’s say we want to write a stupidly inefficient, pointless and incorrect program to compute the factorial of the 50 first integers. We could do it like this:

#include "log.h"

void main()
{
  Log("Computing factorial");

  Log("Starting 50 threads to compute factorials");
  int results[50];
  for (int i=0;i<50;i++)
  {
  	StartThread(ComputeFactorialThread,i,&results[i]);
  }
  Log("Waiting for all threads to finish");
  WaitForAllThreadsBeingDone();

  Log("Showing results");
  int results[50];
  for (int i=0;i<50;i++)
  {
  	std::count "Factorial of " << i << " is equal to " << results[i];
  }
  Log("Quitting application");
}

void ComputeFactorialThread(int threadID,int n,int &result)
{
  Log("Starting thread #" << threadID);
  result=1;
  while (n--)
  {
    Log("Adding value " << n);
  	result+=(n*n);
  }
  Log("Exiting thread #" << threadID);
}

Assuming the code is working (ie: that it does not overflow when computing 50!, that all the missing threading boilerplate is there and that the code actually compute a factorial), what it does is to start 50 threads, the first in charge of computing 1!, the second computing 2!, until the last one computing 50!.

These operations are very trivial, but since we are very insecure programmers we decided to log everything, so before multiplying each new value we decide to log the operation we are going to perform. Obviously it’s never going to be fast, if only because just calling a function to display a text takes more time than multiplying two integers together. But that’s not really the core of the problem: If the Log function was written correctly, it would just use the necessary time to receive the new text entry and return as fast as possible.

Instead every single Log operation is going to:

Try to lock the mutex (or wait until it gets available)
Open the file
Add a new line at the end of the file
Close the file while ensuring everything is really written to disk
Release the mutex
Return to the caller

Thread 1	Thread 2	Thread 3	Thread 4
Wants the lock	Wants the lock	Wants the lock	Wants the lock
Has the lock	Waits	Waits	Waits
Open File
Write To File
Close File
Releases the lock
Working	Has the lock
	Open File
	Write To File
	Close File
	Releases the lock
	Working	Has the lock

When you have 50 threads running on micro-tasks, they will basically spend almost all their time trying to catch the mutex. As soon as it will be freed one other thread will have a chance to catch it while others still wait: Basically you just made your program single-threaded.

The solution

The solution, as often ⁵ is pretty simple. Instead of writing to the file directly in the Log function, just make it write to a queue and have a worker thread regularly empty the queue and perform the actual writing to the disk.

At first it may sound just like deporting the problem somewhere else, but if you think about it this allows some pretty nifty optimizations: If you write each line of text as soon as you receive it, you spend your time opening and closing the file. If you do that in a worker thread instead, then you can write a bunch of entries in the log file in one single operation, that’s a lot more efficient from a system and I/O point of view.

After this change, our new Log function looks like this:

void Log(const char* message)
{
  AquireMutex();
  Push(MainLogQueue,message);
  ReleaseMutex();
}

Note: It is sometime possible to use a “Lock Free” container instead of the Lock/Push/Unlock. As long as the atomicity of the operation is guaranteed it is fine.

And of course we will need our worker thread as well:

void LogWriterThread()
{
  while (IsRunning())
  {
    // Wait a bit
    Wait(DELAY_BETWEEN_WRITES);

    // Swap the queues
    AquireMutex();
    SwapQueue(LocalQueue,MainLogQueue);
    ReleaseMutex();

    // Is there something to write?
    if (!LocalQueue.empty())
    {
      handle=FileOpenForWrite("LOG.TXT");
      FileSeekToEnd(handle);
      while (!LocalQueue.empty())
      {
        message=LocalQueue.pop_front();
        FileWrite(handle,message,strlen(message));
      }
      FileFlush(handle);
      FileClose(handle);
    }
  }
}

If you are wondering about the SwapQueue, it’s an idiom I use very often when doing multi-threading programming. Instead of copying the content of the queue and clearing it, you just swap it out with an empty one: It takes almost no time if your container is implemented correctly. The result is that you keep the lock on the mutex for a very short period of time, so even during the regular moments when you flush the logged entries to disk, you do not lock out your other threads that try to write to the log queue.

The state of the art

I’ve done a small search on the examples of loggers you can find online, and they tend to be pretty bad. Most of them are pretty, seem to handle quite a lot of situations (multiple logs, template meta-programming to handle parameters, automatic creation and deletion of the logger instances, …) but many would behave terribly in real multi-threaded usage. That being said, there are few that seem to use some form of asynchronous file writing, these ones are probably ok to use.

So, if you decided to use a logger you found on the internet, I really encourage you to take a bit of time trying to understand how it was implemented in order to avoid performance issues later on.

Better safe than sorry

Have fun !

Probably because unfortunately the efficient C++ programming style is not really what people got taught at universities. I wish university teachers had some actual experience in the real world instead of repeating the nonsensical crap found in most textbooks. ↩
Obviously, if making your code correct leads to making it inefficiently use your cores then you should not bother trying to thread it at all: Just keep the code nice and simple – and single threaded. ↩
Or std::cout, choose your poison ↩
But this is just minor implementation details, it does not change anything to the base concept and the associated issues. ↩
Some people tend to make things more complex than they really need to be. If you are not being paid by the number of lines of code you write, I advise you to try to write less code. ↩

How fragile is this Internet thing?

30 Jan 2011 at 14:21

Dbug

Uncategorized

1 Comment

Quite a lot of people these days take for granted the ability to communicate with their computers and mobile phones at any time of the day ¹.

People are so used to this apparent dependability that in the odd case when somebody does not answer to a mail, invitation to a party, a greeting card, etc… that it must be that (s)he does not really want to. Well, perhaps they just never did receive the message in the first place ².

One of the thing we’ve seen lately (among other) is that Internet – and the mobile networks as well – were actually quite fragile: A destroyed back-bone here, a revolution there, heavy handled filtering and censorship all over the place: No, you should definitely not consider your ability to use your communication devices today as something rock solid.

Internet was originally designed to be able to work in adverse conditions as a global nuclear war destroying a significant part of the world. The idea was to have as much redundancy as possible.

Internet traffic to and from Egypt on January 27- 28

Internet blackout

Trouble in store

This original intent has obviously been lost somewhere, because a government should not be able (technically) to just press a button to cut entirely its country from the rest of the world.

If you add to the mix the talks about net neutrality, the botnets attacks, the dissemination of supposedly confidential information, the data storage directives, the filtering of pornography (for my own good) or opposition sites, it’s clear that there’s a distinct opposition between what most internet users would like and what their enlightened leaders are pushing for.

2010 in France has seen the successful vote of the Hadopi and Loppsi laws. In the country of the human rights we now have institutional filtering of Internet, tracking of network usage, and a whole lot of other surveillance and control related material.

All over Europe everybody is busy implementing the Data Retention Directive. All over the world lists of dissidents (or potentially troublesome persons) are collected. ³ Nice world we are living in these days.

It’s kind of an interesting twist that during the same week we can follow the Internet black-out of Egypt, the Obama administration proposes a similar system for the USA ⁴.

TOR and Freenet will not help

With the increasing number of case where the freedom of speech has been threatened around the world, people have come with solution to restore the balance, generally using some form of encryption or haystack hiding principles.

Sure, hiding your controversial statements deep in packets, hoping for the government will not find them is nice. But your packets are not going anywhere if the tubes themselves are cut. So no, The Onion Routing, Freenet or Haystack are not going to help people who got their Internet access cut.

The only way you can really fight a low-level communication blackout, is to use alternate means of communication, and most of the time this will involve modifying modern equipment or using older technology.

Going LoFi

Remember how the resistance was sending messages from London to people all over Europe during the second World War? Well, they were simply using radio equipment, and some coded phrases. Anyone with a radio receiver in their house could receive the messages, making it very very hard to find the actual listeners.

The same kind of thing can still be done today, using powerful radio transmitters installed on bordering countries. You can even send data; you are not limited to voice and music. I remember that in France around 1984 there was a radio show broadcasting programs for 8 bit computers: All you had to do is to record the show on your tape deck, and load it later on your microcomputer ⁵.

This does not solve the problem of sending data without being caught, but at least you can get information broadcast from the outside world.

If you want to do the same thing to send data from the cut-out country, your best bet is to get mobile, and have your radio equipment ready to move at the first warning: It’s trivial to locate an emitting station by triangulation. Alternatively, you can decide to broadcast only for very short amounts of time, at random periods of the day. Still dangerous, but should work for a while.

Apparently, some people managed to communicate using their old Dial-up modems, you can read more on LifeHacker. Of course this will fail as well if the land-lines phone network is shut down.

Going WiFi

Now I’m entering in the realm of the guess. I’ve been searching around, but could not find anything about it. I can’t possibly imagine that nobody tried, because the idea is so obvious that I can’t see how it would not work. If you can contribute, you are welcome.

Basically the idea is that today Wifi equipment is nearly ubiquitous in large cities. It’s so widespread that it’s actually a pain to find a place where you are not able to detect at least one Wifi network, and in a previous house I was able to see 9 different ones.

Most of the Wifi hardware (routers or access points) is both programmable and vulnerable: The number of security issues found on the average Wifi equipment is kind of scary. So technically, writing new firmware that would change the way these small devices work should not be much of a problem, and making it easy to install (and possibly transmits from one Wifi device to another automatically) should be doable.

The concept is just that given the high density of Wifi devices, it should be possible to have them maintain some kind of network (effectively a “internet over the air”) using the access points and routers as nodes. At this point all you need is one person managing to get an actual access to the outside world, and then you are back on the real internet, with an admittedly super slow and not reliable network, but at least something that work. Just add some solid encryption and you can relatively safely send and receive messages.

Would this work?

Oh, by the way, have an happy new year 2011!

Edit: Found a bunch of interesting related links:

They probably never experience the thrill of detecting the connection speed of a dial-up connection based on the frequency and funny noises – this double springy sound that tells you that your 56k modem managed to negotiate the x2 protocol ↩
Like one of my sister sending SMS to my father who for some reason can’t receive SMS on his telephone; and my other sister asking me why I’m not sending her mails – then realizing she never actually informed me she had moved to another ISP. ↩
I’m pretty sure that if I started to blog about the American or French Government in violent enough terms (while staying polite of course) soon enough I would be intercepted at some airport for some polite discussion with the immigration offices. Oh and my computer would of course be scanned and confiscated. Actually since my website is hosted on a .org domain by Yahoo! it would most probably be put offline… ↩
But of course it’s to protect the country from cyber threats. ↩
Well, assuming the transmission was correct, that the tape was good, that your mother did not decide it was a good time to start the washing machine, and of course that the computer gods were happy that day ↩

The cooking process theory

15 Dec 2010 at 19:08

Dbug

general

No Comments

These few last weeks I’ve been doing a number of unrelated things: I watched Master Chef on BBC LifeStyle, finished Fallout: New Vegas then started to play Titan Quest, finished a book from 37signals and another from Microsoft Press about project management, and finally spent time reading blog posts on some various famous IT heroes.

In some perverted way it all makes sense.

So, what’s the common idea between these seemingly unrelated elements? Well, the common element is the notion of recipe.

Cooking

A recipe basically is a list of actions to follow in some pretty strict order to achieve a particular result. Following a recipe you could for example manage to cook an egg.

The important point is that even if you follow the recipe accurately, you may still fail to prepare something tasty: Perhaps the type of ingredients available was not exactly matching what the author of the recipe had in mind, your rice should have cooked longer – or perhaps not that much. The temperature of your stove may be slightly different than what the controls indicate, or simply you were not that good at preparing the sauce.

The idea here is that even when you have a very accurate recipe you still need some skills and some brain to actually understand what you are meant to do. Possibly the indication about requiring one litter of olive oil was a typo, perhaps it should have been one decilitre instead. Who knows ?

Watching Master Chef is really a pleasure.

Far from being some kind of sadomasochistic contest orchestrated by a spawn of Satan, you find instead a relaxing setup with people who really try their best to show their ability at cooking, their imagination in how to combine flavors, they have to deal with various real constraints like list of ingredients to use, quantities, time limits, appearance, etc… they are not directly competing against each other and trying to find who else they will vote against. No, they are here to show what they can do and impressed the show’s hosts.

Just watching the results sometimes just makes me want to lick the tv screen.

Titan Quest

So, how does the cooking relates to Titan Quest?

Well, Titan Quest is what you can call a Diablo 2 Clone. Technically you could call it a Diablo 2 rip-off, because really even if there is a difference between inspiration and copy, that one is definitely in the grey zone:

Third person 3d game with fixed view point – Check
Gazillion of items of different quality (normal, magic, rare, set, …) – Check
System of collectible power-ups you can attach to items to give them additional magic properties – Check
High pitched *ping* sound when a magic ring fall on the ground – Check
Town portal – Check
Merchant to sell your crap but that have almost nothing worth buying – Check
Fantasy setup with mythical creatures – Check
Three acts in various countries – Check
Dungeons and caves here and there along a linear path – Check
Side quests unlocking areas – Check
Levels, skill tree, health and energy bars, electric, poison, fire, etc etc… – Check
Groups of random monsters guarding chests in the middle of nowhere – Check
Whole map repopulating when you reload the game – Check
…

I could continue like that probably for a while. Basically I wonder why Blizzard bothered making a new engine for Diablo III, they could just have asked nicely IronLore to gave them the engine – or sue their ass -, because well it’s a nice looking clone. There are of course some differences: It’s using the Mediterranean area for the setup, it’s not scary, it’s way too easy, and the biggest flaw – It’s not super exciting. It’s borderline boring.

So that was an example of cooking: They probably decided to create a game that would be “like diablo” (but better), made a bullet point list of the things they had to have, probably thinking that would be good enough, but in the end it terribly lacks of flavour. It’s not bad, far from it, but it’s just bland.

Obviously Blizzard is still the best company when the objective is to create a Blizzard game.

It’s interesting (I think) to make a comparison here with Fallout: New Vegas which despite being built from the Fallout 3 game, using the same engine, same animations, same graphics (mostly), still managed to feel much more different, and as I previous wrote: Much more rich despite using the same base ingredients.

Basically the difference in the two cases is that IronLore was not able to identify what made Diablo what it was, while Obsidian was able to identify the core values of what a Fallout game should be.

Software Development Methods and Start-ups

The same kind of “recipe” concept exists in the programming domain.

Despite people warning about Silver Bullets for decades, we still have books offering miracle methodologies (Agile, TestDriven, Extreme, …) pushed forward by evangelists and their associated software ecosystem (issue trackers, iterations managers, planning poker games, release managers, continuous integration/building/regression tools, white boards or post-it notes, …); miraculous languages (scripts, web-based, virtual machine based, functional, run-everywhere, easy, fast, magic); programming methods (object oriented programming, design patterns, …); that are going to solve all our past, present and future problems.

And the equivalent exists for how to create your own company and be rich and successful, with lists of do’s and don’ts.

The really annoying thing is that all these authors really believe that what they evangelise is the only true word, that it would work for everybody everywhere, and that if it does not work for you then it’s because you are doing it wrong.

Well, call me whatever you want, but I don’t believe this is true at all.

All these methods are just recipes, tools, that may apply in some particular case but would probably not be suitable in some other.

The difference between a good and a bad programmer for example is not really based on how good he masters a particular domain. What make him good is the set of accumulated knowledge that allows him to know when and how to use something, and when and how to use something else. And possibly when not to do something at all and ask somebody else to do it.

I would not do web programming in C++. It can be done, but for fast changing data C++ is probably not the best choice. On the other hand I would probably not use an interpreted language to do performance computing – except if benchmarks shows that the virtual machine is actually performing as good or better as native code. I would not use COBOL or Fortran as my shell script of choice either. And Visual Basic is probably not the best choice to make portable applications.

I think the biggest deception so far for me has been the whole Agile processes (count in it everything from Scrum to XP). Most people are following a list of things to do (iterations, backlog, velocity, points, whatever), but they are still not doing it right. The important is not all this crap, the important in the end is to deliver what the users need ¹, on time.

Everything else is just a way to get there, and as long as you reach the end objective it does not really matter how you do it.

Proving myself wrong

Just for the sake of doing exactly what I said was wrong with all these people, I will now give my list of things I think should be done and should not be done when you are doing software development!

What you should really have:

A source control system, with a continuous build system that detects the build errors and failed tests, and informs whoever broke them.
Some mix of Unit Testing for the core elements on which everything is built, and some Function/Regression Testing to validate and test the high-level systems and applications.
Your code-base should build at the highest possible warning level, with warning as errors. Else you are going to miss problems.
Everything that can be automated, should be automated.² The more manual operations you have to do, the more you risk to make mistakes or waste time.
If you can find a good existing third party library or tool that matches your needs, just use it instead of writing your own.
Release often, make sure the deployment process is streamlined, make sure that your users don’t waste time when trying your new versions, make sure you can get feedback easily.
Fix the issues (crashes, performance and usability problems) first, add features next.
Have a small team of experienced and motivated people that can work together.
Identify your production and organization bottlenecks and fix them.

What you should avoid:

Apply any particular list of things to do without actually understand why you have to do it, or if it even applies to you at all in first place.
Hire more people to go faster.
Keep bunches of semi-redundant tools and subsystems instead of merging and consolidating them. ³
Changing all your processes at the same time to adopt a new one instead of just incrementally change what does not work.

I could probably add rules like that, but the bottom line is to use your brain and try to be smart.

For example the rule about not adding more people is not an absolute rule, it depends of what you hope to achieve: If it takes 30 minutes to cook a meal, adding more cooks is not going to help you eat earlier. But if you have more cooks, you can instead have some of them work on the main course, other prepare the dessert, while the last ones are doing the starters, or you can have them help preparing a number of meals at the same time.

So yeah, there’s a lot about cooking you can apply to software development.

Which mean the feature they actually need are usable without productivity loss or frustation due to bugs or performance ↩
A good test is to switch off the master electric system, then power on again. Check how long it takes for everything to get back up ↩
If you need a new tool to replace an old tool that does not perform good enough, make sure that the new one can do everything the old one does so you can deprecate and bury the old one. The less thing to maintain the less burden on your shoulders. Same thing for semi duplicated code. ↩

Mini review: Fallout – New Vegas (Obsidian/Bethesda Softworks)

27 Nov 2010 at 20:05

Dbug

computers, games

No Comments

After 71 hours¹ playing Fallout: New Vegas, I’ve finally reached the end of the story.

Mind you, I’m not done with the game: Since you can’t continue to play after reaching the end of the main story line, I loaded a previously saved game to explore parts of the world I did not visit yet (I picked-up the Adventurer Perk just to get the list of all locations), which is about half of the possible locations. Talk about a huge game

Welcome to the Vault 21 hotel!

Flashback

It’s hard to play Fallout: New Vegas without doing comparisons with the previous episodes. How does it compare to the original isometric games (Fallout 1 and 2, and possibly the strategy game Fallout: Brotherhood of Steel), or to the more recent full 3d Fallout 3?

When the Fallout licence was acquired by Bethesda Softworks who soon after announced was working on Fallout 3, the blogs and forums started to look like battlefield.

Some people were expecting Fallout 3 to be awesome, generally from people who loved Oblivion and never really played the original games. Most of older players assumed that Fallout 3 would be a disaster, a “Oblivion with guns”, and certainly not a true Fallout game.

Well, I guess everybody’s point of view was actually correct: Fallout 3 was actually a quite good post-apocalyptic first person adventure game in which you could recognise some core elements of the Fallout universe (super mutants, brotherhood of steel in power armors, nuclear waste, bottle-caps as money unit, etc…) while still not really feeling right as a Fallout game.

Just to make it clear, I’m one of these persons who love Morrowind, and really think that Oblivion is a terribly broken game in a terribly boring setup, so I was expecting the worse for Fallout 3.

When I finally played it ² I found the game actually quite good: All I had to do is just not trying to think of it as a Fallout game.

More of the same?

Comes New Vegas.

I think I would not have found what exactly was missing to Fallout 3 if I did not have the chance to play Fallout: New Vegas. The fact the engine is exactly the same, that the visuals are very similar to the original game, that the GUI and basic mechanism are also verbatim, makes the comparison very easy, and shows the difference between Obsidian and Bethesda when it comes about game design. No suspense here, I think Obsidian really outshines Bethesda.

What’s very obvious from the start is that New Vegas is less forgiving. In Fallout 3 most encounters are non lethal, the consequences of your choices are very obvious on the short term, and everything is manichean. You always know if what you did was GOOD or EVIL. For example the slavers are bad guys. And the survivors in the small village are good guys. No discussion, no polemics, no possibility that you may have it wrong. Life is simple: You are with me or you are against me.

In New Vegas things are not that simple. Your first encounters tend to be your lasts, and you fast discover the virtues of running. There are these characters that are helping you, but you have this nagging feeling that they have their own agenda. Should you really trust them?

Then you have the factions. In Fallout 3 you have few groups and communities spreads over the Washington DC area, but they don’t really have any link with each other. Helping or fighting the Brotherhood of Steel is not going to impact your relations with any other group. It’s really like if Bethesda had each area of the game designed by separate groups of people, each one making its own self contained adventure area on the map.

In New Vegas everything is interconnected. You have the obvious big Factions (like the NCR and the Legion), but as you play you discover a myriad of smaller groups (Boomers, Kings, Vipers, Powder Gangers, etc…) that are loosely related to some other groups. The way you handle them may have large consequences on the way you can solve some of the quests you are doing for some factions. And up to a certain point you can do work for a number of opposing factions at the same time, which give an interesting twist in the way you solve things, instead of just a binary choice: Will I work for this group or for this other group.

The end result is that you feel much more involved in what happens, the world feel alive – with people having real troubles -, and the various branches in the story line keeps you interested.

The sheer number of places, hidden stuff, remote locations where event happens that you would not even have known of if you were not just exploring around is just the icing on the cake.

I don't give a Dam about what you are thinking of my puns.

Bugs, did you say bugs?

Reviewing a game from Bethesda and assuming it’s not buggy, is like reviewing a game from Obsidian and hoping it’s finished³.

Yeah I know, I’m working in the video game industry, been there, done that. Had my share of “sorry it’s not finished but we have to release it now in time for the quarterly reports, we will release patches later” projects. Sucks when it happens.

Anyway, in this particular case, I’ve been lucky enough to play the PC version, and the patches to solve the various animation glitches were released before I started to play, so I’ve not seen the random rotating heads and other scary animation bugs.

What I’ve experienced in 70+ hours of play:
- Two clean exit crash to desktop
- Four freeze when fast travelling
- Quite a log of sluggishness (until I stopped the PipBoy Radio streaming)
- The texture manager sometime refusing to show the high-def textures on the ground
- Ants half buried in the ground
- A soldier busy typing his report on an invisible type writer

That’s about it for the engine bugs. Since I tend to be a serial-quick-saver I was not too much impacted, specially with the game loading quite fast.

There was some more annoying problems still, mostly in the scripting of quests.

In a number of occurrences I had some missing or wrongly positioned quest markers on the map, meaning I had to run around for a while before finding who I was looking after. Some of the dialogues also were not clear, leading me to choose exactly the opposite of what I wanted: very annoying when it is related to the main quest.

One of the quests was particularly badly handled (one involving sabotage), where I guess the idea was to show the player that they did something wrong, but the way is done is so bad that it looks very much like you can fix the problem but that it’s just game being buggy. Since it was a minor quest without any real consequence I just moved along, but that was frustrating.

Final words

Well, I pretty much enjoyed Fallout: New Vegas. I think the game is brilliantly done, with a real dedication in the execution.

I wish I will one day have the opportunity to play another game by Obsidian, but this time one that has not been rushed out, and where the QA department was actually able to spend the time to test things inside-out.

Upgrading the Gamebryo⁴ engine would also not hurt. It’s really showing its age: Objects that jump from the shelves was funny 5 years ago, but since then the expectations for a physics engine have kind of evolved. And remove the invisible walls too, they are very annoying when you explorer.

I can’t avoid making a parallel between Fallout: New Vegas and Dragon Age/Mass Effect.

If you draw a graphic with on one axis the deepness of the experience in terms of choices and consequences and on the second axis the degree of apparent polish in the execution, the result would probably put these games on diametrically opposed positions.

The recent tendencies have been to make dumbed down games where you basically have only fluff decisions that have literally no impact, you could just move forward pressing the fire button or the enter key and the whole game would reach the end. That’s just sad.

Give me more games like Fallout, but let them be polished like Mass Effect

In conclusion: A highly recommended game – if you can live with bugs.

Thanks Steam for keeping statistics about everything. ↩
when it was released on PC as the Game Of The Year version, because Bethesda games are always terribly buggy – so you better wait 6 months for all the critical patches to be released – and they always release downloadable content that finally get bundled together in a nice updated version. ↩
Or reading my blog and not finding grammar/syntax mistakes. ↩
Looks like Emergent is in bad shape, so they may have to do that anyway ↩

Random notes about Community Management

14 Oct 2010 at 19:57

Dbug

computers

No Comments

Foreword

I was really not planning to write anything about Community Management, mostly because I did not consider myself as a Manager, and certainly not a Manager of a Community.

I more or less revised this position few weeks ago when after having released 1337¹ and posted messages on the various relevant message boards and forums we then started to receive messages and comments asking how Defence Force was working, how such a small community managed to release such large and polished games (compared to for example the Thomson and Amstrad communities), and if we could help them by making games on their own machines².!

I cannot talk for Chema or Twilighte, but for my part I have no intention in doing that.

What I can provide is some insights about how we managed to do that. If they really have the intent of doing something similar it’s perfectly doable: All you need is time and method.

What is Defence-Force

I guess I could have started by the dinosaurs and from there describe the chain of events that have lead to the current situation. I will not do that, and instead will start by trying to describe the current situation and what were the critical events and decisions that allowed us to reach this point.

As of today³ Defence-Force is probably the most active Oric resource on the web. There are other support groups like the Club Europe Oric which organize real life user meetings and publishes a monthly magazine, Oric.org which is mostly known as the largest online archive of Oric software, OricGames.com with an active community of persons playing and reviewing games, plus numerous Oric sections on the various generalist retro-communities, but as a general resource used daily, I believe that Defence-Force is now the most active.

So what is Defence-Force? Well, it’s nothing and many things at the same time:

A quite active discussion forum that requires almost no moderation⁴.
An FTP available for users of the forum to share their large related Oric data (like photos and videos of retro-parties, …)
A Subversion repository where programmers can store the source code of their Oric projects, and allows other to learn and participate (for example by optimizing some critical routines)
The maintenance of the OSDK⁵.
A Wiki engine to share some technical knowledge.
Subdomains for community members who have Oric projects that deserve visibility but have no personal websites.
The Oric Library where a significant number of magazines and books can be downloaded.
The #oric chat where people can discuss together in real time.

From a search engine point of view, I admit I’ve been a bit lazy these few last years, if you type “Oric” none of the sites appear on the first page on Google. Now if instead you type “oric forum”, “oric development” or “oric demo”, the first link on the page goes to defence-force. If you type “oric books” the library appears on the first page of results, and “defence force” also appears on the first page of results between the New Zealand and Australian Defence Forces websites.

Could be better; could be worse.

Now of course, we are talking about a Community, and you have to realise that the important is the people, not the infrastructure.

I know plenty of people who wanted to create their own little Community about some particular interest of them who became bitter after failing at achieving it. The thing is, there are some rules to follow. Not complex rules, just rules about how people work. If you don’t understand these rules you will most probably fail.

The human factor

I think the single most important element is that your offering need to address a real need. This need may be obvious or not, but the fact is, your offer needs to bring something to other people. Not to you.

If it brings something to you in the process, sure that’s even better. But if you do something for your own sake, without considering what other people think, you will mostly be disappointed.

The second element to consider, is the trust factor. People want to know that they can trust you, that you are who you say you are, that you are going to be there for a while, that you are going to be reactive to the issues people bring, that your objective is to help others not just be a small king of your mini-empire.

The reason is that people are going to invest some personal time when joining your community, and they probably remember that in the underground of the web lies this gigantic graveyard of animated GIFs infested personal websites, failed opportunities, forgotten start-ups and other crushed hopes.

The last factor is really the quality of the offering. If what you provide is clumsy, ridden with advertisement links, hard to navigate, unreliable, etc… people will not stay, and most probably you will lose any credibility you may have acquired earlier which will make future efforts even more difficult to achieve.

Oh, and it has to be free, so expect to take a financial hit if you do it alone. If you expect people to pay for your services, you are also going to fail – even if you provided awesome services⁶.

So, how to proceed to achieve that?

Making it happen

First, you have to study the existing offering⁷. The existing community can range from one single personal website to something very large with hubs and mailing lists, and it can be from friendly to aggressive (like on some of the home-brew communities were it goes up to the point to the personal attack for using emulators and cease and desist letters sent by attorney because you shared some ROMs for a machine long disappeared).

Staying around for a while without trying to revolutionize the community has some advantages: You get to be known, you get to know the people, you get to know what works and does not work, you can point out what’s missing as well, and if you try to see the situation as an outsider you can even see what people who’ve been in the community for a long while don’t see anymore: The external visibility and how easy for people to find and join the community.

From there you can start to build a small something, offer it to the community, see how people react, fix what needs to be fixed, and see if people actually like it and use it. In the case of Defence Force it all started in 1997 as a personal website to host my own things, but since I had some storage space on my web hosting plan I offered to host some little things like the schematics for a better RGB cable, downloadable books and magazines, etc… When a website has been on-line for 13 years more or less continuously, people tend to trust the owner that the next offering is not going to disappear the next day.

At this point, when people know you, appreciate what you are doing, if you suggest to do something then people will probably accept to discuss about it, do some suggestions on what would be cool to have, etc…

When you have these elements then build/install a small prototype, and show it to some of the people you think are the most active or the ones you feel can be trusted by giving honest feedback, ask for their opinion, discuss the tweaks that may need to be done, ask them to test a bit more, and then when you are confident that everything works you can go for the grand opening: This is valid for forums, wikis, polls, everything online that can crumble due to database configuration, webserver problems, etc… but also for new software. Make sure things are tested and working before deploying: If you release non tested things, it will most probably blow, and people will lose trust.

Putting it all together

This last point is very important: Ultimately the quality of your offering will be perceived as the sum of all the little things put together: You start with some points, and every little detail that does not work removes some points. When the number of points is too low, you have lost.

Here is a list of things that will make you lose points:

A non reliable system which happens to be down most of the time. If your forum is down every two days, if people have two keep re-registering or re-logging, they will be bored very fast.
Something that looks like “pimp my website”: Forget the animated gifs and the awesome flash intros, forget the CSS+canvas madness that works only on the latest version of your favourite browser. Make it simple, clean and compatible. Of course if you happen to be a web-designer who can do something awesome that works everywhere, go for it!
Difficult navigation, non-controlled number of sub-forums, anarchic wiki structure, dead links, etc… People don’t like to get lost, it makes them feel stupid. And people don’t like to feel stupid.
Not having your own domain name. People don’t really care if you are using your ISP⁸‘s allocated personal home page to store all your data, but they will not like it when they will have to change all their bookmarks because you had to move to another ISP for whatever personal reason. As a bonus you get a fix email address that people can still use to contact you year after year after year. Think about it.
Not being reactive to people’s requests, being hard to locate, not answering mails, letting your server down for two days in a row without providing explanations, etc…

Defence-Force.org

In the case of Defence-Force, I decided early to have my own domain name. Since then I changed the web-host for the main website three times, I changed ISP at least 7 times, and I moved to another country, all that without anyone actually noticing any change.

The main website is now at Yahoo!, which has provided a pretty good service for the price for now quite a number of years⁹. I’m using a fraction of the possible bandwidth, and I have still some gigabyte free on the allocated storage. This main hosting is also used for the sub-domains pointing to the games Space:1999, Stormlord and 1337.

The secondary server is at home, on a small Linux machine called Miniserve 2.0¹⁰. It is responsible for hosting the IRC bot/logger and the Subversion repository. The machine is accessed under the name miniserve.defence-force.org using a dyndns daemon and a sub-domain redirect from the main hosting. Of course this works because I have a decent cable line with enough upload capacity to not impact users of the server when I’m using my internet line for my own usage.

Everything is backed up on-line using JungleDisk

How much does that cost?

Well, I guess it depends of how you do it. I’m one of these persons who think you get what you pay for. If you use a free hosting, or if you share a server with some other people, you have to expect issues, and you have to accept these because after all you are using a free service.

In my case the grand total (roughly using current prices rounded) would be something like that:

Domain name, 13 years at $10 per year = $130 (Originally things where way more expensive…)
Web hosting, 13 years of 12 month at $10 per month = $1560
Miniserve 1, about $1000
Miniserve 2, about $1000

So that’s a grand total of $3690 for 13 years, which gives us $283 per year, or $23 per month. That’s more expensive than if it was for free, but I believe it’s stays in the acceptable range considering the amount of control I have on all the parts of the infrastructure.

Have fun, and good luck with your own project!

A new Oric game from Chema, based on Elite but with quite many improvements. ↩
The source code of most of our games is available, so technically they could be ported to other machines by anyone who feels doing it. ↩
September 18, 2010 ↩
I had to moderate TWO messages since I opened the forum in January 2006 ↩
A Software Development Kit specially targeted to the Oric, providing a C compiler, 6502 assembler, BASIC converter, tools to convert pictures, compress files, create virtual tapes and floppy images, and some sample code to get started fast. ↩
I guess a small PayPal button is acceptable, but don’t expect it to pay for your internet line and hosting costs ↩
I’m assuming you are not starting something totally new that nobody knew about before you started it ↩
Internet Service Provider ↩
Except in the two last weeks where the site has been down at least three times without being informed before hand, which made me pretty annoyed. Note for you Yahoo!, you better find what the problem is else I’m going somewhere else. ↩
Miniserve 1.0 died after one year, the new machine seems to work a lot better. ↩

Hacking confessions (part three)

05 Sep 2010 at 12:41

Dbug

computers

3 Comments

First a disclaimer: What I’m writing here now is related to events that happened a long time ago, at a period of time when it was relatively safe to do it, the worse that could happen would have been the expulsion from the school. If you try to do this kind of thing now, you may very well do some jail time and have heavy fines to pay as well.

The beginning of the end

If I’m not mistaken, this article will be the last in the confessions category. After that I stopped experimenting, mostly because I had no good reason for doing them anymore.

Anyway, last time I promised you how I managed to collect the login and passwords of everybody, well that was not a very complicated thing to do, it has been done numerous time, and it will still be done for a while because the principle of showing some interface where you are requested to use a keyboard to enter characters is a very flawed security method.

Some people claim that most of the methods that can be used to collect credentials are based on the fact you had at some point a physical access to the machine used to enter the password, and that it should not really scare you. Well, most of the piracy attempts are done by disgruntled employees, so you should really be scared.

Nowadays people use the fishing method: You receive a mail claiming that there’s some issue on your account (bank account, social network profile, etc …) with a link to a fake login page where people graciously provide all their login information. Now of course your mail client or browser tries to prevent this kind of thing, but there are so many ways you can trick the system that the best security is to not click on a link anyway, and use the official website/login information you were provided with when you created your account in first place.

But that’s now, what I did was in 1991.

Quick Basic, here we are

In the previous part I wrote about all these little programs I wrote to simulate DOS commands. What I did not explain is how I wrote them: Well, it was not very high tech, it was all written in Quick Basic.

QuickBASIC 4.5 Welcome screen

Some people strangely assume that for doing hacking or cracking you need to be a machine code wizard, or a C expert. Well, that’s not true: All you need is something that is good enough to get the job done, and Quick Basic had the advantage of being super easy to learn, and the compiler generated a code good enough for my needs.

I did the same thing for collecting the passwords.

To connect to the Unix server we had to run some commands from the DOS interpreter to bring the AIX login screen. This was a simple text mode application displaying a ugly AIX ascii logo, and then you were asked for your login and password.

What I did was a simple capture of the display, then wrote in Basic a fac simile of this screen. All it would do is to record what was entered in a text file somewhere in my hidden folder, and then pretend to crash by displaying some random hardware error message before disabling itself and quitting.

On the second attempt the real AIX command would run and authenticate the user correctly, so most people would not even signal the issue, assuming it was just a normal hiccup.

With this I was able to collect about every single student’s login and password on the first week at school.

Collecting the root password was a bit more complicated, because the teacher never used the dos machines, she was directly working on the Unix server – the one I wanted to get access to.

Well, all I had to do was a small variant of the program, instead of crashing on any attempt at connecting, it would display an “authentication failure” when my own login and passwords would be entered, and then crash if an attempt at login as “root” would be done.

From this point that was easy: I told my teacher I could not log on the system. She came checking what happened, I showed her what happened when I entered my credentials, she checked on the unix server that I could actually log correctly, then came back to the pc to see if something was wrong by entering the root credentials: Congratulation, you just got owned.

The morality of all that is that you can harden all you want in the operating systems, you can fix as many security breaches you want, the weakest point is still the human factor. If there is a way something can be exploited, it will be.

What about today?

It’s supposed to be more difficult today to do this kind of stunt, because some security measures have been adopted.

At least that’s the theory.

Do you know how you can recognize the real Windows login page from a fake one? That’s easy, all you need is to try to press CTRL+ALT+DELETE: If you see the Windows Task Manager appear, then it means you are not on the real Windows login page, you are just on an application that pretend to be it.

Of course, like me I’m sure you ALWAYS try to press these keys when you log on the machine, when you wake up the computer from the screen saver, etc… well I don’t. I should but I don’t.

How difficult would it be for me to check which screen-saver my colleague at work is using, and while he is gone for the lunch break to just install a small program that just runs the screen saver and when the mouse is moved show a fake authentication screen?

That would be trivial.

Gericom Webgine 1400

Asus eeePC 901

Possible options

Tablets with keyboards?

Android netbook

After market

Post-upgrade feeling

State of Android

Home-automation reloaded

Universal remote controls

My control panel

The actual code

Keen support

The next step

Inspiration

The home-automation concept

The motivation

Json and the metronautes

The complete code

Stamp reuse

Booby-trapped floppies

The usual suspects

Logs, the basic concept

The devil is in the implementation details

Thread safety

Performance and synchronization

A stupid example

The solution

The state of the art

Trouble in store

TOR and Freenet will not help

Going LoFi

Going WiFi

Cooking

Titan Quest

Software Development Methods and Start-ups

Proving myself wrong

Flashback

More of the same?

Bugs, did you say bugs?

Final words

Foreword

What is Defence-Force

The human factor

Making it happen

Putting it all together

Defence-Force.org

The beginning of the end

Quick Basic, here we are

What about today?

Categories

Archives

Tags