At FOSDEM ‘09 I interviewed Rob Savoye (homepage, Wikipedia), primary developer on Gnash (homepage, Wikipedia). He also runs a non profit, Open Media Now.

We had a chat about reverse engineering legalities and some decent metal.

The audio (MP3, OGG, FLAC) and text of this interview is licensed under a Creative Commons Attribution-NonCommercial License (CC-BY-NC).

Transcription was provided by Niall Campbell.

Rob Savoye: So basically I started up a non-profit last year called Open Media Now and we bascially fund raise for a bunch of projects that use patent-free codecs like Gnash flash player also streams Theora and Ogg Vorbis and things like that. I just wrote a clone of the Adobe media server, we do a bunch of stuff like that, and we’re doing a lot of work around legal issues fall from that and codecs and stuff.

David Dolphin: Cool, to go into the reverse engineering a bit more, how did you get into reverse engineering, was it purely you had an itch you wanted to scratch or did you build up to it?

RS: Actually I kind of fell into it backwards, I had a tendancy to work at small startups that had insufficient staff for a lot of things and often cut too many corners, and one person debugging another persons hardware or software could be reverse engineering. I think I kind of got into this stuff literally as a survival skill cos I was always like the systems guy, you’d support packages and you’d find bugs in drivers and you’d have to figure out how it really worked you know, and sometimes observation is better than reality and junk like that. And then I wound up working on the GNU debugger, GDB, I had like the world record in GDB back-ends, and pretty much designer two of the three remote debugging protocols and all that, so I spent a lot more time debugging hardware and weird protocols, I mean, the current one I talked about yesterday is almost my two dozenth, I’m not even sure anymore. I just think it’s really fun, so, and it conveniently turns other people need reverse engineering so I’ve done a lot of reverse engineering for people on contract and it’s kind of amusing.

DD: Is that in terms of community work or commercial work?

RS: Commercial work actually, I used to work for NASA, so I used do to reverse engineering for NASA funnily enough, y’know, taking old code and porting it to Linux, and basically spent a couple of years porting everything they had to Linux for them, and in the process had to reverse engineer a lot of weird stuff like you’re porting some network code and there’s no byte swapping in it because it was running on a big-endian machine so porting it to Linux meant like lots of byte-swapping and data structures that, the documentation was 30 years old, and whoever wrote it is like, y’know, tired, or dead from old age and I spent all this time reverse engineering a lot of really obscure network protocols.

DD: Speaking of reverse engineering I remember hearing a story that Alan Cox, who worked on the Linux Kernel had, where he wanted to get information on a certain bit of code and he was told to talk to his manager, to talk to some other manager, to talk to a global manager, down the chain, and he said ‘Nah, that’s going to take me two days.’ and he rang the guy up in Australia and got an answer in five minutes, and was almost fired, because he didn’t observe and do you see that aswell, that you have you follow procedures and does that bog you down?

RS: Well I don’t work for a big company, so I don’t really have that kind of problem I guess, basically you know, I’m self employed doing whatever the hell I can raise money for but at the same time I mean Alan did is to be honest how you’re supposed to solve your problems, I mean sometimes reverse engineering is like, they talk about hacking, well sometimes it’s more social engineering than it’s actually technical, and often that’s the beauty to be honest these days, of the sort of development community and international conversations and email and all that is you can often talk to the right person who can give you something in five minutes so in a sense blowing off too much process is almost usually a good thing.

DD: Cool, so you’ve worked on the Adobe stuff, have Adobe seen what you’ve done? Have they taken into account, have you been contacted?

RS: I’ve actually been to Adobe. When I first started the Gnash project a bunch of years ago they had concerns, at the same time this was before Adobe had really gotten into open source and so we basically went out there with kind of a dual-pronged focus I guess, one they wanted to ask me some questions and figure out if basically I’d been doing illegal reverse engineering which I was, and it was pretty easy to prove more or less, and at the same time they were kind of curious what it would take to be an active participant in the open source world and much of the problems with flash and the legal licensing they inherited when they bought it from Macromedia. I mean, Adobe used to be customers of mine when I was at Cygnus Support and they’re usually a pretty decent company I mean PDF is open, postscript is open, and part of my pitch to them a couple of years ago was like, just be like Adobe, just make all the specifications for free, I don’t even care if they release their flash player for free I wrote Gnash who cares about their code base, but I want specifications out there so I know what target I’m working to. They’ve been slowly getting around there but they wouldn’t reverse RTMP so I bascially had to reverse engineer that.

DD: Right, and is that mostly a profit thing for themselves? Do they sell servers which support RTMP and they don’t want…

RS: They make a huge amount of money on their servers. In a month I release my clone of the Adobe Media Server.

DD: Very good!

RS: I spent the last six months doing it, the whole reason I reverse engineered RTMP was actually to do this server side flash support since I have a flash virtual machine, as well as client side, and the idea is now we’re working on full flash-based video conferencing support and stuff which is very popular in the professional world, but we’ll be able to use patent free codecs because we’ll have both ends of the network connection. Part of my plan, especially with Open Media Now is to increase adoption of patent-free codecs by building more software infrastructure that actually uses them seamlessly so that you don’t really care.

DD: Of course. Yeah, if someone is creating a site such as Youtube, Google video, whatever, are there ways that they can write their code that it can work under Gnash and Adobe Flash Player?

RS: Oh yeah totally, we actually work a huge amount on compatibility. It’s kind of odd because none of us can run the Adobe tools because of licensing problems, so our sort of volunteer community, it’s kind of their wonderful contribution to Gnash is a lot of them can’t work on Gnash because they’ve signed the license agreement yet they can basically run both on their computer and often our bug reporters and people who hang in there for days on end asking them tedious questions, is incredibly useful I mean it’s one of the fun things about free software is developing in an open space where the community can see when you need help and like ‘oh yeah well the border of this line is three pixels on three sides and two pixels on one side’ and like ‘how do you tell the difference?’ and ‘oh, with a magnifying glass, they look different.’ We get a lot of really funny bug reports like that. So we actually work really hard on even bug for bug compatibility, I mean there are tons and tons of bugs in the Adobe Flash Player, I mean we have test cases that Gnash passes that will crash the Adobe Player left and right. They do a lot of really ugly stuff too that we actually have to emulate because we try to do it, uh, our joke is we’re not bascially cloning Adobe technology, we’re re-implementing your technology, better. I tell my engineers all the time, or guys working on our project, just cos it’s free software doesn’t mean we can’t do REALLY good software.

DD: It sounds a lot like IE6 CSS support type of stuff, like there’s some horrible hacks you have to do, in order to get stuff working properly in both.

RS: Yeah, so for instance you take somebody using a Flash 9 based media player, we do the youtube support fine, dailymotion, a bunch of different people, nowadays really the only trick with Gnash is testing with Gnash, and a lot of people don’t but if they test with Gnash we often find we get contacted by people that actually did the flash movie, it’s usually a one line change in their flash application to work correctly in Gnash and then we’ll usually fix the bug anyway.

DD: Are there things that you would say to Flash developers to avoid using, would you say ‘don’t do this it won’t work with ours yet‘?

RS: A lot of the really ugly stuff is below the level that when you’re programming in Flash and ActionScript you have any control over, some of the uglyness is really stupid, like how they do function calls, and how they lay parameters on the stack, a lot of obscure gunk and in flash it’s just really really ugly. I think the main thing I would say for developers is try stuff with Gnash, and if they find out that ti’s got a bug, we love bug reports from people that are actually flash developers. One of our biggest problems is we don’t have access to professional flash developers at all, they hate Gnash, they think we’re going to destroy the world. The HTML wars of the early 90s or whatever and ‘you got to do the compatible thing’, well no actually we implement the bugs in Adobe Flash player even when their stupid, we put warning messages, “don’t do this”

DD: Is it possible to interface Gnash with development tools, IDEs, is it Dreamweaver that’s used for…?

RS: It’s Flex, actually, we actually maintain the Ming project which is a flash compiler, typically the problem with Ming is it’s mostly original developed for websites so you actually do flash programming in more like C, PHP or C++ and stuff, and we would love to actually see a GUI for our compiler and if we had extra money we probably would hire somebody to do it but nobody’s pitched in the money or volunteered to do this for the hell of it and it’d be a great project really for someone into user interfaces, what we’d want to do is design something simple, a lot of the educational machines using Gnash because of licensing and we run on weird processers that Adobe doesn’t and junk and so we really see that there’s a big need for free flash believe it or not, for educational software. Gnash is really really popular in most of the big Linux distributions, in Spain and stuff, they always have client networks, and Adobe player hangs their entire network pretty often and it turns out that Gnash doesn’t.

DD: Moving away from Adobe a little bit, to reverse engineering in general, what are they big differences say between reverse engineering file formats, reverse engineering network protocols, are their major differences or is there collaboration?

RS: I think there’s huge differences, now I haven’t done file system reverse engineering, most of my background is networking stuff, I think the advantage of networking is that because it’s a network streaming type protocol usually, it’s more limited in the way that you could do it. You make it too complicated and your performance goes to hell basically. Most headers for network protocols are actually very short and simple, typically a handful of fields, some of the gunk I worked on at NASA was like 50 fields and a giant structure and pretty huge. But I actually had documentation and that kind of helped a little.even though the documentation was wrong, but you know, when you know what three quarters of it is, the rest will be easy to figure out. Far from that, look at ODF, OOXML and stuff, some of these things, while I know you could reverse engineer them from scratch, I think it would be much more of a bigger problem, much more complicated, I mean Open Office is still working on weird microsoft compatibility bugs and they are pretty good at it. There’s still a lot of weird stuff, I still get documents all the time in Open Office 3 from Word that don’t look quite right.

DD: For example, one that springs to mind that I’ve come up personally against is symbian phones can take an archive of the entire C: drive, however the archive that it’s in is not a documented format and they only extractor for it is on the phones and the problem being that you can’t unzip it on another phone so if you make an archive of your device you lose your device, you have all your backups but you can’t recover your data unless you get the exact same model again. So it would be ideal to take these archives and have some desktop application to just unzip them.

RS: Archive formats are usually pretty simple, I’ve written lots of packaging code over the years for all sorts of stupid operating systems and packaging schemes, and a lot of times just straight archive backup stuff is very simple, more or less a list of files and it’s usually compressed, and once you figure out what it’s compressed with it’s not that hard, I mean a lot of people are using Zip.

DD: Are there things that make it easy to get into reverse engineering? If someone was going for it what would you point them towards?

RS: Getting a good collection of good, loud music. When it gets really hard, sometimes you just have to turn up the volume. I think, funnily enough, you have to do it in a very low-distraction atmosphere, or distracting yourself in appropriate ways, sometimes when I turn up the music really loud it makes me really focus. It’s kind of hard to get into it’s like learning sculpting or something it can be disappointing and every time you make something you go ‘ugh, this is crap’ and reverse engineering is often that way. You get into something and you go, ‘ugh, I’ll never figure this out.’, that’s really common.

DD: Do you find there’s a fear of reverse engineering? It can be used for constructive things like yourself but also on the other side incredibly illicit things such as cracking applications. Is it viewed in a generally negative light?

RS: I think it slightly depends, most people assume that reverse engineering is violating the law, they don’t realise it’s entirely legal even in the US, and the trick is you have to more or less follow a set of rules, so to speak and to be honest some of those are kind of obscure and it’s not simple, like ‘don’t disassemble someone’s executable.’ that’s not good, that’s bascially what people cracking applications do, and to me there’s no sense of style in that anyway so why would you want to do it that way?

DD: You kind of find yourself tarred with the same brush though. Like the hacking community being seen as people to target banks.

RS: Most of the reverse engineering I’ve been doing lately has been in the flash player, so people actually appreciate it because it makes it work better and they know reverse engineering is the only way to get it there. Most of the people I’ve actually done programming consulting work for reverse engineering it’s bascially been the same way they’re just really glad to find someone willing to do this. It’s sounds funny but sometimes they say ‘you actually like this kind of work?’ and I charge a lot of money for it.

DD: What’s next for you? What’s next on the horizon?

RS: Well the big thing now is I’m about to release my Adobe Media Server project which has been almost a years worth of work, much of it is very slow reverse engineering in the beginning, down to basically writing a server, mimicing how their network works, complete with all the weird timing errors and packet fragmentation, basically getting all that stuff to work. That’s pretty fun it already speaks several network protocols, it does HTTP, it does RTPT is which ActionScript encoded objects using post in HTTP, which is basically what YouTube does and everything else, because it’s cheap, and also there’s RTMP which is the protocol that Adobe uses for their server which costs many many thousands of dollars.

DD: Very good, best of luck to you, hopefully we’ll see you again, are you going to be at any conferences in the next while?

RS: Actually I’ve been to so many conferences in the last few months I’m actually hoping not to go to some for a little while. I was just at LinuxConf and before that I was at Open Source world Spain, EduDebian, I’ve been moving around a little too much lately.

DD: Hopefully after some time off we’ll see you again. Thanks very much.

Tags: fosdem