Why open-source : The LogBook Story

About 4 months back, the Flash team at Comcast Interactive Media released LogBook, our first official open source project. This blog post is a retrospective on the experience. The main reason I am writing this entry is to show that open source serves more than just the developer ego, and a company that allows open source of projects actually benefits from such projects. Hopefully someone can use this entry if they ever need to convince someone on open source projects.

Preamble
As I had mentioned in my pre-launch post, LogBook started as an internal tool that we wrote to debug our Flex applications. Pretty soon this had become an essential part of the application development workflow. We realized there was a real need for a tool like this in the development community and so brought up the idea of open sourcing the application with the development managers and the VP of Engineering. The developer group at CIM had already been participating in the open source community, releasing updates to some Java projects that power our sites, however we had no experience with starting a new open source project. But given our history with open source, selling the idea of open sourcing was a lot easier than you would imagine (special thanks to the legal team at Comcast who got all the legal stuff done so fast, unlike a lot of the stories you hear about projects getting bogged down with legalities)

LogBook was an ideal candidate to get our feet wet with the process of open source projects. The project’s codebase was small (about 15 classes when we started it) was almost completely separated from any other project we worked on. We also knew that the project was not something that grow faster than we were ready for: not a lot of developers enable any kind of logging in their applications and even fewer use the Flex logging framework which is what we released the first build for, but the small scope of the project was by design, being our first open source project, there was a learning curve for us too (I have released quite a bit of code on this blog over the last couple of years but managing a project is a lot different).

With all groups signed off, LogBook was released on Feb 12th, 2008 on Google Code. The project was released under the MIT license. We also started a google group to discuss the evolution of the project. We also made an effort to keep project related conversation in the open, so email exchanges on the LogBook project even within the CIM team happened via the group mail address. Google analytics were also enabled.

LogBook Today

Four months since the project’s release, here is the project’s status:

LogBookStats

Based on the numbers here, with 486 visits to the downloads page and 804 to the main page (which has the direct download link), I would imagine the project has been downloaded around 500 times, which is really cool. The source code itself has been checked out about 216 times. Not bad for what started as a quick debug app between 6 developers ;).

So Why Open Source?
There are a lot of books out there that talk about the advantage of open source projects, and there is no dearth of open source success stories (Apache, Eclipse, etc). However while these examples are great, in our case, LogBook would always be a small project with a few commits every now and then. So why should a company take on the headache of opening up the source code ? Below is a list of the not-so-obvious advantages of releasing code under open source licenses.

1) Much better code quality:
It was really surprising how much better the LogBook code is today than it was when we decided to open source it. A lot of us had put a few snippets of code here and there when it was an internal tool. But when we decided to open the code base, members of the team decided the code was not good enough to open source (Go developer pride 😉 !). Same goes for the user interface as you can see below:

LogBook pre-release:
LogBook prerelease

LogBook post-release:
LogBook post-release

Thanks to the open source project, LogBook is a lot more pleasurable to work.

2) Bug fixes = education
Once bug reports started coming in from other developers, we took some time to review those and see if when we could finish them. But the bugs taught us things that we didn’t know. For example, there was a bug that talked about LogBook hanging when a you tried to clear more than 100 logs from the list. The code looked fine, we were using ArrayCollection.removeAll() so we were baffled for a while. When we replaced that with ArrayCollection.source = new Array(), the performance went back to normal. Details of the bug/fix can be found here but this situation was never discovered internally. Had we not open-sourced LogBook we could have used removeAll() in code in one of our production projects, where CIM might actually lose money/audience if the application hung or something. We have fixed quite a few such bugs and a lot of them have taught us new things. Its almost like having a group of technically savvy QA engineers for free ;).

3) Bigger developer pool
The main reason users give for open sourcing code is to have tap a bigger developer pool than if the project is maintained internally. Our biggest non-Flash-Team contributor was Mat Schaffer, a colleague at CIM . While Mat is not part of the Flash team, he has been interested in Flex and this project let him contribute patches to the LogBook project. The collaboration also taught me more than a couple of things in managing project updates / dependencies, domains that are a lot more mature in the Java world which is Mat’s domain at CIM. For example, I have now started using patches for atomic updates to the LogBook code, which is so much better than running through a number of files trying to update code.

4) Open code + good ideas = easier collaboration
The Flash team gets to work with teams from a lot of other groups / companies and a lot of our projects are around integration. With LogBook, we actually started collaborating better. Once LogBook was out, we actually had a project where we sent the developers from another group the link to LogBook and asked them if they would consider using it for logging events from a black box solution we were getting from them. Since LogBook also scratched their itch as well (they needed to monitor how their code was behaving at runtime too), and since its so easy to integrate LogBook based logging, at the end of the day both teams agreed to logging certain events that either of them could see. Since LogBook was not an internal project, they did not have any problem with integrating code that did not add any dependency on a library owned by us. As we open-source more code, it would be great if other teams we work with use that and contribute to that rather than everyone build their own version of the same functionality.

5) Potential New Hire Litmus Test
Another interesting aspect has been the ability to point potential new hires to code in the LogBook project and see how they understand that. Candidates can also be asked to look at and discuss an open bug on the project, since a lot of our day-to-day work is around bug fixes, its a stronger litmus test than a theoretical algorithm kind of question at an interview. Since LogBook represents the way the team at CIM codes, the candidate’s understanding of the code is a good metric to how well he’ll perform with the rest of the code that is internal to CIM. And if the new hire is not up to snuff, we haven’t shown him any CIM propriety code. We haven’t done this yet but its definitely something we want to try going forward.

The Future of LogBook
LogBook as a project is still going strong. A few months after we released LogBook, we also added the CIM Logging framework to the LogBook project. This allowed pure AS3 projects to use LogBook. We also recently launched a hosted version of LogBook, which is exactly the same as the installed version but doesn’t require the application to be installed.
We are also planning to release more code under the MIT license, and a lot of this will be used in the next version of LogBook. The next feature on the roadmap is User preferences with the ability to save things like recently used log keys so that you dont have to remember them (issue #7 in the issues page). The team tries to implement features and bugfixes as we get time as per our needs, so some features may take longer to implement than others but if you want to join the project and help us steer this along, do join the user group and make some noise ;).

p.s:
And for all you Java/OSGI developers out there, also check out CIM’s cimspringfield open source project. I won’t attempt explaining it cos, well …I cant 😉

My ScrollImage component on Aral’s GAE SWF project :)

So as most of you guys may know, Aral Balkan, one of the biggest names in Flash development, has been working for a while on a reference Google App Engine application called GAE SWF. In the last build, version 1.53, he has included a component that lets users browse and upload an image to their profile page. The user also gets the opportunity to scale and crop the image before he assigns it to his profile. The exciting part is Aral is using my ScrollImage component that he has extended, to handle the scroll, rotate and crop functionality. Aral has also released the updated source code under the MIT license as well, which is the same license that the original code was released in.

I am really excited the ScrollImage is finally out in the wild. I originally wrote it as part of a UI requirement on a pet project that I dropped later so there was no reason to take the component to completion. However now that its out there, I am thinking spending some time adding some configuration options so that its complete (in my book) may not be a bad idea. I will also put the final source code on google code or something where it actually maintains a version history.

In the meanwhile, check out the GAE SWF project here and Aral’s updated code here.

On Flash/Flex build systems and application versioning

The last couple of weeks I have been caught in a couple of conversations around application versioning and build systems for Flash and Flex. Once for example, the context was a list of bugs from a QA build of a Flex project that we could not reproduce, and were not sure of the version of the swf that QA was seeing. We had checked in our latest swf that was part of a Java war that was to be deployed to the site, but there was no way to confirm that the version in QA was the one we had checked in. Another time, we were fixing up a swf that actually did have a version string that was integrated into the context-menu of the swf. However, this version was not tied to any particular logic and the developer was free to increment/decrement versions as he saw fit. And of course there are purists who feel strongly that we should have a build system because all the other “serious developers” have it.

FAIRLY OBVIOUS WARNING: A lot of this post is fairly obvious if you have worked on big projects that have a life of more than 1 or 2 releases (more so if you come from a language with an already established build tools like Java or C#). However this doesn’t seem to be the case with a lot of Flash and Flex developers who are often called to do quick projects. This is more for teams like mine where we work on diverse projects are often jump from one to another. Sometimes we have to work on stuff we did a while back and now there is one new feature thats needed but in the meanwhile dependent libraries have changed.

Because of the lack of any robust, easy-to-setup build and unit testing systems, they have traditionally been considered nice-to-haves but not really available if you have a deadline looming. Ant and FlexUnit are definitely available but why isnt FlexUnit integrated into FlexBuilder with one click “Run Tests” button ? For example, here is an image of JUnit as integrated into Eclipse:

This may be changing slowly though. I think Flex Builder now comes with a template ant build setup in one of the install directories, but its still upto you to set it up on your environment.

So here are a list of issues I deal with on a fairly regular way. I have a few solutions but if anyone has other ideas, please leave me a comment.

1) Identifying a live application:
When you are evolving an application or have QA builds going out fairly regularly, there is a big need to be able to verify what build is in front of the end user / QA engineer. For example, we have had caching issues when even after we have deployed a swf, a few users still see an old version till their browser expires it. So if I get a bug filed on my application, I want to make sure of the version its being filed against. Context menu integrated version strings seem to do this job pretty well, and its something I feel all applications should have. On a recent project this saved quite a bit of time since some network issues were preventing the latest build from actually reaching a test server and it was really easy to verify the build was wrong. One technique we adopted was an idea taken from web services (most webservices are accessed using a url that includes a version, for example facebook’s apis are accessed via a url like http://api.facebook.com/1.0/). So using the same idea, we now compile so that output swf’s name has a version string appended to it. For example we may have player_1.2.3.swf (or release swfs to folders like …/v1_2_3/player.swf). This lets us check version via html and also keeps a history of previous swfs in case we have to revert to one in an emergency, or check if a particular issue in the current live build existed in the previous build as well.

There is a blog post on the SVN based version context menus here

Whats interesting is I have such an easier process when I release code on this blog. With a simple “enable view source” option, All the code gets packed into a downloadable zip file and the context menu item is appended to with the view source option that takes me immediately to the source of the current swf. I wish this concept could be extending to target svn repositories and app versions.

2) Getting back to the source of the code from the swf
The preceding conversation immediately brings to light another requirement: Once I do know what version a bug is on, I should be able to check the code out for that version and build it. One thing I am trying to prevent is a swf that may go into production built of code on a developer’s machine that he may then forget to check in. Ideally the application version should be tied to your source code repository in some way. Although I am nowhere near this yet, ideally, the action of compiling your swf should trigger a checkout from the repository first and the swf should be built of THAT code. And in a perfect world, I would live the SVN revision number to be available in the application version.

This problem becomes a lot more tricky when you may have different versions of an application running in different places. If you are building a generic swf for example (like a photo browser application or a video player that may love on different environments). How do you know which version of the swf is running if the last time you deployed to that environment was like 8 months ago.

3) Controlled Refactoring
Honestly speaking, historically, build systems were always considered nice-to-have for our projects. Developers on a project have a pretty good idea how to get their application to build by hitting the compile icon on Flex Builder. But that breaks down if you have ANY dependency on any other library. For example, there are more than a couple of libraries that we have common among different applications. However if anything needs to be changed on a class in one such library, after the change is made, the developer making the change must ensure that the change didnt break anything on any of the other applications that use that as a dependency. To do that within the context of Flex Builder is virtually impossible, would you open every project and compile it ? At this point it becomes imperative that all your dependent applications have tests that can be triggered with one click when a core class is changed.

The need for Release Engineers
Our other projects have release engineers whose main job among others is to check out code from the repository compile it and then deploy it to the test environment. Here is a quick snippet from Wikipedia:

The responsibility for creating and applying a version numbering scheme into software–and tracking that number back to the specific source files to which it applies–often falls onto the release engineer. Producing or improving automation in software production is usually a goal of the release engineer. Gathering, tracking, and supplying all the tools that are required to develop and build a particular piece of software may be a release engineering task, in order to reliably reproduce or maintain software years after its initial release to customers.

I encourage you to read the entire article here.

These are a few thoughts and ideas I have right now. In the last few weeks we have made a lot of progress towards a mature build and deploy process for Flash based applications. In the meanwhile if any of you readers have a good release system built and that you use regularly, please drop me a comment here and I would love to have a conversation with you on your system. I know there are a few interesting projects out there like antennae, but if there is anything else anyone can recommend, please let me know.

DiggGraphr Lives Again

A while back I had released DiggGraphr, an application that used Digg’s api’s to render the stories on a TreeMap. Then I got caught up with the Adobe AIR mania and the next few versions of DiggGraphr were targeted to AIR rather than the web app. I have to admit that was a bad idea, since the AIR version brought no new functionality to the application that couldn’t have been done on the web. Ah well, it still got me some attention from Techcrunch and I also won a copy of Adobe Creative Suite in a contest that only accepted AIR apps.

In the meanwhile, the web app died online. I think some API changed just slightly, but that wasn’t really surprising. All code left to itself tends to deteriorate as anyone trying to compile code a few months old will tell you. However a few people have asked me about it and I figured I take a few mins and fix the app. Also reduced the crazy red borders that were definitely distracting.

So check out the web app here. If you havent Dugg it yet, do show some Digg love 😉 (the Digg button is right there).
I have no plans to release an AIR version of DiggGraphr, unless someone asks me to.

Let me know what you think.

LogBook, now for pure Actionscript 3 projects

A while back the team at Comcast Interactive Media released LogBook, an AIR based application for logging messages from Flex applications. Today we have released an new swc that enables LogBook based logging for pure AS3 projects. The new swc is based on the CIM logging framework which is very similar to the one in Flex, but with no Flex dependencies. We have also released the asdocs for LogBook and the CIM Logging framework that can be found here. I have a more detailed entry on the CIM flash team blog here.

100 Posts! A retrospective and the best links from my blog.

TADAAAAA ! This post is the 100th blog post on my blog. Its been a fun ride so I figured I wrote a bit on the history of this blog as well as link to my favorite posts here.

Winning the People's choice award at Flash Forward, Austin, 2006
Part of the Comcast Interactive Media Flash Team at Flash Forward, Austin, 2006. The Fan 3.0 won the People’s Choice Award (which is voted for by people on the www and not a panel of judges). A lot has changed since then: The team has changed, Fan 4 is a 100% Flex app and I have longer hair ;).
My first post here was on July 19, 2006. That said, this was my second attempt at blogging, the first attempt was something on blogger, where I posted my opinions on Flash, but I got bored of that real quick as I hardly got any traffic. I started Code Zen with the basic assumption that I will get no traffic at all. I basically wanted a place to keep all the code I worked on and get back to it months later. This was around the time Flex 2 was coming out of beta, and I needed a place to host FlexAmp, my entry to the Flex developer contest (FlexAmp was featured on labs.adobe.com for quite a while and is still featured on the community sample apps section on Adobe.com).Of course once in a while the entries here would be purely opinion, like defending Flex when someone took cheap shots at it, but for the most part, I used this blog to document Flex techniques and gotchas or full blown components (list at the end of this). Some of the examples have even inspired greater work within the community. For example, SuperTabNavigator, one of the most used components on FlexLib, the open source Flex library that seems to be the one with the most traction right now was developed by Doug McCune on top example code on this blog (He of course packaged it into a complete component from the proof of concept I had here).

One of the most memorable moments was getting on the main page of Digg. I had been working on an implementation of a Treemap algorithm, and just to test it out, I used Digg api’s (undocumented at the time) to render the stories in the treemap. I posted the result here and the next morning I had had 904 diggs and comments by some big names including Kevin Lynch, CTO of Adobe and some folk from within Digg itself. The link to the now busted application is here and the Digg page is here. I later repackaged the DiggGraphr application as an AIR application and entered it for the Digg Api contest. I didnt win but I was in the final 10 so I got some Digg T shirts and stuff. The air application was even featured on TechCrunch, one of my favorite blogs.

Around a year or so ago, we adopted Flex as a framework here at Comcast Interactive Media for a lot of our applications. This was definitely big since Flex 2 had just been launched, but I had been working with it since Beta 1. So when I architected and developed the forth version of Comcast Interactive Media’s flagship video portal-esque application in Flex, I blogged about it here. That entry got picked up by some high profile blog as well, including Mashable, ZDNet and Cable 360. That entry also allowed me to strengthen the case for developer blogging within Comcast. That has come to fruition with the recently launched developer blogs at CIM including the Flash/Flex team’s blog here.

Continuing on the cultural revolution within CIM, we recently even started releasing code under some of the most permissive open source licenses. A few months back we released LogBook, an AIR application for logging events from Flex applications under the MIT license. The launch was announced on this blog as well. Ryan Stewart from Adobe blogged about it as well, which was kinda cool :).

Like I said, its been a fun 100, even if it did take me a year and a half ! I am still as passionate about Flash and Flex today as I was earlier if not more. My life of course has changed a bit in the meanwhile, so that may start getting reflected in the content here. I am getting really interested in Ruby and a little intrigued by Processing. Meanwhile, at CIM, my role is changing as I move towards a principal engineering position. That means I spend a lot more time coding these days which is cool ;).

Here is a list of some downloadable code from here:

ScrollImage Flex component: An implementation of ScrollPane for Flex that isn’t based on the Canvas container, making it a lot lighter and efficient.
ImageSlicer: A class for cutting any DisplayObject into a grid and grabbing slices out of it.
MP3Playback Package: A pure AS3 package for playing MP3 sound.: A class for loading content into Flash from WordPress based on XML-RPC.
AS3 WordPress class
TabNavigator with closable/draggable tabs: The FlexLib container has more functionality but this has an explanation and can be extended arbitrarily.

Fixed Treemap code coming soon. I released it earlier but its broke again, so I’ll release it once its fixed.

Thanks all for visiting :).

Flex development within teams: Using patches to update projects

Working in teams on Flex projects requires some discipline, something we at CIM are learning fast as our projects grow beyond individual developers. We have always used version control but traditionally, we have worked with open libraries that all developers had access to. Anyone could commit to the repositories and everyone checked out the latest code the first thing when they began work in the morning. As more and more developers checked code into our core library, the chances of it being in flux escalated. One idea I am now implementing is the concept of release management. Working with patches is one part of this. This idea is being used, for example, in the LogBook open source application that we released earlier last month (and was blogged about by Ryan Stewart as well).

So here is the situation: At no time should we allow untested code to enter LogBook. However, individual projects have features that they need to be built into LogBook. These features need to be committed back into the LogBook repository. Right now, to maintain code quality, I am the only developer that has commit access to LogBook. Developers can write their extensions to LogBook, usually after a conversation we have about the best way to implement this (the conversations all happen on the cimlogbook google group to maintain transparency). Once they have code working, they send me a patch file which I ensure works and doesn’t include hacks for the sake of a feature. Once the patch is confirmed to be good, I commit the patch to the repository.

Working with patches is a new thing for me and I just wanted to blog about this quick. Working within Flex Builder 3 and with the Subclipse plugin installed, creating patches is as easy as right clicking on the source project and selecting Team > Create Patch … .This creates a patch file that basically documents the difference between the committed project and the user’s local files. The developer then sends me the patch file.
Patch Menu on Subclipse

There is a bug in Subclipse that puts absolute paths to local files rather than relative paths and since a developers local file paths may not be the same as mine, we cannot use Subclipse’s apply patch feature out of the box. To apply the patch, making sure I have no local changes in the project, I navigate to the project within my filesystem using the terminal. Once there, I apply the patch by typing

patch -pNumber-of-directories-to-get-to-project < path-to-patch-file

the -pNumber-of-directories-to-get-to-project parameter is basically to remove the folder references to the project directory in the patch file. So for example, if you open a patch file and the first line references the project root as

/Users/comcast/Documents/workspace/cimlogbook/src/LogBook.mxml

you would use -p6 . This removes the /Users/comcast/Documents/workspace/ reference. Once the patch is applied, refresh your project within Flex Builder and you will get the uncommitted changes icon on the changed files/folders. You can synchronize with the repository to see what changed, build the application to make sure it compiles, and then if everything looks good, commit the code to the version control system.

Special thanks to Mat Schaffer for getting me working with patches. I really like this way of committing code since it makes me feel a lot more confident around the libraries I am supposed to be responsible for.

UPDATE:
Another useful tip: Use SVN to generate change logs that can then be published, example:

svn log --revision 217:239 -v

Can PicLens be done in just Flash and Papervision3d ? Here ya go !

PicLens has been one of the new interfaces that has wowed quite a bit of people I know (including myself). But the application is written in native code and needs to be installed on your computer as a plugin to be usable. Off late I have been playing with Papervision3D a bit and I figured it was possible to kind of get the functionality just by using that. Here is the result of my experiments so far. (warning: this demo requires a good internet connection since I am loading a lot of images)

Update: Uploaded a video for users on poorer connections:

Click on the input text field, enter a search term and hit enter:

The code is built on top of the Paged – data classes I had talked about in my last post. So basically new photos are loaded from Flickr as you reach the edge of the currently loaded data set. The code is pretty awful right now since I had to revisit my basic algorithm a couple of times so if you scroll around quite a bit, it may begin to spike your memory or act all weird. I need to implement some recycling of renderers strategy which seems to be advised strongly for such applications. Also I am not completely happy with the tilting algorithm I have right now (to get it really right I need to change the algorithm to do a little more math, right now the transition from tilt to normal is a little jarring). Ah well, that happens later. Right now, I am just stoked this works :).

Photowall: Scroll based data prefetch

Photowall is part 1 of a secret project I am working on (secret only since I am not sure part 2 will work, so I’d rather not declare yet what its about 😉 ). This component on its own is an exercise in data-prefetch. The idea is to load data as the user scrolls to the end of the current set rather than ask the user to click on a “more photos” link. In this case, the component is feeding off Flickr data.

Type in a search word inside the text input and hit enter. Once the first set of data has loaded in, use the scrubber below the photos to scrub through the photos. Note: Data prefetch is triggered on mouse-up rather than mouse-move, I need to implement that in the final version.

/blog/swf/Photowall.swf

Under the hood, I have a class called PagedDataService that computes which page of the remote data the user is on (Flickr apis support paged data, with each call out asking for a certain number of photos and a page offset, so page 1 has 0 to 1 x pagesize photos, page has 2 x pagesize, etc. Right now my pagesize is 60. ). The code also uses a PagePolicy class that the container looks at to see when the more data should be fetched. For example, in this case, I have asked the container to get more photos when the user is about 18 photos (6 columns) away from the last photo in the set.

There is no optimization of the code here yet, so I am not managing the photo renderers that aren’t visible to the user (which would free up memory). Another optimization would be to reuse renderers rather than create new ones like I do for every photo. Finally, a priority queue for the load requests would be great here.

So much for part 1, on to part 2. Stay tuned :).

LogBook 1.2.1 available for download.

LogBook 1.2.1 patch release is now available on googlecode now. The release addresses quite a few bugs and feature requests. The list of patches can be found here.
We are also discussing features for LogBook 1.3 at the Google group for the project. If you have a feature you would like to see added or contribute, join the conversation here. Like Mat said in a previous thread there, the importance of a feature is only as much as the conversation around it.

There are some interesting discussions I have had around LogBook with a couple of folk within Comcast Interactive Media where I work. For example, one of the things we have had to deal with is QA bugs that we cannot reproduce. Running LogBook in the background on QA machines and looking at the logs generated would be an interesting idea to implement, thus making LogBook a part of the QA cycle. The biggest thing that needs is a notion of a session in LogBook which is something we dont have right now. I guess that could be a part of 1.3 or 1.4. Other ideas are welcome.

If you have any code around debugging Flex applications that you would like to contribute to LogBook, do let me know as well.

Links:
Download LogBook 1.2.1
LogBook on Googlecode