Wednesday, August 29, 2012

Crashing an Armenian wedding

From Instagram

Armenia was a blank page in my head. I knew absolutely nothing about it. I didn't even know where it was on the map. I ended up crashing a wedding in Armenia with my friend's family. It was his cousin's wedding. The first two nights I was alone, I had to depart earlier than my friend and his family as it was the only way to secure a cheap ticket, I decided to go only 1 week before the wedding. My decision was a bit abrupt but that's typical me. I tend to just pick up and go, preferably with a large random order to it. There is so much to say about Armenia that I've broken down this post into a few sections to make the post tolerable to read and for readers to pick and choose what they want to read about.

Thanks Armenia, you rocked, literally !
From Instagram

Revisiting the Caucasus as a kernel hacker

From Instagram

I intend on visiting Armenia again some time. I learned quite a lot about Armenian and about the Caucasus but that left me wanting to learn more and experience more. I've been told a lot of great things about not only Armenia but also about other countries nearby. I'm left to think about the possibility of visiting Armenia again and next time climbing the rocks there, and with the idea of even extending this trip to explore Turkey, Iran and going to climb rocks there. As a hacker, I cannot also help but wonder as well about the Caucasus's own position on free software and Linux. Do we have active Free Software hackers? I know of Ozan Çağlayan's work from Turkey, but what else? Do we have any Linux kernel hackers out there? How about companies looking to fund this work? I need a good excuse to go climb rocks int he Caucasus :D

From Instagram

An Armenian wedding

From Instagram
And then the wedding. Wow. What an experience... I'm grateful to the family for having in the end invited this Costarican wedding crasher to experience such a magical personal and cultural moment. The wedding took place in a beautiful green area called Aghveran that showed me how different the landscape can be in Armenia. The ceremony took place in an Orthodox Christian Armenian church in a very small village.

From Instagram

The priest really looked scary, I was not used to this. He had a big hat and some really long white beard.

The car was pretty sweet. That's Vahe, the buddy I am in debt for having invited me to Armenia.

From Instagram

As the celebration in the church ends a local band awaits the bride and groom and folks start dancing.

From Instagram

The father of the bride then looks out and calls for the bride and starts the dance off.
From Instagram

The beautiful bridesmaids:

From Instagram

There then is a big dance on, a meet and greet and then we step to the celebration hall.

From Instagram
Thanks to Armenia and I wish the best of fortune to the bride and groom! Kenaced (Cheers in Armenian)

From Instagram

General tourist advice for Armenia

From Instagram

Yerevan sleeps late -- 11am openings, closes late -- 1am or 2am mobile cell phone service shops, 7am bars. Yerevan parties hard. If you visit Yerevan my preferred hot spot was Cheers Pub a dive bar pub with great music, service and people. I think I spent 4-5 nights at Cheers straight, and yes, I had tried other pubs
like Eden and other nightclubs... They were all "meh". Since Yerevan sleeps late you'll even see whole families walking around with strollers around midnight. At 7pm-11pm you'll see masses take on tons of coffee shops, something you just do not see and we do not have in the US.

From Instagram

Sorry pot heads, not that type of coffee shop. In fact herb is taboo in Yerevan although from what I was told about 80% of the folks likely do smoke it. Don't ask anyone for it, the penalties for it are stupid even tourists, and the typical life of a dealer consists of getting to jail in about 2 years. I'm a dumb tourist so I asked about it, just as I asked about other random stuff. Curiously enough though everyone is a cigarette smoke fiend... I'm not used to it. That made experiences like hanging out at Eden Pub in Yerevan a hazardous chemical experiment for my eyes. That also made much of the beautiful women less attractive to me. And yes, there are tons of gourgous women out there in Yerevan...

From Instagram

In San Francisco you start going out around 10:30pm and everything closes at 2am, in Yerevan you can start heading out at midnight and at what time you come back will vary, but 2am is when things will actually start to get fun. Odly enough Fridays were slow, the rest of the days were amazing but likely a more reasonable explanation is that when we went out to Cheers we brought the party with us. Always a mob of about at least 6-10 people, typically closing Cheers out maybe at 4-6am. It was great.

From Instagram

Touristy thingies I did. Jermuk. Jermuk was bad ass, specially if you like rock stuff. There is this one point a bit after you pass the big bridge towards the right, right before the main entrance to what I think is the waterfall view park entrance. There seems to be canyon there you can hike to... I wanted to go hike there and never come back. Maybe I will... some day.

The bads. Like in all of Europe, service sucks. Everywhere except hotels and Cheers bar.  I stayed at the Best Western Congress Hotel and everything there was amazing except that slow ancient elevator. Rudeness. Local Armenians can be rude, it does not matter if you are a tourist or not. Its just a backwards cultural values that have not moved forward yet. I'm not quite sure how other cultures have resolved this, is this a typical European issue as well? Cabbies will try to rip you off, just get estimates before you ride with them.

From Instagram
From Instagram

Food is amazing, but in my experience, look either for fancy restaurants in Yerevan or bet for the restaurants on the road when on trips on route to places in Jermuk. The best food I had was when we stopped by a restaurant right before entering Jermuk. There you not only get to try the good local fresh food but also great natural spring water.

From Instagram
From Instagram

The market in Armenia is interesting. You can find great small treats but also very interesting unexpected oddities like used medical operation equipment and a whole bunch of chemical lab components. My favorite though, of course, was the local hooker art.

From Instagram

One of my best memories about Yerevan though will be my great conversations with the local folks. Both local Armenians who have never left or those who are coming back from their original departures. I never had a dull conversation with any Armenian and the types of conversations I had were all extremely stimulating and very eye opening.

Running and rock climbing in Armenia

From Instagram

The first two nights that I spent alone in Armenia reminded me of how much I loved not knowing a single soul in a new place and the social challenge aspect of meeting new people. I think I'm addicted to the challenge because since I was young I was always on the up and go. New school, new
neighborhood, new country, new social environment, new culture. Fortunately it was smooth sailing for me, I think it helps that I'm from Costa Rica as that gives my introduction a fun unexpected kick. Where the fuck is Costa Rica?

I'm addicted to active living now and since I have a Tough Mudder coming up towards the end of September I had to bring my running shoes and train a bit. I cannot afford loosing my stamina. I'm also into rock climbing and based on a friend's advice I checked out what the possibilities would be.

Running in Yerevan, Armenia

From Instagram

I got up at 6am for my first run in Armenia. Running in a new city is one of the most rewarding things one can do. You get to explore the city fast, this will help you get around easier in your other trips, you can also easily pick up on certain cultural values and practices, which should help you adapt faster. I believe its possible that you also end up shaking off your jet lag faster given that you are accelerating your metabolism by increasing your oxygen intake at the times your body was previously was used to resting. I ran at 6am and then at 10am on first day. The second day I did a bit more lengthy run, ~6-7 miles and made it an objective to get to an iconic high point in the city, the Yerevan TV tower which turned out to be the highest point in the city. I even found one bus close to the Yerevan TV tower that looked like the bus from Into the Wild (2007).

From Instagram
From Instagram

Both runs allowed me to explore the city quickly in random areas that eventually ended up being recommended tourist destination spots that I was recommended to visit once the family arrived the following days. I had explained I thought I knew where they were talking about when they made recommendations to visit some city places but nevertheless I went again to explore such recommended sites just to confirm. Since I was running really early I was also able to observe easily and safely that Yerevan was a city that slept extremely late, was extremely secure, and when was the best time to avoid walking around due to temperature. Given the empirical gains anytime I visit a new city I will now run it or bike it as soon as I arrive.

From Instagram

Armenia happens to be part of what is called the Caucasus. One of the attributes of being part of the Caucasus is you get tons of water, to the extent you see public water faucets running on 24x7. The amount of faucets around the city makes it a great place to run in the city without the need for you to carry water. Good thing, as I didn't bring my CamelBak.

From Instagram

I only saw one young crowd running near Republic Square, but that's it. There were no other runners in Yerevan. In fact prior to running in Yerevan I was told I'd likely get looked at oddly if I did run in the city. I didn't personally feel awkward but a few old people certainly raised an eyebrow or two as I dashed by. Likely folks do not run as most folks are pretty fit in Yerevan, and also it just has not picked up as part of the culture. Nevertheless culture could be enhanced, I encourage more folks to take the streets and run and take high advantage of all that abundant succulent water.

Rock climbing in Armenia

From Instagram

Rock climbing in Armenia... wow... so much land scape full of routes waiting to be claimed, named, and set. There are mounds of places you can go and rock climb with 0 minute approaches. At least from what I can read online Rock climbing is actually a new sport in Armenia and the only resource for information that I could find about it was a site called Up the Rocks. I tried contacting Mkhitar Mkhitaryan to see if I could arrange a trip. I was quoted $150 for the a trip to Hell's Canyon ~70 Km from Yerevan for the guide work and $10 for each set of shoes and harness, this did not include the price of my transportation and lunch. Frankly that seems a bit steep to me for given that I'm used to just taking my own equipment at this point and I've only heard quoted figures around that price for long multi pitch rock climbing guide around Yosemite. I was hoping to just offer some beers in exchange for some route / map information / gear rental, perhaps next time. In the end it didn't work out for me for this trip but check the video below for what is possible. Other options Mkhitar mentioned was to go to Noravanq Canyon ~100 Km from Yerevan. Personally I saw tons of routes possible with no approach distance (just park your car) on the way to Jermuk. At one point I couldn't resist and just tried to climb a level 5.5 scramble by a river in Jermuk. I only had my sneakers, the rock was a bit dirty, and I could have fallen badly both left or right but I just had to try... What can I say, I'm addicted. I didn't even travel much around Armenia so I'm sure there has to be tons of other great places. If you are in Armenia and are curious about rock climbing I highly recommend you attend / become active with the next Armenian First Ascent Open Festival May 25-31, 2013 and also befriend Mkhitar Mkhitaryan on Facebook and buy him a beer. Better yet, start organizing folk and start practicing for the event.

Monday, August 13, 2012

Automatically backporting the Linux kernel

Users should be able to get support for their hardware if they are on any supported kernel. Linux kernel developers should always prioritize working directly on the Linux kernel and there should be no excuse as to why not do to so. Linux kernel developers should ideally never have to backport anything they do onto older kernels. Linux distributions should share as much as possible if they are already putting some work into backporting. What if we could simply automatically backport the Linux kernel ? What if to automatically backport the Linux kernel we may have to take into consideration some design considerations of the technologies and semantics that would enable this? This  blog post is about all this, where we stand and where we are going.

At the 2012 Linux Collaboration summit in San Francisco I gave a talk about the idea and project objective I have on automatically backporting the Linux kernel. A pictures speaks more than 1000 words so lets try to get directly to shiny pictures and graphs so you have to read less and I can write less. If you haven't yet please go first read my last blog post on optimizing backporting collateral evolutions on the Linux kernel, and then go through the few images below. You can also just review the slides from the automatically backporting the Linux kernel presentation.

This is your Linux backport effort of a collateral evolution on drugs:

If you've been doing backporting or if you've read backported Linux kernel code before though you'll realize that this is what you typically see though. This change in fact would be present for all drivers, and the one above is a change for just one driver. A better idea is to wrap all this code up into a helper routine that could be used over different drivers. Here's what this would look like then:

And this is exactly where we are today with backporting the net_device_ops collateral evolution down to one line for each driver. We do this by implementing the actual helper in a shared module under the project compat. Once we have this implemented you can even express this in SmPL form (see previous blog post on optimizing collateral evolutions) and then we would not have to even code this collateral evolution for other new drivers. In fact, if we added a static inline for netdev_attach_ops() upstream to simply do that assignment to the data structure it would mean we would require 0 lines of code changed to backport this collateral evolution. I really want you to go review the first picture before you make any decisions on how you feel about this.

A few people like the strategy and thanks to all the help of a good large group of contributors today I can say we are striving towards these optimizations slowly. This effort started through the compat and compat-wireless (now renamed to compat-drivers) projects but the number of systems we backport keeps growing and the amount of code we can share keeps growing as well. Below exactly how much code we share is illustrated against the amount of code we are pulling directly from the Linux kernel.

That small green thing -- that's compat, and that red thing between the dark stuff and the green stuff is the delta of patches that we have not yet merged into a generic compat module or are things which we have not yet figured out how to generalize -- this goes into the compat-drivers project. In short: the more we share into a compat module the more subystems we can share at a lower cost and most effort is done automatically.

But not all work is automatic of course. Hauke Mehrtens, Johannes Berg, Bala Shanmugam, Ozan Çağlayan among others have contributed mounds of work to the projects and without their help we would not have such great advancements. Ozan in particular decided to take on backporting the video graphics kernel drivers as part of his Google Summer of Code project and with this making it the 4th subsystem we are working towards integrating into the project, previous to this we only backporte Ethernet, Wireless and Bluetooth subystems. The folks at the Linux Foundation driver backports working group expressed interest in the project at the last Linux Collaboration summit in San Francisco and at their request we will be renaming the project to compat-drivers to put emphasis we are interested in generalizing backporting, and we will eventually be making releases through instead of the wireless front. At this point all relevant documentation has been migrated onto the backports wiki page, please use that page to advance or read the required documentation. We expect to make the first release through as of the v3.7 release cycle.

For now if interested in the project you can refer to the code:
At this point we aim to backport all 4 subystem kernel drivers and making both daily snapshots based on Stephen Rothwell's linux-next.git, and RC releases based on Linus's linux.git sand Greg's stable releases from linux-stable.git and all of these able to compile across 21 older kernels.

If interested in poking around first, you may want to simply clone compat.git and check out the following few scripts:

git clone git://

ckmake is intended to help you cross compile test any external module across 21 older kernels. 

Apart from backporting we realize at times you may need to integrate critical fixes not yet integrated upstream. The project has solution for this that respects and prioritizes Linux upstream development by providing code metrics on each type of delta possible of patches from the birth of a patch all the way up to its integration upstream and the effort required to backport it.

If you want to send patches please subscribe and send patches to the shiny new mailing list: To subscribe send an e-mail with anything on the subject to with this on the subject:

subscribe backports

Read the following e-mails on a follow up of how to confirm subscription. Please read the new backports kernel wiki page on how you are expected to send patches, but please use the new mailing list.

Optimizing backporting collateral evolutions

In November, 2005 the first paper to my knowledge was published that coined the term of "collateral evolutions". This paper, authored by Yoann Padioleau Gilles Muller, and our very own kernel hacker: Julia L. Lawall, formalized what collateral evolutions are with emphasis on Linux kernel development: software evolutions on Linux kernel APIs which require respective Linux device driver code updates. You likely had not heard of the term "collateral evolution" but you likely have seen patches by Julia already and if her name rings a bell you likely are associating it with some odd patch commit log messages which look like this:

@@ identifier ret; expression e,e1,e2,e3,e4,x; @@
( if (\(ret != 0\|ret < 0\) || ...) { ... return ...; } | ret = 0 ) ... when != ret = e1 *x = \(kmalloc\|kzalloc\|kcalloc\|devm_kzalloc\|ioremap\|ioremap_nocache\|devm_ioremap\|devm_ioremap_nocache\)(...); ... when != x = e2 when != ret = e3 *if (x == NULL || ...) { ... when != ret = e4 * return ret; } //

This is all in SmPL (Semantic Patch Language) and the research team who wrote the paper above designed the language specifically to describe collateral evolutions in Linux kernel development. The paper states that about 70% of operating system software consists of device drivers and that 30% of software updates to a Linux kernel consists of addressing collateral evolutions on the Linux kernel. The paper provides some case studies on collateral evolutions, and proposing to work on "Coccinelle" an engine to enable developers evolve the Linux kernel by optimizing on how to express collateral evolutions and applying them.

Its year 2012 and by now there is likely no subsystem in the Linux kernel that has not taken advantage of SmPL. In fact we're a bit far beyond that too in terms of reasearch and how we can apply and use SmPL to evolve the Linux kernel. I'm not going to get into the technical details, instead I'll only provide references for you to do homework if you are interested in the topic but I will provide the conclusions I have and empirical implications that I am observing and foresee for us in Linux kernel development. My interest with SmPL was heightened when trying to move drivers out of the staging area of the Linux kernel and seeing how broadly you could use SmPL. The biggest use case I saw potential for SmPL though was for some work I was focusing on for a while: automatically backporting the Linux kernel.

Everyone and their mother backports code, but they tend to historically have tried backporting their own things: their own driver, their own stacks, and at times for their own Linux distributions and only for a set of supported Linux kernels, and at times forking the Linux kernel and never merging things back upstream. For a while now I have taken a slightly different approach to backporting: backport all drivers on a subsytem, backporting for all Linux distributions with priority to all known supported Linux kernel releases, share as much code as possible, and always prioritize upstream. Turns out that what you have to end up backporting are collateral evolutions. The same SmPL that could be used to help you evolve the Linux kernel with one SmPL patch could therefore in theory be used as a reversed SmPL patch to backport that same collateral evolution. As per review with Julia, this is theoretically possible, and although we have quite a few Linux kernel developers using SmPL to write collateral evolutions of the Linux kernel, not all evolutions are written this way and we should not assume we could convince everyone to do so. Furthermore the learning curve for learning SmPL is steep, its not easy to learn it.

At the 2011 Linux Plumbers conference I met Julia for the first time and at this conference Julia revealed to me the holy grail to the big picture of collateral evolutions and what we'd need to use it for backporting given that not everyone can be expected to write collateral evolutions using SmPL: spdiff. Julia explained to me Jesper Andersen wrote this utility to help you generate SmPL provided you give it two patches which illustrated a collateral evolution. The implications are huge for Linux kernel development. You don't really need to learn SmPL to write SmPL. You don't even need to learn SmPL to evolve a collateral evolution for the Linux kernel for all subystems which require changes within it. It also means that if you backport a collateral evolution for two drivers in the Linux kernel you could in theory backport that collateral evolution for all other Linux kernel drivers.


If true -- and I plan on proving this over time -- the implications are significant for us. It means we can indeed focus more on advancing the Linux kernel faster than ever before but at the same time having to worry less when we need to backport a collateral evolution onto older kernels. As I will explain in later blog posts though not all things are easily backported though -- but fortunately a lot of things are, and with a bit of work and realization of how we can help optimize code to help backport we may be able to backport code even faster over time. If we also evolve the Linux kernel with SmPL with new collateral evolutions it may also mean (although this would need to be proven) less bugs caused by collateral evolutions and obviously also less bugs in backporting as the backports themselves would be the direct inverse of the expressed SmPL