Monday, February 27, 2012

GNU things are GNU - Simplicity of GNU Make

edvard munch - the scream  1893

GNU things are GNU. If you're a hacker you'll understand that this is meant to be a fun interjection. It really depends on who reads this what you'll make of that interjection. As a kernel hacker perhaps you'll cringe and maybe even cry yourself to sleep in fetal position (as I think Johannes Berg [G+] may have today) of the thought of the completion of a project such as the GNU Hurd. As a GNU'er maybe you'll pride yourself with the amount of documentation, portability, and righteousness and of course recursion after the implications of such a statement. If you're not a hacker just understand that GNU has a bible and a church.

As reported by Hauke Mehrtens, I recently broke parallel building in compat, a Linux kernel backport module, when I added autoconf.h-like support support for it. The autoconf.h support allows us to push build time configuration options onto the C and header files we are using at build time. GNU Make typically builds things serially but if you ask it to build things in parallel, by specifying the -j command argument, GNU Make will build all objects through independent tasks and later combine them. For any modern system this is crucial otherwise you system's resources will not be used efficiently at build time and each object will only build until the last one will have finished, even though each one could have been built independently. The component that manages parallel building in GNU Make is called, the jobserver.

I'll confess that I have been avoiding looking into how GNU Make jobserver works given that the man page really doesn't provide much details about its implementation nor does it clarify exactly what that pesky "+" thing is when it fails on using the job server:

hauke@hauke:~/compat-wireless/compat-wireless$ make -j5
make[1]: warning: jobserver unavailable: using -j1.  Add `+' to parent
make rule.

Peter D. Smith is the maintainer of GNU Make and I'm fortunate to have found an entry on his page where he documents in careful detail history and evolution of the implementation details of GNU Make's jobserver. After reading that I now appreciate GNU Make a bit more in consideration for all the "GNU" objectives it strives for, and it being a GNU project utility. Felix Fietkau tells me that this appreciation says more about other GNU utilities than about GNU Make... perhaps he's right but nevertheless I now really like the simplicity of GNU Make's manpage, its apparent simplicity, and its side little optimizations that I have found after mucking around with it a bit more on my hunt to understand its jobserver.

I hate long blog entries so I'm going to now summarize very quickly for hackers my findings, if you're not a hacker, stop reading here.

GNU Masters no FISL 12

The jobserver works using pipes, and the jobserver will barf if its heuristics implementation fails at understanding if a target has a command that may not be another make process (called sub-make process, in GNU Make language). The heuristics implementation is very simple, this means GNU Make is inherently dumb and you need to guide it, the only thing it knows is itself, and to the GNU pride even about recursive calls against itself. The job server will know what to do if a sub-make process is issued with the same binary, to be clear the same make binary -- and this is pretty darn cool. You make emphasis that a sub-process is an explicit GNU Make sub-process and ensure that this propagates properly by using the MAKE variable. It is imperative that sub-processes on a target that are not identical GNU Make sub-processes are annotated as such, you do this by pre-pending a "+" to each command that is not a GNU Make sub-process. Since the jobserver is braindead stupid by design GNU Make will halt the jobserver completely when issuing these specific commands.


What if you have a project that needs some header file but you want that header file to be generated through the build process automatically ? GNU Make is dumb but fortunately smart enough to look for header files that are being included on targets, even if the target is not listed as a dependency to a final required build target. GNU Make is also dumb but smart enough to know that if it had to build a header file that was being included it will call itself again, automatically after building the required target file, in fact it will not fatally fail in this case even if you asked it to fail (by not specifying that the include file can be missing by prefexing the include command with the "-"). This is really cool and shows some real GNU recursive philosophy in mind.

Now to the stupid part. When trying to optimize your Makefiles take some time to review all the ways that you have left GNU Make go astray and be stupid by looking at the make -d output. What I found was that there were tons of implicit checks on targets that needed to be generated, this included even the god damn Makefile itself and possible derivatives of it. I counted 18 stupid derivatives. The simple solution is to optimize build time by adding to .PHONY even your own Makefile.

If you'd like to see examples you can review my jobserver and general GNU Make optimization patches to compat. These patches illustrate all these optimizations at the Linux kernel module level, along with some build time optimizations for external Linux kernel modules. I hope someone out there appreciates the recursion between the link to this blog post on the patches posted and the linking from this blog post to those patches. I wonder what Google calls this when scraping the web, there must be a name for it.

Friday, February 10, 2012

Using iperf with Linux containers

At times you may want to hack on a driver and test it on the same machine without having to require a switch in the middle, or even another box. In my case I wanted to test a new Atheros Ethernet driver we're working on, alx, with a 1 GB/s link through my free PCI-E slot and hook it up to my with an RJ-45 to my laptop's own internal 1 GB Intel Ethernet card, that uses the e1000e driver.

If you try assign IP addresses to these guys, and use 'iperf -s' on one and 'iperf -c' on another you'll end up transmitting using the loopback won't actually get metrics you want on the device.

To work around this you may be inclined at first to work with chroot, but that won't give you the separation you want with hardware. There are some ARP tricks you can work on to accomplish this, and another alternative is to use some -E iperf flag which is not merged. I also tried iperf -B but that didn't work either. In the end I decided to give Linux containers a shot and it worked nicely, albeit, with a bit of work too. Below is a few thing you can use to follow to get your setup with it.

First, go and get yourself a debootstrap setup for Linux containers and use a simple twist to that guide by using the lxc "phys" type for the network device as illustrated below. After this just cp -a the debootstrap'd thingy to another dir to get yourself a second box. You may be able to share the same files for a lxc run, but I haven't tested this. Then create two lxc.conf files, one for each of your containers which will use their own dedicated network device. I use /etc/lxc/sid-01.conf and /etc/lxc/sid-02.conf, below is my /etc/lxc/sid-01.conf:
lxc.tty = 6
lxc.pts = 1024
lxc.rootfs = /var/lib/lxc/sid-01
lxc.cgroup.devices.deny = a
# /dev/null and zero
lxc.cgroup.devices.allow = c 1:3 rwm
lxc.cgroup.devices.allow = c 1:5 rwm
# consoles
lxc.cgroup.devices.allow = c 5:1 rwm
lxc.cgroup.devices.allow = c 5:0 rwm
lxc.cgroup.devices.allow = c 4:0 rwm
lxc.cgroup.devices.allow = c 4:1 rwm
# /dev/{,u}random
lxc.cgroup.devices.allow = c 1:9 rwm
lxc.cgroup.devices.allow = c 1:8 rwm
lxc.cgroup.devices.allow = c 136:* rwm
lxc.cgroup.devices.allow = c 5:2 rwm
# rtc
lxc.cgroup.devices.allow = c 254:0 rwm
# network = phys = up = eth1 = = eth1 = 00:FF:12:34:56:78
For your sid-02.conf just change the to eth4 or whatever, and also change the IP address and hw address slightly. That's all. After this running 'iperf -s' on one box and 'iperf -c' on the other should produce the results you expect.

From I am mcgrof's smirking revenge

All the fun things you can get done now with just one box :)

Saturday, February 04, 2012

Surfing in the bay area

Rust and Surf # 2 - San Francisco

While learning to surf in Costa Rica I broke a surf board. The good thing about this story is the guy at the shop, Ozzie Hoppe from Adrift Surf Shop at Playa Esterillos, was really friendly and only charged me what the rental damage demanded, $60, but also, after learning that I lived in San Francisco explained he lived there before -- and explained that surfing is just as good in the bay area as it was in Costa Rica! He eneded up writing down in piece of paper a bunch of places he recommended to surf at in the bay area and in this blog post I share that with you all.

San Francisco

  • Bolinas: Good during big swell NW or small south can get big on bigger south swells - medium tide.
  • Ocean Beach North End: Kelly's Cove, only good on small days, under 4ft - medium-high tide.
  • Rockaway - good on smaller swells, low tide
  • Linda Mar: In Pacifica, on south end of beach, mid to high tide can get crowded. Good on medium to small days.
  • Princeton Jetty: In half moon bay. Mid to high tide. Good on medium sized south swell or big NW.

Santa Cruz

  • Small - Medium Days: The Hook on the east side. Capitola on the east side. Manresa - south of the hook (beach break).
  • Big Days: Cowells on the west side, inside of Steamer's Lane.
Surf Supplies

He recommend to check out clean line surf site to buy surf gear. He recommended to try out suits by Kelly's Cove, on 46th Ave or Mollusk, then you'd just buy the part on the site. He pointed out all brand have the same size, so this should work well.

I have yet to try any of these places but am pretty excited to try them.

Intro to surfing in Costa Rica

For the 2012 new years I visited my home country, Costa Rica, for my cousin's wedding. The wedding took place in Puerto Viejo. It was my first time at a wedding in the beach, it was great, and I think.. I overdressed.

From I am mcgrof's smirking revenge

I've been meaning to learn to surf and last time I visited CR, I tried surfing for the first time in Tamarindo. I only got the chance to surf once though, this time I was determined, I would learn. In San Francisco I met a friend who recommended I spend at least 5 days surfing, otherwise I wouldn't get it. So I set my vacation goal: surf 5 days in 5 different beaches. Below are some accounts of these adventures and some recommendations.

Although I started in Puerto Viejo, I didn't get a chance to surf there, due to time constraints. Me and two cousins decided to venture out to Santa Teresa, in Guanacaste right after the wedding. I never told them about my objective but was pleased to see them more than willing to learn to surf as well. I didn't rent a car this time, and instead just trooped it by public transportation. We took the Santa Teresa bus from the "Coca Cola" bus stop in San Jose. Its only $10, and takes about 4-5 hours. We arrived there on a Tuesday January 3, 2012. Low and behold, it was high season and we had a hard time finding a place to stay at. We ended up just crashing at a Hostel, it came out to about $17 / night. If you are visiting Santa Teresa -- or any beach in CR -- take cash out before going. There really aren't many ATM machines around and once you find one you'd be very displeased to be charged $5 per ATM transaction.

From I am mcgrof's smirking revenge

Santa Teresa was nice... and a hell of a great place to start learning to surf in Costa Rica. Ideal. We got lucky. It cost about $5 per 3 hours or $10 for the day to rent a board. Go with what they recommend at first if you are learning. You'll get something that looks like a yacht, and heavy as a log. On the first day me and my cousins only got one board and we took turns.

From I am mcgrof's smirking revenge

I give tons of credit to my cousin David who on his first day managed to stand up on the board. I was advised it would take a few days, maybe even 5 to stand up on the board for the first time. Both my cousin David and Edgar actually seemed to be naturals to the sport, they picked up on it quick. Santa Teresa was nice to learn to surf as it has really long water beach entries, the waves will take you in for a while so you can try to ride on them for a while.

From I am mcgrof's smirking revenge

You also get to pick whether or not to go for larger waves that break way deep or waves that break half way in. You can expect a really good wave about every 1-2 minutes. The sunset was really awesome in Santa Teresa, and the night scene lived up to our expectations. We ended up just resting after our first day though and only trooped it out to hang by the Carmen on the second night.

From I am mcgrof's smirking revenge

On the second day we ended up in Playa Carmen and also got two boards, a medium type of board and then a more regular smaller looking board. The beach was a bit rocky, but the waves were just as nice. Playa Carmen was easier to hang around in though given that the only ATM in the area was there. If you can choose though and you want to learn to surf I'd recommend to just stay at the end in Santa Teresa. We're all glad we tried the different boards. I couldn't advance much on the smaller board but my cousin Edgar did manage to catch a good wave and stood up on it for a while. The medium board seems more appropriate after 2-3 days of learning.

From I am mcgrof's smirking revenge

Overall both Santa Teresa and Playa Carmen were a great place to learn. I highly recommend it. This area also attracts a lot of surf tourists and as such you'll also see a lot of surf instructors teaching their young padawans. You can take advantage of this, just as we did, and listen closely to what the surf instructors were telling their apprentices. After surfing go find yourself a good Sopa Negra, a Costarican style bean soup.

From I am mcgrof's smirking revenge

After the first two days of trying I was hooked... and right after that went on a road trip with my father to Dominical. I read online about the beaches there and read they were rough but really good for surfing. My third surf beach in this trip would be Playa Dominical. This one kicked my ass and I'm lucky to have survived trying to surf at this shit. The beach was really rocky and the waves short and big. The waves were also coming in pretty fast, instead of the rate of a good big wave every 1-2 minutes as in Santa Teresa, they were coming in at the rate of about every 5 seconds! If you went under a wave just to skip it, you had to be ready to either take on or skip another big wave. If you can't swim well, you'll fucking drown ;)

From I am mcgrof's smirking revenge

After Dominical we stopped at Playa Esterillos Oeste. This one was very similar to Dominical but had less frequent waves and in my opinion manageable. Only thing though is I ended up breaking my first surf board at Esterillos. I couldn't believe it, I simply went under a wave and as I am coming up I hear a pop! I didn't know what it was until I felt the weight of the board had changed and felt something was missing. The top had gone off into the water. I managed to recover it and take it back to the surf shop I rented it at. The guy at the shop explained the odds of this were about 10/100 and it seems I had agreed to pay them $60 in case of this happening. They didn't charge me the entire board, that would have been about $200-$300 it seems. Odd or not after talking to a few folks I've learned my lesson, this board likely was already popped and had a cheap repair done to it. Before renting a board you should check to see if its painted, and if its been damaged before. It should be pretty hard to pop a board.

After my road trip I couldn't get enough of surfing and decided to make one last trip to the beach on my own, to complete my 5 day surfing excursion in Costa Rica. I couldn't go so far this time as I had to leave the next day.. so I decided to try Playa Hermosa. Playa Hermosa is right next to Playa Jaco, and Jaco is the more popular tourist destination. Getting to Playa Hermosa or Jaco is also pretty easy, you just take a bus for less than $10 at the "Coca Cola" bus stop in Chepe (San Jose). I arrived to Play Hermosa happy and excited but without any cash... and guess what? Turns out there were no fucking ATM machines in all of Playa Hermosa and I had no other option but to go back to Playa Jaco to spend the night. I decided to stay and try surfing in Jaco...

From I am mcgrof's smirking revenge

Jaco was terrible to surf at. I was up by 6:30am pumped to surf really early and the first rental place opened at 8:30am. Apart from that the waves were really small and the rate at which you would get a good one would be about every 15 minutes or more... Perhaps it was for the better, now I just want to surf more.

From I am mcgrof's smirking revenge

I had a blast in Costa Rica and I'm glad I got to learn to surf with my cousins. Perhaps when I visit next they'll have a few tricks to show me. I'm pretty happy with my progress, I managed to stand up well 4 times and ride the wave all the way through. Up next I'll disclose a few places I was recommended to surf at in the bay area.

This blog post, and my blog title name change, is heavily inspired by my good friend Nakeyshia's blog on active living.