As part of my home automation system, I need to connect "high-level" systems (such as the primary Linux server) down into the underlying hardware systems around the house. The PIC16F688 microcontrollers that run those systems are seriously low-level. Thus, I've chosen to place all of them onto an I2C bus(*) driven by an Arduino. Why? ... the Arduino has enough capability to mount an Ethernet port on the "high" side, and has built-in I2C support for the "low" side. It is a great mapping device between the complex systems, and the hardware systems.
With the hardware selected, and the wiring selected, it came down to protocol. How can that Linux server talk to the Arduino, as a proxy to the individual microcontroller critters splattered across the household? ... Naturally, HTTP was my first choice, as it allows all kinds of languages, libraries, and scripts to perform the work, and even some basic interaction via a full-on browser.
Digging into HTTP servers for the Arduino... Ugh. The landscape is very disappointing. Much of the code is poorly engineered, or the code is designed for serving web pages. During my search, I ran into a colleague's TinyWebServer. I'd call it the best out there for web serving (well-designed and well-coded), but is still overpowered for my purpose: mapping a protocol down to simple hardware interactions.
As a result, I designed a small library to construct a simple API endpoint on the Arduino. The application can register dozens of endpoints to process different types of client requests (62 endpoints are easy, more if you want to seek out the critical edges of allowed-characters in URIs). Each endpoint can accept a number of bytes as parameters, and can return zero, or an unlimited set of bytes to the client.
I have yet to "package" the Endpoint system, so any requests and changes are most welcome! I've got some documentation to write, along with incorporating feedback from others' and my own usage.
Would love to get some feedback! ==> http://goo.gl/O5GJ3g
(*) and yes, I know an I2C bus is designed for 0.5m meter runs, rather than a whole house; bus accelerators, speed compensation, and other approaches "should" manage it. I'll report on my success/failure in a future post.
Friday, August 15, 2014
Friday, August 01, 2014
CanaKit PIC Programmer Reset
Earlier tonite, I scorched one of my 16F688 PICs in a stupid move. To see if it was recoverable/usable, I dropped the sucker into my CanaKit programmer and ran 'pk2cmd -P' on my Mac to check whether it could see/detect the PIC.
Bad move.
"No PICkit 2 found." ... and the red BUSY light just started flashing.
It took some research because CanaKit has almost zero documentation. The short answer is the programmer is a clone of Microchip's PICkit 2 Development Programmer/Debugger. The red flashy means the board couldn't load its firmware.
Solution:
Download the V2.32 firmware from the above page. Unzip the file. Then load it onto the board:
$ pk2cmd -D/path/to/firmware/PK2V023200.hex
It'll load the firmware, verify it, then reset the device. No more angry LED!
(I hope those who run into a similar problem will find this blog post, to more quickly reach a solution)
Bad move.
"No PICkit 2 found." ... and the red BUSY light just started flashing.
It took some research because CanaKit has almost zero documentation. The short answer is the programmer is a clone of Microchip's PICkit 2 Development Programmer/Debugger. The red flashy means the board couldn't load its firmware.
Solution:
Download the V2.32 firmware from the above page. Unzip the file. Then load it onto the board:
$ pk2cmd -D/path/to/firmware/PK2V023200.hex
It'll load the firmware, verify it, then reset the device. No more angry LED!
(I hope those who run into a similar problem will find this blog post, to more quickly reach a solution)
Friday, June 06, 2014
LED lighting control boards
Woot! I just ordered a set of PCBs for lighting control in the new house. Wifey would kill me if we had no lights, so the priority kinda jumped on this one ... (*Wifey comments: I wouldn't kill you. However, I consider it prudent to go to Home Depot and buy some cheapo mechanical switches just in case)
One of the more unique things about my (upcoming) house is that the bulk of our lighting are LED 24 VDC cans/pots/recessed lamps(*). Ceilings fans, chandeliers, and watercloset light/fans are standard high-voltage AC, but the bulk of our electrical fixtures are using these nifty low-voltage lamps. One neat thing is that the lamps don't require AC/DC stepdown transformers, so their cost is dropped, their reliability increases, and their heatprint is lowered.
The wiring becomes both simpler, and more complex in ways, and possibly cheaper depending on your choices. Simpler because you can run standard 14 gauge stranded speaker wire (cheap; flexible around those corners) instead of the heavier gauge Romex stuff. The lower gauge wire can save you quite a bit on copper costs, and you can lose the heavy insulation and concerns of high-voltage wires in your walls. But much more complicated because you need special hardware to run them ... a repair is no longer just a run to your local Home Depot.
In my case, much of the wiring in the house has been "home-run" back to my server room, so I ended up doubling my copper/install costs. All the control is localized to that room, which also means I don't have actual "switches" in my house, but just sensors (see Capacitive Touch Wall Switches). This leads to "how do I control these lighting circuits?"
Thus, my custom PWM Control Boards. They have an embedded microcontroller which can "do anything" with the nine (9) output channels. A higher-level "upstream" controller will tell it what actions to perform, when, and how, communicating via TTL Serial or I2C or what-have-you. The boards sink current, at whatever voltage (so I will also use these for my 12VDC LED light strips).
I've sent the PCBs off to ITEAD for production. Between now and when they arrive, I'll finish the microcontroller work that I've been working on. Assemble some parts from Mouser, plug it all together, and LIGHTS!
Theoretically.
Will report back, as I make progress...
(*) I decided not to name/link the manufacturer until I get experience and form an opinion. Just email me to ask, if you're interested.
One of the more unique things about my (upcoming) house is that the bulk of our lighting are LED 24 VDC cans/pots/recessed lamps(*). Ceilings fans, chandeliers, and watercloset light/fans are standard high-voltage AC, but the bulk of our electrical fixtures are using these nifty low-voltage lamps. One neat thing is that the lamps don't require AC/DC stepdown transformers, so their cost is dropped, their reliability increases, and their heatprint is lowered.
The wiring becomes both simpler, and more complex in ways, and possibly cheaper depending on your choices. Simpler because you can run standard 14 gauge stranded speaker wire (cheap; flexible around those corners) instead of the heavier gauge Romex stuff. The lower gauge wire can save you quite a bit on copper costs, and you can lose the heavy insulation and concerns of high-voltage wires in your walls. But much more complicated because you need special hardware to run them ... a repair is no longer just a run to your local Home Depot.
In my case, much of the wiring in the house has been "home-run" back to my server room, so I ended up doubling my copper/install costs. All the control is localized to that room, which also means I don't have actual "switches" in my house, but just sensors (see Capacitive Touch Wall Switches). This leads to "how do I control these lighting circuits?"
Thus, my custom PWM Control Boards. They have an embedded microcontroller which can "do anything" with the nine (9) output channels. A higher-level "upstream" controller will tell it what actions to perform, when, and how, communicating via TTL Serial or I2C or what-have-you. The boards sink current, at whatever voltage (so I will also use these for my 12VDC LED light strips).
I've sent the PCBs off to ITEAD for production. Between now and when they arrive, I'll finish the microcontroller work that I've been working on. Assemble some parts from Mouser, plug it all together, and LIGHTS!
Theoretically.
Will report back, as I make progress...
(*) I decided not to name/link the manufacturer until I get experience and form an opinion. Just email me to ask, if you're interested.
Monday, September 09, 2013
Capacitive Sensing Wall Switches
For my new house, I wanted to go with an unobtrusive, "no controls" style. There are no thermostats, no pool controller, no security panels, and no standard toggle/paddle switches for the lights. Instead, I'm going for a designer style piece of colored glass with capacitive touch sensors behind it. Think of your smart phone's touch capability, and that will be my light switches.
The "switches" are really just sensors, as the signals are delivered to the main house controller where the actual lighting management occurs. The sensor pad can detect touches, gestures, and multi-taps to provide different lighting requests, based on time of day and ambient light. The switches also include optional hookups to an IR/motion sensor, temperature measurement, and all switches have RGB backlighting.
All my work will be Open Source, so I've started the documentation for the wall switches. I have a couple custom PCBs that I'm working on, and (eventually) a whole mess of microcontroller code for the PIC16F688.
I'll keep updating the wiki pages, and committing changes to my repository. I don't have commit emails going anywhere, but Google Code supports feeds for wiki and source changes if you'd like to track the changes. Of course, I'll keep posting here when interesting milestones occur!
The "switches" are really just sensors, as the signals are delivered to the main house controller where the actual lighting management occurs. The sensor pad can detect touches, gestures, and multi-taps to provide different lighting requests, based on time of day and ambient light. The switches also include optional hookups to an IR/motion sensor, temperature measurement, and all switches have RGB backlighting.
All my work will be Open Source, so I've started the documentation for the wall switches. I have a couple custom PCBs that I'm working on, and (eventually) a whole mess of microcontroller code for the PIC16F688.
I'll keep updating the wiki pages, and committing changes to my repository. I don't have commit emails going anywhere, but Google Code supports feeds for wiki and source changes if you'd like to track the changes. Of course, I'll keep posting here when interesting milestones occur!
Monday, August 12, 2013
Bluetooth Household
For my home automation project, I chose to go with a fully-wired approach since I'm building from scratch. I get to run wires wherever needed, providing (hopefully) more reliability and an ability to troubleshoot problems. Certainly, the parts cost will be much lower than an RF solution.
But with that said, I met Eric Migicovsky a couple weeks ago (Founder/CEO of Pebble). He came up with a great idea: use a Pebble watch as a control mechanism. Sure, I'll have phones, tablets, infrared remotes, and various sensors... but something on my wrist? Always handy? Very cool idea! With multiple Bluetooth base stations, I can even detect signal strength and triangulate a user's position in the house, in order to provide context-sensitive menus and button controls. If you're in the home theater, then "Pause" is going to be a handy watch button, when that drink needs a refill! Given that I'm writing the app, I can even provide Wifey with her own customized watch experience.
To that end, I started doing some research on Bluetooth, and on the Pebble SDK. The first thing to pop up was the need to use Bluetooth Low Energy (aka Bluetooth 4.0, BLE, or Bluetooth Smart [Ready]) rather than the older Bluetooth 2.x or 3.x protocols. BLE allows for interactions without pairing, which is important for roaming about the house, with multiple base stations. The Pebble hardware supports BLE, but it seems that the SDK doesn't (yet) allow for applications to deliver messages to one/more/available base stations. My hope is to work with the Pebble guys to see where/how to make that available to (my) home automation application.
The second part of the problem is the development of the base stations for my house. There are inexpensive Bluetooth/USB dongles (about US$11) that can speak BLE. I've got a few Raspberry Pi boards around the house, with previously-unused USB ports. A little searching seems to indicate the dongles are supported under Linux.
These dongles seem to present themselves as an HID device (eg. keyboard, mouse, etc), and can be switched to a [Bluetooth] Host Controller Interface (HCI). I haven't dug in deeply on this stuff yet, but I do have a Fitbit dongle on my Mac OS. The Fitbit (Flex) speaks BLE, so it seemed appropriate to experiment with.
Working with HID seemed harsh, until I found hidapi. The API is very clean and simple. As a Python programmer, bindings were the next step. Ran across Cython-HIDAPI, which sucks: forked copy of HIDAPI and heavyweight Cython-based bindings (given the ugly, I'm not gonna provide link-love).
Answer: I wrote a ctypes-based binding for hidapi. My first, undocumented draft landed at just 143 lines of Python. Of course, I've checked it in, along with a sample script.
And after all that, my Fitbit dongle is purely a USB device (calling hid_open() fails). Sigh.
I've got more research to do, and maybe ordering a dongle for experimentation (see Adafruit, or various on Amazon). Maybe I can interact with the Fitbit dongle through USB rather than HID. Who knows. But once I figure the base station thing out, I can track Pebble watches, Fitbits, and other Bluetooth devices throughout my house.
But with that said, I met Eric Migicovsky a couple weeks ago (Founder/CEO of Pebble). He came up with a great idea: use a Pebble watch as a control mechanism. Sure, I'll have phones, tablets, infrared remotes, and various sensors... but something on my wrist? Always handy? Very cool idea! With multiple Bluetooth base stations, I can even detect signal strength and triangulate a user's position in the house, in order to provide context-sensitive menus and button controls. If you're in the home theater, then "Pause" is going to be a handy watch button, when that drink needs a refill! Given that I'm writing the app, I can even provide Wifey with her own customized watch experience.
To that end, I started doing some research on Bluetooth, and on the Pebble SDK. The first thing to pop up was the need to use Bluetooth Low Energy (aka Bluetooth 4.0, BLE, or Bluetooth Smart [Ready]) rather than the older Bluetooth 2.x or 3.x protocols. BLE allows for interactions without pairing, which is important for roaming about the house, with multiple base stations. The Pebble hardware supports BLE, but it seems that the SDK doesn't (yet) allow for applications to deliver messages to one/more/available base stations. My hope is to work with the Pebble guys to see where/how to make that available to (my) home automation application.
The second part of the problem is the development of the base stations for my house. There are inexpensive Bluetooth/USB dongles (about US$11) that can speak BLE. I've got a few Raspberry Pi boards around the house, with previously-unused USB ports. A little searching seems to indicate the dongles are supported under Linux.
These dongles seem to present themselves as an HID device (eg. keyboard, mouse, etc), and can be switched to a [Bluetooth] Host Controller Interface (HCI). I haven't dug in deeply on this stuff yet, but I do have a Fitbit dongle on my Mac OS. The Fitbit (Flex) speaks BLE, so it seemed appropriate to experiment with.
Working with HID seemed harsh, until I found hidapi. The API is very clean and simple. As a Python programmer, bindings were the next step. Ran across Cython-HIDAPI, which sucks: forked copy of HIDAPI and heavyweight Cython-based bindings (given the ugly, I'm not gonna provide link-love).
Answer: I wrote a ctypes-based binding for hidapi. My first, undocumented draft landed at just 143 lines of Python. Of course, I've checked it in, along with a sample script.
And after all that, my Fitbit dongle is purely a USB device (calling hid_open() fails). Sigh.
I've got more research to do, and maybe ordering a dongle for experimentation (see Adafruit, or various on Amazon). Maybe I can interact with the Fitbit dongle through USB rather than HID. Who knows. But once I figure the base station thing out, I can track Pebble watches, Fitbits, and other Bluetooth devices throughout my house.
Tuesday, April 16, 2013
Building omxplayer on a Raspberry Pi
The past couple days, I set aside my PIC work and concentrated on setting up a Raspberry Pi ("RPi"). I've got a couple of these, and will use them as video streamers for televisions in my house.
There is quite a bit of documentation for getting an RPi set up, so I won't repeat that here. My current focus is on getting video streaming working. An obvious candidate is RaspBMC, but I was looking for something very bare-bones to simply put a video onto the HDMI output. I ran across PyPlex which seemed right up my alley: Python and effectively an interface-less control of the video.
Yah. Well. Then I look at the setup/build requirements. twisted-web? pexpect? Seriously? Somebody has made this much more complicated than it should be. Whatever happened to just using BaseHTTPServer and the subprocess module?
Digging in, I find it is using omxplayer underneath. No wonder they're using pexpect -- there is a tty/keyboard interface to omxplayer. (of course, pty might be simpler than pexpect, but whatever) So this PyPlex thing starts up a web service and then controls omxplayer via a tty connection. I'm not seeing reliability and responsiveness here. And a lot of code, to boot.
Tearing off another layer of the onion, I start looking at omxplayer. Sigh. Requirements hell yet again. GCC 4.7. Boost. ffmpeg. Oh, and it is generally set up for cross-compilation rather than building on the RPi. This isn't a bad concept in general, as the RPi is no speed demon. But the build only takes a long time because they chose ffmpeg, whereas the Raspbian distribution uses libav. (these two libraries are reasonably similar, as libav forked from ffmpeg rather nastily a couple years ago)
So I'm looking at this giant pile of C++ code with a bunch of crazy requirements, which would take hours to build on my RPi. This is the wonderful state of video on the RPi. Sigh.
Well... I found a post by Keith Wright where he talks about building (a tweaked fork) of omxplayer on Raspbian. Much better, but the instructions still have crazy oddities about reconfiguring RAM, sudo to build in strange filesystem locations, and hey! fun! building ffmpeg from scratch again. Sigh. A guy just can't get any love around here.
Being the good geek that I am... this just wasn't something that I want to put up with. I want to build this sucker on my RPi, using standard tooling and libraries that come on Raspbian.
First up, I started from huceke/omxplayer rather than Keith's because it is much newer. But I did grab the Makefile.include from Keith, as it was sane for building on the RPi. Adjusted some of the paths to point to the installed items. Then, I had to install the following packages on the RPi: libpcre3-dev, libavcodec-dev, libavdevice-dev, libavfilter-dev, libavformat-dev, libboost-dev. As I started working through getting omxplayer built, I ran into a bug in a system header.
In /opt/vc/include/interface/vmcs_host/vcgencmd.h, line 33 needs to be changed to:
Next up, I had to hack away, tweak, and otherwise put a bit of pain on the omxplayer sources. Some hacks were easy, but others likely broke stuff (I'm not sure if subtitles work any more). Hard to tell. A/V code is not easy, and not something that I'm familiar with.
You can find all of my changes in my omxplayer fork. Clone that to your RPi, install the necessary packages, and hit "make". No system reconfiguration. No sudo. No hours of ffmpeg building. No GCC 4.7 requirement.
Clone. Make.
Go have fun, and watch some movies!
(my next step is to tear off the user interface bits, and shift to a simpler, pure-C library which I can wrap/control from Python)
There is quite a bit of documentation for getting an RPi set up, so I won't repeat that here. My current focus is on getting video streaming working. An obvious candidate is RaspBMC, but I was looking for something very bare-bones to simply put a video onto the HDMI output. I ran across PyPlex which seemed right up my alley: Python and effectively an interface-less control of the video.
Yah. Well. Then I look at the setup/build requirements. twisted-web? pexpect? Seriously? Somebody has made this much more complicated than it should be. Whatever happened to just using BaseHTTPServer and the subprocess module?
Digging in, I find it is using omxplayer underneath. No wonder they're using pexpect -- there is a tty/keyboard interface to omxplayer. (of course, pty might be simpler than pexpect, but whatever) So this PyPlex thing starts up a web service and then controls omxplayer via a tty connection. I'm not seeing reliability and responsiveness here. And a lot of code, to boot.
Tearing off another layer of the onion, I start looking at omxplayer. Sigh. Requirements hell yet again. GCC 4.7. Boost. ffmpeg. Oh, and it is generally set up for cross-compilation rather than building on the RPi. This isn't a bad concept in general, as the RPi is no speed demon. But the build only takes a long time because they chose ffmpeg, whereas the Raspbian distribution uses libav. (these two libraries are reasonably similar, as libav forked from ffmpeg rather nastily a couple years ago)
So I'm looking at this giant pile of C++ code with a bunch of crazy requirements, which would take hours to build on my RPi. This is the wonderful state of video on the RPi. Sigh.
Well... I found a post by Keith Wright where he talks about building (a tweaked fork) of omxplayer on Raspbian. Much better, but the instructions still have crazy oddities about reconfiguring RAM, sudo to build in strange filesystem locations, and hey! fun! building ffmpeg from scratch again. Sigh. A guy just can't get any love around here.
Being the good geek that I am... this just wasn't something that I want to put up with. I want to build this sucker on my RPi, using standard tooling and libraries that come on Raspbian.
First up, I started from huceke/omxplayer rather than Keith's because it is much newer. But I did grab the Makefile.include from Keith, as it was sane for building on the RPi. Adjusted some of the paths to point to the installed items. Then, I had to install the following packages on the RPi: libpcre3-dev, libavcodec-dev, libavdevice-dev, libavfilter-dev, libavformat-dev, libboost-dev. As I started working through getting omxplayer built, I ran into a bug in a system header.
In /opt/vc/include/interface/vmcs_host/vcgencmd.h, line 33 needs to be changed to:
#include "interface/vmcs_host/linux/vchost_config.h"I've filed a pull request to github:raspberrypi/firmware to fix this. Not sure if that is the Right place (that code may come from upstream?), but hopefully somebody will see it.
Next up, I had to hack away, tweak, and otherwise put a bit of pain on the omxplayer sources. Some hacks were easy, but others likely broke stuff (I'm not sure if subtitles work any more). Hard to tell. A/V code is not easy, and not something that I'm familiar with.
You can find all of my changes in my omxplayer fork. Clone that to your RPi, install the necessary packages, and hit "make". No system reconfiguration. No sudo. No hours of ffmpeg building. No GCC 4.7 requirement.
Clone. Make.
Go have fun, and watch some movies!
(my next step is to tear off the user interface bits, and shift to a simpler, pure-C library which I can wrap/control from Python)
Sunday, April 14, 2013
PIC Programming on Mac OS X
Lately, I've begun working on home automation to wire up my entire house with all kinds of goodies. The hobbyist in me, and with an attempt to be frugal, I'm skipping off-the-shelf solutions and building my own. A friend of mine calls me crazy, that it will reduce the value of my house, etc. Whatever. This is some fun stuff!
A big part of these systems is wiring together sensors and activators. You need something to control all of these things. There are a gazillion various solutions, with the obvious ones being an Arduino or Raspberry Pi and their GPIO pins (among many other features they have). I decided on a layered approach with "small brains" connected to the sensor, managing the specifics, then communicating upstream to a larger brain. I'll likely talk about the upstream bits in a later post, but this one is dedicated to the small brain: the PIC 16F688 microcontroller.
I grew up on a 6502 microprocessor. Graduated to a 68000 when it arrived in the Mac 128k. And after that, never really worried about machine/assembly code. As I looked around for microcontrollers, I ran into the 16F688 on SparkFun. This is a crazy chip -- the number of features packed into a tiny 14-pin DIP is simply incredible compared to where I came from. A couple key features that pointed me at this chip: UART on board, and about $2 per part. This allows me to do stuff such as communicate to serial sensors (such as the Zilog ePIR IR sensor), use bit-banging to communicate to I2C sensors (such as the MPR121 capacitive touch sensor), measure voltages for security systems and temperature (TMP36), ... and all in a tiny and cheap package.
Next up is programming the dang thing. I've got a programmer and my Macbook. This post will document the steps needed to get some code running on the PIC. (to help others, and if I have to retrace these steps in the future)
First, you will need the following software packages installed:
A big part of these systems is wiring together sensors and activators. You need something to control all of these things. There are a gazillion various solutions, with the obvious ones being an Arduino or Raspberry Pi and their GPIO pins (among many other features they have). I decided on a layered approach with "small brains" connected to the sensor, managing the specifics, then communicating upstream to a larger brain. I'll likely talk about the upstream bits in a later post, but this one is dedicated to the small brain: the PIC 16F688 microcontroller.
I grew up on a 6502 microprocessor. Graduated to a 68000 when it arrived in the Mac 128k. And after that, never really worried about machine/assembly code. As I looked around for microcontrollers, I ran into the 16F688 on SparkFun. This is a crazy chip -- the number of features packed into a tiny 14-pin DIP is simply incredible compared to where I came from. A couple key features that pointed me at this chip: UART on board, and about $2 per part. This allows me to do stuff such as communicate to serial sensors (such as the Zilog ePIR IR sensor), use bit-banging to communicate to I2C sensors (such as the MPR121 capacitive touch sensor), measure voltages for security systems and temperature (TMP36), ... and all in a tiny and cheap package.
Next up is programming the dang thing. I've got a programmer and my Macbook. This post will document the steps needed to get some code running on the PIC. (to help others, and if I have to retrace these steps in the future)
First, you will need the following software packages installed:
- pk2cmd (download from Microchip)
This uses a standard configure; make; make install - gputils (see their SourceForge project)
I used "make mac105", then symlink'd "pk2cmd" and "PK2DeviceFile.dat" into my PATH
Note: I did not setuid-root on the binary (as the docs seem to suggest). It seems to operate fine with my login id.
When I plug my programmer into the USB port and run "pk2cmd -P" it detects my PIC16F688. Woot!
And for a basic "Hello World" to test my setup, I wrote a "blink an LED" program. Download that and type "make install" and it should assemble and load the program onto the PIC sitting in your programmer. Yank out the PIC, wire up RA0 to an LED tied to Vdd (I used a 680Ω resistor), and apply power. The LED should blink at you.
Not that hard!
If it doesn't? Well. Not like you can bring up Visual Studio and debug this thing. The program works for me, and the wiring is dead simple, so I wouldn't know where to point you.
Next up: switch my blinky program to use the chip's Sleep Mode and interrupts [rather than a busy loop]. Less power consumption!
Wednesday, March 07, 2012
Oroppas, by St Clement
So it was finally time to open and drink my flight of St Clement's Oroppas wine (info on 2007 bottling). It's been waiting too long, so the wife and I decided to cook a wonderful dinner and start popping open bottles.
My oldest bottle was from 1995. Waiting 17 years is certainly too long, but what's done is done. The 1995 still had a lot of flavor, rich notes, but with a very short finish (as expected).
Sunday night, we opened up the first four bottles (1995, 1996, 1997, and 1998). Since these bottles were old, I figured they wouldn't be strong representatives of Oroppas. So... I also opened up a 2007 Oroppas; the tasting notes said it was just getting ready to drink. The 2007 was our "control" bottle to really show the bold, smooth flavors of St Clement's Oroppas series.
As I mentioned, the 1995 was still good, but with a short finish. The 1996 was tasty, starting to show some of Oroppas' deeper flavor.
Tuesday night, we opened the last two... the 2001 and 2002. This pair was very similar to Monday night's bottles: one smooth and full-bodied, and the other was a bit less. The 2001 was excellent. It had all the favor, smoothness, fruit and berry, and richness expected. The 2002 was a bit weaker on the fruit and the tannins were starting to creep in.
Okay... Wednesday evening, we opened a 2004 Oroppas as the final bottle in this tasting. It was hiding behind the flight, in the picture above (along with the 2007). Based on tasting all of these wines, I have to say: the 2004 is the absolute best of the entire pack. It has all of the fruit, boldness, smooth flavor, and only a little tannin. I would suggest that holding Oroppas for right around eight years (from the vintage date) is its ideal. (well... based on my tastes)
Overall, please remember that I'm talking comparisons here. All of these bottles were very tasty. There is just no way to call any of these bad wines. As expected, the 1995 had weaker flavor yet no tannins. Moving forward in time, the flavor definitely improved, but the true Oroppas boldness did not show up until about 1998 or so. Around 2001 or 2002, a light shade of tannins started to arrive. The 2004 seemed to be the peak in the bottles that I had.
I've been a club member of St Clement for 10 or 15 years. Their whites, reds, and specialities like the Oroppas are all fabulous wines. Their winery is a big Victorian up on a hill; it is beautiful, with a wonderful view over Napa Valley. I'm definitely a fan, and this flight has been an awesome experience. I highly recommend their wines, and a visit to their wonderful property.
Cheers!
 |
| Flight of Oroppas, 1995 through 2002 |
Sunday night, we opened up the first four bottles (1995, 1996, 1997, and 1998). Since these bottles were old, I figured they wouldn't be strong representatives of Oroppas. So... I also opened up a 2007 Oroppas; the tasting notes said it was just getting ready to drink. The 2007 was our "control" bottle to really show the bold, smooth flavors of St Clement's Oroppas series.
Strangely enough, the 1997 was a step backwards. The wine was a bit sharp and acidic, unlike the big bold flavor of the 2007. Even the 1996 demonstrated some of that boldness. Given the usual success of Napa wines from 1997, I was quite surprised. Thankfully, moving onwards to the 1998 put everything back on track. The 1998 was a good representatve of that Oroppas flavor and style. Bold and smooth, with lots of rich flavor and a great mouth feel. The 2007 had even more flavor, but still contained some of the rough tannins of a young wine; the 1998 had none of that roughness and much (though not all) of the flavor. I will probably wait at least two or three years before opening more of the 2007, to age away some of the tannins.
By mid-evening on Monday, we had finished off the 1995, 1996, and 1997. We kept around some of the 1998 and the 2007 for comparison, as I popped open the 1999 and 2000. The progression from 1998 was quite obvious with the 1999 wine -- the flavor and finish just got bigger. But the 2000 was missing the strong berry and fruit undertones, leaving just a woody finish. The 1999 was the clear winner in these few years.
Tuesday night, we opened the last two... the 2001 and 2002. This pair was very similar to Monday night's bottles: one smooth and full-bodied, and the other was a bit less. The 2001 was excellent. It had all the favor, smoothness, fruit and berry, and richness expected. The 2002 was a bit weaker on the fruit and the tannins were starting to creep in.
Okay... Wednesday evening, we opened a 2004 Oroppas as the final bottle in this tasting. It was hiding behind the flight, in the picture above (along with the 2007). Based on tasting all of these wines, I have to say: the 2004 is the absolute best of the entire pack. It has all of the fruit, boldness, smooth flavor, and only a little tannin. I would suggest that holding Oroppas for right around eight years (from the vintage date) is its ideal. (well... based on my tastes)
Overall, please remember that I'm talking comparisons here. All of these bottles were very tasty. There is just no way to call any of these bad wines. As expected, the 1995 had weaker flavor yet no tannins. Moving forward in time, the flavor definitely improved, but the true Oroppas boldness did not show up until about 1998 or so. Around 2001 or 2002, a light shade of tannins started to arrive. The 2004 seemed to be the peak in the bottles that I had.
I've been a club member of St Clement for 10 or 15 years. Their whites, reds, and specialities like the Oroppas are all fabulous wines. Their winery is a big Victorian up on a hill; it is beautiful, with a wonderful view over Napa Valley. I'm definitely a fan, and this flight has been an awesome experience. I highly recommend their wines, and a visit to their wonderful property.
Cheers!
Sunday, March 04, 2012
Lots of Stuff
I spent a good amount of time in front of my television. Watching shows (cable and Netflix streaming) or playing video games. The TV occupies a large portion of my life. Good? Bad? Who knows. But that isn't the topic for today...
Lately, I've been fascinated watching the show Hoarders on A&E. The show is like a train wreck -- you just can't stop watching. It is a bit sad once you truly understand that hoarding is a psychological disorder, but it is so hard to stop watching. The things that people collect, that get hoarded, the condition of the house, etc ... there is always something new on the show.
But here is where the "Stuff" from my post title comes in. There are a number of shows on that focus on "stuff". Hoarders is one show, but TLC has a similar show named Hoarding: Buried Alive. The show title is a bit crazy, given that it actually happens.
It doesn't stop there. Those two shows are about people collecting. But there are shows on the other side of the equation, too. Hoarding is about acquiring, but the stuff (sometimes) needs to go away, too. Storage Wars is about lockers that get auctioned off when people don't pay their bill. In many cases, the lockers were owned by hoarders, and all kinds of awesome stuff is found in there.
And then you have the show, American Pickers. The show is extremely fascinating. Mike and Frank find lots of stuff in peoples' hoards, but they concentrate on old items, and the history behind them. As a big History Channel fan, American Pickers is an interesting lens into history. Old cars, music legends, bicycles, cars, and other memorabilia.
The stuff that people accumulate is incredibly fascinating, and these shows provide a broad view into history, people, and an endless variety of "stuff".
Lately, I've been fascinated watching the show Hoarders on A&E. The show is like a train wreck -- you just can't stop watching. It is a bit sad once you truly understand that hoarding is a psychological disorder, but it is so hard to stop watching. The things that people collect, that get hoarded, the condition of the house, etc ... there is always something new on the show.
But here is where the "Stuff" from my post title comes in. There are a number of shows on that focus on "stuff". Hoarders is one show, but TLC has a similar show named Hoarding: Buried Alive. The show title is a bit crazy, given that it actually happens.
It doesn't stop there. Those two shows are about people collecting. But there are shows on the other side of the equation, too. Hoarding is about acquiring, but the stuff (sometimes) needs to go away, too. Storage Wars is about lockers that get auctioned off when people don't pay their bill. In many cases, the lockers were owned by hoarders, and all kinds of awesome stuff is found in there.
And then you have the show, American Pickers. The show is extremely fascinating. Mike and Frank find lots of stuff in peoples' hoards, but they concentrate on old items, and the history behind them. As a big History Channel fan, American Pickers is an interesting lens into history. Old cars, music legends, bicycles, cars, and other memorabilia.
The stuff that people accumulate is incredibly fascinating, and these shows provide a broad view into history, people, and an endless variety of "stuff".
Monday, October 31, 2011
Installing Zabbix on Mac OS (Leopard)
My friend Sam Ruby dabbles in a lot of technology, and he tends to do writeups on his blog as he experiments with the stuff. I figured to take a page from his book, and share my own issues/troubles getting Zabbix up and running on my MacBook (running Leopard).
I grabbed the 1.8.8 tarball and unpacked it. For my scenario, I needed the server, the agent, and the frontend (but not the proxy). For simplicity in testing, and because I don't need to monitor bunches o' boxes, I decided to go with SQLite for the database. Zabbix uses the standard "configure/make/make install" pattern, so no hassle so far.
Burp. The compilation failed. Investigating, I found that I needed to apply the patch from ZBX-4085. The build completed, so started to look at the frontend.
The Frontend is written in PHP, which is natively available (along with Apache) on my laptop. With some configuration, I got the frontend to load in my browser. There is a click-through GPL license (huh?) and then a really awesome page that checks your setup. I quickly realized that the builtin PHP was not going to work. Sigh.
I've got MacPorts installed on my laptop, so I just continued with that. Homebrew is all the new rage with the kids, but it doesn't have builtin recipes for PHP. There are a few out on the 'net, but I really didn't want to monkey with that stuff.
Lots of packages were needed: php5, php5-gd, php5-mbstring, php5-sockets, php5-sqlite3, sqlite3. A hojillion dependencies were installed, including another copy of Apache (sigh).
Reloading the setup page, it continued to say SQLite wasn't installed. Looking at the frontend source, it was using a function named sqlite3_open(). With some investigation, I found an email describing the SQLite interfaces for PHP. Zabbix was using an unmaintained version. Rather than monkeying with that, I just edited the code to use the preferred PHP SQLite interface, and filed issue ZBX-4289 to push my changes upstream.
Finally, I needed to tweak /opt/local/etc/php5/php.ini for the recommended Zabbix settings (after copying php.ini-development to php.ini). This included some timezone settings, timeouts, upload sizes, etc. The Zabbix setup page is quite good about guiding you here.
So I created my initial SQLite .db file based on the instructions from the manual and pointed the Zabbix configuration page at it (taking a moment to realize it wanted the pathname put into the database field of the form). The test connection worked and then Zabbix saved the configuration file into frontends/php/conf/zabbix.conf.php. It looks like there is a "download" option for that configuration file, which I presume appears when the conf directory is not writeable. The Apache server (running from MacPorts now, using the MacPorts PHP) was running as myself, so it had no problem writing that configuration file.
Next up: wrestling with the zabbix-server. The first annoying problem was that you cannot give it a configuration file in the current directory. It fails trying to lock "." for some dumb reason. Solution: pass an absolute path to the custom configuration file (the default is in /etc or somesuch, which I didn't want to monkey with). Getting the server running was very frustrating because it spawns multiple processes which communicate using shared memory. It kept failing with errors about not being able to allocate the shared memory segments. After some research, I found that Mac OS defaults to some pretty small limits. Given that I wasn't about to reconfigure my kernel (using sysctl and some recipes I found on the web), I went to rejigger all the various cache sizes in the zabbix_server.conf file.
It ended up that I had to drop all the sizes to their minimum 128k setting: CacheSize, HistoryCacheSize, TrendCacheSize, HistoryTextCacheSize. Each were set to 131072. Finally, the server started. Whew.
When I returned to the frontend to "Finish" the installation and bring up the console... it hung. No response from the server. Huge sigh. With a bunch of investigation, I found that something was holding an exclusive lock on the whole damned SQLite file. Nothing else could write to it (and it seems the frontend likes to test its writability by creating/dropping a dummy table).
Fuck. Time to scrap the whole damned "simple SQLite" idea. Fine... I've used MySQL before, so I went with that. Back to MacPorts to install MySQL, the server, and the PHP driver for MySQL. Then I fired it up, created a "zabbix" user, loaded in all the tables, and zapped the zabbix.conf.php file to trigger reconfiguration (after noting to restart Apache to pick up the PHP changes).
The frontend looked happy now, so I tweaked the server's configuration file for MySQL and restarted the server. No workee. Damn. Forgot to reconfigure the server using --with-mysql=/opt/local/lib/mysql5/bin/mysql_config. After reconfiguring, the link failed with unsatisfied references to iconv(), iconv_open(), and iconv_close(). The MySQL interface in the server needs these for some UTF-8 conversions. The builtin Mac OS libiconv should work, but my MacPorts copy of libiconv was interfering, and these functions are named libiconv(), libiconv_open(), and libiconv_close(). My patience was ending, so I was not about to delve into autoconf bullshit and conditional compilation and all that. I simply edited src/libs/zbxcommon/str.c to call the libiconv* versions of the functions. The compile and link succeeded, and I re-installed the newly built server.
Yay! The server restarted, and the website loads up with a nifty little default console.
After a day to get this sucker installed, now I gotta start figuring out how to use it. Oh, joy.
I hope this post will help some future person treading these waters. Good luck!
ps. I may have missed some steps or packages to install or whatever. YMMV, but I think that I've got most of it down. Zabbix is supposed to be some hotness, and I do like its custom agent capability. But hoo-wee. Not a simple package to bring up (I hope it will be easier on a recent Ubuntu, than it was on my creaky Leopard install).
I grabbed the 1.8.8 tarball and unpacked it. For my scenario, I needed the server, the agent, and the frontend (but not the proxy). For simplicity in testing, and because I don't need to monitor bunches o' boxes, I decided to go with SQLite for the database. Zabbix uses the standard "configure/make/make install" pattern, so no hassle so far.
Burp. The compilation failed. Investigating, I found that I needed to apply the patch from ZBX-4085. The build completed, so started to look at the frontend.
The Frontend is written in PHP, which is natively available (along with Apache) on my laptop. With some configuration, I got the frontend to load in my browser. There is a click-through GPL license (huh?) and then a really awesome page that checks your setup. I quickly realized that the builtin PHP was not going to work. Sigh.
I've got MacPorts installed on my laptop, so I just continued with that. Homebrew is all the new rage with the kids, but it doesn't have builtin recipes for PHP. There are a few out on the 'net, but I really didn't want to monkey with that stuff.
Lots of packages were needed: php5, php5-gd, php5-mbstring, php5-sockets, php5-sqlite3, sqlite3. A hojillion dependencies were installed, including another copy of Apache (sigh).
Reloading the setup page, it continued to say SQLite wasn't installed. Looking at the frontend source, it was using a function named sqlite3_open(). With some investigation, I found an email describing the SQLite interfaces for PHP. Zabbix was using an unmaintained version. Rather than monkeying with that, I just edited the code to use the preferred PHP SQLite interface, and filed issue ZBX-4289 to push my changes upstream.
Finally, I needed to tweak /opt/local/etc/php5/php.ini for the recommended Zabbix settings (after copying php.ini-development to php.ini). This included some timezone settings, timeouts, upload sizes, etc. The Zabbix setup page is quite good about guiding you here.
So I created my initial SQLite .db file based on the instructions from the manual and pointed the Zabbix configuration page at it (taking a moment to realize it wanted the pathname put into the database field of the form). The test connection worked and then Zabbix saved the configuration file into frontends/php/conf/zabbix.conf.php. It looks like there is a "download" option for that configuration file, which I presume appears when the conf directory is not writeable. The Apache server (running from MacPorts now, using the MacPorts PHP) was running as myself, so it had no problem writing that configuration file.
Next up: wrestling with the zabbix-server. The first annoying problem was that you cannot give it a configuration file in the current directory. It fails trying to lock "." for some dumb reason. Solution: pass an absolute path to the custom configuration file (the default is in /etc or somesuch, which I didn't want to monkey with). Getting the server running was very frustrating because it spawns multiple processes which communicate using shared memory. It kept failing with errors about not being able to allocate the shared memory segments. After some research, I found that Mac OS defaults to some pretty small limits. Given that I wasn't about to reconfigure my kernel (using sysctl and some recipes I found on the web), I went to rejigger all the various cache sizes in the zabbix_server.conf file.
It ended up that I had to drop all the sizes to their minimum 128k setting: CacheSize, HistoryCacheSize, TrendCacheSize, HistoryTextCacheSize. Each were set to 131072. Finally, the server started. Whew.
When I returned to the frontend to "Finish" the installation and bring up the console... it hung. No response from the server. Huge sigh. With a bunch of investigation, I found that something was holding an exclusive lock on the whole damned SQLite file. Nothing else could write to it (and it seems the frontend likes to test its writability by creating/dropping a dummy table).
Fuck. Time to scrap the whole damned "simple SQLite" idea. Fine... I've used MySQL before, so I went with that. Back to MacPorts to install MySQL, the server, and the PHP driver for MySQL. Then I fired it up, created a "zabbix" user, loaded in all the tables, and zapped the zabbix.conf.php file to trigger reconfiguration (after noting to restart Apache to pick up the PHP changes).
The frontend looked happy now, so I tweaked the server's configuration file for MySQL and restarted the server. No workee. Damn. Forgot to reconfigure the server using --with-mysql=/opt/local/lib/mysql5/bin/mysql_config. After reconfiguring, the link failed with unsatisfied references to iconv(), iconv_open(), and iconv_close(). The MySQL interface in the server needs these for some UTF-8 conversions. The builtin Mac OS libiconv should work, but my MacPorts copy of libiconv was interfering, and these functions are named libiconv(), libiconv_open(), and libiconv_close(). My patience was ending, so I was not about to delve into autoconf bullshit and conditional compilation and all that. I simply edited src/libs/zbxcommon/str.c to call the libiconv* versions of the functions. The compile and link succeeded, and I re-installed the newly built server.
Yay! The server restarted, and the website loads up with a nifty little default console.
After a day to get this sucker installed, now I gotta start figuring out how to use it. Oh, joy.
I hope this post will help some future person treading these waters. Good luck!
ps. I may have missed some steps or packages to install or whatever. YMMV, but I think that I've got most of it down. Zabbix is supposed to be some hotness, and I do like its custom agent capability. But hoo-wee. Not a simple package to bring up (I hope it will be easier on a recent Ubuntu, than it was on my creaky Leopard install).
Monday, August 15, 2011
Blast from the past: removing the GIL
Way back in 1996, I created a patch to remove the GIL from Python's interpreter (version 1.4!). Dave Beazley just picked up the patch and tore it apart, and writing a fantastic blog post. It is quite nostalgic for me, from back in the day when I was working at Microsoft on their electronic commerce efforts.
[ I commented on Dave's post; it provides some context that you may also be interested in reading ]
[ I commented on Dave's post; it provides some context that you may also be interested in reading ]
Monday, November 29, 2010
Open Languages are Not Required
I just posted again to Apache Asserts on Computerworld UK: Open Languages are Not Required.
And please note that I'm speaking primarily to enterprise (internal) software developers, who are the vast majority of developers on the planet. They shouldn't really have to worry about the language that they use for their development. Having an open language is critical for us FLOSS developers, but that is an entirely separate discussion. (hat tip to webmink, to clarify my point here)
Note: the publish date is wrong (says last month); dunno what's up with that.
Update: corrected link after the publish date was fixed.
And please note that I'm speaking primarily to enterprise (internal) software developers, who are the vast majority of developers on the planet. They shouldn't really have to worry about the language that they use for their development. Having an open language is critical for us FLOSS developers, but that is an entirely separate discussion. (hat tip to webmink, to clarify my point here)
Note: the publish date is wrong (says last month); dunno what's up with that.
Update: corrected link after the publish date was fixed.
Friday, October 29, 2010
Are You An Open Source Friend?
The Apache Software Foundation was invited to find some people for Computerworld UK to write for a new blog named "Apache Asserts". Myself and a few others were selected to post our thoughts on open source, the enterprise, and whatever else we may find interesting.
My first post has been published... check it out!
My first post has been published... check it out!
Sunday, August 15, 2010
Android uses Java? Um... no
I've seen a lot of misinformation over the weekend, talking about the Oracle/Google lawsuit. Many of these blog posts and article talks about how "Android uses Java". Heh. That simply isn't true.
Android applications are written using the Java programming language. True. But those applications run on the Dalvik virtual machine. Not the Java virtual machine. Source code is owned/copyright by the author and is entirely unbound from any intellectual property concerns based around the syntax/grammar of that language.
Class libraries? Not Java either. Much of the core libraries come from Apache Harmony, and the rest are libraries that Google wrote. Given that Apache was never provided access to the Java Compatibility Kit, Harmony is not labeled as "Java-certified". Also note that Harmony is a clean-room implementation of the Java class libraries.
So, people: stop saying that Android "uses Java". It doesn't.
(obviously, some of these various components may trample on Oracle's patents; I have no idea, and that is an entirely separate question)
Android applications are written using the Java programming language. True. But those applications run on the Dalvik virtual machine. Not the Java virtual machine. Source code is owned/copyright by the author and is entirely unbound from any intellectual property concerns based around the syntax/grammar of that language.
Class libraries? Not Java either. Much of the core libraries come from Apache Harmony, and the rest are libraries that Google wrote. Given that Apache was never provided access to the Java Compatibility Kit, Harmony is not labeled as "Java-certified". Also note that Harmony is a clean-room implementation of the Java class libraries.
So, people: stop saying that Android "uses Java". It doesn't.
(obviously, some of these various components may trample on Oracle's patents; I have no idea, and that is an entirely separate question)
Wednesday, August 04, 2010
Outcomes
Nights out with a friend can be quite interesting. Especially if they are single and "looking". I've found there are generally three possible outcomes with these nights out:
- The Cock-Block.
Your friend is trying to hook up or otherwise get especially friendly with somebody, but you monopolize the "target's" attention in some way to distract them from your friend's intent. Obviously, this outcome is "poor", unless you're some kind of dickhead that doesn't want your friend departing early with the target. Quite selfish, to try and keep them out with you. Of course, there are all sorts of minor rules variants here, that are rather crass: e.g if you're both interested in the target, who steps forward, who holds back? It's simply best to avoid this scenario because it never turns out well.
- The Wingman.
Oh yah. We all know this one... the friend who props up the other and makes them ever more desirable. Talk up their strengths, ensure that the person-of-interest gets excited to know more about your friend. This is the ideal outcome, especially if they make some kind of lasting connection.
- The Bus-Tosser.
This isn't nearly as bad as the Cock-Block, but your friend isn't going to be all that happy with you. At least for a short while. This is where you think your friend is interested in somebody, so you move into Wingman mode. Provide lots of opportunity for the two to talk and hang out, provide some good commentary, etc. Like any good Wingman would do. But afterwards, you find out your friend was not interested. At all. This is the "thanks for throwing me under the bus" maneuver, putting your friend into harms way. Especially if the purported target is interested and giving undue attention to your friend. ... Thankfully, in the long run, this provides lots of laughable material for how you sucked as a myopic Wingman.
I think the best answer all around is to simply go out and have a great time with your friend. Anything that will involve a possible third person can fall into a poor outcome, or simply distract from an awesome evening with a friend.
Wednesday, May 19, 2010
svn stash
This is the third of (at least four) posts in my miniseries about Subversion's next-generation working copy library. See the introduction and what we're doing to fix things.
Once we have this fancy new code, it will provide a stable and robust base for building new features. The DVCS systems have done a great job exploring new areas and needs of version control users. One feature in particular is called stashing or shelving (see "git stash" and "hg shelve").
For those not familiar with the stash concept: consider the scenario where you've been doing some work, and a high-priority bug arrives, needing to be fixed right away. Classically, a Subversion user would check out a fresh working copy, fix the bug, perform the commit, and go back to their work in the original working copy. Instead, when using stash, it takes all of your current work and sets it aside, leaving you with an unchanged working copy, ready for your bug-fix work. After your commit, you retrieve the changes that were stashed. The presumption here, of course, is that stashing is a much faster and simpler operation than setting up a new working copy.
We'll be able implement this feature quite easily using the WC-NG datastore. It will take just a few operations:
Recovering the changes from the stash is effectively running a big "svn merge" operation. The merge is required because you may have made other changes to the working copy (your bug-fix) and/or updated to the latest revision.
Other features, such as multiple stashes, management of those changes, applying subsets, and whatnot would be added, too. The feature set has not (yet) been designed, so I have no idea what is required or how we would present this to our users. We'll definitely be looking at git and hg as we explore the needs around stashing/shelving.
"When?" is your next question, I'm sure :-) ... Well, we're releasing WC-NG in Subversion 1.7. That will probably happen this fall. We want to get those changes out the door since that will mark 18 months of development time. WC-NG is a feature in itself, and we want to get it into people's hands without further delays [waiting for additional features]. After that, I'm interested in adding stash support (and a "checkpoint" feature (described in my next post)). So let's say stashing will appear in 1.8 which should be released around this time next year.
Once we have this fancy new code, it will provide a stable and robust base for building new features. The DVCS systems have done a great job exploring new areas and needs of version control users. One feature in particular is called stashing or shelving (see "git stash" and "hg shelve").
For those not familiar with the stash concept: consider the scenario where you've been doing some work, and a high-priority bug arrives, needing to be fixed right away. Classically, a Subversion user would check out a fresh working copy, fix the bug, perform the commit, and go back to their work in the original working copy. Instead, when using stash, it takes all of your current work and sets it aside, leaving you with an unchanged working copy, ready for your bug-fix work. After your commit, you retrieve the changes that were stashed. The presumption here, of course, is that stashing is a much faster and simpler operation than setting up a new working copy.
We'll be able implement this feature quite easily using the WC-NG datastore. It will take just a few operations:
- preserve all metadata about local changes
- place a copy of each locally-modified into pristine storage, recording their SHA-1 key with the stashed metadata
- revert all local changes
Recovering the changes from the stash is effectively running a big "svn merge" operation. The merge is required because you may have made other changes to the working copy (your bug-fix) and/or updated to the latest revision.
Other features, such as multiple stashes, management of those changes, applying subsets, and whatnot would be added, too. The feature set has not (yet) been designed, so I have no idea what is required or how we would present this to our users. We'll definitely be looking at git and hg as we explore the needs around stashing/shelving.
"When?" is your next question, I'm sure :-) ... Well, we're releasing WC-NG in Subversion 1.7. That will probably happen this fall. We want to get those changes out the door since that will mark 18 months of development time. WC-NG is a feature in itself, and we want to get it into people's hands without further delays [waiting for additional features]. After that, I'm interested in adding stash support (and a "checkpoint" feature (described in my next post)). So let's say stashing will appear in 1.8 which should be released around this time next year.
Wednesday, May 05, 2010
Heading to Berlin!
In June, in Berlin, elego is hosting a "Subversion Day", along with workshops and a hackathon/sprint. And with great thanks to elego, I will be able to attend and contribute to the event. I'll be in Berlin from June 9th through the 14th.
As always, I'm looking forward to meeting up with my fellow Subversion developers, but there are quite a few others in Berlin that I want to spend time with. Torsten, Valerie, Erik -- I'm looking at you! Julian: road trip from Münster? Who else am I missing? Torsten says that he'll arrange for one of the regular Apache-people dinners. Want to join?
As always, I'm looking forward to meeting up with my fellow Subversion developers, but there are quite a few others in Berlin that I want to spend time with. Torsten, Valerie, Erik -- I'm looking at you! Julian: road trip from Münster? Who else am I missing? Torsten says that he'll arrange for one of the regular Apache-people dinners. Want to join?
Monday, April 26, 2010
WC-NG Changes
In my last post, I described how libsvn_wc had become brittle and hard to manage. The WC-NG process is working to solve that problem, and though we're not yet done, I believe we're on the right path.
The basic question that needed answering is, "where did we go wrong?" While version control is a hard problem (especially if you version directories!), it does not inherently lead to a brittle library. Somewhere, we had gone wrong in the design, the data model, or simply the implementation.
Before I had started working on the problem (almost) two years ago, one of the Subversion developers (Erik Hülsmann, I believe) laid out his thoughts for a next-generation library. In those notes, he postulated on what I now call the Three-Tree Model:
This was the key insight. In our "wc-1" implementation, the svn_wc_entry_t structure blended all three trees together. Making a change to that structure could have been operating on any of the three trees depending on its flags. Its checksum field could correspond to a checked-out file, or a locally-copied file. To determine, you had to look at the schedule field and the copied field. And hell will rain upon you, should you mess up the flags or forget to check one.
For WC-NG, we have built a new data storage system with an API designed around this three-tree model. This has isolated our storage mechanism behind a solid encapsulation (wc-1 code had too much knowledge of the old "entries" storage model). Operations are now understandable: "copy nodes in the restructuring tree" instead of "set entry->schedule".
This new storage subsystem could produce an entire post on its own. It is radically different from the prior model (a single .svn subdir at the root of the working copy and SQLite-based storage). This is causing huge challenges in upgrades/migrations to the new format, and backwards compatibility for our classic APIs.
Another radical change was our move to using absolute paths to refer to items. The prior model used an "access baton" which implied a relative directory, along with a path relative to that baton. These relative batons and paths caused enormous problems because it led to the question, "relative to what?" In most cases, the answer was "the operating system's current working directory," which is a terrible basis for a deterministic API. In switching to absolute paths, this rendered the access batons obsolete. Since they were a core part of the public API for libsvn_wc (not to mention the widespread internal changes!), this has had a huge impact on the API and its users (such as Subversion's libsvn_client library and its command-line tools).
These two items (data model and absolute paths) are the core changes in WC-NG. The ripple effect from just these two items is immense. We will need to rewrite almost every one of the 40,000 lines of code in the library. And given our incremental approach, many of those will be changed multiple times. We're a solid year into this (although we saw downtime last fall due to our move to the Apache Software Foundation), and we probably have another several months of basic grunt work ahead of us. Stabilization and testing will put our 1.7 release into late summer or possibly this fall.
I could really go on and on about this stuff, but I hope this post provides some basic background on the WC-NG efforts. Please feel free to post any questions (I have no idea what aspects you may want to hear more about!), and I'll work on answering them.
The basic question that needed answering is, "where did we go wrong?" While version control is a hard problem (especially if you version directories!), it does not inherently lead to a brittle library. Somewhere, we had gone wrong in the design, the data model, or simply the implementation.
Before I had started working on the problem (almost) two years ago, one of the Subversion developers (Erik Hülsmann, I believe) laid out his thoughts for a next-generation library. In those notes, he postulated on what I now call the Three-Tree Model:
- the tree you checked out
- the above tree, plus structural changes (add, delete, move, copy)
- the above tree, plus content changes (file edits, property edits)
This was the key insight. In our "wc-1" implementation, the svn_wc_entry_t structure blended all three trees together. Making a change to that structure could have been operating on any of the three trees depending on its flags. Its checksum field could correspond to a checked-out file, or a locally-copied file. To determine, you had to look at the schedule field and the copied field. And hell will rain upon you, should you mess up the flags or forget to check one.
For WC-NG, we have built a new data storage system with an API designed around this three-tree model. This has isolated our storage mechanism behind a solid encapsulation (wc-1 code had too much knowledge of the old "entries" storage model). Operations are now understandable: "copy nodes in the restructuring tree" instead of "set entry->schedule".
This new storage subsystem could produce an entire post on its own. It is radically different from the prior model (a single .svn subdir at the root of the working copy and SQLite-based storage). This is causing huge challenges in upgrades/migrations to the new format, and backwards compatibility for our classic APIs.
Another radical change was our move to using absolute paths to refer to items. The prior model used an "access baton" which implied a relative directory, along with a path relative to that baton. These relative batons and paths caused enormous problems because it led to the question, "relative to what?" In most cases, the answer was "the operating system's current working directory," which is a terrible basis for a deterministic API. In switching to absolute paths, this rendered the access batons obsolete. Since they were a core part of the public API for libsvn_wc (not to mention the widespread internal changes!), this has had a huge impact on the API and its users (such as Subversion's libsvn_client library and its command-line tools).
These two items (data model and absolute paths) are the core changes in WC-NG. The ripple effect from just these two items is immense. We will need to rewrite almost every one of the 40,000 lines of code in the library. And given our incremental approach, many of those will be changed multiple times. We're a solid year into this (although we saw downtime last fall due to our move to the Apache Software Foundation), and we probably have another several months of basic grunt work ahead of us. Stabilization and testing will put our 1.7 release into late summer or possibly this fall.
I could really go on and on about this stuff, but I hope this post provides some basic background on the WC-NG efforts. Please feel free to post any questions (I have no idea what aspects you may want to hear more about!), and I'll work on answering them.
Tuesday, April 13, 2010
What is Subversion's WC-NG?
When I started working on the Subversion project (again) back in August 2008, I wanted to do something that was interesting, technically challenging, and important to the project. For many years, the developers had been complaining about the "working copy" (WC) library. This library was one of the first that we worked on back in June 2000, and had grown (ahem) "organically" over the following eight years. By "organically", I mean it had become a rat's nest of brittle code. Hard to work with, not fun to modify, and difficult as hell to build new features reliably. Over such a lengthy time frame, most actively-developed code tends to end up like this, unless you work real hard against it.
In 2000, we didn't even know all the requirements for the library. Nobody had ever done versioning for directories. Just files. In fact, I think that Subversion may (still) be the only version control system (VCS) out there which treats a directory as a first-class object. It is a very difficult problem, along with being able to work with only pieces of your repository (which leads to "mixed-revision" working copies; something that distributed VCS systems like Git and Mercurial don't have to deal with, much to their enjoyment!).
So we started the library and figured things out as we went. Then it was too slow, so we added stuff to make it work faster. Then we added more features. And revamped some stuff to make it go faster again. More features. And even more.
By this time, the library had become brittle. Adding a feature usually broke something else. There were too many considerations, and internal layering/hiding was not present. Everything could, and did, manipulate a public structure (called svn_wc_entry_t). If you didn't do it right, then something broke. And there was some very deep and hard to understand relationships in the handling of data in that structure. Forward progress was being stifled.
The developers had been talking about fixing the WC library for years, but most of them had other priorities. I had no such baggage, and the WC problem had everything I was looking for: interesting problems to fix, challenging to accomplish, and very important to Subversion's future. Some people had already written up some thoughts on a next generation of the WC library, calling it "WC-NG". After I started digging in, and some other developers joined, the project took on the WC-NG title in earnest and in day-to-day use.
WC-NG is Subversion's name for an entirely new working copy library. We have a new design, and we're incrementally rebuilding the library towards this new design. Due to stringent backwards-compatibility requirements, and the complexity of the system, we cannot simply "rewrite from scratch". This effort is the current focus of our upcoming 1.7 release, and it will provide a Subversion client that will be vastly faster, much more robust and capable, and provide a solid foundation for new features.
In future posts, I'll provide some more detail about WC-NG's design (and how the original WC was broken). I also want to talk about a couple of these new features that will be implemented upon this new foundation. Stay tuned!
In 2000, we didn't even know all the requirements for the library. Nobody had ever done versioning for directories. Just files. In fact, I think that Subversion may (still) be the only version control system (VCS) out there which treats a directory as a first-class object. It is a very difficult problem, along with being able to work with only pieces of your repository (which leads to "mixed-revision" working copies; something that distributed VCS systems like Git and Mercurial don't have to deal with, much to their enjoyment!).
So we started the library and figured things out as we went. Then it was too slow, so we added stuff to make it work faster. Then we added more features. And revamped some stuff to make it go faster again. More features. And even more.
By this time, the library had become brittle. Adding a feature usually broke something else. There were too many considerations, and internal layering/hiding was not present. Everything could, and did, manipulate a public structure (called svn_wc_entry_t). If you didn't do it right, then something broke. And there was some very deep and hard to understand relationships in the handling of data in that structure. Forward progress was being stifled.
The developers had been talking about fixing the WC library for years, but most of them had other priorities. I had no such baggage, and the WC problem had everything I was looking for: interesting problems to fix, challenging to accomplish, and very important to Subversion's future. Some people had already written up some thoughts on a next generation of the WC library, calling it "WC-NG". After I started digging in, and some other developers joined, the project took on the WC-NG title in earnest and in day-to-day use.
WC-NG is Subversion's name for an entirely new working copy library. We have a new design, and we're incrementally rebuilding the library towards this new design. Due to stringent backwards-compatibility requirements, and the complexity of the system, we cannot simply "rewrite from scratch". This effort is the current focus of our upcoming 1.7 release, and it will provide a Subversion client that will be vastly faster, much more robust and capable, and provide a solid foundation for new features.
In future posts, I'll provide some more detail about WC-NG's design (and how the original WC was broken). I also want to talk about a couple of these new features that will be implemented upon this new foundation. Stay tuned!
Monday, March 29, 2010
Version Control
Last week, I spent some time in NYC with friends of mine talking about Subversion. The conversion focused around the long-term vision and roadmap. I'll post more on that soon, along with some specific ideas on how I'd like to build some features that other version control (VC) systems have demonstrated as useful and demanded by users.
For this post, I wanted to share a discussion document written by Martin Fowler. This is one of the best, level-headed comparisons between Subversion, Git, and Mercurial. I believe these are the Big Three VC systems that the industry will be using over the next decade, and to see a useful discussion, absent of rhetoric, is very encouraging.
For this post, I wanted to share a discussion document written by Martin Fowler. This is one of the best, level-headed comparisons between Subversion, Git, and Mercurial. I believe these are the Big Three VC systems that the industry will be using over the next decade, and to see a useful discussion, absent of rhetoric, is very encouraging.
Subscribe to:
Posts (Atom)