Death to Folder Hierarchies

Receiving materials from a classmate has made the issue of folder hierarchies more salient to me. In these comprehensive exam materials, articles are arranged by topic, which makes sense, but before that there is a dichotomous hierarchy imposed on the topics: industrial and organizational (I dislike this false distinction in my field, but that’s another story). These higher order folders only have folders in them. This creates more drilling down every time I need to access articles. This creates the decision whether the topic I need is in “I” or “O.” Then there are subfolders within topics, so I need to remember whether a topic is I or O AND whether it subsumed by another topic. This creates a lot of cognitive load.

Folder hierarchies seem to be a less and less relevant concept in modern computing, but they persist. Advanced searching and tagging obviate the need for deep folder hierarchies. Even rigorous files naming can help this when used with something like Spotlight. Though file browsers (Finder, Windows Explorer), still reinforce the folder paradigm AND duplicates for files. A better design would be a behind the scenes database design, like iTunes or iPhoto. Most people don’t think about where their iTunes music is living in the File system, but it does keep it organized for transparency. I can imagine a Finder that more prominently brings in tags and “playlists” (smart or otherwise) that allow dynamic sorting of files without creating duplicates. I would only need one copy of articles that are used in a class, for the comp exam, and in my research. Hard drives are huge, but this would be more more efficient in space and organizing. A master database would manage this, as iTunes does, all behind the scenes. Tags, searching, and smart folders will create as needed organization without forcing the user to have to enforce and remember an organization. Everything will be searchable. The organization can be changed as often as needed. Finish the comp? Just remove the smart folder, but the files (and tags) are still there for later reference. Leopard’s Finder does have smart folders based on Spotlight, but it’s very limited and doesn’t allow the kind of folder free organization I crave.

UPDATE: I guess this topic is in the air as the excellent UI blog, ignore the code, just posted on this topic. By the way, I love the recent redesign to his blog.

Follow Up to SPSS Woes

I figured what was going on with the inaccurate scatter plot from the Chart Builder. For some reason, the variables were being treated as Nominal, despite the variable view showing them as Scale!

variables

This may be an artifact of an import from SAS, but the Legacy Dialogs and Descriptives treat both variables as if they are Scale. Even though, I was able to get a correct graph by changing the variables to Nominal and back to Scale, I am left just as befuddled how this even happened in the first place. It seems that you may be better off using Excel (or Numbers!) for graphing.

This error really calls into question most analyses I (and others) run in SPSS. Considering that much of the published research in psychology relies on the accuracy of SPSS in reporting statistics, I am very concerned.

Moral of the story: Use SAS or R. Unfortunately, I can’t use SAS (it hasn’t had a Mac version is OS 9). Maybe it is high time to give R a serious look, but the lack of GUI makes for a huge learning curve.

What is wrong with the Chart Builder in SPSS?

I am the lab instructor for a graduate psychology statistics course. For my class’s homework, they were required to figure out if a linear regression transformation was warranted for a sample data set. When assessing the skewness of a distribution, scatterplots are invaluable. Unfortunately, SPSS does not render them consistently even within itself. The data set is very skewed and requires natural log transformations for both X and Y to not violate assumptions of linear regression.

Consider first the chart editor, the option SPSS prefers by hiding the “legacy dialogs” inside a submenu. I suspect that eventually these will be altogether removed from the menus, leaving the commands available via syntax for backwards compatibility. The following syntax generated this graph:

GGRAPH
/GRAPHDATASET NAME="graphdataset" VARIABLES=BODY BRAIN MISSING=LISTWISE REPORTMISSING=NO
/GRAPHSPEC SOURCE=INLINE.
BEGIN GPL

SOURCE: s=userSource(id("graphdataset"))
DATA: BODY=col(source(s), name("BODY"), unit.category())
DATA: BRAIN=col(source(s), name("BRAIN"), unit.category())
GUIDE: axis(dim(1), label("BODY"))
GUIDE: axis(dim(2), label("BRAIN"))
ELEMENT: point(position(BODY*BRAIN))

END GPL.

chartbuilder

It looks like a linear relationship, right? Wrong! The scale is grossly off. The Body variable actually goes up over 6000. The true relationship is shown through the hidden (supposedly deprecated) legacy dialogs. Which is rendered with much more parsimonious syntax:

GRAPH
/SCATTERPLOT(BIVAR)=BODY WITH BRAIN
/MISSING=LISTWISE.

legacy

Just for yucks, I copied the data in Numbers, a consumer spreadsheet app from Apple. It even does a better job of displaying the data than SPSS’s chart builder.

numbers1

I am left with a lot of questions and few answers. Why did the chart builder cut off those points at the upper range that radically skew the data set? Why is the syntax for the chart builder 4 times as long as the old charting syntax? Why can’t the chart builder do as accurate a graph as a “baby” spreadsheet app? What is going on here?

My new reconstruction of SMiLE

I recently did a new reconstruction (remix or completion if you will) of  The Beach Boys’s SMiLE from vintage 1967 tracks into a cohesive piece. You can find a FLAC version here. If you are unfamiliar with SMiLE see here and here. This is my third crack at doing this. The first time was before Brian Wilson Presents SMiLE sometime in 2003. I found some random constructions all in mp3 and put it together from that. My second attempt was after Brian Wilson Presents SMiLE using mp3 versions of the Good Vibrations boxset and tracks from my first attempt and a bit better understanding of DAWs (I was still using Acid at that point). That version was pretty good (a hybrid between my first version and the 2004 version), but it always annoyed me that it was in mp3, so now I have done a lossless version. I have tracked down the best sounding versions (all lossless) that I could. Now I have and can use Logic, so the result flows and has a bit more polish (aside from the rough recording quality on some pieces).

I take some cues from the 2004 SMiLE and other constructions, but have left out awkward vocal overdubs, using clips from the modern recording, and digital pitch correction (please don’t make the Beach Boys sound like Kanye West!). I respect the effort of PurpleChick, but am not a fan of her version. I wanted He Gives Speeches, more Bicycle Rider, an ending with Surf’s Up, among other pieces not included with 2004 recording. The end results is not a slavish recreation of Brian Wilson Presents SMiLE with vintage tracks nor a purely historical recreation of what might have been had the album seen release in ‘67. This is personal and reflects my avant-garde aesthetics (but they aren’t hard to find in the original tracks), a fondness for instrumentals/extended pieces, and a bit of revisionist history.

Here are some notes by track. The source(s) for each track are in parentheses.

  1. Our Prayer / Gee (Good Vibrations Box). I like the pairing and transitions on Brian Wilson Presents SMiLE. All sourced from the Good Vibrations Box, so it all sounds great (especially Our Prayer).
  2. Heroes and Villains (Smiley Smile Bonus Track). I love this version (the Cantina version). It’s weirder than the released version: love the tape feedback in the middle. It fits SMiLE more than the single version.
  3. Do You Like Worms (Good Vibrations Box). This is nice and stretched out.
  4. Barnyard (SMiLE Vigotone). Has the vocals/animal sounds and decent sound, though it’s probably one of the worst songs from a fidelity standpoint. There is not much to be done here.
  5. Old Master Painter / You are my Sunshine (SMiLE Vigotone + Mark Linnet Mix). The Old Master Painter sounded better on the Vigotone and Sunshine better on the Mark Linnet Mix. The vocals are much more audible. The sax ending better on Vigotone. All crossfaded seamlessly.
  6. He Gives Speeches (Mok SMiLE). I’m not sure where he got this version, but it sounds great. I like the slapback delay on the handclaps.
  7. Wonderful (Good Vibrations Box). Wonderful sound for a wonderful song.
  8. Child is Father of the Man (Mok SMiLE). Not sure where he got this version, but it sounds better than the Vigotone and has quite a bit more on the end. I did some editing to remove a vocal section I didn’t like towards the end. These pieces are so wonderfully modular that you can do this.
  9. Cabinessence (Good Vibrations Box). This is my favorite version of this song. I love the droning trombone.
  10. Bicycle Rider (Good Vibrations Box + SMiLE Vigotone). This is my experimental track; a home for all the avant-garde or drugged out (depending on your bias) snippets. I incorporate  George Fell into his French Horn (I had to include some of it!) and bits of Heroes and Villains with the  Bicylce Rider Theme. I apologize for the creative liberties, but this is my reconstructions.
  11. Good Vibrations (Smiley Smile). The original and best. I toyed with the idea of using the stretched out instrumental version from the sessions, but why mess with perfection? The best way to end side one, as likely would have had it been released in ‘67.
  12. Look (Mark Linnet Mix). No vocals, but I like this as an instrumental. I did a bit of editing to take a section that interrupted the flow towards the end. This piece is full of abrupt cuts, so it works. I think mine is much smoother.
  13. Vegetables (Good Vibrations Box). This is a nice version. It may not be my favorite, but it sounds great. The ending piano segues nicely into…
  14. I Want to be Around / Workshop (SMiLE Vigotone). This the best sound I’ve heard for this one. I don’t like the vocals on Brian Wilson Presents SMiLE. It works much better as an instrumental, much weirder.
  15. Holiday (Mark Linnet Mix). Another instrumental. The sound is pretty good, better than a lot of versions I’ve heard. The piano towards the end is pretty cool. I’m not sure that I like the vocal version of this one either.
  16. Windchimes (Good Vibrations Box). Great song, great sound. If only it had the great drum part of Brian Wilson Presents SMiLE, though the piano ending is fantastic.
  17. Mrs. O’Leary’s Cow (Mok SMiLE + Mark Linnet Mix). This may be my favorite piece on SMiLE. The tasty intro from Mok just makes it all the weirder. There are versions I like ever so slightly better (like on Archeaology, but this one sounds the best).
  18. I Love to  Say Dada (Unsurpassed Masters Vol. 17 – SMiLE Sessions + Good Vibrations Box). The water chant comes from syncing the backing organ drone take with a wonderful stereo vocal take. The I Love to Say Dada comes Good Vibrations. I prefer the instrumental and wordless vocals over the lyrics.
  19. Can’t Wait Too Long (Smiley Smile). This is not a true SMiLE track as it was recorded later, but it’s wonderful and fits the feel of the end. The production and vocals are just to die for. The outro is fantastic. It was included on the Mark Linnet tapes, as well.
  20. Surf’s Up (Mok SMiLE). The arrangement is stretched out and suitably baroque to end the album and the sound is great. I feel Brian Wilson Presents SMiLE did this song a disservice by not ending with it. The solo Brian version is still my favorite, but this fits best. Though I love it and it sounds great, the solo piano is a bit too austere to fit with the rest of the album.

10,000 Hours (Yeah, right, Malcolm!)

Malcolm Gladwell’s new book Outliers makes the audacious claim that 10,000 hours of practice is the magic number for obtaining expertise (from Daring Fireball)

Having done a fair amount of reading on expertise (though it is not my research area), I can say the 10,000 hours figure for expertise attainment is arbitrary and quite problematic. Psychological research generally shuns these magic numbers because they oversimplify reality. I don’t fault Gladwell for wanting to simplify the dizzying complexity of research in this area, but I do take issue with his tendency to make claims that are not warranted from the research, yet claiming support from the research. This is quite disingenuous for someone with such clout. More on Gladwell, later…

The 10,000 hours figure does not coincide with the cognitive psychology research into expertise. The rule of thumb is that 10 years of effortful practice is needed for expertise. I will stress that this rule of thumb (or average tendency) is not the same as a magic number because there is considerable variance in human behavior contributed by existing experience (e.g., expertise in another area) and natural ability, among other things. I have played saxophone for over 10 years, but I do not have the natural ability and have not put forth effortful practice enough to become an expert. Thus, I am still a dilettante after easily 10,000 hours playing and 10 years of practice. Others have become expert saxophonists in less time. That does not change that the average tendency is still about 10 years.

The figure of 10,000 hours does not match up with the 10 year standard supported by the research. Say someone devotes 40 hours a week to practice (as if it were a full time job) that comes out to 4.8 years. Devoting 30 hours a week comes out to 6.4 years. Neither of these are reasonably close to 10 years. I’m not saying that it is impossible to achieve expertise in less than 10 years, but it is the average amount of time needed.

Lest you think Mr. Gruber believes this figure: when I sent a version of this post, he replied with a quote from Merlin Mann: “If I were half as smart as Malcolm Gladwell, I’d already have statistics and a clever name for my theory that he’s mostly full of shit.” (from here).