Tuesday, November 4, 2008

Database Mining: The New Ward Heeler

The Republican Party has led in database development for the last 20 years, mostly with the work of former direct mailer Richard Viguerie, with his Moral Majority and Reagan contacts. I remember reading about it in industry journals at the time.

What Obama's campaign has done is made this kind of database scientific. There is no contact that is not followed up on through any of the following means: text message, email, phone, facebook, or twitter. This has been the Democrat's strategic advantage since he entered the race. The way his campaign has stayed on top of technology throughout has been impressive.

When everything is said and done after this campaign, everyone will point out the massive GOTV effort by the Democrats will have provided the edge. But this edge, of people walking their states, would not have been possible without the social networking and database contacts.

I probably received, in the last 2 months, 8 entreaties to phone bank or canvass in battleground states near me EVERY DAY. I had to look to do this in 2004.

Sunday, November 2, 2008

FiveThirtyEight.com: Regressions for Politics

Number two on my list (previous post) of technological innovations to political discourse and campaigning this year is regression modeling of polls. The first one that I saw doing something like this, though far more simplistically is realclearpolitics.com (RCP). Basically what they did is took an average of all the polls, without assigning any kind of weighting. This was good to give us a nice and easy quick look at where the polls stood for the election, and washing out, for the most part, the effects of left- or right-lean of polls.

I would check RCP during the primary season occasionally, just to get the sense of where things stood, but they often didn't get the right trends. Around the middle of the primary season, I came across the website fivethirtyeight.com.  At first, I was very apprehensive about the model. My initial reaction was that there would be a propagation of errors that basically rendered the results useless. But the genius of the regression model, wasn't just the polling but the way these polls were weighted, both with lean and with recency.

If you'd like to check out the FiveThirtyEight FAQ, please do so; I will not repeat work that is already excellently explained even if you're not mathematically inclined. I think many of us who were watching this campaign intently, and liked the numbers aspect, really came on board around the time of the North Carolina and Indiana primaries. It was at that point that I put 538 on my bookmarks and checked it daily. You will also notice that I now use the 538 widget on my right rail. 

A note about FiveThirtyEight.com is that the statistician in question is Nate Silver, of the baseball stats site, BaseballProspectus.com. Most of you know that I also have a baseball-related blog, Sabermetrati.blogspot.com, and am a big fan of Nate Silver's work on BP, which I reference regularly. I did not realize that it was the same Nate Silver until around early June, and it definitely lended more credence to his fine work on FiveThirtyEight.

My favorite part of 538 is the win percentage. I have been following that number since Obama officially won the nomination. It now stands at 94% chance of Obama winning the election, with a peak about a week ago of 98%.

Tuesday will prove the accuracy of Nate Silver's model; it should be interesting.

Brave New World on November 4th?

The time is finally upon us. After nearly two years of campaigning we are about to elect a new president. This election has kept me so wrapped up in all the technology that I have not even had a chance to add my voice to it. Here are some of the amazing technologies that will forever be useful in future  campaigns:
  • Social Media Networking
  • Regression modeling of Polls
  • Get Out The Vote (GOTV) database mining
  • Voting Machine Fraud Blogging
  • Trash Emailing
  • YouTube Moments
  • Mobile Contact
While some of these items are completely non-partisan in nature and merely helped to inform us of campaign status (Poll regression modeling, Voting Machine Fraud blogging), the others were utilized to better advantage in getting out the message for the candidates. Who won this battle? Let's take a closer look.

Social Media Networking
The Obama campaign was on FaceBook, MySpace, Twitter, and LinkedIn before any of the other candidates during the long primary. I was not on the Obama bandwagon early, but I was impressed that I was being contacted in these areas long before anyone else. Any of us on FaceBook saw how many people were joining the Obama groups steadily over the course of the last two years. How many Obama buttons, Obama Lightsabers, etc. have we received during this time--too many to count?

On LinkedIn, Senator Obama asked my advice on topics facing his campaign and topics facing the country. I didn't fall for it: I knew he didn't send this personally nor do I think he read all of the comments. But I did send in my response when I thought I had something to say about a topic, and I received an automated response--that's better than when I was looking for a job and I would almost never get even the most cursory of automated responses ("Thank you for your interest, we will be contacting you if your resume meets our requirements.").

Between all of these social media items, this might constitute, as Mary Butler Twittered in my feed, his new communications channel after he's elected (IF he's elected). An unfiltered feed to the huge network he's built up and one that can get his message about his agenda out before the Limbaughs and Hannities of the world can lie about.

Are we witnessing the beginning of a whole new way of communicating in the Internet age between our government and the electorate?

Monday, September 29, 2008

It Happens Every Fall...

It is that time of the year where baseball is foremost in this blogger's mind. Yes, I am following the election, to the point of "election fatigue" and I have many things to say about technology and politics. In fact, I will be covering the action over at my favorite site since the primaries, fivethirtyeight.com. But that will come a little slower in this coming month, particularly with the Cubs in the playoffs...

In any case, I hope you hop over to Sabermetrati to keep up with all of the sabermetrically-inclined baseball news. A sneek peak: it could be a Red Sox vs. Cubs All Time All Futility World Series!

Thursday, September 4, 2008

Question: Why Are Republicans Graded on a Curve?

Why are Republicans graded on a curve? Everyone in the Mainstream Media has said, "Oh, well McCain's not a good public speaker" and then said the same about Sarah Palin, basically the idea that she is inexperienced and will not do well.

Yet, when these tepid speeches are completed, why do the Media go into paroxysms about how much better they did than anyone thought? Particularly if you're a Republican, and you don't believe in quotas, or "grading on a curve", than why are McCain and Palin accorded such kudos for speeches that pale in comparison to even the worst Democratic speaker last week?

Just a question...

Monday, September 1, 2008

Standing in the Shadow of the Birthplace of Democracy

BOSTON - To take this picture, I stood at the spot where the Boston Massacre sparked a nascent movement for liberty, sovereignty, and freedom from colonial rule.

As I stood there, I imagined what it must have been like to be standing the exact same spot in 1776 with the rest of the assembled masses as the first reading of the Declaration of Independence in public. That was the technology of the day: public oratory and pamphleteering, the olden day version of YouTube and Blogging (sans the instantaneous global reach).

After Senator Obama's speach last Thursday night, I'd say oratory is still big in the ever-more United States of America. After walking the Freedom Trail, I felt a part of our revolutionary history.

To see more pictures from the Freedom Trail taken by The Wife and I, follow this thread to Flickr. My particular favorite is the Boston Massacre Monument, which is in the Boston Commons, the oldest Public Park in the USA.

For a Fenway Park story, go to http://sabermetrati.blogspot.com.

Sunday, August 10, 2008

Trash Emailing = eTrashMail

I'm sure many of you have come across one of these "trash emails" trashing Barack Obama:
I recently received the first one about how Obama's tax plan is supposed to work. It was sent to me and my friends by a friend making fun of it. He in turn received it from what appears to be a very long thread. This thread is started by a VP at Bear Sterns.

His name was clear, his office number was there, and his title was clear. I will not reprint any of it, though, because I called him and asked him about it. Whether he sent it willfully as a show of support for John McCain, or as he told me, as an "I can't believe what people are saying about Barack Obama, look at this..." I don't think it appropriate to punish him further. He told me how many people have called or emailed him. His supervisor isn't happy with the notoriety, either.

This just goes to show the new playing field unfolding in the campaign arena: no longer does the negative campaigning come in the form of a negative ad, but it is a full-court press of emails (spam, actually!), YouTube videos, and blog posts. But the method in question today is that of the trash-talking email, or eTrashMail.

The eTrashMail format is one where blatant lies seem to be used but with the appearance that they are coming from an unquestionable source. For instance, in the one above, the source about taxes was coming from a VP at Bear-Stearns (yes, I know they are not exactly sporting a spotless reputation these days...).

eTrashMail can only mean that there will be only one winner this election season: the advertisers on Snopes.com! If you want to see who is the clear eTrashMail winner (meaning who's attacking the most), see the truth about Barack Obama and the same for John McCain. Obviously, the right wing smear machine is working overtime...

Wednesday, July 16, 2008

Mid-Season Baseball Prediction cross-posting alert...

See my mid-season prediction for the baseball season on http://www.bigshoulderssports.com.

Saturday, July 12, 2008

The Great AT&T and Apple iFiasco of 2008

Yesterday morning, I took an early lunch at 11am to go see if the iPhone was worth getting, and if the line was short, so I walked the block from my office to the Chelsea AT&T store. A quick calculation of about 30 people in front of me, at least 6 salespeople, at 15 minutes per transaction (on average from what was being said on the webs) meant that I'd be taken care of in between 1 and 1.5 hours. I waited 37 minutes and was the 16th person in line, when an AT&T employee came out to announce that only 5 iPhones were left, all 8GB. I turned and went right back to my office.

I had been stymied, like the 100 people behind me (and 11 in front).

Over the course of the day, my RSS feeds on the tech sites and blogs I frequent started revealing the story (mostly on CNet's excellent coverage): Apple and AT&T (but mostly Apple) had completely screwed up what was poised to be a summer blockbuster of major proportions, and during a recession, no less. When I left to go back to the office, there were approximately 100 people behind me. Let's do a quick tally of how much was lost, at minimum, before Noon yesterday, for a single AT&T store's gross profits on the day: 111 x $199 (8GB iPhone) + $36 (activation fee) = $26,085. That obviously does not include the people who would have joined the line at Noon. Let's say this happened roughly equally around the country at all the AT&T stores (according to CNet, there's 1,800), that would mean an extremely conservative loss of approximately $46,935,000. In one day. Before Lunch.

On top of that, when I called to see if the other local AT&T stores had iPhones in stock, both told me that they were out and that all the AT&T stores were out and might, or might not, be restocked on Saturday, Monday, or the week after. That doesn't sound very good for AT&T or Apple.

On Wall Street, AT&T stock lost over .58% by the end of the day. Apple was down 2.28%.

Now for Apple's story: it is clear that they did not follow proper Business Process Engineering standards. Nor did they follow proper business procedures.

If I'm an Apple or AT&T executive, based on last year's successful iPhone launch, I would prepare, at minimum, to handle at least as much as last year (about 700,000 iPhones sold from 7/1-7/3 last year). Then I would also plan to handle at least that many concurrently at the moment I made the software upgrade available. Why would they not do something like the following: Day 1, release iPhone 3G in the North American market; Day 8, release software upgrade for iPhone 1 on Apple's Web site (or through automatic update) for the same market; Day 15: Release to Western/Eastern Markets in a phased manner (Europe first day, then Asia, then Australia, then South America, etc.); Day 22, release software upgrade to the rest of the world.

I haven't yet seen the bad sales figures for Apple, nor do I know what they predicted they would do. I can be fairly certain that they will fall far short of expectations for this opening weekend due to this iDebacle. We'll see how low things go when the markets open on Monday. I don't know what the total loss will be when all is said and done, but it will be in the high millions, no doubt, when taking into account lost revenue from the iPhones that weren't sold, the AT&T high rate data plans that weren't started (and compounded by the fact that people will continue to make money for their competitors while using their old phones!) and iPhone apps that won't be bought now, later, or never. This is without even mentioning what will happen to the Apple brand and can't-miss aura.

I wonder if AT&T is secretly happy: the next iPhone might have more telecomm partners (which is what usually happens to exclusive deals in the industry after 1 or 2 years--see Q, Motorola), and perhaps won't sell quite as many as they might have, had they been successful on this launch. Conspiracy theory, anyone? Hmm...

Saturday, May 3, 2008

It's Social-Profit, Not Non-Profit!

In my quest to determine ROI in a charitable organization environment, I quickly came to the conclusion that applying the standard formulas used in for-profit businesses will not fit. Being a “Non-“ profit automatically means that a return on investment will be “nothing”. Since that is clearly not the case with any charitable organization, the “profit” must lie elsewhere. Reading the Good to Great Non-Profit version by Jim Collins gave me the idea that the profit generated by these agencies is actually social. He describes a social entrepreneur as someone trading in social benefit, rather than monetary benefit. That is, the efforts of so many volunteers and staff all working towards a social goal are generating a strong current of social profit that must be measurable.

It is within this measurement of social profit that I am stuck on. Sure, one easy measurement is the number of clients served. But that is more a function of the money we have, either through government grants or private donations. In almost all cases, if we had more money, we would be able to serve more clients.

Additionally, there are other social forces at work, some of which cannot be measured, although some can. For instance, we know that in the case of GMHC, where we have been fighting HIV/AIDS for over 25 years, the epidemic has changed many times, but one key fact has always remained, prevention (condom use, essentially) has prevented an uncountable number of people from being infected. But we’ll never know what the denominator is, and therefore cannot claim that we have prevented x amount of new infections.

What else can we do? This is again where I get stuck. I know that GMHC is generating social profit with 200 dedicated staff, and several thousand volunteers. So we can certainly total up the person-hours worked and certainly come up with a derivative formula that can measure something. But this would not help me with ROI because I still have the problem that it is money coming in that is driving the number of person-hours.

Perhaps another way to attack this problem is to try to come up with a “stock market” of Social Profit Organizations. The Social Profit Index could have several components:
• Private Funds to Market ratio
• Government Funds to Market ratio
• Volunteer to Market ratio
• Percentage of raising a dollar
• Black “Ink” Measure
• Concrete Social Profit as measured by clients served

Now we are reaching a point where the CIO/IT Director can start to get some numbers to plug into a model. For instance, once we have a Social Profit Index, or SPI, we can take the cost of any other administrative function, plug it into the model, get the return on the investment.

Additionally, we can look at only those areas of interest that we might want to impact, or areas that would be impacted differently with different types of technology. So, for example, we can measure ROI on raising private funds, or requesting government grants on the one hand, but also measure the impact of IT, specifically, on the SPI. We can finally begin to measure the validity of new projects, whether they fit the mission or are an expansion to the mission. In the next section, we’ll explore a way to measure these numbers across organizations in a meaningful way. Also, this might be used to provide an actual stock market Web site, where people could research their favorite charities and decide how best to spend their money. This could then in turn contribute yet another significant piece of information, a “trading value” that would be included on the site.

Wednesday, March 19, 2008

Networking, Networking, and, uh, Networking...

As I attended various discussions at the NTEN Conference () today, Networking seemed to be the word of the day:


  • Networking as in the standard way an IT geek uses it, which is the Local Area Network or Wide Area Network. This is important because for networking to be done in other ways, a good solid network infrastructure needs to be used in an organization. This will become more apparent as the the other type of "networking" starts to be used...

  • Networking as in Social Networking, or as the more geek-inclined among us call it, Interactive Web or Web 2.0. Web 2.0 requires that an organization that attempts to use this type of technology in fundraising and outreach needs to be very secure, with a solid network architecture that can handle both the bandwidth requirements (internal AND external) and the real-time communication and collaboration requirements.

  • Networking, as in meeting your colleagues from around the country and listening to what they're doing and how they might be resolving the same issues that you might have. This is always the best way to hear about what others are doing and discovering what works for them and allowing you to think about how it might work for your Social Profit organization.



I have also been working on how best to utilize Web 2.0 at GMHC for the last year. One thing that I've noticed from my research is that an agency really needs to examine it's mission and core values to determine how or whether to use this technology. My mantra in IT (and in all enterprise processes for that matter) is "how does this forward the mission?" If the answer is positive, then the next question is "how can this be implemented with best practices for the most impact and greatest value to the organization?" It is at this point that the team that will implement the process convenes and produces the project.



By the way, I want to compliment the NTEN team for their innovative use of just such Interactive Web technologies. This blog post, via the tag, is being collected with a number of other blog posts on the event using technorati. Also, although I did get sick of the constant "tweats" on my cell phone, their use of Twitter has been a great success and a wonderful idea. This is a technology to reckon with in the future, as you may have seen it just earlier today being used successfully to gather war protesters in real-time.

Sunday, February 10, 2008

Punk Rock, From The Jam, and Ubiquitous Technology

Last night, my wife and I went to see From The Jam at the Blender Theatre at Gramercy. This is two-thirds of The Jam, missing only Paul Weller. I last saw The Jam in 1982 at the Aragon in Chicago, and it was a transcendent experience: Paul Weller's guitar-playing was incandescent and inspiring, Bruce Foxton's bass exploded melodically, and the thunderous drumming of Rick Buckler almost caused my knees to buckle. Last night wasn't 1982, but it was still amazing and on occasion, I was 18 again.

Before I get to the point about technology, let me mention how politics enters the picture here. The Jam was nowhere near as famous in the States as their contemporaries, The Clash, but for me, from about 1978/1979 to 1984, though I listened to other music, nearly nothing remained on my turntable as long as the The Clash, Gang of Four, and The Jam. Although I'd always been interested in politics, both in the US and abroad, it seemed that these three bands really catalyzed and gave voice to the feelings that I had, and shaped those politics to what they are today, in no small part.

As the unmistakeable riff of In The City started, and the old punk rock blood cells started to emanate from the heart into the rest of my body, I looked around the audience. A lot of grey hair, bald spots, and digital CAMERAS/CAMERAPHONES being held aloft! I just remember thinking, "Boy, do I wish I had these kind of Star-Trek-like gadgets back in the day!" Not that it marred the experience, but I would see them out very frequently, with the now ever-present flash capturing every moment of our lives, like a live blog.

This was then further brought to the fore as they began This Is The Modern World and I now found myself thinking rather than just relaxing and enjoying the show. How prescient this song was back in 1978 when it came out. And this became a theme throughout the show that was different from that 1982 show because the world was different, and the changing meaning of "modern" came up. How so:

  • The thick cloud and stench of cigarette and marijuana smoke was no longer there, as it used to be concert de rigeur

  • At least half the audience took pictures and crude YouTube videos with their digital cameras and iPhones, and we'll all be able to relive the show by the time this blog posts (to wit: Mary's Flickr Page)

  • My ears will not be ringing for a week, even though the sound was loud, but balanced and undistorted

  • That A-Bomb in Wardour St. has a very different meaning today to an American than it did then



What was the same from 1982?

  • At a crossroads of political power between progressives and reactionaries

  • Brits and US in two unnecessary wars (then: Argentina, Grenada; today: Iraq for both), and as sung then and now by Foxton: "It was done underneath the flag of democracy..."

  • That the audience sang along and knew the words and timing as well as the band



What a weird confluence of events that brought us to this show: hanging out at the Cake Shop in LES with Mary finding a copy of Setting Sons on vinyl two weeks ago and purchasing it; then my friend John Dodge calling me last week to tell me he saw From The Jam in Chicago and that it was surprisingly good; and then seeing it also in my New Yorker's Upcoming Events section. I sometimes tire of what Mary and I call "Geezer Shows", but this one was well worth it, even without the erstwhile Paul Weller.

Here we are Down In the Tube Station at Midnight with my watch as proof!

Wednesday, January 30, 2008

Those Pesky Diebold Machines...

Recently, some stories have come up regarding the Diebold voting machines, this time favoring Senator Clinton against Senator Obama in the New Hampshire primary. While I believe there is a statistical rationale to being able to find wacky voting results (see Elections, U.S. Presidential, years 2000 and 2004), I did not believe that this was the case in New Hampshire. An excellent Salon Blog recently covered this topic before

I could get to it and quoted a just-completed study by two sociologists. They found that precincts that tended to use optical scan (Diebolds) were demographically different from those that did hand counts, and that was the reason for the differential between the candidates.


This is exactly what I suspected, and it only serves to prove the point that Americans have lost trust in one of our most fundamental systems--that which is supposed to ensure fair and honest elections. We need to be able to see the perils of an unbalanced voting system:


  • Will our poorest precincts have lesser technology, more prone to error than the richer precincts around the country?

  • Should there be a standard voting machine, so that no matter where any person might go, they'll be able to use the same machine that they're used to?

  • Will standardization of back-up systems, to allow for a hand-count, for instance become the requirement of a voting machine?

  • Should we work on an Internet-based voting system to maximize the number of people who vote?



Certainly that last one will be important in the future, but it will need to happen, provided the proper security is ensured. It might be at that point that we finally realize a true democracy in this country.

Monday, January 28, 2008

Technology in the World of Social Profit Organizations

If you're reading this, you've probably read my profile and see that I'm the Director, Information Systems at GMHC, which is a charitable organization. I do not like the sobriquets of "charitable" organization or "non-profit", because essentially, we are neither. We do not provide charity nor do we work for nothing. I prefer to tell people that I work in the Social-Profit sector. GMHC, and other organizations like ours generate a great deal of social profit; not only do our immediate clients benefit, but society in general most definitely profits from our work.

Since coming to GMHC, I have thought a lot about how I can measure the return on investment (ROI) of the work that my team and I contribute, in both human resource terms and purely hardware and software terms. Should I measure what our annual budget says we spend and amortize all of the hardware that was here and look at it in comparison to our revenues? Should I total the cost of running this department and create an index based on a per-hour basis that I can apply to the time saved to the end-user in our organization? I don't like any of those, because even though they have their applications in the For-[money]Profit sector, they don't truly capture what IT does in the Social-Profit sector. Or for that matter, what the true social profit is.

I am going to continue to explore this and will report my findings on this blog. Stay tuned for more Social Profit!