Ed Felten recently wrote about honor systems, and I think he absolutely nailed one of the reasons the Caltech Honor System has been successful:
Caltech has ... a student culture that features less student versus student competitiveness than you might expect. Competition there tends to be student versus crushing workload.
The Caltech Honor System is enforced by an undergraduate committee called the Board of Control, of which I was a member my junior year and Chairman my senior year. Most of the academic cases I saw involved students who had cracked under pressure (usually external pressure, such as from parents). Often they reported themselves out of guilt days before the cheating was detected by their professors. Paradoxically, many were already getting an 'A' in the class without cheating, but essentially self-destructed.
College is a game like any other. The participants enjoy different aspects of it. Some just enjoy playing nicely with others. Some enjoy winning, at various costs. Some enjoy breaking the rules; others enjoy making them. For an honor system to be successful, it has to have clear benefits for everyone playing the game, regardless of their goals -- and not only the students, but also the faculty and staff (whose perspectives I'll not discuss in this article).
Cheating at Caltech is mostly an exercise in futility. If you understand what you're doing then you don't need to cheat, and if you don't then cheating is like trying to pretend you speak a foreign language in front of native speakers. Good luck faking that! The answers are rarely to be found in books or your peers, who are generally just as lost as you are. If a professor is lazy, you might find answers in previous years' exams. You might also be able to team up with a graduate student teaching assistant. Either way, your work is readily identifiable as plagiarism -- believe me, I saw several such cases.
Cheating really only helps your GPA, but almost nothing you could want at Caltech (other than the grade itself) depends on your grades. Awards don't depend on your GPA: I won the institute's highest undergraduate award, and my GPA was a pitiful 2.8, barely a 'B minus'. Jobs don't depend on your GPA: Employment comes from establishing contacts with your professors and peers, and they're all evaluating you on your raw intelligence and competence, not your grades. Again, good luck faking that! Graduating does depend on your GPA (somewhat), but Caltech makes dropping a course a much easier way to escape a failing grade. If you're failing a required course, you're going to re-evaluate your major. If you're failing a core requirement, you're going to re-evaluate the choice to attend Caltech at all. Indeed, most of the academic cases I saw involved freshmen and their core classes, and for some of them expulsion was a visible relief.
As Ed wrote, at Caltech, it's really you versus the exam (and the exam's probably going to win). I don't remember ever running out of time on an exam, but I do remember often having a lot of time left and way too many unanswered questions.
So for undergraduates, the expected value of cheating at Caltech is nearly zero, against the risk of severe consequences (including expulsion). But even if cheating at Caltech might have some reward, the Caltech Honor System would still appeal to the undergraduates. Among other reasons, the Caltech Honor System is a social contract with your peers; breaking it is tantamount to declaring yourself an outsider. But part of the appeal of Caltech &emdash; a large part of why you're there and not at a less expensive, less demanding university &emdash; is that you're finally surrounded by people like you. The Honor System helps you fit in, and helps you feel secure.
All this is not to say the system's perfect. It depends pretty heavily on the individuals who enforce it, especially the Dean of Undergraduate Students (who chooses to enact or override the Board's recommendations) and the Chairman and Secretary of the Board (who choose which cases receive a full board hearing). The students tend to judge each other pretty harshly, perhaps more harshly than the faculty would. Because of the (necessary) secrecy surrounding the Board, the community at large doesn't have a lot of insight into how it really works, and so the system relies heavily on trust. Obviously some cases go unreported and undetected.
As Chairman of Caltech's Board of Control, the problem that bothered me the most was a flaw I've noticed in most judicial systems. When convicted of an offense, the guilty who completely own up to their mistakes tend to receive some leniency, while the innocent who vehemently deny their guilt (even in the face of a guilty verdict) tend to receive the harshest sentences. On the one hand, this is completely sensible; a guilty person who accepts their guilt is clearly more ready to be a functioning, trustworthy member of society than a guilty person who is still in denial. On the other hand, this means that a wrongly convicted innocent person will often be punished above and beyond the already terribly mistaken verdict. And what alternatives are available to the innocent convict, other than appeal? (Or plea bargains, which generally don't exist in the academic context.)