Should you participate in EconMatch?

Question

I was recently asked by a friend whether or not her student should participate in EconMatch.  My colleague, Umut Dur, and I thought about this for a bit, and we'd like to share our initial thoughts.  To be clear, this is not a complete analysis and we welcome criticism.  Also, this is not meant to criticize the creators of EconMatch who we believe had good intentions.  But as said by Roth " Market design involves a responsibility for detail, a need to deal with all of a market’s complications, not just its principle features. "  We believe that any system proposed for job market matching needs to be carefully analyzed in every detail.

EconMatch addresses a clear problem.  There are only a limited number of fly-outs and so departments often do not fly out candidates deemed unattainable and similarly sometimes fly out candidates that are unattainable and end up not hiring anyone.  This clearly has the potential for inefficiency, and EconMatch aims to address this through a recommendation process based on matching theory.

EconMatch makes recommendations for fly-outs, not an actual match.  But we will point out a couple of problems with the match that the recommendation is based on.  We consider two cases - case 1 there is pressure for you to accept the recommendation from the match and so the assignment is "binding - and case 2, the assignment is just a suggestion.

Case 1 - the assignment is binding - here we are imagining a system where if you participate in EconMatch and are assigned to a school, then there is an expectation that you accept the job offer.  In this case, you cannot know which preferences to submit to EconMatch if not all schools participate.  For example, if you rank schools A, B, and C, and B does not participate in the match, then until you know whether or not B offers you a job, you cannot know whether you should tell EconMatch that A is your only acceptable school or whether you prefer A to C but both are acceptable.  If you tell the "truth: (submit A,C) then this would obviously be a big mistake if you are not matched with A and B does end up making you an offer.  For any matching theorists, the problem here is that you don't actually know what your true preferences are because you don't know your outside option.  If it's B, then your true preferences are just A.  If it's not B, then the true preferences are A,C.

Case 2 - the assignment is not binding.  In this case, it's hard to even know what an assignment is.  It is more of a suggestion that this school should make an offer to this candidate.  But it is not at all clear that this is a good suggestion.  Again, the problem is partial participation.  Suppose there are three schools, A, B, and C, and three students, i, j, and k.  All students rank the schools A>B>C.  School A either wants to hire i or no one but only they know which one.  Schools B and C both rank i>j>k.  School A does not participate in the match.  Since A does not participate, the match will assign i to B and j to C.  Should i accept?  No, i should wait to see if she gets an offer from A.  Should j accept?  No.  j should also wait to see if i gets an offer.  If she does, then j will get an offer from B.  An analogous situation would occur for the schools if not all of their candidates participate.  Also notice that, if j accepts without waiting for i's decision it is possible for B to end up unassigned, i.e. this situation is also problematic for the colleges.

Finally, we would like to point out that not participating in the match can be beneficial for schools (and candidates).  Suppose there are two schools, A and B, and two students, i and j. A ranks i > j, B ranks j > i, i ranks B > A, and j ranks A > B.  If everyone participates, then i is matched with A while j is matched with B.  Suppose A chooses not to participate.  Now the match assigns j to B.  i is left unassigned and now A is able to hire i.  Therefore, A gets her favorite candidate by opting out of the match.

There are other reasons why we think EconMatch would be problematic (timing!).  But our main concern is that if we were a school (or a candidate), we wouldn't know the best way to rank our options or whether to follow the systems recommendation.  For these reasons, we would not recommend that people participate.  The key point is that full participation cannot be guaranteed, and without full participation, Econmatch might hurt some of the participants.

Again, these are preliminary thoughts, and we would love to hear what other people think.

Good luck to all the job candidates,

Thayer Morrill and Umut Dur

PS.  We believe the recent (and growing!) literature on sequential/parallel matching systems is very relevant to this problem.  There are great papers by Andersson, Dogan, Ekmekci, Ertemel, Kesten, Manjunanth, Turhan, Yenmez, Westkamp..... and Umut on this.

Comments

It would be nice if everyone gave it a try to see what happens, because these comments certainly do highlight the need for all to participate.

Answer 1

I would recommend NOT to use that website:
- There is virtually no information about who is behind it.
- No precise description of the "matching" algorithm, the claims found in the FAQ section are dubious.

More worrisome is:
- The security level of the website is dismal, if not dangerous (e.g., passwords are not encrypted, no https connection, etc.).
- No user agreement. Also, the only data policy that can be found is that the data will not be communicated to other parties. But nothing is said about what the people behind the website will do with the data they collect. If some academic work is intended (which wouldn't be surprising) I would like to see an IRB.

Answer 2

I would like to echo what Guillaume wrote. At this time, I have a number of concerns regarding EconMatch.

On a high level, I think EconMatch is aiming to solve an interesting problem, i.e., the coordination failure on the job market. And personally, I find the idea of a using a recommender system to solve this problem quite exciting. However, I do not think that EconMatch is ready for field deployment .

As Guillaume, I am also concerned that there is no information on the website regarding (a) who designed the algorithm, (b) who is behind this effort, and (c) who is running the website/algorithm. However, this is hugely important to build "trust in the market institution" itself. How should the participants trust the mechanism, if they do not even know who they are giving their private information to? The security concerns mentioned by Guillaume come on top of this.

My second major concern is about the mechanism itself. Guillaume already mentioned that some of the claims on the FAQ page are dubious. My particular concern is that there is no analysis of the effects of the mechanism if only some (e.g., a small minority) of the applicants and departments participate in the mechanism. The participants won't know who else has participated, so it will be difficult if not impossible for them to interpret any result of the mechanism. This is problematic, because the recommendations hugely depend on the participation levels on both sides of the market. Thus, the departments and the candidates must effectively guess who is participating, to gauge what to do with the recommendations. How should we know that this will improve coordination? Perhaps it will make things worse? [Thayer and Umut made a very similar point in their post.]

Long story short: while I think this is an interesting research direction, I think it is too early to put this mechanism into practice, and using a mechanism like this too early may even be harmful.

Answer 3

We're grateful for the broad interest in our project to help improve the function of the economics job market, the result of which can help candidates and employers find well-suited matches. We've had an outpouring of enthusiasm for the service from users and professionals across the academy.

In our view, the match signal we provide needs to be compared to the status quo, rather than the ideal thought experiment.

The key is to notice that, in the status quo, employers have an information problem created by the fact that candidates cannot credibly signal their preferences for several reasons (in addition to not being able to commit to cheap talk, they don't even know their opportunity set).

The match signal usefully aggregates information on preferences and attainability which allows candidates to credibly signal interest while giving employers useful information about which candidates (among users) are the best candidates likely willing to accept an offer.

We agree that the match signal is improved as more parties use the platform. It is for this reason we find it disappointing that some have discouraged participation, though participation is the solution to the problem they describe.

We're interested in inviting collaboration, and we have privately invited the writers here to provide their suggestions for the platform.

For the future, we're working on a tool that mitigates the effects of partial participation by predicting the preferences of parties who do not participate. In this way, the output can provide a better prediction of which candidates are attainable for a given employer if preferred employers don't participate.